tommy0103andClaude Opus 4.8
e3e61cc7ab
Implement the full ADR-0006 plan: three-layer separation of transaction
correctness, retry policy, and cross-process writer coordination.
Layer 1 — scripts/tx.ts (transaction correctness):
- runWriteTransaction executes work exactly once; no internal retry.
- BEGIN IMMEDIATE takes the write lock up front (avoids SQLITE_BUSY_SNAPSHOT).
- Guarded rollback: checks inTransaction() via adapter before attempting
ROLLBACK; never masks the primary exception.
- WriteTxDiagnostics attached to errors: phase, code, label,
rollbackSucceeded, rollbackError, transactionActive.
- Binding adapters (betterSqliteTransactionAdapter, nodeSqliteTransactionAdapter)
mapping better-sqlite3's `.inTransaction` and node:sqlite's `.isTransaction`.
- configureConnection centralizes WAL + synchronous + busy_timeout.
Layer 2 — scripts/write-coordinator.ts (retry policy):
- runRetryableWriteTransaction: bounded retry with total time budget.
- Only retries when the transaction confirmed ended (transactionActive=false)
and the error is SQLITE_BUSY during work/commit phase.
- BEGIN-phase BUSY = abort entire build (isBeginBusyFailure); the caller
returns `{ deferred: true, reason: 'writer_busy' }` instead of waiting.
- hasUnusableTransaction detects a still-active transaction after failure;
aborts the build immediately, never retries.
Layer 3 — scripts/writer-lease.ts (cross-process coordination):
- acquireWriterLease: dedicated writer.lock.sqlite with busy_timeout=0 +
BEGIN IMMEDIATE. Non-blocking attempt; bounded wait with retryDelayMs.
- writerLockPathFor derives lock path from the target DB path.
- Lease held for the entire build; released on completion or failure.
- Lock DB uses DELETE journal (not WAL); crash/close auto-releases.
- All consumers obey: skill acquires at build start (returns deferred if
unavailable); app daemon (via worker) acquires for its build cycle.
Build semantics changes:
- affectedSessionIds updated only after successful commit.
- BuildIndexResult gains skipped/skippedFiles for observability.
- Skill finalize failure now fails the build (was silently warned).
- Checkpoint changed to PASSIVE (TRUNCATE reserved for maintenance/exit).
- Skill buildIndex returns { deferred, reason } on lease contention;
indexer-service reschedules the build (deferredRetryMs) without publishing
a heartbeat (so the build-deferred state is visible to cross-process
arbitration).
- Service publishes heartbeat immediately on start() for correct arbitration.
Tests:
- tests/write-transaction.test.mjs: single-shot execution, diagnostics
propagation, auto-rolled-back transaction detected, rollback failure
captured as metadata, BEGIN IMMEDIATE semantics.
- tests/writer-lease.test.mjs: acquire/release, contention returns null,
bounded wait with release during budget.
- tests/app-writer-lease.test.mjs: better-sqlite3 adapter integration.
- tests/app-rollback-guard.test.mjs: rewritten — transient BUSY recovered
by coordinator, persistent BUSY skips file, begin-busy aborts build,
live-transaction aborts build, phantom affectedSessionIds prevented.
- tests/daemon-arbitration.test.mjs: skill defers to fresh app heartbeat,
builds when heartbeat is stale.
- tests/app-indexer-service.test.mjs: new cases for deferred-retry
scheduling and immediate heartbeat on start.
- app/tests/electron-concurrency.mjs + child: dual-child IPC structure for
real better-sqlite3 contention (holder acquires lock → build child starts
→ delayed release → result collected; persistent contention bounded).
ADR-0006 updated to reflect the implemented design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
4.1 KiB
Obelisk
Obelisk is explicit memory infrastructure for coding agents: it indexes local Claude Code and Codex transcripts into a queryable SQLite evidence layer, and a CodeAct runtime lets an agent write a small query, run it, and answer from real session history. This glossary pins the terms that are specific to Obelisk; it is not a spec.
Runtime interface
Runtime interface:
The public contract, expressed as four verbs — build, search(text),
query(code), attune(code). Skill, CLI, and MCP are transports over this same
shape; none of them add their own retrieval surface.
Avoid: API, tool surface
CodeAct:
The interaction style where an agent writes JavaScript that runs inside the
query(code) sandbox and returns JSON, rather than calling many fine-grained
tools. This is Obelisk's core design choice.
Avoid: tool-calling, function-calling
Helper:
A convenience accessor available only inside the query(code) sandbox
(overview, search, context, sql, memories, …). Helpers are never
promoted to an external tool surface.
Indexing
Provider adapter:
A pure per-source module (claude, codex, later opencode, pi, …) that discovers a
source's transcript files and parses one into a stream of records. It never opens
or writes a database; adding a source means adding one adapter. The shared pure
parse/discover helpers live in scripts/parsing.mjs, which imports only
node:fs/path/os — deliberately node:sqlite-free so the compiled providers can be
consumed by the app (whose Electron runtime has no node:sqlite).
Avoid: parse core, parser, ingest
Record: One normalized row destined for the index (session, message, tool call, tool result, summary, subagent, workflow, …), emitted by a provider adapter before any persistence happens.
Persist layer:
The single shared, provider- and binding-agnostic writer that consumes records
from any adapter and writes them into an injected SQLite handle inside a
transaction. The binding is injected — node:sqlite (skill/CLI) or
better-sqlite3 (app) — so there is one persist implementation, not one per
binding.
Avoid: writer, sink, DAO
Daemon indexing mode: Continuous incremental indexing driven by a long-lived process (the desktop app, later a CLI daemon) that watches transcript directories and keeps the index fresh as files change. Avoid: watcher mode, live indexing
Passive pull mode: On-demand incremental indexing performed by the skill when there is no active daemon: an invocation of the runtime brings the index up to date, then answers. Avoid: lazy indexing, on-read indexing
index_state:
The bookkeeping table shared by both indexing modes. It records, per transcript
path, the last-seen mtime and lines_processed (enabling resume-from-line
incremental indexing), plus heartbeat/last-build markers used for daemon
arbitration.
Daemon arbitration:
The policy by which the passive pull mode detects a fresh daemon from the
__app_heartbeat__ marker and skips every skill-side mutation, including schema
setup, indexing, checkpointing, and attune. The heartbeat alone means “the
daemon should write”; __app_last_successful_build__ records coverage/freshness,
not ownership. Both indexing modes use the same persist layer.
Writer lease:
The hard cross-process safety mutex behind daemon arbitration. A writer holds
BEGIN IMMEDIATE on .obelisk/writer.lock.sqlite for the complete mutation;
manual rebuild holds it through build, target-database replacement, and reopen.
The heartbeat expresses policy, while the writer lease prevents overlapping
writes during races, stale heartbeats, or processes from different versions.
Memory
Queryable session memory: The evidence layer — real sessions, messages, tool calls, subagents, workflows — that an agent queries on demand. Obelisk deliberately does this instead of implicit/ambient memory. Avoid: implicit memory, ambient memory, auto-recall
Approved durable memory:
Human-approved conclusions persisted as markdown plus a registry record, via
attune(code) calling remember()/forget(). Auditable and revocable.
Avoid: long-term memory, vector memory