Implement the full ADR-0006 plan: three-layer separation of transaction
correctness, retry policy, and cross-process writer coordination.
Layer 1 — scripts/tx.ts (transaction correctness):
- runWriteTransaction executes work exactly once; no internal retry.
- BEGIN IMMEDIATE takes the write lock up front (avoids SQLITE_BUSY_SNAPSHOT).
- Guarded rollback: checks inTransaction() via adapter before attempting
ROLLBACK; never masks the primary exception.
- WriteTxDiagnostics attached to errors: phase, code, label,
rollbackSucceeded, rollbackError, transactionActive.
- Binding adapters (betterSqliteTransactionAdapter, nodeSqliteTransactionAdapter)
mapping better-sqlite3's `.inTransaction` and node:sqlite's `.isTransaction`.
- configureConnection centralizes WAL + synchronous + busy_timeout.
Layer 2 — scripts/write-coordinator.ts (retry policy):
- runRetryableWriteTransaction: bounded retry with total time budget.
- Only retries when the transaction confirmed ended (transactionActive=false)
and the error is SQLITE_BUSY during work/commit phase.
- BEGIN-phase BUSY = abort entire build (isBeginBusyFailure); the caller
returns `{ deferred: true, reason: 'writer_busy' }` instead of waiting.
- hasUnusableTransaction detects a still-active transaction after failure;
aborts the build immediately, never retries.
Layer 3 — scripts/writer-lease.ts (cross-process coordination):
- acquireWriterLease: dedicated writer.lock.sqlite with busy_timeout=0 +
BEGIN IMMEDIATE. Non-blocking attempt; bounded wait with retryDelayMs.
- writerLockPathFor derives lock path from the target DB path.
- Lease held for the entire build; released on completion or failure.
- Lock DB uses DELETE journal (not WAL); crash/close auto-releases.
- All consumers obey: skill acquires at build start (returns deferred if
unavailable); app daemon (via worker) acquires for its build cycle.
Build semantics changes:
- affectedSessionIds updated only after successful commit.
- BuildIndexResult gains skipped/skippedFiles for observability.
- Skill finalize failure now fails the build (was silently warned).
- Checkpoint changed to PASSIVE (TRUNCATE reserved for maintenance/exit).
- Skill buildIndex returns { deferred, reason } on lease contention;
indexer-service reschedules the build (deferredRetryMs) without publishing
a heartbeat (so the build-deferred state is visible to cross-process
arbitration).
- Service publishes heartbeat immediately on start() for correct arbitration.
Tests:
- tests/write-transaction.test.mjs: single-shot execution, diagnostics
propagation, auto-rolled-back transaction detected, rollback failure
captured as metadata, BEGIN IMMEDIATE semantics.
- tests/writer-lease.test.mjs: acquire/release, contention returns null,
bounded wait with release during budget.
- tests/app-writer-lease.test.mjs: better-sqlite3 adapter integration.
- tests/app-rollback-guard.test.mjs: rewritten — transient BUSY recovered
by coordinator, persistent BUSY skips file, begin-busy aborts build,
live-transaction aborts build, phantom affectedSessionIds prevented.
- tests/daemon-arbitration.test.mjs: skill defers to fresh app heartbeat,
builds when heartbeat is stale.
- tests/app-indexer-service.test.mjs: new cases for deferred-retry
scheduling and immediate heartbeat on start.
- app/tests/electron-concurrency.mjs + child: dual-child IPC structure for
real better-sqlite3 contention (holder acquires lock → build child starts
→ delayed release → result collected; persistent contention bounded).
ADR-0006 updated to reflect the implemented design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reported as "cannot rollback - no transaction is active" — a secondary error.
SQLite auto-rolls back certain failures (SQLITE_BUSY / SQLITE_BUSY_SNAPSHOT,
disk full), after which the per-file loop's unguarded ROLLBACK in its catch
threw over the real error and aborted the whole build instead of skipping just
the bad file.
Stopgap only (see docs/adr/0006 for the planned full fix):
- Add safeRollback(db) in both indexers: it swallows only the rollback's own
error, so the true cause surfaces. Per-file failures are logged and the build
continues; the finalize failure still propagates.
- Give the skill's node:sqlite connection an explicit PRAGMA busy_timeout=5000
(it has no default). The app adds none: better-sqlite3 already defaults to
5000ms, so busy_timeout is NOT the root-cause fix and is not treated as one.
Add tests/app-rollback-guard.test.mjs: injects a DB that faithfully reproduces
"a write auto-rolls back the txn, then ROLLBACK errors" and asserts the build
survives (bad file skipped, other file indexed). Revert-checked: without the
guard the test fails with the exact "cannot rollback - no transaction is active".
docs/adr/0006 records the real fix (shared runWriteTransaction, single-writer
coordination, BEGIN IMMEDIATE, whole-transaction retry, PASSIVE checkpointing)
as deferred, two-phase work — and why bumping busy_timeout is not it.
Verified: suite 124/124, typecheck clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The skill's `--build` (always a force build) cleared `index_state` and
re-indexed existing files, but never removed rows for transcripts that no
longer exist on disk, so deleted sessions accumulated forever (the dogfood
showed 328 indexed sessions vs 283 current files). A force build is meant to
be a clean rebuild, matching what the app already does.
Drop every derived table (messages, tool_calls, tool_results, sessions,
summaries, subagents, workflows, workflow_agents) in the force path, then
re-index from the current files. `memories` is the durable, human-approved
layer and is never cleared; messages_fts is repopulated by the existing
'rebuild' command in finalize.
Add a test that builds two sessions, deletes one transcript, force-rebuilds,
and asserts the stale session is purged while a seeded memory survives.
Verified the test is discriminating: without the fix it reports
['gone','keep'] instead of ['keep'].
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move all pure parse/discover helpers (message extraction, project-path, codex
helpers, discovery) out of db.mjs/indexer.mjs into scripts/parsing.mjs, which
imports only node:fs/path/os. Providers now import from parsing.mjs, so the
provider import graph no longer transitively loads node:sqlite — a prerequisite
for the app (Electron/Node without node:sqlite) to consume the compiled core.
Verbatim move, no behavior change; indexer.mjs 840→209 lines. 119/119.
Complete the skill-side provider migration: codex now goes through a pure adapter
and the shared persist layer, and the two original monolithic indexers are gone.
New:
- scripts/providers/codex.ts — pure codex adapter. Full-reparse (buffers the whole
file) because the event_msg↔response_item dedup needs whole-file, bidirectional
knowledge; emits SessionRecord with countMode 'total'. Handles guardian threads
(→ delete-session), agent spawns/tool calls (→ tool_call/subagent), token_count
(patched onto the message record) and task_complete (→ message-turn-duration).
Contract:
- SessionRecord.countMode ('total' | 'delta') tells persist whether to replace or
accumulate message_count — claude is line-incremental (delta), codex full-reparse
(total). SubagentRecord non-key fields are optional; persist merges them
column-wise with COALESCE. MessageTurnDurationRecord.turn_duration_ms is nullable.
Orchestration:
- buildIndex's codex branch parses via the adapter and writes via persist. An
unchanged file is skipped but still swept for stale guardian rows (routed through
persist as a delete-session), preserving prior behavior.
Cleanup:
- Remove the now-unused indexJsonl, indexCodexJsonl, deleteCodexThreadRows and
upsertCodexSubagent — their semantics now live in the adapters + persist.
indexer.mjs drops from ~840 to 428 lines. Codex pure helpers stay exported for
codex.ts and the guardian sweep (physical move deferred to the app-side reorg).
- Migrate the upsert drift test off indexJsonl to the claude.parse + persist path,
keeping the rowid-stability and count-replace regression guards.
Tests: tests/codex-parse.test.mjs (record-stream golden: dedup, tools, token patch,
turn-duration, guardian→delete) and tests/codex-index.test.mjs (full buildIndex
path: fresh build + incremental full-reparse, total-count replace, no duplicates).
Verified equivalent on the real ~/.obelisk index: codex messages 82476 and
subagents 522 identical before/after, zero guardian leakage; real incremental
confirmed (touch a codex file → reparsed idempotently, unchanged files skipped).
lint + typecheck clean, 119/119.
buildIndex's claude branch now parses via providers/claude.ts and writes via
persist.ts instead of the inlined indexJsonl. Behavior is equivalent — the full
buildIndex integration suite (runtime.test.mjs) stays green, 116/116 — and a
force rebuild of the real ~/.obelisk index (327 sessions, 119k messages)
reproduced identical session counts with project_path fully populated.
codex, indexSubagentMeta, workflows, history and the project_path pass are
untouched. indexJsonl is now unused by buildIndex (kept for its drift test;
removed once codex is migrated).
Adds tests/incremental-index.test.mjs: verifies resume/accumulate through the
full buildIndex path (append new lines to an indexed session → incremental
build resumes from the cursor, message_count accumulates, no duplicates). The
30s shouldSkipBuild debounce is cleared in-test so the incremental run fires.
claude.parse mirrors indexJsonl line-for-line but yields IndexRecords (no db).
Adds MessageTurnDurationRecord op. Purely additive — buildIndex still uses the
old path. 111/111 green.
scripts/indexer.mjs indexJsonl used INSERT OR REPLACE (rowid/FTS churn) and
carried message_count forward on full re-scan; align to app's ON CONFLICT upsert
+ count reset so the two indexers no longer silently diverge. Regression test added.
Add `source` column to sessions and messages ('claude' | 'codex').
Discover and parse Codex JSONL files from ~/.codex/sessions/, mapping
Codex thread/item structures to the same schema (messages, tool_calls,
tool_results, subagents). Move DB to ~/.obelisk/ with legacy migration.
Add rebuild-to-temp-then-swap for safe full rebuilds. On the app side:
source filter toggle, collapsible untitled session fold, configurable
codexDir in Settings, and a dev script. Update SKILL.md and query
helpers to expose source fields and accept source filter opt.
Extract schema DDL into scripts/schema.sql shared between CLI and app.
Add an in-process chokidar-based indexer-service that watches ~/.claude/projects
for JSONL changes, debounces, and triggers background rebuilds via a worker
thread. Rename Usage view to Activity, flesh out MemoryDetail and SubagentDetail
views, and refine App.vue layout/routing. The main process now starts/stops the
indexer lifecycle and notifies renderer windows on index updates.
Introduce an Electron app with session browser, memory list, and usage
views (vanilla JS + Vue scaffolding). On the data layer: add content_type
and is_meta to messages for transcript control-plane filtering, introduce
FTS5-backed memory recall with safe tokenization, support memory archival
via forget() through the renamed --attune runtime, and expose anchors on
memory records.
The old slug-to-path conversion (replace hyphens with slashes) was
lossy and wrong for paths containing hyphens. Now infers project_path
from the most-frequent observed cwd across session messages, falling
back to slug decoding only when no cwd data exists. Adds
refreshSessionProjectPaths() to backfill existing sessions on rebuild.
lightweight workflowTree, build debounce, minor fixes
- Index phase, label, model, state, duration, tokens per workflow agent
- Index duration, total_tokens, status, name per workflow run
- workflowTree returns parsed result + agent summaries instead of
dumping all messages
- 30s debounce on buildIndex to avoid repeated directory scans
- Fix broken BASH_EXIT_PAT (SQLite LIKE has no character classes)
- Fix SKILL.md step numbering, document FTS5 hyphen limitation
Index the is_error boolean from JSONL tool_result blocks into a new
column, replacing the old ERROR_PATS text-matching approach that
produced ~90% false positives. Bash exit code pattern kept as fallback.
Prepares for adding Codex JSONL support — new indexers can be added
without touching the query layer.
- db.mjs: schema, openDb, truncation helpers, shared utilities
- indexer.mjs: file discovery, JSONL parsing, SQLite writes
- query.mjs: all query API functions (search, context, raw, etc.)
- runtime.mjs: thin CLI entry point + VM script executor
tommy0103andClaude Opus 4.8