fix: coordinate sqlite index writers

Implement the full ADR-0006 plan: three-layer separation of transaction
correctness, retry policy, and cross-process writer coordination.

Layer 1 — scripts/tx.ts (transaction correctness):
- runWriteTransaction executes work exactly once; no internal retry.
- BEGIN IMMEDIATE takes the write lock up front (avoids SQLITE_BUSY_SNAPSHOT).
- Guarded rollback: checks inTransaction() via adapter before attempting
  ROLLBACK; never masks the primary exception.
- WriteTxDiagnostics attached to errors: phase, code, label,
  rollbackSucceeded, rollbackError, transactionActive.
- Binding adapters (betterSqliteTransactionAdapter, nodeSqliteTransactionAdapter)
  mapping better-sqlite3's `.inTransaction` and node:sqlite's `.isTransaction`.
- configureConnection centralizes WAL + synchronous + busy_timeout.

Layer 2 — scripts/write-coordinator.ts (retry policy):
- runRetryableWriteTransaction: bounded retry with total time budget.
- Only retries when the transaction confirmed ended (transactionActive=false)
  and the error is SQLITE_BUSY during work/commit phase.
- BEGIN-phase BUSY = abort entire build (isBeginBusyFailure); the caller
  returns `{ deferred: true, reason: 'writer_busy' }` instead of waiting.
- hasUnusableTransaction detects a still-active transaction after failure;
  aborts the build immediately, never retries.

Layer 3 — scripts/writer-lease.ts (cross-process coordination):
- acquireWriterLease: dedicated writer.lock.sqlite with busy_timeout=0 +
  BEGIN IMMEDIATE. Non-blocking attempt; bounded wait with retryDelayMs.
- writerLockPathFor derives lock path from the target DB path.
- Lease held for the entire build; released on completion or failure.
- Lock DB uses DELETE journal (not WAL); crash/close auto-releases.
- All consumers obey: skill acquires at build start (returns deferred if
  unavailable); app daemon (via worker) acquires for its build cycle.

Build semantics changes:
- affectedSessionIds updated only after successful commit.
- BuildIndexResult gains skipped/skippedFiles for observability.
- Skill finalize failure now fails the build (was silently warned).
- Checkpoint changed to PASSIVE (TRUNCATE reserved for maintenance/exit).
- Skill buildIndex returns { deferred, reason } on lease contention;
  indexer-service reschedules the build (deferredRetryMs) without publishing
  a heartbeat (so the build-deferred state is visible to cross-process
  arbitration).
- Service publishes heartbeat immediately on start() for correct arbitration.

Tests:
- tests/write-transaction.test.mjs: single-shot execution, diagnostics
  propagation, auto-rolled-back transaction detected, rollback failure
  captured as metadata, BEGIN IMMEDIATE semantics.
- tests/writer-lease.test.mjs: acquire/release, contention returns null,
  bounded wait with release during budget.
- tests/app-writer-lease.test.mjs: better-sqlite3 adapter integration.
- tests/app-rollback-guard.test.mjs: rewritten — transient BUSY recovered
  by coordinator, persistent BUSY skips file, begin-busy aborts build,
  live-transaction aborts build, phantom affectedSessionIds prevented.
- tests/daemon-arbitration.test.mjs: skill defers to fresh app heartbeat,
  builds when heartbeat is stale.
- tests/app-indexer-service.test.mjs: new cases for deferred-retry
  scheduling and immediate heartbeat on start.
- app/tests/electron-concurrency.mjs + child: dual-child IPC structure for
  real better-sqlite3 contention (holder acquires lock → build child starts
  → delayed release → result collected; persistent contention bounded).

ADR-0006 updated to reflect the implemented design.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/adr/0006-write-transaction-rollback-and-concurrency.md

@@ -1,76 +1,75 @@
 # Write-transaction rollback safety and SQLite concurrency
 
 **Context.** The app surfaced `Obelisk index build failed: cannot rollback - no
-transaction is active`. That message is a *secondary* error: SQLite auto-rolls
-back certain failures (notably `SQLITE_BUSY` / `SQLITE_BUSY_SNAPSHOT`, also disk
-full), after which the per-file build loop's unguarded `db.exec('ROLLBACK')` in
-its `catch` threw over the real error and aborted the whole build instead of
-skipping just the offending file. The underlying trigger is concurrency: the app
-runs a daemon indexer, manual rebuilds, and read queries against one WAL
-database, and the skill's passive-pull build can write the same database from a
-separate process.
+transaction is active`. That text was a secondary cleanup failure. SQLite had
+already ended the transaction, then the catch block's unguarded `ROLLBACK`
+threw over the primary exception and turned a skippable per-file failure into a
+whole-build failure. The masked exception was not preserved, so contention
+(`SQLITE_BUSY` / `SQLITE_BUSY_SNAPSHOT`) is the leading explanation rather than
+a proven historical fact. It is plausible because daemon builds, manual
+rebuilds, skill passive-pull indexing, heartbeat writes, and reads share one WAL
+database.
 
-`busy_timeout` is **not** the root-cause fix and must not be treated as one.
-better-sqlite3's constructor already defaults `timeout` to 5000ms, so the app hit
-`SQLITE_BUSY` *despite* a 5s wait — which points at `SQLITE_BUSY_SNAPSHOT` from
-deferred (read-then-write) transactions, a snapshot conflict that `busy_timeout`
-does not wait on. Only `BEGIN IMMEDIATE` plus whole-transaction retry addresses
-that.
+`busy_timeout` alone is not a correctness fix. In particular,
+`SQLITE_BUSY_SNAPSHOT` is not made safe by waiting longer, and retrying only the
+failed statement can replay part of a transaction.
 
-**Decision.** Split the work into a cheap stopgap now and a correctness/
-concurrency fix later.
+**Decision.** Use one transaction primitive plus two explicit coordination
+layers.
 
-*Stopgap (done).* Both indexers use a guarded rollback (`safeRollback`): a
-cleanup rollback swallows only its own error and never masks the primary one.
-Per-file failures are logged and the build continues; the finalize failure still
-propagates. The skill's connection (`node:sqlite`, which has **no** default busy
-timeout) gets an explicit `PRAGMA busy_timeout = 5000`; the app adds no such
-pragma because better-sqlite3 already defaults to 5000ms — adding it there was
-redundant and removed, precisely so nobody reads it as "the fix".
+- `scripts/tx.ts` owns the binding-agnostic `runWriteTransaction(db, work)`.
+  Adapters expose transaction state from better-sqlite3's `inTransaction` and
+  node:sqlite's `isTransaction`. The primitive performs `BEGIN IMMEDIATE`, runs
+  `work` exactly once, commits, and attempts rollback only when the binding says
+  a transaction is active or its state is unknown. Cleanup never masks the
+  primary exception. Diagnostics record phase, SQLite code, rollback outcome,
+  transaction state, label, and attempts.
+- Retry is an upper-layer policy in `scripts/write-coordinator.ts`, never hidden
+  inside the transaction primitive. Only an idempotent whole transaction that
+  failed during work/commit with `SQLITE_BUSY*` and is confirmed inactive may be
+  retried. The default is three attempts within a one-second budget with short
+  backoff. BEGIN contention is deferred to the build scheduler; an active or
+  unknown post-error transaction aborts the build.
+- Per-file failures remain warnings and are reported in `skippedFiles`; finalize
+  failures propagate. `affectedSessionIds` is updated only after the relevant
+  commit. Force cleanup is one atomic, retryable transaction, and finalize is
+  likewise retried as a complete idempotent transaction.
+- A fresh `__app_heartbeat__` is policy ownership: while it is fresh, the skill
+  opens no write connection and performs no migration, schema setup, checkpoint,
+  index build, or `attune`. `__app_last_successful_build__` remains an
+  observability/freshness marker and is not required for ownership. The skill
+  checks ownership again after acquiring the hard lease to close the TOCTOU
+  window. Search/query connections are read-only.
+- A dedicated `.obelisk/writer.lock.sqlite` provides the cross-process safety
+  mutex on every platform. Acquisition is `BEGIN IMMEDIATE` with non-blocking or
+  bounded waiting; release is idempotent. App builds and heartbeats, skill builds
+  and attune, app schema/legacy migrations and memory mutations, and manual
+  rebuild all participate. Manual rebuild's main process owns the lease across
+  worker build, atomic target replacement, and database reopen; the worker uses
+  the explicit `caller-held` mode.
+- The app's in-process indexer service permits one build at a time. A lease
+  deferral retains changed paths and schedules a short retry without announcing
+  a successful build. Service start publishes the ownership heartbeat
+  immediately, then refreshes it periodically.
+- Index-writer and skill read connections use an explicit 250 ms SQLite busy
+  timeout inside the larger bounded coordination budget. The long-lived app
+  query connection retains a 5 s timeout; heartbeat is deliberately non-blocking
+  (`0 ms`) so it never stalls the Electron main thread. Builds use
+  `BEGIN IMMEDIATE`. Routine checkpointing is `PASSIVE`; blocking `TRUNCATE` is
+  reserved for explicit maintenance.
 
-*Planned full fix (deferred, two phases).*
+**Verification.** Fast tests inject auto-rollback and BUSY failures to prove the
+primary error is preserved, retry replays the whole transaction, persistent
+per-file failure is skipped, force cleanup is atomic, and affected-session state
+is commit-aware. The Electron harness uses real Electron-ABI better-sqlite3 and
+two child processes: one holds the SQLite writer lease until signalled, while
+the other runs synchronous `buildIndex`. It verifies both release-within-budget
+success and bounded `writer_busy` deferral. Separate arbitration tests prove a
+heartbeat-only daemon marker keeps query and attune paths read-only.
 
-Phase 1 — transaction semantics:
-- A shared, binding-agnostic `runWriteTransaction(db, work)` (same injection
-  model as `persist`): BEGIN → work → COMMIT, guarded rollback on failure,
-  original exception always preserved. Both app and skill call it, so their
-  behaviour is identical.
-- Per-file callers catch and continue; finalize does **not** swallow — a finalize
-  failure fails the build (the skill currently only warns; that changes).
-- In-memory state such as `affectedSessionIds` is updated **only after** a
-  successful COMMIT (today the app adds the id before COMMIT, so a failed commit
-  can report a wrong affected set).
-- Structured diagnostics: `phase` (begin/file-write/commit/finalize/checkpoint),
-  SQLite `code`, file path, whether rollback succeeded, whether a txn is still
-  active; surface the skipped-file count in the build result rather than only
-  `console.warn` (no silent coverage gaps).
-
-Phase 2 — concurrency:
-- A stable concurrency test against real Electron `better-sqlite3` (daemon +
-  rebuild + skill writer), not an injected fake BUSY. Note this cannot run under
-  standalone `node --test` (better-sqlite3 is Electron-ABI); it needs an
-  Electron-hosted harness. The fast injected-BUSY unit test is kept as a
-  lower-level guard for `runWriteTransaction`, alongside it.
-- Serialize all index writes through a single writer: an in-process
-  `BuildCoordinator`, plus the existing cross-process daemon arbitration
-  (`__app_heartbeat__` markers) so the skill defers to a live daemon. Single
-  writer = both layers together.
-- `BEGIN IMMEDIATE` to take the write lock up front and avoid
-  `SQLITE_BUSY_SNAPSHOT` on read-then-write.
-- Bounded, short-backoff retry of the **whole** transaction (not the single
-  failed statement) on `SQLITE_BUSY` / `SQLITE_BUSY_SNAPSHOT`. This relies on the
-  per-file work being idempotent (upsert/replace + delete-session cascade), which
-  it is — an invariant the retry depends on.
-- Stop forcing `wal_checkpoint(TRUNCATE)` after every build; prefer a `PASSIVE`
-  checkpoint on idle, with `TRUNCATE` reserved for maintenance/exit.
-- Centralized connection configuration (explicit 5000ms for `node:sqlite` — a
-  real behaviour change; explicit for better-sqlite3 too, acknowledging its
-  default is already 5000ms).
-
-**Consequences.** With only the stopgap in place, contention no longer crashes a
-build, but a conflicting file is *skipped* (non-fatal) and its data is briefly
-missing until the next build — acceptable as a stopgap because the guard keeps
-the index consistent and self-healing. The true fix is tracked as the two-phase
-plan above. A future contributor should not "fix" the concurrency by bumping or
-re-adding `busy_timeout`; the direction is a shared transaction module, single
-writer, `BEGIN IMMEDIATE`, and whole-transaction retry.
+**Consequences.** Heartbeat and lease have deliberately different jobs: the
+heartbeat decides who should write, while the lease guarantees writers cannot
+overlap when policy information races or is stale. A single bad transcript can
+still be skipped so the index self-heals on a later build; structural/finalize
+failures remain visible. Longer timeouts must not replace the transaction and
+ownership rules recorded here.