obelisk/docs/adr/0001-parse-core-and-persist-layers.md

Indexing is a registry of pure provider adapters over one shared persist layer

Revised 2026-07-08. The first draft framed the parse layer as a single "parse core" with "two thin persist layers, one per binding." That was wrong on both axes and is corrected below: the parse layer is a registry of per-provider adapters (driven by the multi-provider roadmap), and there is one shared persist layer, not one per binding.

Context. Obelisk had two divergent full indexers — the former scripts/indexer.mjs (node:sqlite, the former skill-embedded runtime) and app/indexer.js (better-sqlite3, Electron app) — that duplicated the same Claude and Codex JSONL parsing and had silently diverged in write semantics (INSERT OR REPLACE vs ON CONFLICT DO UPDATE, message-count accumulation). Two forces shape the fix: (1) the roadmap will add more transcript sources — opencode, pi, and others — so the parse layer must be pluggable, not one monolith; (2) node:sqlite and better-sqlite3 share the same prepare/run/get/all API, so persistence is already nearly binding-agnostic and does not need a per-binding implementation.

Decision. Split indexing along two orthogonal axes.

• Provider axis — a registry of pure adapters. Each source (Claude Code, Codex, Kimi Code, later Pi, …) is a provider adapter implementing one complete boundary: serializable descriptor metadata, watchRoots(root), discover(context) → IndexUnit[], parse(unit, cursor) → Iterable<Record>, and raw(lookup). An IndexUnit is deliberately not a file abstraction: Kimi uses one session directory containing state plus multiple agent wire logs. An adapter is pure: it emits normalized records and never touches a database. Adding a source means adding one adapter and registering it; nothing else changes. parse exposes an iterator as its common interface and streams when the provider semantics permit it. An adapter may buffer one complete IndexUnit when correctness requires whole-unit semantics — for example, Codex duplicate reconciliation or Kimi context.undo / context.clear replay. Each adapter maps its own resume/change semantics onto the existing mtime and lines_processed cursor pair in index_state.
• Persist axis — one shared orchestration. A single provider-agnostic, binding-agnostic layer consumes records from any adapter and writes them: incremental index_state bookkeeping, FTS maintenance, and the canonical upsert (ON CONFLICT(uuid) DO UPDATE) write semantics reconciled from the drift on 2026-07-08. The database handle is injected, so node:sqlite (CLI) and better-sqlite3 (app) run the same code — there is no per-binding persist layer.

Two indexing modes share all of the above and differ only in trigger: daemon mode (the app, and potentially a future CLI daemon, watches and keeps the index fresh) and passive pull mode (a CLI command indexes on invocation when no daemon is active). They never write concurrently — passive mode detects a fresh daemon via heartbeat markers in index_state (daemon arbitration).

Consequences. Golden tests anchor on each adapter's parse output (feed fixture JSONL, assert the yielded record sequence) — independent of binding and persistence. The app's richer changed-path discovery becomes a discover strategy injected into the shared orchestration, not a fork of it. The Electron main process migrates to ESM (ADR-0003) to import the shared core. The real work is disentangling the currently interleaved parse-and-write inside indexJsonl / indexCodexJsonl into (pure adapter parse) + (shared persist).

The SQLite schema and normalized IndexRecord union are the stable center of the design. Provider-only concepts are either projected lossily into that language or ignored; they do not add provider columns or tables. The registry, not provider switches, drives both indexers, watcher roots, persisted source roots, source catalog/UI labels and colors, and raw-record routing. Adding Pi therefore changes the Pi adapter, its registration, and its conformance tests; the shared schema, persist layer, indexers, settings, query API, and renderer do not acquire Pi-specific branches.