obelisk/docs/adr/0001-parse-core-and-persist-layers.md

Indexing is a registry of pure provider adapters over one shared persist layer

Revised 2026-07-08. The first draft framed the parse layer as a single "parse core" with "two thin persist layers, one per binding." That was wrong on both axes and is corrected below: the parse layer is a registry of per-provider adapters (driven by the multi-provider roadmap), and there is one shared persist layer, not one per binding.

Context. Obelisk had two divergent full indexers — the former scripts/indexer.mjs (node:sqlite, the former skill-embedded runtime) and app/indexer.js (better-sqlite3, Electron app) — that duplicated the same Claude and Codex JSONL parsing and had silently diverged in write semantics (INSERT OR REPLACE vs ON CONFLICT DO UPDATE, message-count accumulation). Two forces shape the fix: (1) the roadmap will add more transcript sources — opencode, pi, and others — so the parse layer must be pluggable, not one monolith; (2) node:sqlite and better-sqlite3 share the same prepare/run/get/all API, so persistence is already nearly binding-agnostic and does not need a per-binding implementation.

Decision. Split indexing along two orthogonal axes.

• Provider axis — a registry of pure adapters. Each source (Claude Code, Codex, Kimi Code, Pi, …) is a provider adapter implementing one complete boundary: serializable descriptor metadata, watchRoots(root), discover(context) → IndexUnit[], parse(unit, cursor) → Iterable<Record>, and raw(lookup). An IndexUnit is deliberately not a file abstraction: Kimi uses one session directory containing state plus multiple agent wire logs. An adapter is pure: it emits normalized records and never touches a database. Discovery receives a read-only view of the provider's already-indexed session paths. A full-reparse adapter can therefore attach retractSessionIds to a replacement or tombstone unit without querying SQLite itself. Retraction and replacement records commit in the same unit transaction, so a failed parse preserves the last complete snapshot. Provider identity may therefore be richer than a wire-level ID. Pi, whose explicit session IDs are project-local, deterministically namespaces the header ID by the normalized header cwd; source paths remain provenance rather than identity, so copying or moving a transcript does not rename the session. Adding a source means adding one adapter and registering it; nothing else changes. parse exposes an iterator as its common interface and streams when the provider semantics permit it. An adapter may buffer one complete IndexUnit when correctness requires whole-unit semantics — for example, Codex duplicate reconciliation, Kimi context.undo / context.clear replay, or Pi tree projection. Each adapter maps its own resume/change semantics onto an opaque cursor stored exactly in index_state.cursor; mtime and lines_processed remain compatibility/index-inspection columns. A provider-wide replay is a destructive snapshot boundary. Discovery reports any source location it could not enumerate; destructive rebuilds fail closed when such a report exists. Cleanup, parsing, persistence, FTS rebuild and marker publication commit in one transaction, so a parse failure cannot replace the last-good index. The emitted TranscriptRecord stream is also the input to provider-independent session detail assembly; see ADR-0007.
• Persist axis — one shared orchestration. A single provider-agnostic, binding-agnostic layer consumes records from any adapter and writes them: incremental index_state bookkeeping, FTS maintenance, and the canonical upsert (ON CONFLICT(uuid) DO UPDATE) write semantics reconciled from the drift on 2026-07-08. The database handle is injected, so node:sqlite (CLI) and better-sqlite3 (app) run the same code — there is no per-binding persist layer.

Two indexing modes share all of the above and differ only in trigger: daemon mode (the app, and potentially a future CLI daemon, watches and keeps the index fresh) and passive pull mode (a CLI command indexes on invocation when no daemon is active). They never write concurrently — passive mode detects a fresh daemon via heartbeat markers in index_state (daemon arbitration).

Consequences. Golden tests anchor on each adapter's parse output (feed fixture JSONL, assert the yielded record sequence) — independent of binding and persistence. The app's richer changed-path discovery becomes a discover strategy injected into the shared orchestration, not a fork of it. The Electron main process migrates to ESM (ADR-0003) to import the shared core. The real work is disentangling the currently interleaved parse-and-write inside indexJsonl / indexCodexJsonl into (pure adapter parse) + (shared persist).

The normalized TranscriptRecord union is the stable center of the design, and SQLite is one serialization adapter for it. Provider-only concepts are either projected lossily into that language or ignored. A genuinely shared concept may extend the canonical language by explicit decision: summary-generating model calls, for example, carry the same normalized input/output usage as ordinary messages. The registry, not provider switches, drives both indexers, watcher roots, persisted source roots, source catalog/UI labels and colors, and raw-record routing. Adding Pi therefore adds no Pi-specific branch to the shared schema, persist layer, indexers, settings, query API, or renderer; the provenance and summary-usage additions are provider-neutral contracts.