feat(indexer): define source-agnostic Provider + IndexRecord contract (Phase 5a)

Provider adapters own discovery/change-detection/cursoring and emit records;
one shared binding-agnostic persist consumes them. Records map 1:1 to schema
tables. Revises ADR-0001 to the provider-registry model.
This commit is contained in:
tommy0103
2026-07-08 18:10:04 +08:00
parent 51597f42f0
commit 338dae413d
2 changed files with 249 additions and 26 deletions
+43 -26
View File
@@ -1,31 +1,48 @@
# Indexing splits into a shared parse core and binding-specific persist layers
# Indexing is a registry of pure provider adapters over one shared persist layer
> Revised 2026-07-08. The first draft framed the parse layer as a single "parse
> core" with "two thin persist layers, one per binding." That was wrong on both
> axes and is corrected below: the parse layer is a *registry of per-provider
> adapters* (driven by the multi-provider roadmap), and there is *one* shared
> persist layer, not one per binding.
**Context.** Obelisk had two divergent full indexers — `scripts/indexer.mjs`
(791 lines, `node:sqlite`, for the skill/runtime) and `app/indexer.js` (1170
lines, `better-sqlite3`, for the Electron app) — both parsing the same Claude and
Codex JSONL into the same schema. Keeping two implementations contradicts the
"single source of truth / infra" goal, but forcing one SQLite binding is also
bad: `node:sqlite` is zero-native-dep and ideal for a clone-and-run skill
artifact, while `better-sqlite3` is the battle-tested choice inside Electron.
(`node:sqlite`, skill/runtime) and `app/indexer.js` (`better-sqlite3`, Electron
app) — that duplicated the same Claude and Codex JSONL parsing and had silently
diverged in write semantics (`INSERT OR REPLACE` vs `ON CONFLICT DO UPDATE`,
message-count accumulation). Two forces shape the fix: (1) the roadmap will add
more transcript sources — opencode, pi, and others — so the parse layer must be
*pluggable*, not one monolith; (2) `node:sqlite` and `better-sqlite3` share the
same `prepare/run/get/all` API, so persistence is *already* nearly
binding-agnostic and does not need a per-binding implementation.
**Decision.** Unify at the logic layer, not the binding layer. Extract a pure
**parse core** (`jsonl -> records`) with no database dependency, and keep two
thin **persist layers** (`node:sqlite` for skill/CLI, `better-sqlite3` for the
app) that consume the same records. The parse core is a **streaming iterator**
(`parseJsonl(file, fromLine)` yielding records), not a batched `records[]`, to
preserve the existing memory-friendly line-by-line indexing and the
`lines_processed` resume-from-line semantics recorded in `index_state`.
**Decision.** Split indexing along two orthogonal axes.
**Two indexing modes** share this parse core and differ only in trigger:
**daemon indexing mode** (app/CLI watches and keeps the index fresh) and
**passive pull mode** (skill indexes on invocation when no daemon is active).
They never write concurrently because the passive mode detects a fresh daemon via
heartbeat markers in `index_state` (**daemon arbitration**) and skips its own
indexing.
- **Provider axis — a registry of pure adapters.** Each source (claude, codex,
later opencode, pi, …) is a provider adapter implementing
`discover(opts) → files` and `parse(file, fromLine) → Iterable<Record>`. An
adapter is *pure*: it emits normalized records and never touches a database.
Adding a source means adding one adapter and registering it; nothing else
changes. `parse` is a streaming iterator, preserving memory-friendly indexing
and the `lines_processed` resume-from-line semantics in `index_state`.
- **Persist axis — one shared orchestration.** A single provider-agnostic,
binding-agnostic layer consumes records from any adapter and writes them:
incremental `index_state` bookkeeping, FTS maintenance, and the canonical
**upsert** (`ON CONFLICT(uuid) DO UPDATE`) write semantics reconciled from the
drift on 2026-07-08. The database handle is *injected*, so `node:sqlite`
(skill/CLI) and `better-sqlite3` (app) run the same code — there is no
per-binding persist layer.
**Consequences.** Golden tests anchor on the parse core: feed a fixture JSONL,
assert the yielded record sequence — independent of binding. The refactor's real
work is disentangling parse from persist inside the current indexers, where
prepared-statement writes are today interleaved into the parse loop. The app's
richer incremental-discovery logic must be folded *into* the shared parse core,
not dropped.
**Two indexing modes** share all of the above and differ only in trigger:
**daemon mode** (app/CLI watches and keeps the index fresh) and **passive pull
mode** (skill indexes on invocation when no daemon is active). They never write
concurrently — passive mode detects a fresh daemon via heartbeat markers in
`index_state` (**daemon arbitration**).
**Consequences.** Golden tests anchor on each adapter's `parse` output (feed
fixture JSONL, assert the yielded record sequence) — independent of binding and
persistence. The app's richer changed-path discovery becomes a `discover`
strategy injected into the shared orchestration, not a fork of it. The Electron
main process migrates to ESM (ADR-0003) to import the shared core. The real work
is disentangling the currently interleaved parse-and-write inside `indexJsonl` /
`indexCodexJsonl` into (pure adapter parse) + (shared persist).