diff --git a/docs/adr/0001-parse-core-and-persist-layers.md b/docs/adr/0001-parse-core-and-persist-layers.md index aabc2cc..79efdf1 100644 --- a/docs/adr/0001-parse-core-and-persist-layers.md +++ b/docs/adr/0001-parse-core-and-persist-layers.md @@ -1,31 +1,48 @@ -# Indexing splits into a shared parse core and binding-specific persist layers +# Indexing is a registry of pure provider adapters over one shared persist layer + +> Revised 2026-07-08. The first draft framed the parse layer as a single "parse +> core" with "two thin persist layers, one per binding." That was wrong on both +> axes and is corrected below: the parse layer is a *registry of per-provider +> adapters* (driven by the multi-provider roadmap), and there is *one* shared +> persist layer, not one per binding. **Context.** Obelisk had two divergent full indexers — `scripts/indexer.mjs` -(791 lines, `node:sqlite`, for the skill/runtime) and `app/indexer.js` (1170 -lines, `better-sqlite3`, for the Electron app) — both parsing the same Claude and -Codex JSONL into the same schema. Keeping two implementations contradicts the -"single source of truth / infra" goal, but forcing one SQLite binding is also -bad: `node:sqlite` is zero-native-dep and ideal for a clone-and-run skill -artifact, while `better-sqlite3` is the battle-tested choice inside Electron. +(`node:sqlite`, skill/runtime) and `app/indexer.js` (`better-sqlite3`, Electron +app) — that duplicated the same Claude and Codex JSONL parsing and had silently +diverged in write semantics (`INSERT OR REPLACE` vs `ON CONFLICT DO UPDATE`, +message-count accumulation). Two forces shape the fix: (1) the roadmap will add +more transcript sources — opencode, pi, and others — so the parse layer must be +*pluggable*, not one monolith; (2) `node:sqlite` and `better-sqlite3` share the +same `prepare/run/get/all` API, so persistence is *already* nearly +binding-agnostic and does not need a per-binding implementation. -**Decision.** Unify at the logic layer, not the binding layer. Extract a pure -**parse core** (`jsonl -> records`) with no database dependency, and keep two -thin **persist layers** (`node:sqlite` for skill/CLI, `better-sqlite3` for the -app) that consume the same records. The parse core is a **streaming iterator** -(`parseJsonl(file, fromLine)` yielding records), not a batched `records[]`, to -preserve the existing memory-friendly line-by-line indexing and the -`lines_processed` resume-from-line semantics recorded in `index_state`. +**Decision.** Split indexing along two orthogonal axes. -**Two indexing modes** share this parse core and differ only in trigger: -**daemon indexing mode** (app/CLI watches and keeps the index fresh) and -**passive pull mode** (skill indexes on invocation when no daemon is active). -They never write concurrently because the passive mode detects a fresh daemon via -heartbeat markers in `index_state` (**daemon arbitration**) and skips its own -indexing. +- **Provider axis — a registry of pure adapters.** Each source (claude, codex, + later opencode, pi, …) is a provider adapter implementing + `discover(opts) → files` and `parse(file, fromLine) → Iterable`. An + adapter is *pure*: it emits normalized records and never touches a database. + Adding a source means adding one adapter and registering it; nothing else + changes. `parse` is a streaming iterator, preserving memory-friendly indexing + and the `lines_processed` resume-from-line semantics in `index_state`. +- **Persist axis — one shared orchestration.** A single provider-agnostic, + binding-agnostic layer consumes records from any adapter and writes them: + incremental `index_state` bookkeeping, FTS maintenance, and the canonical + **upsert** (`ON CONFLICT(uuid) DO UPDATE`) write semantics reconciled from the + drift on 2026-07-08. The database handle is *injected*, so `node:sqlite` + (skill/CLI) and `better-sqlite3` (app) run the same code — there is no + per-binding persist layer. -**Consequences.** Golden tests anchor on the parse core: feed a fixture JSONL, -assert the yielded record sequence — independent of binding. The refactor's real -work is disentangling parse from persist inside the current indexers, where -prepared-statement writes are today interleaved into the parse loop. The app's -richer incremental-discovery logic must be folded *into* the shared parse core, -not dropped. +**Two indexing modes** share all of the above and differ only in trigger: +**daemon mode** (app/CLI watches and keeps the index fresh) and **passive pull +mode** (skill indexes on invocation when no daemon is active). They never write +concurrently — passive mode detects a fresh daemon via heartbeat markers in +`index_state` (**daemon arbitration**). + +**Consequences.** Golden tests anchor on each adapter's `parse` output (feed +fixture JSONL, assert the yielded record sequence) — independent of binding and +persistence. The app's richer changed-path discovery becomes a `discover` +strategy injected into the shared orchestration, not a fork of it. The Electron +main process migrates to ESM (ADR-0003) to import the shared core. The real work +is disentangling the currently interleaved parse-and-write inside `indexJsonl` / +`indexCodexJsonl` into (pure adapter parse) + (shared persist). diff --git a/scripts/providers/types.ts b/scripts/providers/types.ts new file mode 100644 index 0000000..655c9e6 --- /dev/null +++ b/scripts/providers/types.ts @@ -0,0 +1,206 @@ +// Phase 5 target contract (see docs/adr/0001). +// +// The indexing layer splits along two orthogonal axes: +// - Provider axis: pure per-source adapters (claude, codex, later opencode, +// pi, …) that discover their own work and parse it into records. A source is +// NOT assumed to be a single JSONL file — an adapter may read a SQLite store, +// a directory tree, etc. So discovery, change-detection, and resume cursoring +// are all adapter-owned and format-specific. +// - Persist axis: one shared, provider- and binding-agnostic orchestration +// that consumes the records and writes them (index_state, FTS, upsert). +// +// This file defines only the shapes crossing that boundary. Record fields mirror +// the columns in scripts/schema.sql; keep them in sync. Types only — no runtime +// code — so consumers must import with `import type`. + +// Opaque per-unit resume/watermark token. The orchestration stores it verbatim +// (in index_state) and hands it back on the next run; ONLY the adapter that +// produced it interprets it. A JSONL adapter might encode `"${mtime}:${lines}"`; +// a SQLite-backed adapter might encode a rowid or timestamp high-water mark. +export type Cursor = string | null; + +// One unit of work an adapter has discovered. It is not necessarily a file: for +// a file-based source `key` is the path; for a DB-backed source it might be +// `"${dbPath}#${internalId}"`. `meta` carries adapter-private data (e.g. the +// resolved file path or source handle) that the orchestration passes back to +// parse() untouched. +export interface IndexUnit { + /** Stable identity used as the index_state cursor key. */ + key: string; + /** Session id this unit indexes into. */ + sessionId: string; + /** Project slug (dash-encoded path), when the source exposes one. */ + project?: string; + /** Set for subagent transcripts, whose messages carry an agent id. */ + isSubagent?: boolean; + agentId?: string; + /** Adapter-private payload, opaque to the orchestration. */ + meta?: unknown; +} + +/** Context the orchestration provides to discovery. */ +export interface DiscoverContext { + /** Look up the cursor persisted for a unit key on a previous run. */ + lastCursor(key: string): Cursor; + /** When set (daemon changed-path mode), restrict discovery to these paths. */ + changedPaths?: string[]; +} + +/** Discriminated union of everything an adapter's parse can emit. Each record + * kind maps to one schema table (see scripts/schema.sql); `delete-session` is + * the exception — a retraction op, not a table. Sources without a table + * (history.jsonl, codex session_index.jsonl) are not records: adapters fold them + * into the SessionRecord they already emit. */ +export type IndexRecord = + | SessionRecord + | MessageRecord + | ToolCallRecord + | ToolResultRecord + | SummaryRecord + | SubagentRecord + | WorkflowRecord + | WorkflowAgentRecord + | DeleteSessionRecord; + +export interface MessageRecord { + kind: 'message'; + uuid: string; + session_id: string; + type: string; + parent_uuid: string | null; + timestamp: string | null; + role: string | null; + text: string | null; + content_type: string | null; + is_meta: 0 | 1; + model: string | null; + is_sidechain: 0 | 1; + agent_id: string | null; + input_tokens: number | null; + output_tokens: number | null; + cwd: string | null; + skill: string | null; + source: string; +} + +export interface ToolCallRecord { + kind: 'tool_call'; + id: string; + message_uuid: string; + session_id: string; + name: string; + input_json: string; + file_path: string | null; +} + +export interface ToolResultRecord { + kind: 'tool_result'; + tool_use_id: string; + message_uuid: string; + session_id: string; + content: string; + file_path: string | null; + is_error: 0 | 1; +} + +export interface SummaryRecord { + kind: 'summary'; + id: string; + session_id: string; + timestamp: string | null; + source: string; + content: string; +} + +export interface SubagentRecord { + kind: 'subagent'; + agent_id: string; + session_id: string; + parent_tool_use_id: string | null; + agent_type: string | null; + description: string | null; + duration_ms: number | null; + total_tokens: number | null; +} + +// A workflow run. `agent_count` is intentionally absent: it is a derived +// aggregate (COUNT of workflow_agents for this run) that persist computes, since +// the agents may be indexed on different runs than the workflow metadata. +export interface WorkflowRecord { + kind: 'workflow'; + run_id: string; + session_id: string; + task_id: string | null; + script: string | null; + result_json: string | null; + timestamp: string | null; + duration_ms: number | null; + total_tokens: number | null; + status: string | null; + workflow_name: string | null; +} + +// One workflow agent. A single row is contributed by TWO independent units, in +// any order: the subagent .meta.json unit fills agent_type/description; the +// workflow run json unit fills phase/label/model/state/duration_ms/tokens/ +// tool_calls. So every optional field a unit does not know is omitted, and +// persist merges column-wise (ON CONFLICT(agent_id) DO UPDATE SET +// col=COALESCE(excluded.col, col)). All contributors MUST use the same unified +// agent_id key so the merge lands on the same row. +export interface WorkflowAgentRecord { + kind: 'workflow_agent'; + agent_id: string; + run_id: string; + session_id: string; + agent_type?: string | null; + description?: string | null; + phase?: string | null; + label?: string | null; + model?: string | null; + state?: string | null; + duration_ms?: number | null; + tokens?: number | null; + tool_calls?: number | null; +} + +// Retraction op (not a table). The adapter emits this when a previously-indexed +// session must be removed — e.g. a Codex guardian/auto-review thread. Persist +// executes the cascade delete across all tables for that session. +export interface DeleteSessionRecord { + kind: 'delete-session'; + sessionId: string; +} + +// Session-level aggregate. Emitted once, after the unit's records are produced, +// because started_at/ended_at/message_count are computed across the stream. +// title/ended_at may be enriched by the adapter from source-specific auxiliary +// files (claude history.jsonl, codex session_index.jsonl); persist upserts with +// fill-if-null (COALESCE) so those never clobber a value already present. +// project_path is NOT set here — the orchestration's global pass derives it from +// persisted message cwds (refreshSessionProjectPaths). +export interface SessionRecord { + kind: 'session'; + id: string; + title: string | null; + project: string | null; + started_at: string | null; + ended_at: string | null; + git_branch: string | null; + version: string | null; + message_count: number; + jsonl_path: string; + source: string; +} + +// A transcript source. Pure: it never touches the Obelisk database. It owns its +// own discovery, change-detection, and resume cursoring, because those are +// format-specific (file mtime, DB watermark, …). `parse` is a generator that +// yields records for one unit and RETURNS the new cursor to persist. +export interface Provider { + /** Stable source tag stored on rows, e.g. 'claude' | 'codex'. */ + readonly name: string; + /** Discover units needing (re)indexing, using stored cursors to detect change. */ + discover(ctx: DiscoverContext): IndexUnit[]; + /** Stream records for one unit resuming from `cursor`; return the new cursor. */ + parse(unit: IndexUnit, cursor: Cursor): Generator; +}