feat(indexer): define source-agnostic Provider + IndexRecord contract (Phase 5a)
Provider adapters own discovery/change-detection/cursoring and emit records; one shared binding-agnostic persist consumes them. Records map 1:1 to schema tables. Revises ADR-0001 to the provider-registry model.
This commit is contained in:
tommy0103
@@ -1,31 +1,48 @@
|
|||||||
# Indexing splits into a shared parse core and binding-specific persist layers
|
# Indexing is a registry of pure provider adapters over one shared persist layer
|
||||||
|
|
||||||
|
> Revised 2026-07-08. The first draft framed the parse layer as a single "parse
|
||||||
|
> core" with "two thin persist layers, one per binding." That was wrong on both
|
||||||
|
> axes and is corrected below: the parse layer is a *registry of per-provider
|
||||||
|
> adapters* (driven by the multi-provider roadmap), and there is *one* shared
|
||||||
|
> persist layer, not one per binding.
|
||||||
|
|
||||||
**Context.** Obelisk had two divergent full indexers — `scripts/indexer.mjs`
|
**Context.** Obelisk had two divergent full indexers — `scripts/indexer.mjs`
|
||||||
(791 lines, `node:sqlite`, for the skill/runtime) and `app/indexer.js` (1170
|
(`node:sqlite`, skill/runtime) and `app/indexer.js` (`better-sqlite3`, Electron
|
||||||
lines, `better-sqlite3`, for the Electron app) — both parsing the same Claude and
|
app) — that duplicated the same Claude and Codex JSONL parsing and had silently
|
||||||
Codex JSONL into the same schema. Keeping two implementations contradicts the
|
diverged in write semantics (`INSERT OR REPLACE` vs `ON CONFLICT DO UPDATE`,
|
||||||
"single source of truth / infra" goal, but forcing one SQLite binding is also
|
message-count accumulation). Two forces shape the fix: (1) the roadmap will add
|
||||||
bad: `node:sqlite` is zero-native-dep and ideal for a clone-and-run skill
|
more transcript sources — opencode, pi, and others — so the parse layer must be
|
||||||
artifact, while `better-sqlite3` is the battle-tested choice inside Electron.
|
*pluggable*, not one monolith; (2) `node:sqlite` and `better-sqlite3` share the
|
||||||
|
same `prepare/run/get/all` API, so persistence is *already* nearly
|
||||||
|
binding-agnostic and does not need a per-binding implementation.
|
||||||
|
|
||||||
**Decision.** Unify at the logic layer, not the binding layer. Extract a pure
|
**Decision.** Split indexing along two orthogonal axes.
|
||||||
**parse core** (`jsonl -> records`) with no database dependency, and keep two
|
|
||||||
thin **persist layers** (`node:sqlite` for skill/CLI, `better-sqlite3` for the
|
|
||||||
app) that consume the same records. The parse core is a **streaming iterator**
|
|
||||||
(`parseJsonl(file, fromLine)` yielding records), not a batched `records[]`, to
|
|
||||||
preserve the existing memory-friendly line-by-line indexing and the
|
|
||||||
`lines_processed` resume-from-line semantics recorded in `index_state`.
|
|
||||||
|
|
||||||
**Two indexing modes** share this parse core and differ only in trigger:
|
- **Provider axis — a registry of pure adapters.** Each source (claude, codex,
|
||||||
**daemon indexing mode** (app/CLI watches and keeps the index fresh) and
|
later opencode, pi, …) is a provider adapter implementing
|
||||||
**passive pull mode** (skill indexes on invocation when no daemon is active).
|
`discover(opts) → files` and `parse(file, fromLine) → Iterable<Record>`. An
|
||||||
They never write concurrently because the passive mode detects a fresh daemon via
|
adapter is *pure*: it emits normalized records and never touches a database.
|
||||||
heartbeat markers in `index_state` (**daemon arbitration**) and skips its own
|
Adding a source means adding one adapter and registering it; nothing else
|
||||||
indexing.
|
changes. `parse` is a streaming iterator, preserving memory-friendly indexing
|
||||||
|
and the `lines_processed` resume-from-line semantics in `index_state`.
|
||||||
|
- **Persist axis — one shared orchestration.** A single provider-agnostic,
|
||||||
|
binding-agnostic layer consumes records from any adapter and writes them:
|
||||||
|
incremental `index_state` bookkeeping, FTS maintenance, and the canonical
|
||||||
|
**upsert** (`ON CONFLICT(uuid) DO UPDATE`) write semantics reconciled from the
|
||||||
|
drift on 2026-07-08. The database handle is *injected*, so `node:sqlite`
|
||||||
|
(skill/CLI) and `better-sqlite3` (app) run the same code — there is no
|
||||||
|
per-binding persist layer.
|
||||||
|
|
||||||
**Consequences.** Golden tests anchor on the parse core: feed a fixture JSONL,
|
**Two indexing modes** share all of the above and differ only in trigger:
|
||||||
assert the yielded record sequence — independent of binding. The refactor's real
|
**daemon mode** (app/CLI watches and keeps the index fresh) and **passive pull
|
||||||
work is disentangling parse from persist inside the current indexers, where
|
mode** (skill indexes on invocation when no daemon is active). They never write
|
||||||
prepared-statement writes are today interleaved into the parse loop. The app's
|
concurrently — passive mode detects a fresh daemon via heartbeat markers in
|
||||||
richer incremental-discovery logic must be folded *into* the shared parse core,
|
`index_state` (**daemon arbitration**).
|
||||||
not dropped.
|
|
||||||
|
**Consequences.** Golden tests anchor on each adapter's `parse` output (feed
|
||||||
|
fixture JSONL, assert the yielded record sequence) — independent of binding and
|
||||||
|
persistence. The app's richer changed-path discovery becomes a `discover`
|
||||||
|
strategy injected into the shared orchestration, not a fork of it. The Electron
|
||||||
|
main process migrates to ESM (ADR-0003) to import the shared core. The real work
|
||||||
|
is disentangling the currently interleaved parse-and-write inside `indexJsonl` /
|
||||||
|
`indexCodexJsonl` into (pure adapter parse) + (shared persist).
|
||||||
|
|||||||
@@ -0,0 +1,206 @@
|
|||||||
|
// Phase 5 target contract (see docs/adr/0001).
|
||||||
|
//
|
||||||
|
// The indexing layer splits along two orthogonal axes:
|
||||||
|
// - Provider axis: pure per-source adapters (claude, codex, later opencode,
|
||||||
|
// pi, …) that discover their own work and parse it into records. A source is
|
||||||
|
// NOT assumed to be a single JSONL file — an adapter may read a SQLite store,
|
||||||
|
// a directory tree, etc. So discovery, change-detection, and resume cursoring
|
||||||
|
// are all adapter-owned and format-specific.
|
||||||
|
// - Persist axis: one shared, provider- and binding-agnostic orchestration
|
||||||
|
// that consumes the records and writes them (index_state, FTS, upsert).
|
||||||
|
//
|
||||||
|
// This file defines only the shapes crossing that boundary. Record fields mirror
|
||||||
|
// the columns in scripts/schema.sql; keep them in sync. Types only — no runtime
|
||||||
|
// code — so consumers must import with `import type`.
|
||||||
|
|
||||||
|
// Opaque per-unit resume/watermark token. The orchestration stores it verbatim
|
||||||
|
// (in index_state) and hands it back on the next run; ONLY the adapter that
|
||||||
|
// produced it interprets it. A JSONL adapter might encode `"${mtime}:${lines}"`;
|
||||||
|
// a SQLite-backed adapter might encode a rowid or timestamp high-water mark.
|
||||||
|
export type Cursor = string | null;
|
||||||
|
|
||||||
|
// One unit of work an adapter has discovered. It is not necessarily a file: for
|
||||||
|
// a file-based source `key` is the path; for a DB-backed source it might be
|
||||||
|
// `"${dbPath}#${internalId}"`. `meta` carries adapter-private data (e.g. the
|
||||||
|
// resolved file path or source handle) that the orchestration passes back to
|
||||||
|
// parse() untouched.
|
||||||
|
export interface IndexUnit {
|
||||||
|
/** Stable identity used as the index_state cursor key. */
|
||||||
|
key: string;
|
||||||
|
/** Session id this unit indexes into. */
|
||||||
|
sessionId: string;
|
||||||
|
/** Project slug (dash-encoded path), when the source exposes one. */
|
||||||
|
project?: string;
|
||||||
|
/** Set for subagent transcripts, whose messages carry an agent id. */
|
||||||
|
isSubagent?: boolean;
|
||||||
|
agentId?: string;
|
||||||
|
/** Adapter-private payload, opaque to the orchestration. */
|
||||||
|
meta?: unknown;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Context the orchestration provides to discovery. */
|
||||||
|
export interface DiscoverContext {
|
||||||
|
/** Look up the cursor persisted for a unit key on a previous run. */
|
||||||
|
lastCursor(key: string): Cursor;
|
||||||
|
/** When set (daemon changed-path mode), restrict discovery to these paths. */
|
||||||
|
changedPaths?: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Discriminated union of everything an adapter's parse can emit. Each record
|
||||||
|
* kind maps to one schema table (see scripts/schema.sql); `delete-session` is
|
||||||
|
* the exception — a retraction op, not a table. Sources without a table
|
||||||
|
* (history.jsonl, codex session_index.jsonl) are not records: adapters fold them
|
||||||
|
* into the SessionRecord they already emit. */
|
||||||
|
export type IndexRecord =
|
||||||
|
| SessionRecord
|
||||||
|
| MessageRecord
|
||||||
|
| ToolCallRecord
|
||||||
|
| ToolResultRecord
|
||||||
|
| SummaryRecord
|
||||||
|
| SubagentRecord
|
||||||
|
| WorkflowRecord
|
||||||
|
| WorkflowAgentRecord
|
||||||
|
| DeleteSessionRecord;
|
||||||
|
|
||||||
|
export interface MessageRecord {
|
||||||
|
kind: 'message';
|
||||||
|
uuid: string;
|
||||||
|
session_id: string;
|
||||||
|
type: string;
|
||||||
|
parent_uuid: string | null;
|
||||||
|
timestamp: string | null;
|
||||||
|
role: string | null;
|
||||||
|
text: string | null;
|
||||||
|
content_type: string | null;
|
||||||
|
is_meta: 0 | 1;
|
||||||
|
model: string | null;
|
||||||
|
is_sidechain: 0 | 1;
|
||||||
|
agent_id: string | null;
|
||||||
|
input_tokens: number | null;
|
||||||
|
output_tokens: number | null;
|
||||||
|
cwd: string | null;
|
||||||
|
skill: string | null;
|
||||||
|
source: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ToolCallRecord {
|
||||||
|
kind: 'tool_call';
|
||||||
|
id: string;
|
||||||
|
message_uuid: string;
|
||||||
|
session_id: string;
|
||||||
|
name: string;
|
||||||
|
input_json: string;
|
||||||
|
file_path: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ToolResultRecord {
|
||||||
|
kind: 'tool_result';
|
||||||
|
tool_use_id: string;
|
||||||
|
message_uuid: string;
|
||||||
|
session_id: string;
|
||||||
|
content: string;
|
||||||
|
file_path: string | null;
|
||||||
|
is_error: 0 | 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SummaryRecord {
|
||||||
|
kind: 'summary';
|
||||||
|
id: string;
|
||||||
|
session_id: string;
|
||||||
|
timestamp: string | null;
|
||||||
|
source: string;
|
||||||
|
content: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SubagentRecord {
|
||||||
|
kind: 'subagent';
|
||||||
|
agent_id: string;
|
||||||
|
session_id: string;
|
||||||
|
parent_tool_use_id: string | null;
|
||||||
|
agent_type: string | null;
|
||||||
|
description: string | null;
|
||||||
|
duration_ms: number | null;
|
||||||
|
total_tokens: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// A workflow run. `agent_count` is intentionally absent: it is a derived
|
||||||
|
// aggregate (COUNT of workflow_agents for this run) that persist computes, since
|
||||||
|
// the agents may be indexed on different runs than the workflow metadata.
|
||||||
|
export interface WorkflowRecord {
|
||||||
|
kind: 'workflow';
|
||||||
|
run_id: string;
|
||||||
|
session_id: string;
|
||||||
|
task_id: string | null;
|
||||||
|
script: string | null;
|
||||||
|
result_json: string | null;
|
||||||
|
timestamp: string | null;
|
||||||
|
duration_ms: number | null;
|
||||||
|
total_tokens: number | null;
|
||||||
|
status: string | null;
|
||||||
|
workflow_name: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// One workflow agent. A single row is contributed by TWO independent units, in
|
||||||
|
// any order: the subagent .meta.json unit fills agent_type/description; the
|
||||||
|
// workflow run json unit fills phase/label/model/state/duration_ms/tokens/
|
||||||
|
// tool_calls. So every optional field a unit does not know is omitted, and
|
||||||
|
// persist merges column-wise (ON CONFLICT(agent_id) DO UPDATE SET
|
||||||
|
// col=COALESCE(excluded.col, col)). All contributors MUST use the same unified
|
||||||
|
// agent_id key so the merge lands on the same row.
|
||||||
|
export interface WorkflowAgentRecord {
|
||||||
|
kind: 'workflow_agent';
|
||||||
|
agent_id: string;
|
||||||
|
run_id: string;
|
||||||
|
session_id: string;
|
||||||
|
agent_type?: string | null;
|
||||||
|
description?: string | null;
|
||||||
|
phase?: string | null;
|
||||||
|
label?: string | null;
|
||||||
|
model?: string | null;
|
||||||
|
state?: string | null;
|
||||||
|
duration_ms?: number | null;
|
||||||
|
tokens?: number | null;
|
||||||
|
tool_calls?: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Retraction op (not a table). The adapter emits this when a previously-indexed
|
||||||
|
// session must be removed — e.g. a Codex guardian/auto-review thread. Persist
|
||||||
|
// executes the cascade delete across all tables for that session.
|
||||||
|
export interface DeleteSessionRecord {
|
||||||
|
kind: 'delete-session';
|
||||||
|
sessionId: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Session-level aggregate. Emitted once, after the unit's records are produced,
|
||||||
|
// because started_at/ended_at/message_count are computed across the stream.
|
||||||
|
// title/ended_at may be enriched by the adapter from source-specific auxiliary
|
||||||
|
// files (claude history.jsonl, codex session_index.jsonl); persist upserts with
|
||||||
|
// fill-if-null (COALESCE) so those never clobber a value already present.
|
||||||
|
// project_path is NOT set here — the orchestration's global pass derives it from
|
||||||
|
// persisted message cwds (refreshSessionProjectPaths).
|
||||||
|
export interface SessionRecord {
|
||||||
|
kind: 'session';
|
||||||
|
id: string;
|
||||||
|
title: string | null;
|
||||||
|
project: string | null;
|
||||||
|
started_at: string | null;
|
||||||
|
ended_at: string | null;
|
||||||
|
git_branch: string | null;
|
||||||
|
version: string | null;
|
||||||
|
message_count: number;
|
||||||
|
jsonl_path: string;
|
||||||
|
source: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
// A transcript source. Pure: it never touches the Obelisk database. It owns its
|
||||||
|
// own discovery, change-detection, and resume cursoring, because those are
|
||||||
|
// format-specific (file mtime, DB watermark, …). `parse` is a generator that
|
||||||
|
// yields records for one unit and RETURNS the new cursor to persist.
|
||||||
|
export interface Provider {
|
||||||
|
/** Stable source tag stored on rows, e.g. 'claude' | 'codex'. */
|
||||||
|
readonly name: string;
|
||||||
|
/** Discover units needing (re)indexing, using stored cursors to detect change. */
|
||||||
|
discover(ctx: DiscoverContext): IndexUnit[];
|
||||||
|
/** Stream records for one unit resuming from `cursor`; return the new cursor. */
|
||||||
|
parse(unit: IndexUnit, cursor: Cursor): Generator<IndexRecord, Cursor>;
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user