feat(core): add first-class Pi session indexing (#23)

Pi cannot be read as another linear JSONL stream. Its history is a tree with
a durable leaf, orphan roots, branch summaries, and two compaction forms, so
the active context is something the format states rather than something line
order implies. The adapter keeps those semantics inside itself and projects
the result into the existing canonical tables.

Sessions are keyed by (normalized header cwd, header id) rather than by path,
because Pi's --session-id lookup is project-local: two projects may reuse an
id, while a move or an identical copy is still one session. Discovery covers
both layouts Pi writes and fingerprints each file by mtime, ctime, size and
inode, so a rewrite that preserves mtime is not read as unchanged.

Abandoned branches are preserved rather than dropped. Visibility becomes
three-state -- visible, inactive, hidden -- and helpers return only visible
rows until includeInactive asks for the superseded path, labeling every row
so a caller knows which it holds. Usage counts all three, because an
abandoned call still spent tokens; message_count reports only the visible
transcript.

A committed MIT-licensed oracle transcribed from Pi 0.83.0 pins the context
algorithms, and a fixed-seed differential runs 512 generated sessions against
it on every test run. Schema changes are additive.
This commit is contained in:
SaladDay
2026-08-04 23:33:01 +08:00
committed by GitHub
parent 1941e64572
commit 2589384e68
63 changed files with 7796 additions and 374 deletions
+30 -11
View File
@@ -21,19 +21,36 @@ binding-agnostic and does not need a per-binding implementation.
**Decision.** Split indexing along two orthogonal axes.
- **Provider axis — a registry of pure adapters.** Each source (Claude Code,
Codex, Kimi Code, later Pi, …) is a provider adapter implementing one complete
Codex, Kimi Code, Pi, …) is a provider adapter implementing one complete
boundary: serializable descriptor metadata, `watchRoots(root)`,
`discover(context) → IndexUnit[]`, `parse(unit, cursor) → Iterable<Record>`,
`discover(context) → IndexUnit[]`,
`parse(unit, cursor) → Iterable<Record>`,
and `raw(lookup)`. An `IndexUnit` is deliberately not a file abstraction: Kimi
uses one session directory containing state plus multiple agent wire logs. An
adapter is *pure*: it emits normalized records and never touches a database.
Discovery receives a read-only view of the provider's already-indexed session
paths. A full-reparse adapter can therefore attach `retractSessionIds` to a
replacement or tombstone unit without querying SQLite itself. Retraction and
replacement records commit in the same unit transaction, so a failed parse
preserves the last complete snapshot.
Provider identity may therefore be richer than a wire-level ID. Pi, whose
explicit session IDs are project-local, deterministically namespaces the
header ID by the normalized header cwd; source paths remain provenance rather
than identity, so copying or moving a transcript does not rename the session.
Adding a source means adding one adapter and registering it; nothing else
changes. `parse` exposes an iterator as its common interface and streams when
the provider semantics permit it. An adapter may buffer one complete
`IndexUnit` when correctness requires whole-unit semantics — for example,
Codex duplicate reconciliation or Kimi `context.undo` / `context.clear`
replay. Each adapter maps its own resume/change semantics onto the existing
`mtime` and `lines_processed` cursor pair in `index_state`. The emitted
Codex duplicate reconciliation, Kimi `context.undo` / `context.clear`
replay, or Pi tree projection. Each adapter maps its own resume/change
semantics onto an opaque cursor stored exactly in `index_state.cursor`;
`mtime` and `lines_processed` remain compatibility/index-inspection columns.
A provider-wide replay is a destructive snapshot boundary. Discovery reports
any source location it could not enumerate; destructive rebuilds fail closed
when such a report exists. Cleanup, parsing, persistence, FTS rebuild and
marker publication commit in one transaction, so a parse failure cannot
replace the last-good index.
The emitted
`TranscriptRecord` stream is also the input to provider-independent session
detail assembly; see ADR-0007.
- **Persist axis — one shared orchestration.** A single provider-agnostic,
@@ -60,9 +77,11 @@ is disentangling the currently interleaved parse-and-write inside `indexJsonl` /
The normalized `TranscriptRecord` union is the stable center of the design, and
SQLite is one serialization adapter for it. Provider-only concepts are either
projected lossily into that language or ignored. The registry,
not provider switches, drives both indexers, watcher roots, persisted source
roots, source catalog/UI labels and colors, and raw-record routing. Adding Pi
therefore changes the Pi adapter, its registration, and its conformance tests;
the shared schema, persist layer, indexers, settings, query API, and renderer do
not acquire Pi-specific branches.
projected lossily into that language or ignored. A genuinely shared concept may
extend the canonical language by explicit decision: summary-generating model
calls, for example, carry the same normalized input/output usage as ordinary
messages. The registry, not provider switches, drives both indexers, watcher
roots, persisted source roots, source catalog/UI labels and colors, and
raw-record routing. Adding Pi therefore adds no Pi-specific branch to the
shared schema, persist layer, indexers, settings, query API, or renderer; the
provenance and summary-usage additions are provider-neutral contracts.
@@ -10,11 +10,25 @@ the direct parse path to drift from the persisted path.
**Decision.** Every provider adapter emits a canonical `TranscriptRecord`
stream. The adapter owns all source-specific interpretation: duplicate raw
events, stable identities, tool relationships, message classification, and
visibility. `visibility` is separate from `is_meta`: hidden transport context is
omitted from session detail, while visible system evidence can remain a metadata
card. Presentation-sensitive concepts are explicit canonical fields: tool calls
carry a presentation class, Skill instructions carry a content type, and
workflows carry their parent tool-call identity.
visibility. Messages and summaries carry provider-normalized visibility.
`visibility` is separate from `is_meta`. `visible` is current evidence.
`inactive` is physical evidence that the provider explicitly attests was
superseded; default queries omit it, and supported helpers may return it only
with `includeInactive: true`. `hidden` is display-suppressed or transport-only
material and no standard helper returns it. Session detail and the desktop app
remain visible-only, while visible system evidence can remain a metadata card.
Presentation-sensitive concepts are explicit canonical fields: tool calls carry
a presentation class, Skill instructions carry a content type, and workflows
carry their parent tool-call identity. Summaries carry normalized input/output
usage when their provider performed a separate model call; cached input is
folded into input usage by the provider, as it is for messages. Visibility does
not erase accounting: aggregate usage includes model calls that were later
abandoned.
Provider capability determines whether `inactive` is meaningful. Pi attests
supersession through branch, leaf, and compaction state. Kimi undo/clear can
attest it, but preserving that history is separate work. Claude transcripts do
not attest rewind or current-leaf state, and Codex sessions do not branch.
The Core `assembleSessionDetail(input)` module is the only session-detail seam.
It accepts either a provider's complete transcript stream from a fresh parse