feat(core): add first-class Pi session indexing (#23)

Pi cannot be read as another linear JSONL stream. Its history is a tree with
a durable leaf, orphan roots, branch summaries, and two compaction forms, so
the active context is something the format states rather than something line
order implies. The adapter keeps those semantics inside itself and projects
the result into the existing canonical tables.

Sessions are keyed by (normalized header cwd, header id) rather than by path,
because Pi's --session-id lookup is project-local: two projects may reuse an
id, while a move or an identical copy is still one session. Discovery covers
both layouts Pi writes and fingerprints each file by mtime, ctime, size and
inode, so a rewrite that preserves mtime is not read as unchanged.

Abandoned branches are preserved rather than dropped. Visibility becomes
three-state -- visible, inactive, hidden -- and helpers return only visible
rows until includeInactive asks for the superseded path, labeling every row
so a caller knows which it holds. Usage counts all three, because an
abandoned call still spent tokens; message_count reports only the visible
transcript.

A committed MIT-licensed oracle transcribed from Pi 0.83.0 pins the context
algorithms, and a fixed-seed differential runs 512 generated sessions against
it on every test run. Schema changes are additive.

skill-doc/references/api-reference.md

@@ -57,14 +57,15 @@ Full-text search across all indexed message text using FTS5.
 | `opts.after` | `string` | ISO lower bound on message timestamp |
 | `opts.before` | `string` | ISO upper bound on message timestamp |
 | `opts.cwd` | `string` | SQL `LIKE` filter over `messages.cwd` |
-| `opts.source` | `string` | `"claude"`, `"codex"`, or omitted/all |
+| `opts.source` | `string` | Provider ID such as `"claude"`, `"codex"`, `"kimi"`, or `"pi"` |
 | `opts.includeMeta` | `boolean` | Include `is_meta=1` rows, default false |
+| `opts.includeInactive` | `boolean` | Include provider-attested superseded rows, default false |
 
 Returns:
 
 ```js
 Array<{
-  message: { uuid, text, content_type, is_meta, role, timestamp, model, cwd, source },
+  message: { uuid, text, content_type, is_meta, role, timestamp, model, cwd, visibility, source },
   session: { id, title, project, started_at, source },
   rank,
   context
@@ -72,6 +73,7 @@ Array<{
 ```
 
 `context` is temporal neighbor context in the same session, not a parent chain.
+Hits and neighbors carry `visibility`.
 Use `context(uuid)` or `trace(uuid)` for causal/parent-chain expansion. Lower
 FTS rank sorts earlier; prefer returned order unless deliberately inspecting
 FTS ranking.
@@ -81,13 +83,14 @@ malformed (for example a hyphenated term like `foo-bar`) does not error: it
 falls back to safe per-token quoting — the same tokenization `memories()` uses —
 so ordinary text never crashes the query.
 
-#### `context(uuid)`
+#### `context(uuid, opts?)`
 
 Full indexed context around one message.
 
 | Param | Type | Description |
 | --- | --- | --- |
 | `uuid` | `string` | Message UUID |
+| `opts.includeInactive` | `boolean` | Include a superseded target and ancestors, default false |
 
 Returns:
 
@@ -95,9 +98,11 @@ Returns:
 { message, parentChain, session, subagent, workflow } | null
 ```
 
-`parentChain` contains ancestors, not temporal neighbors. If the message belongs
+`message` and every returned ancestor carry `visibility`. `parentChain`
+contains ancestors, not temporal neighbors. If the message belongs
 to a subagent or workflow agent, `subagent` or `workflow` is populated when the
-metadata exists.
+metadata exists. Hidden targets always return `null`, and hidden ancestors are
+always omitted.
 
 #### `sql(query, ...params)`
 
@@ -176,7 +181,7 @@ Returns:
     projects,
     sessions,
     memories,
-    sources: [{ source: 'claude' | 'codex', session_count, last_session_at }]
+    sources: [{ source: 'claude' | 'codex' | 'kimi' | 'pi', session_count, last_session_at }]
   }
 }
 ```
@@ -195,11 +200,13 @@ Session rows ordered by `ended_at` descending. Passing a number is treated as
 | `opts.before` | `string` | ISO upper bound on `started_at` |
 | `opts.limit` | `number` | Max rows, default 50 |
 | `opts.branch` | `string` | Exact git branch |
-| `opts.source` | `string` | `"claude"`, `"codex"`, or omitted/all |
+| `opts.source` | `string` | Provider ID such as `"claude"`, `"codex"`, `"kimi"`, or `"pi"`; omit for all |
 | `opts.sessionId` | `string` | Exact session ID |
 | `opts.sessions` | `string[]` | Restrict to session IDs |
 
 Returns `Array<session_row>`.
+`message_count` describes the visible canonical transcript; inactive and hidden
+records do not increase it.
 
 #### `recent(n?)`
 
@@ -222,6 +229,7 @@ is treated as `sessionId`; passing a number is treated as `limit`.
 | `opts.branch` | `string` | Exact source session branch |
 | `opts.source` | `string` | Provider filter through joined session |
 | `opts.limit` | `number` | Max rows, default 100 |
+| `opts.includeInactive` | `boolean` | Include superseded summaries, default false |
 
 Returns:
 
@@ -230,7 +238,10 @@ Array<summary_row & { session_title, project }>
 ```
 
 `summaries.source` is the summary kind, such as `away_summary`; it is not the
-provider source.
+provider source. `input_tokens` and `output_tokens` contain normalized usage
+when the provider performed a separate model call for that summary.
+Rows carry `visibility`. Inactive summaries describe work that was tried and
+then superseded. Hidden summaries are never returned.
 
 #### `memories(opts?)`
 
@@ -265,11 +276,13 @@ full content.
 
 ## Structural Expansion Helpers
 
-#### `trace(uuid)`
+#### `trace(uuid, opts?)`
 
 Walk the `parent_uuid` chain from a message to the conversation root.
 
-Returns `Array<message>` ordered root-first.
+Pass `{ includeInactive: true }` to follow a superseded path. Returns labeled
+messages ordered root-first. A hidden target returns an empty array, and hidden
+ancestors are omitted.
 
 #### `thread(sessionId, opts?)`
 
@@ -279,30 +292,34 @@ Messages in a session ordered by timestamp.
 | --- | --- | --- |
 | `sessionId` | `string` | Session ID |
 | `opts.includeMeta` | `boolean` | Include injected/control-plane rows, default false |
+| `opts.includeInactive` | `boolean` | Include superseded messages, default false |
 
 Returns `Array<message>`. Use `thread()` as a last resort; prefer targeted
 search/context or compact SQL projections.
 
 #### `raw(uuid, opts?)`
 
-Windowed access to the original JSONL line for one indexed message. Use this
-when indexed text, tool inputs, or tool results were truncated and you need the
-raw source.
+Windowed access to the source record for one indexed message, normally its
+original JSONL line. Use this when indexed text, tool inputs, or tool results
+were truncated and you need the raw source. Pi returns the selected source
+message object for both direct and retained-tail storage, so one physical
+compaction line never exposes other retained messages.
 
 | Param | Type | Description |
 | --- | --- | --- |
 | `uuid` | `string` | Message UUID |
 | `opts.offset` | `number` | Character offset into the JSONL line, default 0 |
 | `opts.limit` | `number` | Max characters, default 10000 |
+| `opts.includeInactive` | `boolean` | Allow a superseded target, default false |
 
 Returns:
 
 ```js
-{ text, totalLength, offset, limit, hasMore } | null
+{ text, totalLength, offset, limit, hasMore, visibility } | null
 ```
 
 `raw()` resolves main-session, subagent, workflow-agent, and Codex JSONL paths
-from indexed metadata.
+from indexed metadata. Hidden targets always return `null`.
 
 ---
 
@@ -369,6 +386,7 @@ well as `Edit`/`Write`.
 | `opts.before` | `string` | ISO upper bound |
 | `opts.source` | `string` | Provider filter |
 | `opts.limit` | `number` | Max rows, default 200 |
+| `opts.includeInactive` | `boolean` | Include superseded tool evidence, default false |
 
 Returns:
 
@@ -376,7 +394,8 @@ Returns:
 Array<{
   toolCall: { id, message_uuid, name, input_json },
   session: { id, title, project },
-  timestamp
+  timestamp,
+  visibility
 }>
 ```
 
@@ -396,11 +415,12 @@ the failure. Passing a string is treated as `sessionId`.
 | `opts.before` | `string` | ISO upper bound on result message timestamp |
 | `opts.source` | `string` | Provider filter |
 | `opts.limit` | `number` | Max rows, default 50 |
+| `opts.includeInactive` | `boolean` | Include superseded failures and neighbors, default false |
 
 Returns:
 
 ```js
-Array<{ toolCall, result, session, nextMessages }>
+Array<{ toolCall, result, session, nextMessages, visibility }>
 ```
 
 Use SQL for precise counts and grouping; treat `failures()` as compact evidence,

skill-doc/references/pitfalls.md

@@ -39,6 +39,7 @@ sql(`
   JOIN sessions s ON s.id = m.session_id
   WHERE s.project LIKE ?
     AND m.text LIKE ?
+    AND COALESCE(m.visibility, 'visible') = 'visible'
   ORDER BY m.timestamp DESC
   LIMIT 10
 `, '%quiet-zero%', '%workflow-script%')

skill-doc/references/query-patterns.md

@@ -349,6 +349,7 @@ for (const { facet, terms } of learnedFacets) {
     JOIN sessions s ON s.id = m.session_id
     WHERE m.session_id IN (${sessionIds.map(() => '?').join(',')})
       AND m.text IS NOT NULL
+      AND COALESCE(m.visibility, 'visible') = 'visible'
       AND (${clauses})
     ORDER BY m.timestamp
     LIMIT 3
@@ -448,6 +449,7 @@ const before = sql(
   `SELECT uuid, role, timestamp, substr(text,1,200) AS snippet
    FROM messages
    WHERE session_id=? AND timestamp<?
+     AND COALESCE(visibility, 'visible') = 'visible'
    ORDER BY timestamp DESC LIMIT 3`,
   s.session_id,
   s.timestamp
@@ -456,6 +458,7 @@ const after = sql(
   `SELECT uuid, role, timestamp, substr(text,1,200) AS snippet
    FROM messages
    WHERE session_id=? AND timestamp>?
+     AND COALESCE(visibility, 'visible') = 'visible'
    ORDER BY timestamp ASC LIMIT 3`,
   s.session_id,
   s.timestamp
@@ -532,6 +535,7 @@ const counts = sql(`
   JOIN messages m ON m.uuid = tr.message_uuid
   JOIN sessions s ON s.id = tr.session_id
   WHERE tr.is_error = 1
+    AND COALESCE(m.visibility, 'visible') = 'visible'
     AND s.project LIKE ?
   GROUP BY tc.name
   ORDER BY failure_count DESC, last_failure_at DESC
@@ -551,6 +555,7 @@ const examples = sql(`
   JOIN messages m ON m.uuid = tr.message_uuid
   JOIN sessions s ON s.id = tr.session_id
   WHERE tr.is_error = 1
+    AND COALESCE(m.visibility, 'visible') = 'visible'
     AND s.project LIKE ?
   ORDER BY m.timestamp DESC
   LIMIT 8
@@ -580,6 +585,7 @@ const groups = sql(`
   JOIN messages m ON m.uuid = tr.message_uuid
   JOIN sessions s ON s.id = tr.session_id
   WHERE tr.is_error = 1
+    AND COALESCE(m.visibility, 'visible') = 'visible'
     AND s.project LIKE ?
   GROUP BY s.id
   ORDER BY last_failure_at DESC
@@ -598,6 +604,7 @@ const examples = sql(`
   JOIN messages m ON m.uuid = tr.message_uuid
   JOIN sessions s ON s.id = tr.session_id
   WHERE tr.is_error = 1
+    AND COALESCE(m.visibility, 'visible') = 'visible'
     AND s.project LIKE ?
   ORDER BY m.timestamp DESC
   LIMIT 12
@@ -694,6 +701,7 @@ const row = sql(`
   SELECT uuid, length(text) AS indexed_len
   FROM messages
   WHERE length(text) >= 10000
+    AND COALESCE(visibility, 'visible') = 'visible'
   LIMIT 1
 `)[0];
 if (!row) return null;

skill-doc/references/retrieval-semantics.md

@@ -37,7 +37,7 @@ Project-like fields are distinct:
 - `memories.project`: stored project slug copied onto registered memory records.
 - `sessions.project_path`: absolute session path derived from message `cwd` when available; slug decoding is only a fallback.
 - `messages.cwd`: working directory at message time.
-- `sessions.source` / `messages.source`: transcript provider, currently `claude` or `codex`.
+- `sessions.source` / `messages.source`: transcript provider: `claude`, `codex`, `kimi`, or `pi`.
 - helper `project`: SQL `LIKE` over `sessions.project`, not exact membership.
 - helper `source`: optional provider filter. Omit it unless provenance matters.
 
@@ -85,6 +85,9 @@ Ordering and context are semantic:
 - `fileHistory()` is oldest first.
 - `search().context` is temporal neighbors in one session, not causal context.
 - `context(uuid)` and `trace(uuid)` are for parent-chain/causal expansion.
+  They return only current evidence by default. Use `includeInactive: true` for
+  a Pi path that was tried and then superseded; hidden records remain
+  unavailable.
 
 ### Evidence Before Conclusion
 

skill-doc/references/schema.md

@@ -14,11 +14,14 @@ exist.
 
 ## Source Model
 
-Obelisk stores Claude Code and Codex transcripts in the same schema.
+Obelisk stores Claude Code, Codex, Kimi Code, and Pi transcripts in the same
+schema.
 
 - Claude rows use `source='claude'`.
 - Codex rows use `source='codex'`; root session and message IDs are prefixed
   with `codex:`.
+- Kimi Code rows use `source='kimi'`.
+- Pi rows use `source='pi'`; session IDs are prefixed with `pi:`.
 - Omit `source` filters unless provider provenance matters.
 - Codex child threads are represented through `subagents`; Codex may not have
   Claude-style workflow rows.
@@ -59,9 +62,9 @@ One row per root session.
 | `started_at`, `ended_at` | ISO timestamps |
 | `git_branch` | Branch at session time |
 | `version` | Provider CLI/app version |
-| `message_count` | Indexed user + assistant messages |
+| `message_count` | Visible canonical messages; inactive and hidden records are excluded |
 | `jsonl_path` | Source JSONL path |
-| `source` | `claude` or `codex` |
+| `source` | Provider ID: `claude`, `codex`, `kimi`, or `pi` |
 
 ### `messages`
 
@@ -77,6 +80,7 @@ Core evidence table.
 | `text` | Extracted text, truncated to 10k chars |
 | `content_type` | `text`, `thinking`, `tool_use`, `tool_result`, or `unknown` |
 | `is_meta` | 1 for injected/control-plane messages |
+| `visibility` | `visible` for current evidence, `inactive` for provider-attested superseded history, `hidden` for display-suppressed or transport-only material |
 | `model` | Assistant model name |
 | `is_sidechain` | Retry/branch marker |
 | `agent_id` | Subagent/workflow agent ID |
@@ -84,7 +88,7 @@ Core evidence table.
 | `cwd` | Working directory at message time |
 | `skill` | Skill that generated the response, if known |
 | `turn_duration_ms` | Wall-clock duration for the turn |
-| `source` | `claude` or `codex` |
+| `source` | Provider ID: `claude`, `codex`, `kimi`, or `pi` |
 
 `content_type='tool_use'` is only a marker. Tool-call details live in
 `tool_calls`. `content_type='tool_result'` marks provider-emitted tool-result
@@ -131,6 +135,8 @@ Session summary rows.
 | `timestamp` | Summary timestamp |
 | `source` | Summary kind, such as `away_summary`; not provider source |
 | `content` | Summary text |
+| `visibility` | Same `visible` / `inactive` / `hidden` contract as messages |
+| `input_tokens`, `output_tokens` | Model usage when summary generation was a separate provider call |
 
 ### `subagents`
 
@@ -207,6 +213,7 @@ Indexer progress and sentinel state.
 | `jsonl_path` | Source path or synthetic sentinel key |
 | `mtime` | Last indexed mtime |
 | `lines_processed` | Incremental line cursor |
+| `cursor` | Exact opaque provider cursor; legacy rows fall back to `mtime:lines_processed` |
 
 Sentinel keys include `__last_build__`, `__app_heartbeat__`,
 `__app_last_successful_build__`, `__indexer_owner_app__`, and
@@ -250,6 +257,7 @@ FROM tool_calls tc
 JOIN messages m ON m.uuid = tc.message_uuid
 JOIN sessions s ON s.id = tc.session_id
 WHERE s.project LIKE ?
+  AND COALESCE(m.visibility, 'visible') = 'visible'
 ORDER BY m.timestamp DESC
 LIMIT 20;
 ```
@@ -262,6 +270,7 @@ FROM tool_results tr
 JOIN tool_calls tc ON tc.id = tr.tool_use_id
 JOIN messages m ON m.uuid = tr.message_uuid
 WHERE tr.is_error = 1
+  AND COALESCE(m.visibility, 'visible') = 'visible'
 ORDER BY m.timestamp DESC
 LIMIT 20;
 ```
@@ -274,6 +283,7 @@ FROM messages m
 JOIN sessions s ON s.id = m.session_id
 WHERE s.project LIKE ?
   AND COALESCE(m.is_meta, 0) = 0
+  AND COALESCE(m.visibility, 'visible') = 'visible'
 ORDER BY m.timestamp DESC
 LIMIT 20;
 ```
@@ -322,7 +332,6 @@ Common indexed filters:
   absolute path when known; `messages.cwd` is per-message working directory.
 - Memory rows are archived with `deleted_at`; do not recall archived memories.
 - Indexed text and JSON fields are truncated to 10k chars. Use `raw()` from
-  `references/api-reference.md` when a specific message needs the original JSONL
-  line.
+  `references/api-reference.md` when a specific message needs its source record.
 - Prefer SQL-side `COUNT`, `GROUP BY`, `MAX`, `ORDER BY`, and `LIMIT` over
   returning large row sets and hand-counting in the final answer.