feat(core): add first-class Pi session indexing (#23)

Pi cannot be read as another linear JSONL stream. Its history is a tree with
a durable leaf, orphan roots, branch summaries, and two compaction forms, so
the active context is something the format states rather than something line
order implies. The adapter keeps those semantics inside itself and projects
the result into the existing canonical tables.

Sessions are keyed by (normalized header cwd, header id) rather than by path,
because Pi's --session-id lookup is project-local: two projects may reuse an
id, while a move or an identical copy is still one session. Discovery covers
both layouts Pi writes and fingerprints each file by mtime, ctime, size and
inode, so a rewrite that preserves mtime is not read as unchanged.

Abandoned branches are preserved rather than dropped. Visibility becomes
three-state -- visible, inactive, hidden -- and helpers return only visible
rows until includeInactive asks for the superseded path, labeling every row
so a caller knows which it holds. Usage counts all three, because an
abandoned call still spent tokens; message_count reports only the visible
transcript.

A committed MIT-licensed oracle transcribed from Pi 0.83.0 pins the context
algorithms, and a fixed-seed differential runs 512 generated sessions against
it on every test run. Schema changes are additive.
This commit is contained in:
SaladDay
2026-08-04 23:33:01 +08:00
committed by GitHub
parent 1941e64572
commit 2589384e68
63 changed files with 7796 additions and 374 deletions
+15 -6
View File
@@ -14,11 +14,14 @@ exist.
## Source Model
Obelisk stores Claude Code and Codex transcripts in the same schema.
Obelisk stores Claude Code, Codex, Kimi Code, and Pi transcripts in the same
schema.
- Claude rows use `source='claude'`.
- Codex rows use `source='codex'`; root session and message IDs are prefixed
with `codex:`.
- Kimi Code rows use `source='kimi'`.
- Pi rows use `source='pi'`; session IDs are prefixed with `pi:`.
- Omit `source` filters unless provider provenance matters.
- Codex child threads are represented through `subagents`; Codex may not have
Claude-style workflow rows.
@@ -59,9 +62,9 @@ One row per root session.
| `started_at`, `ended_at` | ISO timestamps |
| `git_branch` | Branch at session time |
| `version` | Provider CLI/app version |
| `message_count` | Indexed user + assistant messages |
| `message_count` | Visible canonical messages; inactive and hidden records are excluded |
| `jsonl_path` | Source JSONL path |
| `source` | `claude` or `codex` |
| `source` | Provider ID: `claude`, `codex`, `kimi`, or `pi` |
### `messages`
@@ -77,6 +80,7 @@ Core evidence table.
| `text` | Extracted text, truncated to 10k chars |
| `content_type` | `text`, `thinking`, `tool_use`, `tool_result`, or `unknown` |
| `is_meta` | 1 for injected/control-plane messages |
| `visibility` | `visible` for current evidence, `inactive` for provider-attested superseded history, `hidden` for display-suppressed or transport-only material |
| `model` | Assistant model name |
| `is_sidechain` | Retry/branch marker |
| `agent_id` | Subagent/workflow agent ID |
@@ -84,7 +88,7 @@ Core evidence table.
| `cwd` | Working directory at message time |
| `skill` | Skill that generated the response, if known |
| `turn_duration_ms` | Wall-clock duration for the turn |
| `source` | `claude` or `codex` |
| `source` | Provider ID: `claude`, `codex`, `kimi`, or `pi` |
`content_type='tool_use'` is only a marker. Tool-call details live in
`tool_calls`. `content_type='tool_result'` marks provider-emitted tool-result
@@ -131,6 +135,8 @@ Session summary rows.
| `timestamp` | Summary timestamp |
| `source` | Summary kind, such as `away_summary`; not provider source |
| `content` | Summary text |
| `visibility` | Same `visible` / `inactive` / `hidden` contract as messages |
| `input_tokens`, `output_tokens` | Model usage when summary generation was a separate provider call |
### `subagents`
@@ -207,6 +213,7 @@ Indexer progress and sentinel state.
| `jsonl_path` | Source path or synthetic sentinel key |
| `mtime` | Last indexed mtime |
| `lines_processed` | Incremental line cursor |
| `cursor` | Exact opaque provider cursor; legacy rows fall back to `mtime:lines_processed` |
Sentinel keys include `__last_build__`, `__app_heartbeat__`,
`__app_last_successful_build__`, `__indexer_owner_app__`, and
@@ -250,6 +257,7 @@ FROM tool_calls tc
JOIN messages m ON m.uuid = tc.message_uuid
JOIN sessions s ON s.id = tc.session_id
WHERE s.project LIKE ?
AND COALESCE(m.visibility, 'visible') = 'visible'
ORDER BY m.timestamp DESC
LIMIT 20;
```
@@ -262,6 +270,7 @@ FROM tool_results tr
JOIN tool_calls tc ON tc.id = tr.tool_use_id
JOIN messages m ON m.uuid = tr.message_uuid
WHERE tr.is_error = 1
AND COALESCE(m.visibility, 'visible') = 'visible'
ORDER BY m.timestamp DESC
LIMIT 20;
```
@@ -274,6 +283,7 @@ FROM messages m
JOIN sessions s ON s.id = m.session_id
WHERE s.project LIKE ?
AND COALESCE(m.is_meta, 0) = 0
AND COALESCE(m.visibility, 'visible') = 'visible'
ORDER BY m.timestamp DESC
LIMIT 20;
```
@@ -322,7 +332,6 @@ Common indexed filters:
absolute path when known; `messages.cwd` is per-message working directory.
- Memory rows are archived with `deleted_at`; do not recall archived memories.
- Indexed text and JSON fields are truncated to 10k chars. Use `raw()` from
`references/api-reference.md` when a specific message needs the original JSONL
line.
`references/api-reference.md` when a specific message needs its source record.
- Prefer SQL-side `COUNT`, `GROUP BY`, `MAX`, `ORDER BY`, and `LIMIT` over
returning large row sets and hand-counting in the final answer.