diff --git a/README.md b/README.md index 238bf59..b41d0ea 100644 --- a/README.md +++ b/README.md @@ -94,19 +94,27 @@ Don't invent a rigid query DSL either. Agents can write code. So Obelisk gives them a small local query runtime over your past work. -The agent has a two-tier API. Most questions only need the simple layer: +The agent starts from a small core API, then uses structured shortcuts and +references only when the question needs them: -**Simple API** — taught directly in the skill prompt: +**Core primitives** — the main CodeAct surface: - `search(text)` — FTS5 full-text search, returns matches with surrounding context - `context(uuid)` — full story around a message (parent chain, subagent/workflow metadata) - `sql(query, ...params)` — raw SQL for anything else -**Advanced API** — agent reads `references/schema.md` on demand: +**Structured shortcuts** — session, summary, subagent, workflow, file-history, +failure, raw-window, and parent-chain helpers over the same SQLite data. -- `trace()` · `thread()` · `subagents()` · `workflows()` · `workflowTree()` · `fileHistory()` · `failures()` · `recent()` +**References** — agent reads on demand when the task needs deeper structure: -The design is progressive disclosure: the agent doesn't see the full schema until it needs it. +- `references/schema.md` — full SQLite schema and API reference +- `references/query-patterns.md` — copyable CodeAct recipes for common retrieval tasks +- `references/pitfalls.md` — scope, FTS, ordering, compact/raw, and field-name traps + +The design is progressive disclosure with guardrails: the main skill keeps the +core contract and high-risk pitfalls visible, while longer recipes and the full +schema stay out of the first prompt until the agent needs them. ## What gets indexed @@ -129,7 +137,9 @@ Full-text search via FTS5 covers message text across every layer, while the SQLi ├── scripts/ │ └── runtime.mjs # Indexer + query runtime (400 lines, zero deps) └── references/ - └── schema.md # Full table schema + advanced API + query patterns + ├── schema.md # Full table schema + advanced API reference + ├── query-patterns.md # Copyable retrieval recipes + └── pitfalls.md # Scope, FTS, ordering, and compactness traps ``` ## Implementation Notes @@ -145,4 +155,3 @@ Zero npm dependencies. Uses Node 22's built-in node:sqlite with FTS5. The entire ## License MIT @tommy0103 - diff --git a/SKILL.md b/SKILL.md index b93f2c8..1057165 100644 --- a/SKILL.md +++ b/SKILL.md @@ -12,186 +12,188 @@ allowed-tools: # obelisk -Searches and queries your Claude Code session history stored in `~/.claude/`. -A SQLite index with FTS5 full-text search covers all sessions, subagent conversations, and workflow agent runs. -You write JS query snippets that run in a sandboxed VM against the indexed data, then parse the JSON output. +Search and query Claude Code session history stored in `~/.claude/`. +Obelisk indexes sessions, messages, tool calls, tool results, summaries, +subagents, workflows, workflow agents, parent chains, and raw JSONL lines into +SQLite + FTS5. + +Obelisk is a CodeAct memory layer: write a small JS query, run it locally, read +the JSON, then answer. Do not turn history into a flat document or browse entire +sessions by default. ## Quick Start -The base directory for this skill is provided as `$SKILL_DIR` at invocation time (shown as "Base directory for this skill: ..."). +The skill directory is provided as `$SKILL_DIR` at invocation time. -**Fast keyword search** (no script needed): +Fast keyword search: ```bash node $SKILL_DIR/scripts/runtime.mjs --search "keyword" ``` -**Custom query** (write a JS snippet, run it): +Custom query: -1. Write a query to a temp file (e.g. `/tmp/q.mjs`) -2. Run: `node $SKILL_DIR/scripts/runtime.mjs --query /tmp/q.mjs` -3. Parse the JSON stdout and answer the user +1. Write a bounded JS query to a temp file, for example `/tmp/q.mjs`. +2. Run: -The query file body is executed inside `(async () => { ... })()` with the API below available as globals. The last expression is returned as JSON. Use `return` to emit results. + ```bash + node $SKILL_DIR/scripts/runtime.mjs --query /tmp/q.mjs + ``` -## API +3. Parse JSON stdout and answer with concise evidence. -### search(text, opts?) +The query file runs inside `(async () => { ... })()`. Use `return` to emit JSON. -Full-text search across all messages (user, assistant, subagent, workflow agent). +## Reference Triggers -Returns: `[{ message: {uuid, text, role, timestamp, model, cwd}, session: {id, title, project, started_at}, rank, context: [...surrounding messages] }]` +Use progressive disclosure, but do not guess. -opts: `{ limit, sessionId, project, after, before, cwd }` +- Read `references/schema.md` before raw `sql()` unless the needed table/column relationship is already explicit here. +- Read `references/query-patterns.md` for workflow trees, failed tool counts or failure groups, broad development-history synthesis, file history synthesis, summary neighbors, subagent recall, raw windows, and empty-result handling. +- Read `references/pitfalls.md` when a scoped result is empty or tiny, when a query may over-fetch, when a term is hyphenated, when project scope is ambiguous, or when helper row fields are unclear. -`rank` is the FTS5 relevance score (negative; closer to 0 = more relevant). Use it to judge result quality and stop early when results become irrelevant. +If a helper row shape is unclear, first run a tiny scoped query and return +`Object.keys(row)` or a compact sample. Do not invent field names. -### sessions(opts?) +## Core API -Query sessions with filters. Returns session rows ordered by `ended_at` descending. +### `search(text, opts?)` -opts: `{ project, after, before, limit, branch, sessionId, sessions }` +Full-text search across main messages, subagent messages, and workflow-agent +messages. + +Returns: ```js -sessions({ project: '%quiet-zero%' }) -sessions({ after: '2026-06-01', branch: 'main', limit: 5 }) +[{ message: { uuid, text, role, timestamp, model, cwd }, + session: { id, title, project, started_at }, + rank, + context }] ``` -### context(uuid) +`context` here means temporal neighbors: nearby messages in the same session by +timestamp. It is not the parent chain. Use `context(uuid)` or `trace(uuid)` for +causal/parent-chain context. -Full story around a message: the message itself, parent chain, session info, subagent/workflow metadata. +Opts: `{ limit, sessionId, project, after, before, cwd }`. -Returns: `{ message, parentChain, session, subagent, workflow }` +`project` is a SQL `LIKE` filter over `sessions.project`, not an exact project +identity. Results are already ordered by FTS5 rank; lower rank sorts earlier. +Prefer returned order over manually interpreting numeric rank unless you are +deliberately using FTS5 semantics. -### recent(n?) +### `context(uuid)` -Shorthand for `sessions({ limit: n })`. Latest n sessions (default 10). - -### sql(query, ...params) - -Raw SQL. Use `?` placeholders. Returns array of row objects. - -**Before writing your first SQL query, read `references/schema.md` for the full table schema, column names, and relationships.** Don't guess column names — the schema is your source of truth. - -**Schema-safe SQL pattern:** when aggregating event tables, join to the table that actually owns the metadata instead of inventing columns. For example, `tool_calls` does **not** own timestamps; join `messages m ON m.uuid = tc.message_uuid` for `m.timestamp`, and join `sessions s ON s.id = tc.session_id` for project/session filters. Prefer SQL-side `GROUP BY`/`COUNT`/`MAX` with `LIMIT`, and return compact evidence rows with stable IDs rather than raw event records. - -Tables: `sessions`, `messages`, `tool_calls`, `tool_results`, `subagents`, `workflows`, `workflow_agents`, `messages_fts` - -### Other APIs - -All list-returning functions accept a common filter opts object: `{ project, after, before, limit, sessionId, sessions }`. For backward compatibility, passing a string is treated as `sessionId`, a number as `limit`. - -- `trace(uuid)` -- full parent chain from root to message -- `thread(sessionId)` -- all messages in a session, ordered by time -- `subagents(opts?)` -- subagent metadata + message counts. opts: `{ sessionId, project, limit }` -- `workflows(opts?)` -- workflow runs with duration, tokens, status. opts: `{ sessionId, project, after, before, limit }` -- `workflowTree(runId)` -- workflow metadata + parsed result + agents with phase/label/tokens/duration (no messages; use `sql()` with `agent_id` to drill into a specific agent) -- `fileHistory(filePath, opts?)` -- every Edit/Write/Read on a file. opts: `{ after, before, limit }` -- `failures(opts?)` -- tool calls that returned errors, with surrounding context. opts: `{ sessionId, project, after, before, limit }` -- `summaries(opts?)` -- session summaries (away recaps, compaction summaries). opts: `{ sessionId, project, after, before, limit, sessions }`. Returns: `[{ id, session_id, timestamp, source, content, session_title, project }]`. Use `source` for values like `away_summary`; use `content` for the summary text. -- `raw(uuid, opts?)` -- windowed access to the original JSONL line (bypasses index truncation) - -### Retrieval strategy - -**For helper field/schema confirmation questions:** use the relevant helper under the user's explicit scope with a small `limit`, then return only `Object.keys(row)` or a projection of the fields being verified plus short snippets. Do not invent alias fields when helper docs/rows use different names (for summaries, use `source`, `content`, `session_id`, `project`; not `text` or `summary_type`). Include stable evidence IDs such as `id`, `session_id`, or `uuid` in the final answer. - -**Respect explicit scopes and empty results.** If the user asks for a specific project/session/file/time range, keep every query inside that scope. When a scoped helper call such as `summaries({ project, limit })` returns `[]`, report no results; do not broaden to all projects or all summaries unless the user asks for fallback. - -**Never pull an entire session.** Navigate incrementally: - -1. `sessions({ project: '...' })` or `recent()` — find relevant sessions -2. `summaries({ project: '...' })` — read session summaries to judge relevance (cheapest) -3. `search()` — find specific messages matching a query -4. When you find a relevant message and want more context, expand from that point: - - **Horizontally**: use `sql()` to fetch neighboring messages by timestamp - ```js - sql('SELECT uuid,role,text FROM messages WHERE session_id=? AND timestamp>? ORDER BY timestamp LIMIT 5', sid, msg.timestamp) - ``` - - **Vertically**: use `trace(uuid)` to walk up the parent chain, or `context(uuid)` to see subagent/workflow relationships -4. `raw(uuid, opts?)` — recover truncated content from a specific message -5. `thread(sessionId)` — full session dump, **last resort only** - -**File history queries:** `fileHistory()` can include many `Read` rows and large tool inputs. For questions about how a file changed, filter to `Edit`/`Write` before returning, cap the filtered list to the requested evidence count, and return compact evidence records only. - -### raw(uuid, opts?) - -Some indexed fields (tool call inputs, tool results) are truncated to 10k chars. `raw()` reads the original JSONL line to recover the full content. - -Returns: `{ text, totalLength, offset, limit, hasMore }` - -opts: `{ offset: 0, limit: 10000 }` — character window into the raw JSONL line. +Returns the full story around one indexed message: ```js -// First window -const r = raw(messageUuid) -// r.text = first 10k chars of the original JSONL line -// r.totalLength = full line length -// r.hasMore = true if more content remains - -// Scroll forward -const r2 = raw(messageUuid, { offset: 10000, limit: 10000 }) +{ message, parentChain, session, subagent, workflow } ``` -## Examples +Use this after `search()` finds a promising message. It is the usual way to +expand vertically from one evidence point without dumping the whole session. -### "上次怎么修 auth 的" +### `sql(query, ...params)` + +Raw SQL with `?` placeholders. Returns array rows. + +Before writing non-trivial SQL, read `references/schema.md`. Common safe joins: + +- `tool_calls` does not have timestamps. Join `messages m ON m.uuid = tc.message_uuid`. +- `tool_results` does not have timestamps. Join `messages m ON m.uuid = tr.message_uuid`. +- For project/session filters, join `sessions s ON s.id = .session_id`. +- Prefer SQL-side `GROUP BY`, `COUNT`, `MAX`, `ORDER BY`, and `LIMIT` over hand-counting in the final answer. + +Tables: `sessions`, `messages`, `tool_calls`, `tool_results`, `summaries`, +`subagents`, `workflows`, `workflow_agents`, `messages_fts`. + +## Structured Helpers + +These helpers are convenience accessors over the same SQLite structure. They do +not replace `sql()`; use `sql()` when you need an exact aggregation or a join +the helper does not expose. + +All list helpers accept a bounded `limit`. Many also accept: +`{ project, after, before, sessionId, sessions, branch }`. Check the schema or a +tiny sample before relying on less common filters. + +- `sessions(opts?)` -- session rows, newest first. `project` is a SQL `LIKE` pattern. +- `recent(n?)` -- shorthand for recent sessions. +- `summaries(opts?)` -- summary rows, newest first: `{ id, session_id, timestamp, source, content, session_title, project }`. +- `subagents(opts?)` -- subagent metadata plus `messageCount`. +- `workflows(opts?)` -- workflow runs, newest first. +- `workflowTree(runId)` -- workflow row plus parsed `result` and `agents`; may include bulky `script` and `result_json`, so project compact fields. +- `fileHistory(filePath, opts?)` -- Read/Edit/Write tool calls for a file, oldest first; includes many `Read` rows. +- `failures(opts?)` -- failed tool results with tool/session context, newest first. +- `trace(uuid)` -- parent chain from root to message. +- `thread(sessionId)` -- full session messages; last resort only. +- `raw(uuid, opts?)` -- windowed access to the original JSONL line. + +## Retrieval Contract + +Keep queries scoped, bounded, and structural. + +- Preserve explicit project/session/file/time scopes. Empty or tiny scoped results are real results; do not broaden unless the user asks. +- Treat project scope as three distinct semantics: exact `sessions.project` slug, exact `sessions.project_path`, or fuzzy `LIKE` search. Use `sql()` for exact slug/path membership; helper `project` means fuzzy `LIKE`. +- Start with cheap locators: `sessions()`, `summaries()`, `search()`, or a small SQL query. +- Expand incrementally with `context()`, `trace()`, neighbor SQL, or `raw()` windows. +- Return compact evidence with stable IDs (`session_id`, `uuid`, `tool_call_id`, `run_id`, `agent_id`) and short snippets. +- Avoid `thread()` unless the user explicitly asks for a full transcript or all smaller probes are insufficient. +- Keep runtime JSON small. Do not return all sessions, all summaries, all tool calls, complete workflow trees, full raw messages, or whole tool results. +- When counting or aggregating, compute counts in SQL or in the query script and return those counts. Do not hand-count from long rows in prose. +- For recent failures or "which tasks failed" questions, aggregate by session/task and return counts plus sparse examples. Do not return raw failure rows. +- For broad "how did X evolve / what did we do / what problems happened" history synthesis, use a bounded facet sweep from `references/query-patterns.md`. For concept recall, session lookup, or exact term recall, keep compact `search()` first. + +High-frequency field contracts: + +- `summaries()` uses `source` and `content`, not `summary_type` or `text`. +- `search().context` is temporal neighbor context, not causal or parent-chain context. +- `fileHistory()` includes `Read`; filter to `Edit`/`Write` for causal change history. +- `workflowTree()` may expose raw `script` and `result_json`; omit them unless the user asks for raw workflow details. +- `fileHistory()` is ordered oldest first. For recent file changes, use SQL with `ORDER BY m.timestamp DESC`. +- FTS5 tokenizes hyphens and treats `search(text)` as raw `MATCH` syntax. For `workflow-script`, search the quoted tokenized phrase such as `"workflow script"` or use SQL `LIKE` for exact hyphen matching. + +## Minimal Patterns + +Search, then expand one promising hit: ```js -const hits = search('auth fix') -return hits.slice(0, 5).map(h => ({ - session: h.session.title, - date: h.session.started_at, - message: h.message.text?.slice(0, 200) -})) +const hits = search('auth fix', { limit: 5 }); +if (!hits.length) return []; +return hits.slice(0, 3).map(h => ({ + session_id: h.session.id, + session_title: h.session.title, + uuid: h.message.uuid, + snippet: h.message.text?.slice(0, 240), +})); ``` -### "最近在做什么" +Check helper fields before assuming names: ```js -return sessions({ limit: 10 }).map(s => ({ title: s.title, project: s.project, date: s.started_at })) +const rows = summaries({ project: '%quiet-zero%', limit: 1 }); +return rows.length ? Object.keys(rows[0]) : []; ``` -### "哪些文件被反复修改" +Fetch message neighbors without a full thread: ```js -return sql(` - SELECT file_path, COUNT(*) as n FROM tool_calls - WHERE name IN ('Edit','Write') AND file_path IS NOT NULL - GROUP BY file_path HAVING n > 3 ORDER BY n DESC LIMIT 20 -`) +const hit = search('runtime query', { limit: 1 })[0]; +return sql( + `SELECT uuid, role, timestamp, substr(text,1,240) AS snippet + FROM messages + WHERE session_id=? AND timestamp>=? + ORDER BY timestamp LIMIT 6`, + hit.session.id, + hit.message.timestamp +); ``` -### "这个项目的 workflow 跑过几次" - -```js -return workflows({ project: '%quiet-zero%' }).map(w => ({ - run: w.run_id, agents: w.agent_count, time: w.timestamp -})) -``` - -### "上次跑 experiment 用了多少 token" - -```js -const hits = search('experiment') -if (!hits.length) return 'No experiment sessions found' -const sid = hits[0].session.id -return sql('SELECT SUM(input_tokens) as input, SUM(output_tokens) as output FROM messages WHERE session_id = ?', sid) -``` - -### "追踪一下那个决策是怎么做的" - -```js -const hits = search('the decision query here') -if (!hits.length) return 'Nothing found' -return context(hits[0].message.uuid) -``` +See `references/query-patterns.md` for longer recipes. ## Notes -- First run builds the index (~5s for ~100 sessions). Subsequent runs are incremental. -- DB location: `~/.claude/obelisk.sqlite` -- Subagent and workflow agent conversations are fully indexed and searchable. -- Query scripts run in a sandboxed VM context -- no file system or network access from inside scripts. -- Text is truncated to 10k chars per message during indexing. -- FTS5 search supports standard SQLite FTS syntax: `"exact phrase"`, `term1 AND term2`, `term1 OR term2`, `term1 NOT term2`. -- FTS5 tokenizes on hyphens. To search for `SkillOpt-outputs`, use `"skillopt outputs"` (replace hyphen with space, wrap in quotes for phrase match). For exact match on hyphenated strings, use `sql()` with LIKE instead. +- First run builds the index. Later runs update incrementally. +- DB location: `~/.claude/obelisk.sqlite`. +- Query scripts run in a sandboxed VM with no filesystem or network access from inside the script. +- Indexed text and stored tool inputs/results are truncated to 10k chars. Use `raw(uuid, { offset, limit })` for specific JSONL windows. diff --git a/references/pitfalls.md b/references/pitfalls.md new file mode 100644 index 0000000..d28e389 --- /dev/null +++ b/references/pitfalls.md @@ -0,0 +1,171 @@ +# Obelisk Pitfalls + +Use this when a query may over-fetch, when a scoped query returns few or zero +rows, or when helper fields are unclear. + +## Scope Is A Contract + +If the user gives a project, session, file, or time range, keep every query +inside that scope. Do not broaden because a scoped result is small. + +There are three different project-like scopes: + +- `sessions.project`: stored Claude Code project slug. +- `sessions.project_path`: reconstructed absolute project path. +- `messages.cwd`: working directory for a specific message. + +`project` filters in helpers are SQL `LIKE` patterns over `sessions.project`. +`%quiet-zero%` can match benchmark or generated workspaces that merely contain +that string. Prefer exact `sql()` filters when the user asks for exact project +membership: + +```js +sql(` + SELECT id, title, project, project_path, ended_at + FROM sessions + WHERE project_path = ? + ORDER BY ended_at DESC + LIMIT 20 +`, '/Users/tomiya/Code/quiet-zero') +``` + +Use fuzzy project search only when the task is discovery or the user explicitly +asked to search broadly. If you broaden, make the broadening visible in the +returned evidence. + +Self-noise examples to filter when the user asks for real historical sessions: + +- `SkillOpt-outputs` +- `obelisk_train` +- `obelisk-eval` + +## FTS5 Hyphens And Syntax + +`search(text)` passes text to FTS5 `MATCH`. Hyphenated terms can be parsed as +operators or separate tokens, and special characters can raise FTS syntax +errors. + +For `workflow-script`, use a quoted tokenized phrase: + +```js +search('"workflow script"', { limit: 10 }) +``` + +Use exact phrases for phrase semantics, separate terms for token semantics, and +SQL `LIKE` for literal punctuation. Do not silently fallback from a scoped FTS +query to all sessions. + +For exact hyphen matching, use SQL `LIKE` on `messages.text` under a scope: + +```js +sql(` + SELECT m.uuid, s.id AS session_id, s.title, substr(m.text,1,240) AS snippet + FROM messages m + JOIN sessions s ON s.id = m.session_id + WHERE s.project LIKE ? + AND m.text LIKE ? + ORDER BY m.timestamp DESC + LIMIT 10 +`, '%quiet-zero%', '%workflow-script%') +``` + +`rank` is already applied by `ORDER BY rank`; lower rank sorts earlier in this +runtime. Prefer returned order over comparing "closer to zero" manually. + +## Context Is Not Always Causal + +`search().context` returns temporal neighbors: nearby messages by timestamp in +the same session. It is useful for quick orientation, but it is not the parent +chain and may cross side branches, subagents, or workflow boundaries. + +Use: + +- `context(uuid)` for message, parent chain, session, subagent, and workflow. +- `trace(uuid)` for just the parent chain. +- SQL timestamp neighbors for horizontal expansion inside one session. + +## Ordering Defaults Matter + +Some helpers return newest first; others do not. + +- `sessions()` returns newest sessions first. +- `summaries()` returns newest summaries first. +- `workflows()` returns newest workflows first. +- `failures()` returns newest failures first, but should still be treated as an evidence helper rather than a precise count helper. +- `fileHistory()` orders by message timestamp ascending. If the user asks for recent changes, use SQL explicitly: + +```js +sql(` + SELECT tc.id, tc.name, tc.file_path, m.timestamp, s.id AS session_id, s.title + FROM tool_calls tc + JOIN messages m ON m.uuid = tc.message_uuid + JOIN sessions s ON s.id = tc.session_id + WHERE tc.file_path = ? + AND tc.name IN ('Edit', 'Write', 'NotebookEdit') + ORDER BY m.timestamp DESC + LIMIT 20 +`, '/absolute/path/to/file') +``` + +## Compact Vs Raw + +Default to compact evidence. Raw/full access is a conscious escalation. + +- `workflowTree()` may include `script`, `result_json`, parsed `result`, and all agents. Project only the fields needed for the answer. +- `thread(sessionId)` dumps a whole session; use it only as a last resort. +- `raw(uuid)` can recover long original JSONL lines; use small windows and cite `totalLength`/`hasMore`. +- Tool results and tool inputs can be large. Return short snippets. + +## Field Names To Avoid Guessing + +Common wrong guesses: + +- Summaries: use `source` and `content`; do not use `summary_type` or `text`. +- Tool result timestamps: `tool_results` has no timestamp. Join `messages`. +- Tool call timestamps: `tool_calls` has no timestamp. Join `messages`. +- Workflow agent message counts: `workflowTree()` returns `messageCount` for agents. + +When uncertain: + +```js +const rows = summaries({ limit: 1 }); +return rows.length ? Object.keys(rows[0]) : []; +``` + +## Counting Must Be Structural + +If the user asks "how many", "counts", "top N", or "group by", compute it in SQL +or in the query script and return the computed data. Do not infer counts from +visible snippets in prose. + +Good: + +```js +sql(` + SELECT tc.name, COUNT(*) AS n + FROM tool_results tr + JOIN tool_calls tc ON tc.id = tr.tool_use_id + WHERE tr.is_error = 1 + GROUP BY tc.name + ORDER BY n DESC + LIMIT 10 +`) +``` + +Bad: + +```js +const rows = failures({ limit: 20 }); +return rows; // then count by eye in the final answer +``` + +## Empty Results + +An empty array is often the correct answer. + +When the user asks for a scoped project/file/session or an exact sentinel: + +1. Run the scoped query. +2. Return `[]` or compact counts. +3. Say no matching prior result was found. +4. Do not call `recent()`, all-project `summaries()`, or `thread()` as fallback unless the user asks. diff --git a/references/query-patterns.md b/references/query-patterns.md new file mode 100644 index 0000000..9b441ce --- /dev/null +++ b/references/query-patterns.md @@ -0,0 +1,352 @@ +# Obelisk Query Patterns + +These are copyable CodeAct patterns for `runtime.mjs --query` scripts. They are +not new APIs. Adapt them to the user's scope and return compact evidence. + +## Bounded Search To Context + +Use `search()` to locate candidates, then expand only the strongest hits. + +```js +const hits = search('"runtime query"', { project: '%quiet-zero%', limit: 8 }); +return hits.slice(0, 5).map(h => { + const c = context(h.message.uuid); + return { + session_id: h.session.id, + session_title: h.session.title, + uuid: h.message.uuid, + timestamp: h.message.timestamp, + snippet: h.message.text?.slice(0, 240), + parentChain: (c?.parentChain || []).slice(-3).map(m => ({ + uuid: m.uuid, + role: m.role, + snippet: m.text?.slice(0, 120), + })), + }; +}); +``` + +## Facet Sweep For Broad History + +Use this only for broad synthesis questions such as "how did X evolve", "what +did we do on X", or "what problems happened". Do not use it for concept recall, +exact session lookup, exact term recall, or tasks that ask for compact search +hits. + +Keep the sweep small: 3-4 facets, `limit: 3` per facet, and at most 12 compact +evidence rows. + +```js +const name = 'obelisk'; +const facets = [ + 'runtime CLI script', + 'schema SQLite FTS', + 'skill API helper docs', + 'test failure problem', +]; + +const rows = []; +for (const facet of facets) { + for (const h of search(`${name} ${facet}`, { project: '%quiet-zero%', limit: 3 })) { + rows.push({ facet, h }); + } +} + +const seen = new Set(); +return rows + .filter(({ h }) => { + const key = h.message.uuid || `${h.session.id}:${h.message.timestamp}`; + if (seen.has(key)) return false; + seen.add(key); + return true; + }) + .slice(0, 12) + .map(({ facet, h }) => ({ + facet, + session_id: h.session.id, + session_title: h.session.title, + project: h.session.project, + uuid: h.message.uuid, + timestamp: h.message.timestamp, + snippet: h.message.text?.slice(0, 180), + })); +``` + +## Summary Rows And Neighbors + +Use `source`, `content`, `session_id`, `project`, and `session_title`. + +```js +const rows = summaries({ project: '%quiet-zero%', limit: 8 }); +return rows.map(s => ({ + id: s.id, + session_id: s.session_id, + session_title: s.session_title, + project: s.project, + source: s.source, + timestamp: s.timestamp, + snippet: s.content?.slice(0, 240), +})); +``` + +To inspect messages around one summary: + +```js +const s = summaries({ project: '%quiet-zero%', limit: 1 })[0]; +if (!s) return { results: [] }; +const before = sql( + `SELECT uuid, role, timestamp, substr(text,1,200) AS snippet + FROM messages + WHERE session_id=? AND timestamp? + ORDER BY timestamp ASC LIMIT 3`, + s.session_id, + s.timestamp +); +return { summary: s, before, after }; +``` + +## File History Synthesis + +`fileHistory()` contains reads as well as writes. For "why/how did this file +change", scan a bounded `Edit`/`Write` set first, then return only compact +evidence. Do not choose the answer from only the first few rows if the question +asks for evolution. + +```js +const rows = fileHistory('/absolute/path/to/file', { limit: 100 }); +const writes = rows.filter(r => ['Edit', 'Write'].includes(r.toolCall?.name)); +const reads = rows.filter(r => r.toolCall?.name === 'Read'); + +const bySession = new Map(); +for (const r of writes) { + const sid = r.session.id; + const group = bySession.get(sid) || { + session_id: sid, + session_title: r.session.title, + project: r.session.project, + write_edit_count: 0, + first_timestamp: r.timestamp, + last_timestamp: r.timestamp, + themes: [], + evidence: [], + }; + group.write_edit_count++; + group.first_timestamp = group.first_timestamp < r.timestamp ? group.first_timestamp : r.timestamp; + group.last_timestamp = group.last_timestamp > r.timestamp ? group.last_timestamp : r.timestamp; + const snippet = String(r.toolCall.input_json || '').slice(0, 360); + if (group.themes.length < 8) group.themes.push(snippet); + if (group.evidence.length < 3) { + group.evidence.push({ + tool: r.toolCall.name, + tool_id: r.toolCall.id, + timestamp: r.timestamp, + snippet, + }); + } + bySession.set(sid, group); +} + +return { + counts: { reads: reads.length, writes_edits: writes.length }, + sessions: [...bySession.values()].slice(0, 10), +}; +``` + +## Failed Tool Counts + +For precise counts, aggregate in SQL. Do not hand-count long result rows in the +final answer. + +```js +const counts = sql(` + SELECT + tc.name AS tool_name, + COUNT(*) AS failure_count, + MAX(m.timestamp) AS last_failure_at + FROM tool_results tr + JOIN tool_calls tc ON tc.id = tr.tool_use_id + JOIN messages m ON m.uuid = tr.message_uuid + JOIN sessions s ON s.id = tr.session_id + WHERE tr.is_error = 1 + AND s.project LIKE ? + GROUP BY tc.name + ORDER BY failure_count DESC, last_failure_at DESC + LIMIT 20 +`, '%quiet-zero%'); + +const examples = sql(` + SELECT + tr.tool_use_id, + tc.name AS tool_name, + m.timestamp, + s.id AS session_id, + s.title AS session_title, + substr(tr.content, 1, 180) AS error_snippet + FROM tool_results tr + JOIN tool_calls tc ON tc.id = tr.tool_use_id + JOIN messages m ON m.uuid = tr.message_uuid + JOIN sessions s ON s.id = tr.session_id + WHERE tr.is_error = 1 + AND s.project LIKE ? + ORDER BY m.timestamp DESC + LIMIT 8 +`, '%quiet-zero%'); + +return { counts, examples }; +``` + +## Failure Investigation Groups + +For questions like "recent failed tool calls", "which tasks failed", or "group +failures by task/session", group structurally and return sparse examples. Use +SQL for counts; treat `failures()` as an evidence helper, not a precise counter. + +```js +const project = '%quiet-zero%'; + +const groups = sql(` + SELECT + s.id AS session_id, + s.title AS session_title, + s.project, + COUNT(*) AS failure_count, + MAX(m.timestamp) AS last_failure_at + FROM tool_results tr + JOIN tool_calls tc ON tc.id = tr.tool_use_id + JOIN messages m ON m.uuid = tr.message_uuid + JOIN sessions s ON s.id = tr.session_id + WHERE tr.is_error = 1 + AND s.project LIKE ? + GROUP BY s.id + ORDER BY last_failure_at DESC + LIMIT 10 +`, project); + +const examples = sql(` + SELECT + tr.tool_use_id AS tool_call_id, + tc.name AS tool_name, + s.id AS session_id, + m.timestamp, + substr(tr.content, 1, 180) AS error_snippet + FROM tool_results tr + JOIN tool_calls tc ON tc.id = tr.tool_use_id + JOIN messages m ON m.uuid = tr.message_uuid + JOIN sessions s ON s.id = tr.session_id + WHERE tr.is_error = 1 + AND s.project LIKE ? + ORDER BY m.timestamp DESC + LIMIT 12 +`, project); + +return { groups, examples }; +``` + +## Workflow Tree Compact View + +Find the run with `workflows()` under scope, then project `workflowTree()` into +compact fields. Do not return raw `script`, `result_json`, or the full tree. + +```js +const runs = workflows({ project: '%quiet-zero%', limit: 30 }); +const target = runs.find(w => + /session[-_ ]journal/i.test(`${w.workflow_name || ''} ${w.task_id || ''} ${w.run_id || ''}`) +); +if (!target) { + return { + found: false, + candidates: runs.slice(0, 8).map(w => ({ + run_id: w.run_id, + workflow_name: w.workflow_name, + timestamp: w.timestamp, + agent_count: w.agent_count, + })), + }; +} + +const tree = workflowTree(target.run_id); +return { + run_id: target.run_id, + workflow_name: target.workflow_name, + status: tree?.status ?? target.status, + timestamp: tree?.timestamp ?? target.timestamp, + agent_count: tree?.agent_count ?? tree?.agents?.length ?? target.agent_count, + agents: (tree?.agents || []).map(a => ({ + agent_id: a.agent_id, + phase: a.phase, + label: a.label, + state: a.state, + tokens: a.tokens, + messageCount: a.messageCount, + })), +}; +``` + +## Subagent Metadata Recall + +Use `subagents()` for metadata. Do not expand transcripts unless the user asks. + +```js +const rows = subagents({ project: '%quiet-zero%', limit: 50 }); +return rows + .filter(r => /obelisk/i.test(`${r.description || ''} ${r.agent_type || ''}`)) + .map(r => ({ + agent_id: r.agent_id, + agent_type: r.agent_type, + description: r.description, + session_id: r.session_id, + messageCount: r.messageCount, + total_tokens: r.total_tokens, + })); +``` + +## Empty Result Without Fallback + +If the user asks for an exact sentinel, scoped project, or exact file, an empty +result is valid. Report it; do not broaden automatically. + +```js +const needle = 'obelisk-impossible-sentinel-20260602'; +const hits = search(`"${needle.replace(/-/g, ' ')}"`, { limit: 10 }); +const real = hits.filter(h => { + const scope = `${h.session?.project || ''} ${h.message?.cwd || ''}`; + return !/SkillOpt[-/. ]outputs|obelisk_train|obelisk-eval/i.test(scope); +}); +return real.map(h => ({ + session_id: h.session.id, + session_title: h.session.title, + project: h.session.project, + uuid: h.message.uuid, + snippet: h.message.text?.slice(0, 200), +})); +``` + +## Raw Window + +Use `raw()` only after identifying a specific message UUID. + +```js +const row = sql(` + SELECT uuid, length(text) AS indexed_len + FROM messages + WHERE length(text) >= 10000 + LIMIT 1 +`)[0]; +if (!row) return null; +const first = raw(row.uuid, { offset: 0, limit: 4000 }); +return { + uuid: row.uuid, + indexed_len: row.indexed_len, + totalLength: first?.totalLength, + hasMore: first?.hasMore, + text: first?.text?.slice(0, 500), +}; +``` diff --git a/references/schema.md b/references/schema.md index 70e99b9..d3f78b9 100644 --- a/references/schema.md +++ b/references/schema.md @@ -211,12 +211,22 @@ Full-text search across all message text using FTS5. | `text` | `string` | FTS5 query (terms, phrases, prefix) | | `opts.limit` | `number` | Max results (default 20) | | `opts.sessionId` | `string` | Restrict to one session | -| `opts.project` | `string` | Restrict to a project slug | +| `opts.project` | `string` | SQL `LIKE` pattern over `sessions.project` | | `opts.after` | `string` | ISO 8601 lower bound on timestamp | | `opts.before` | `string` | ISO 8601 upper bound on timestamp | | `opts.cwd` | `string` | Filter by working directory (supports LIKE) | -**Returns:** `Array<{ message, session, rank, context }>` where `context` is the 6 nearest messages by timestamp. `rank` is the FTS5 relevance score (negative; closer to 0 = more relevant). +**Scope note:** `sessions.project` is the stored Claude Code project slug, +`sessions.project_path` is the reconstructed absolute project path, and +`messages.cwd` is the working directory at message time. Helper `project` +filters are fuzzy `LIKE` filters over `sessions.project`. For exact project +membership, use `sql()` with `s.project = ?` or `s.project_path = ?`. + +**Returns:** `Array<{ message, session, rank, context }>` where `context` is the +6 nearest messages by timestamp in the same session. It is temporal neighbor +context, not a parent chain. `rank` is the FTS5 relevance score used by +`ORDER BY rank`; lower values sort earlier, so treat the returned order as the +relevance order unless you are deliberately using FTS5 ranking details. ```js const hits = search('MCTS exploration'); @@ -285,7 +295,7 @@ All subagent spawns, with message counts. For backward compatibility, passing a | Param | Type | Description | |-------|------|-------------| | `opts.sessionId` | `string` | Restrict to one session | -| `opts.project` | `string` | Filter by project slug (LIKE) | +| `opts.project` | `string` | SQL `LIKE` pattern over `sessions.project` | | `opts.limit` | `number` | Max results (default 100) | **Returns:** `Array<{ ...subagent_row, messageCount }>`. @@ -302,7 +312,7 @@ Workflow executions. For backward compatibility, passing a string is treated as | Param | Type | Description | |-------|------|-------------| | `opts.sessionId` | `string` | Restrict to one session | -| `opts.project` | `string` | Filter by project slug (LIKE) | +| `opts.project` | `string` | SQL `LIKE` pattern over `sessions.project` | | `opts.after` | `string` | ISO 8601 lower bound on timestamp | | `opts.before` | `string` | ISO 8601 upper bound on timestamp | | `opts.limit` | `number` | Max results (default 100) | @@ -338,6 +348,9 @@ All tool calls that touched a specific file, across every session. **Returns:** `Array<{ toolCall, session, timestamp }>`. +Default order is oldest first (`ORDER BY m.timestamp`). For recent file changes, +use raw SQL with `ORDER BY m.timestamp DESC`. + ```js const edits = fileHistory('/Users/tomiya/Code/quiet-zero/src/mcts.ts', { after: '2026-05-28' }); return edits.map(e => ({ tool: e.toolCall.name, session: e.session.title, time: e.timestamp })); @@ -350,13 +363,15 @@ Tool calls whose results contain error patterns (`Error`, `ENOENT`, `failed`, `p | Param | Type | Description | |-------|------|-------------| | `opts.sessionId` | `string` | Restrict to one session | -| `opts.project` | `string` | Filter by project slug (LIKE) | +| `opts.project` | `string` | SQL `LIKE` pattern over `sessions.project` | | `opts.after` | `string` | ISO 8601 lower bound | | `opts.before` | `string` | ISO 8601 upper bound | | `opts.limit` | `number` | Max results (default 50) | **Returns:** `Array<{ toolCall, result, session, nextMessages }>`. +Default order is newest first by the result message timestamp. + ```js const fails = failures({ project: '%quiet-zero%', limit: 10 }); return fails.map(f => ({ tool: f.toolCall?.name, error: f.result.content?.slice(0, 200) })); @@ -379,7 +394,7 @@ Query sessions with filters. For backward compatibility, passing a number is tre | Param | Type | Description | |-------|------|-------------| -| `opts.project` | `string` | Filter by project slug (supports LIKE, e.g. `'%quiet-zero%'`) | +| `opts.project` | `string` | SQL `LIKE` pattern over `sessions.project` | | `opts.after` | `string` | ISO 8601 lower bound on `started_at` | | `opts.before` | `string` | ISO 8601 upper bound on `started_at` | | `opts.limit` | `number` | Max results (default 50) | @@ -389,6 +404,9 @@ Query sessions with filters. For backward compatibility, passing a number is tre **Returns:** `Array` ordered by `ended_at` descending. +For exact slug/path membership, use raw SQL with `project = ?` or +`project_path = ?`. + ```js const qz = sessions({ project: '%quiet-zero%', limit: 5 }); return qz.map(s => ({ title: s.title, branch: s.git_branch, ended: s.ended_at })); diff --git a/scripts/query.mjs b/scripts/query.mjs index b8772aa..c98ab59 100644 --- a/scripts/query.mjs +++ b/scripts/query.mjs @@ -140,7 +140,7 @@ function createQueryApi(db) { const join = needsJoin ? 'LEFT JOIN sessions s ON s.id=tr.session_id' : ''; const errorCond = `(tr.is_error = 1 OR tr.content LIKE '${BASH_EXIT_PAT}')`; const allParams = [...filterParams, limit]; - const rows = db.prepare(`SELECT tr.* FROM tool_results tr ${join} LEFT JOIN messages rm ON rm.uuid=tr.message_uuid WHERE ${errorCond} AND ${where} LIMIT ?`).all(...allParams); + const rows = db.prepare(`SELECT tr.* FROM tool_results tr ${join} LEFT JOIN messages rm ON rm.uuid=tr.message_uuid WHERE ${errorCond} AND ${where} ORDER BY rm.timestamp DESC LIMIT ?`).all(...allParams); return rows.map(r => { const tc = db.prepare('SELECT * FROM tool_calls WHERE id=?').get(r.tool_use_id); const session = db.prepare('SELECT * FROM sessions WHERE id=?').get(r.session_id);