diff --git a/README.md b/README.md index df39032..837ee47 100644 --- a/README.md +++ b/README.md @@ -108,9 +108,9 @@ references only when the question needs them: - `context(uuid)` — full story around a message (parent chain, subagent/workflow metadata) - `sql(query, ...params)` — read-only SQL for structured queries -**Structured shortcuts** — session, memory, summary, subagent, workflow, -file-history, failure, raw-window, and parent-chain helpers over the same -SQLite data. +**Structured shortcuts** — overview, session, memory, summary, subagent, +workflow, file-history, failure, raw-window, and parent-chain helpers over the +same SQLite data. **References** — agent reads on demand when the task needs deeper structure: diff --git a/SKILL.md b/SKILL.md index aaa5c6e..c80ba35 100644 --- a/SKILL.md +++ b/SKILL.md @@ -122,6 +122,7 @@ All list helpers accept a bounded `limit`. Many also accept: `{ project, after, before, sessionId, sessions, branch }`. Check the schema or a tiny sample before relying on less common filters. +- `overview(opts?)` -- compact orientation map. Returns current cwd/project if knowable, global project counts, and current-project recent sessions plus memory records. It is a map, not evidence. - `sessions(opts?)` -- session rows, newest first. `project` is a SQL `LIKE` pattern. - `recent(n?)` -- shorthand for recent sessions. - `summaries(opts?)` -- summary rows, newest first: `{ id, session_id, timestamp, source, content, session_title, project }`. @@ -140,9 +141,11 @@ tiny sample before relying on less common filters. Keep queries scoped, bounded, and structural. - Scope First: classify the locator as scope, artifact, or semantic. Use the narrowest structural locator before FTS; empty scoped results are valid unless the user asks to broaden. +- Orient When Needed: use `overview()` when the current project or available scopes are unclear. It is a navigation map; confirm facts with `memories()`, `search()`, helpers, or `sql()`. - Plan Before Probe: for conclusion, broad history, failure investigation, or file evolution, write a bounded retrieval script instead of spending turns on intermediate results. - Structure Before Text: compute counts, joins, grouping, dedupe, and projection in SQL or JS; keep runtime JSON compact, ideally under 10k-12k chars for synthesis tasks. - Evidence Before Conclusion: return compact evidence with stable IDs (`session_id`, `uuid`, `tool_call_id`, `run_id`, `agent_id`) and short snippets, then synthesize in the final answer. +- Persist Durable Conclusions: after answering, if retrieval produced a durable conclusion that future sessions are likely to reuse and `memories()` does not already cover it, explicitly offer to write a memory. Keep the offer brief. Do not write the markdown file or run `--remember` until the user approves. If field, context, ordering, FTS, or helper semantics affect the query, read `references/retrieval-semantics.md` before coding. If a query errors, read @@ -163,6 +166,12 @@ prior conclusions relevant to the current task. Like other list helpers, passing a string is treated as `sessionId`, and passing a number is treated as `limit`. Read the file at `path` for full content. +Good memory candidates include design decisions, project conventions, abandoned +alternatives, repeated failure causes, workflow patterns, and conclusions +synthesized across multiple raw evidence points. Do not propose memory for +one-off lookups, uncertain findings, or conclusions already covered by existing +memories. + **Writing memories:** after a retrieval produces a conclusion worth persisting, propose writing a memory file. The user must approve. Flow: diff --git a/references/query-patterns.md b/references/query-patterns.md index 298df8a..e905f50 100644 --- a/references/query-patterns.md +++ b/references/query-patterns.md @@ -4,6 +4,37 @@ These are copyable CodeAct patterns for `runtime.mjs --query` scripts, plus one `--remember` registration pattern. They are not new APIs. Adapt them to the user's scope and return compact evidence. +## Orient Before Retrieval + +Use `overview()` when the current project or available scopes are unclear. Treat +the result as a map, not evidence; follow up with `memories()`, `search()`, +helpers, or `sql()` for facts. + +```js +const map = overview({ limit: 6 }); +return { + current: map.current, + current_project: map.current_project && { + project: map.current_project.project, + session_total: map.current_project.session_total, + sessions: map.current_project.sessions.map(s => ({ + id: s.id, + title: s.title, + branch: s.git_branch, + ended_at: s.ended_at, + })), + memory_total: map.current_project.memory_total, + memories: map.current_project.memories.map(m => ({ + id: m.id, + path: m.path, + summary: m.summary?.slice(0, 240), + })), + }, + projects: map.projects.slice(0, 8), + totals: map.totals, +}; +``` + ## Bounded Search To Context Use `search()` to locate candidates, then expand only the strongest hits. diff --git a/references/retrieval-semantics.md b/references/retrieval-semantics.md index fc3091f..117095d 100644 --- a/references/retrieval-semantics.md +++ b/references/retrieval-semantics.md @@ -11,10 +11,16 @@ Classify the user's request before choosing tools. | User signal | Locator mode | Start with | Avoid first | |-------------|--------------|------------|-------------| +| unclear project/session landscape | orientation | `overview()` | treating overview rows as evidence | | project name/path, session, cwd, file, time range | scope | `sessions()`, exact SQL on `project_path`, `sessionId`, `fileHistory()` | broad FTS | | workflow, subagent, tool call, summary, edit | artifact | `workflows()`, `subagents()`, `summaries()`, `tool_calls`, `tool_results` | all-session search | | concept, conclusion, design history, vague memory | semantic | `memories({ query })`, `search()`, summaries, bounded facet sweep | session dumps | +`overview()` is a navigation map: current cwd/project if knowable, global +project counts, and recent current-project session/memory entry points. Use it +when scope is unclear, then query the memory or raw session layer for evidence. +It does not guess the current session. + One-shot retrieval is not all-shot retrieval. A query script may perform multiple steps, but the first locator should be the narrowest semantic fit. If a scope locator finds the relevant project/session/file, do not also run broad FTS @@ -94,6 +100,15 @@ For semantic questions, build a task-local evidence view: Then synthesize the conclusion in the final answer. Do not pretend the raw evidence view is itself a stored Obelisk entity. +After synthesis, check whether the conclusion should become a memory. Offer to +write one when the result is durable, likely to help future sessions, and not +already covered by `prior_memories`. Good candidates include design decisions, +project conventions, abandoned alternatives, repeated failure causes, workflow +patterns, and conclusions synthesized across multiple raw evidence points. Do +not propose memory for one-off lookups, uncertain findings, or duplicate +coverage. The offer is only a proposal: write the markdown file and run +`--remember` only after user approval. + ## Text Search Semantics `search(text)` passes text to SQLite FTS5 `MATCH`. diff --git a/references/schema.md b/references/schema.md index c1277d0..f74c921 100644 --- a/references/schema.md +++ b/references/schema.md @@ -416,6 +416,85 @@ const last5 = recent(5); return last5.map(s => ({ title: s.title, project: s.project_path, ended: s.ended_at })); ``` +#### `overview(opts?)` + +Compact orientation map for choosing the next retrieval scope. It is not an +evidence helper: it does not return snippets, full messages, or markdown file +contents. Passing a string is treated as `project`, and passing a number is +treated as the current-project session `limit`. + +| Param | Type | Description | +|-------|------|-------------| +| `opts.project` | `string` | Project slug or SQL `LIKE` pattern to use as the current project scope | +| `opts.limit` | `number` | Max recent sessions in `current_project.sessions` (default 8) | +| `opts.projectLimit` | `number` | Max rows in the global `projects` map (default 20) | +| `opts.memoryLimit` | `number` | Max memory records in `current_project.memories` (default 100) | + +If `opts.project` is absent, `overview()` tries to identify the current project +from `process.cwd()` against `sessions.project_path`, then from exact +`messages.cwd` matches. It does not guess the current session. + +**Returns:** + +```js +{ + current: { + cwd, + project: { + project, + project_path, + source: 'opts' | 'cwd_project_path' | 'cwd_messages', + confidence: 'exact' | 'inferred' | 'unknown' + } | null + }, + current_project: { + project, + project_path, + session_total, + sessions: [ + { id, title, project, project_path, started_at, ended_at, git_branch, message_count } + ], + memory_total, + memories: [ + { id, path, summary, session_id, project, created_at } + ] + } | null, + projects: [ + { + project, + project_path, + session_count, + memory_count, + last_session_at, + last_memory_at, + recent_branches + } + ], + totals: { projects, sessions, memories } +} +``` + +Use `current_project.sessions` as recent entry points only. `session_total` and +`memory_total` tell you whether the returned arrays are complete enough for the +task. Confirm facts with `memories()`, `search()`, other helpers, or `sql()`. + +```js +const map = overview({ limit: 5 }); +return { + current: map.current, + sessions: map.current_project?.sessions.map(s => ({ + id: s.id, + title: s.title, + ended_at: s.ended_at, + })), + memories: map.current_project?.memories.map(m => ({ + id: m.id, + path: m.path, + summary: m.summary, + })), +}; +``` + #### `sessions(opts?)` Query sessions with filters. For backward compatibility, passing a number is treated as `limit`. diff --git a/scripts/query.mjs b/scripts/query.mjs index 20e4802..12152c1 100644 --- a/scripts/query.mjs +++ b/scripts/query.mjs @@ -40,6 +40,13 @@ function createQueryApi(db) { return db.prepare(sql).all(...p); }; + const normalizeOverviewOpts = (optsOrScalar) => { + if (optsOrScalar == null) return {}; + if (typeof optsOrScalar === 'string') return { project: optsOrScalar }; + if (typeof optsOrScalar === 'number') return { limit: optsOrScalar }; + return optsOrScalar; + }; + const search = (text, opts = {}) => { const { limit = 20, sessionId, project, after, before, cwd } = opts; let where = 'WHERE mf.text MATCH ?'; @@ -181,6 +188,173 @@ function createQueryApi(db) { return db.prepare(`SELECT su.*, s.title as session_title, s.project FROM summaries su LEFT JOIN sessions s ON s.id=su.session_id WHERE ${where} ORDER BY su.timestamp DESC LIMIT ?`).all(...params); }; + const overview = (optsOrScalar) => { + const opts = normalizeOverviewOpts(optsOrScalar); + const cwd = process.cwd(); + const sessionLimit = opts.limit ?? 8; + const projectLimit = opts.projectLimit ?? 20; + const memoryLimit = opts.memoryLimit ?? 100; + + const projectDescriptor = (row, source, confidence) => row ? ({ + project: row.project, + project_path: row.project_path || null, + source, + confidence, + }) : null; + + const latestProjectByPattern = (pattern) => { + const fromSessions = db.prepare(` + SELECT project, project_path + FROM sessions + WHERE project LIKE ? + ORDER BY COALESCE(ended_at, started_at) DESC + LIMIT 1 + `).get(pattern); + if (fromSessions) return fromSessions; + return db.prepare(` + SELECT project, NULL AS project_path + FROM memories + WHERE project LIKE ? + ORDER BY created_at DESC + LIMIT 1 + `).get(pattern); + }; + + const resolveCurrentProject = () => { + if (opts.project) { + const row = latestProjectByPattern(opts.project); + const confidence = row ? (/[%_]/.test(opts.project) ? 'inferred' : 'exact') : 'unknown'; + return projectDescriptor(row || { project: opts.project, project_path: null }, 'opts', confidence); + } + + const paths = db.prepare(` + SELECT project, project_path, MAX(COALESCE(ended_at, started_at)) AS last_seen + FROM sessions + WHERE project IS NOT NULL AND project_path IS NOT NULL AND project_path != '' + GROUP BY project, project_path + `).all(); + const byProjectPath = paths + .filter(r => cwd === r.project_path || cwd.startsWith(r.project_path + path.sep)) + .sort((a, b) => b.project_path.length - a.project_path.length || String(b.last_seen || '').localeCompare(String(a.last_seen || '')))[0]; + if (byProjectPath) return projectDescriptor(byProjectPath, 'cwd_project_path', 'exact'); + + const byMessageCwd = db.prepare(` + SELECT s.project, s.project_path, MAX(m.timestamp) AS last_seen + FROM messages m + LEFT JOIN sessions s ON s.id=m.session_id + WHERE m.cwd = ? AND s.project IS NOT NULL + GROUP BY s.project, s.project_path + ORDER BY last_seen DESC + LIMIT 1 + `).get(cwd); + if (byMessageCwd) return projectDescriptor(byMessageCwd, 'cwd_messages', 'inferred'); + + return null; + }; + + const projects = db.prepare(` + WITH names AS ( + SELECT project FROM sessions WHERE project IS NOT NULL GROUP BY project + UNION + SELECT project FROM memories WHERE project IS NOT NULL GROUP BY project + ), + session_stats AS ( + SELECT project, COUNT(*) AS session_count, MAX(COALESCE(ended_at, started_at)) AS last_session_at + FROM sessions + WHERE project IS NOT NULL + GROUP BY project + ), + memory_stats AS ( + SELECT project, COUNT(*) AS memory_count, MAX(created_at) AS last_memory_at + FROM memories + WHERE project IS NOT NULL + GROUP BY project + ) + SELECT + n.project, + ( + SELECT s2.project_path + FROM sessions s2 + WHERE s2.project = n.project AND s2.project_path IS NOT NULL + ORDER BY COALESCE(s2.ended_at, s2.started_at) DESC + LIMIT 1 + ) AS project_path, + COALESCE(ss.session_count, 0) AS session_count, + COALESCE(ms.memory_count, 0) AS memory_count, + ss.last_session_at, + ms.last_memory_at + FROM names n + LEFT JOIN session_stats ss ON ss.project = n.project + LEFT JOIN memory_stats ms ON ms.project = n.project + ORDER BY COALESCE(ss.last_session_at, ms.last_memory_at) DESC + LIMIT ? + `).all(projectLimit).map(row => { + const branches = db.prepare(` + SELECT git_branch + FROM sessions + WHERE project = ? AND git_branch IS NOT NULL AND git_branch != '' + GROUP BY git_branch + ORDER BY MAX(COALESCE(ended_at, started_at)) DESC + LIMIT 5 + `).all(row.project).map(r => r.git_branch); + return { ...row, recent_branches: branches }; + }); + + const currentProject = resolveCurrentProject(); + let current_project = null; + if (currentProject?.project) { + const sessionTotal = db.prepare('SELECT COUNT(*) AS c FROM sessions WHERE project = ?').get(currentProject.project)?.c || 0; + const sessionsForProject = db.prepare(` + SELECT id, title, project, project_path, started_at, ended_at, git_branch, message_count + FROM sessions + WHERE project = ? + ORDER BY COALESCE(ended_at, started_at) DESC + LIMIT ? + `).all(currentProject.project, sessionLimit); + const memoryTotal = db.prepare('SELECT COUNT(*) AS c FROM memories WHERE project = ?').get(currentProject.project)?.c || 0; + const memoriesForProject = db.prepare(` + SELECT id, path, summary, session_id, project, created_at + FROM memories + WHERE project = ? + ORDER BY created_at DESC + LIMIT ? + `).all(currentProject.project, memoryLimit); + current_project = { + project: currentProject.project, + project_path: currentProject.project_path, + session_total: sessionTotal, + sessions: sessionsForProject, + memory_total: memoryTotal, + memories: memoriesForProject, + }; + } + + const totalProjects = db.prepare(` + SELECT COUNT(*) AS c + FROM ( + SELECT project FROM sessions WHERE project IS NOT NULL GROUP BY project + UNION + SELECT project FROM memories WHERE project IS NOT NULL GROUP BY project + ) + `).get()?.c || 0; + const totalSessions = db.prepare('SELECT COUNT(*) AS c FROM sessions').get()?.c || 0; + const totalMemories = db.prepare('SELECT COUNT(*) AS c FROM memories').get()?.c || 0; + + return { + current: { + cwd, + project: currentProject, + }, + current_project, + projects, + totals: { + projects: totalProjects, + sessions: totalSessions, + memories: totalMemories, + }, + }; + }; + const resolveJsonlPath = (messageUuid) => { const msg = db.prepare('SELECT session_id, agent_id FROM messages WHERE uuid=?').get(messageUuid); if (!msg) return null; @@ -251,7 +425,7 @@ function createQueryApi(db) { return db.prepare(`SELECT mem.* FROM memories mem ${join} WHERE ${where} ORDER BY mem.created_at DESC LIMIT ?`).all(...params); }; - return { sql: q, search, context, trace, thread, subagents, workflows, workflowTree, fileHistory, failures, sessions, recent, summaries, raw, memories }; + return { sql: q, search, context, trace, thread, subagents, workflows, workflowTree, fileHistory, failures, sessions, recent, summaries, raw, memories, overview }; } function createRememberApi(db) {