tommy0103
b52f57b538
Memory layer is now English-indexed: memories() query terms and remember() summaries must be English. Adds a runtime assertion that rejects CJK text in both paths, guiding the agent to translate non-English user requests before querying or writing memories. Ensures consistent retrieval regardless of conversation language.
134 lines
6.3 KiB
Markdown
134 lines
6.3 KiB
Markdown
# Obelisk Retrieval Semantics
|
|
|
|
Read this before designing a non-trivial query. This is the query design frame;
|
|
`pitfalls.md` is only the debug checklist.
|
|
|
|
## Four Principles
|
|
|
|
### Scope First
|
|
|
|
Classify the user's request before choosing tools.
|
|
|
|
| User signal | Locator mode | Start with | Avoid first |
|
|
|-------------|--------------|------------|-------------|
|
|
| unclear project/session landscape | orientation | `overview()` | treating overview rows as evidence |
|
|
| project name/path, session, cwd, file, time range | scope | `sessions()`, exact SQL on `project_path`, `sessionId`, `fileHistory()` | broad FTS |
|
|
| workflow, subagent, tool call, summary, edit | artifact | `workflows()`, `subagents()`, `summaries()`, `tool_calls`, `tool_results` | all-session search |
|
|
| concept, conclusion, design history, vague memory | semantic | `memories({ query })`, `search()`, summaries, bounded facet sweep | session dumps |
|
|
|
|
`overview()` is a navigation map: current cwd/project if knowable, global
|
|
project counts, and recent current-project session/memory entry points. Use it
|
|
when scope is unclear, then query the memory or raw session layer for evidence.
|
|
It does not guess the current session.
|
|
|
|
For a new task, the first pass normally starts with `overview({ limit: 6 })`
|
|
unless the user gave an exact session ID, message UUID, or absolute file path.
|
|
Broad synthesis and progress-summary tasks should start from
|
|
`references/query-patterns.md`, not raw SQL.
|
|
|
|
One-shot retrieval is not all-shot retrieval. A query script may perform
|
|
multiple steps, but the first locator should be the narrowest semantic fit. If a
|
|
scope locator finds the relevant project/session/file, do not also run broad FTS
|
|
unless scoped evidence is insufficient and `query_plan` says why.
|
|
|
|
Project-like fields are distinct:
|
|
|
|
- `sessions.project`: stored Claude Code project slug.
|
|
- `memories.project`: stored project slug copied onto registered memory records.
|
|
- `sessions.project_path`: absolute session path derived from message `cwd` when available; slug decoding is only a fallback.
|
|
- `messages.cwd`: working directory at message time.
|
|
- helper `project`: SQL `LIKE` over `sessions.project`, not exact membership.
|
|
|
|
For exact project membership, prefer helper filters or a scoped first pass when
|
|
they are expressive enough; use `sql()` with `s.project = ?` or
|
|
`s.project_path = ?` when you need exact membership across a join or
|
|
aggregation. Empty or tiny scoped results are valid results; do not broaden
|
|
unless the user asks or your `query_plan` explicitly marks a fallback.
|
|
|
|
### Plan Before Probe
|
|
|
|
For conclusion, broad history, failure investigation, or file evolution tasks,
|
|
prefer a retrieval script over interactive probing.
|
|
|
|
Good shape:
|
|
|
|
1. locate candidates with scope/artifact/semantic locators;
|
|
2. expand only selected hits;
|
|
3. dedupe and group in the script;
|
|
4. return compact evidence rows plus counts and limits.
|
|
|
|
If a second detail pass is needed, derive filters or facets from the first pass:
|
|
candidate sessions, discovered vocabulary, files, tools, timestamps, or
|
|
decisions. Prefer a learned faceted detail pass over `LIMIT 25` session windows.
|
|
If vocabulary is still unclear, use a small filtered window and say so in
|
|
`query_plan`.
|
|
|
|
### Structure Before Text
|
|
|
|
Use the database shape before asking the model to read text. This means
|
|
structured helpers and compact JS shaping first; raw SQL only when it expresses
|
|
the needed join, grouping, or exact schema-level check better than helpers.
|
|
|
|
- Count and aggregate in SQL or JS (`GROUP BY`, `COUNT`, `MAX`, `ORDER BY`, `LIMIT`).
|
|
- Join metadata from the owner table instead of inventing fields.
|
|
- Project compact rows; do not return whole sessions, complete workflow trees, full raw messages, or entire tool results.
|
|
- Keep synthesis runtime JSON around 10k-12k chars when possible.
|
|
- For recent failures, aggregate by session/task and return sparse examples.
|
|
- For file evolution, filter `fileHistory()` to `Edit`/`Write`, group by session or phase, and return short deltas.
|
|
|
|
Ordering and context are semantic:
|
|
|
|
- `sessions()`, `memories()`, `summaries()`, `workflows()`, and `failures()` are newest first.
|
|
- `fileHistory()` is oldest first.
|
|
- `search().context` is temporal neighbors in one session, not causal context.
|
|
- `context(uuid)` and `trace(uuid)` are for parent-chain/causal expansion.
|
|
|
|
### Evidence Before Conclusion
|
|
|
|
Obelisk's raw session layer stores original structure, not precompiled claims:
|
|
sessions, messages, summaries, tool calls/results, files, subagents, workflows,
|
|
parent chains, and raw JSONL windows. The memory layer can store
|
|
human-approved markdown conclusions, but treat them as prior notes to compare
|
|
against raw evidence when correctness matters.
|
|
|
|
For semantic questions, build a task-local evidence view:
|
|
|
|
```js
|
|
{
|
|
query_plan: { mode, scope, facets, limits },
|
|
prior_memories: [
|
|
{ id, path, session_id, created_at, summary }
|
|
],
|
|
evidence: [
|
|
{ type, id, session_id, timestamp, facet, snippet }
|
|
],
|
|
omitted: 0
|
|
}
|
|
```
|
|
|
|
Memory recall is English-indexed: translate non-English user requests into
|
|
concise English query terms before calling `memories({ query })`. Memory
|
|
summaries registered with `remember()` are also English, regardless of the
|
|
conversation language.
|
|
|
|
Then synthesize the conclusion in the final answer. Do not pretend the raw
|
|
evidence view is itself a stored Obelisk entity.
|
|
|
|
After synthesis, check whether the conclusion should become a memory. Offer to
|
|
write one when the result is durable, likely to help future sessions, and not
|
|
already covered by `prior_memories`. Good candidates include design decisions,
|
|
project conventions, abandoned alternatives, repeated failure causes, workflow
|
|
patterns, and conclusions synthesized across multiple raw evidence points. Do
|
|
not propose memory for one-off lookups, uncertain findings, or duplicate
|
|
coverage. The offer is only a proposal: write the markdown file and run
|
|
`--remember` only after user approval.
|
|
|
|
## Text Search Semantics
|
|
|
|
`search(text)` passes text to SQLite FTS5 `MATCH`.
|
|
|
|
- Hyphens tokenize: for `workflow-script`, use `"workflow script"` or SQL `LIKE` for literal punctuation.
|
|
- Special characters may produce FTS syntax errors; simplify or quote the FTS query under the same scope.
|
|
- Exact phrase, token search, and literal punctuation are different semantics.
|
|
- Results are ordered by `ORDER BY rank`; lower rank sorts earlier. Prefer returned order over "closer to zero" comparisons.
|