obelisk/skill-doc/references/retrieval-semantics.md

# Obelisk Retrieval Semantics

Read this before designing a non-trivial query. This is the query design frame;
`pitfalls.md` is only the debug checklist.

## Four Principles

### Scope First

Classify the user's request before choosing tools.

| User signal | Locator mode | Start with | Avoid first |
|-------------|--------------|------------|-------------|
| unclear project/session landscape | orientation | `overview()` | treating overview rows as evidence |
| project name/path, session, cwd, file, time range | scope | `sessions()`, exact SQL on `project_path`, `sessionId`, `fileHistory()` | broad FTS |
| workflow, subagent, tool call, summary, edit | artifact | `workflows()`, `subagents()`, `summaries()`, `tool_calls`, `tool_results` | all-session search |
| concept, conclusion, design history, vague memory | semantic | `memories({ query })`, `search()`, summaries, bounded facet sweep | session dumps |

`overview()` is a navigation map: current cwd/project if knowable, global
project counts, and recent current-project session/memory entry points. Use it
when scope is unclear, then query the memory or raw session layer for evidence.
It does not guess the current session.

For a new task, the first pass normally starts with `overview({ limit: 6 })`
unless the user gave an exact session ID, message UUID, or absolute file path.
Broad synthesis and progress-summary tasks should start from
`references/query-patterns.md`, not raw SQL.

One-shot retrieval is not all-shot retrieval. A query script may perform
multiple steps, but the first locator should be the narrowest semantic fit. If a
scope locator finds the relevant project/session/file, do not also run broad FTS
unless scoped evidence is insufficient and `query_plan` says why.

Project-like fields are distinct:

- `sessions.project`: provider-normalized project slug.
- `memories.project`: stored project slug copied onto registered memory records.
- `sessions.project_path`: absolute session path derived from message `cwd` when available; slug decoding is only a fallback.
- `messages.cwd`: working directory at message time.
- `sessions.source` / `messages.source`: transcript provider, currently `claude` or `codex`.
- helper `project`: SQL `LIKE` over `sessions.project`, not exact membership.
- helper `source`: optional provider filter. Omit it unless provenance matters.

For exact project membership, prefer helper filters or a scoped first pass when
they are expressive enough; use `sql()` with `s.project = ?` or
`s.project_path = ?` when you need exact membership across a join or
aggregation. Empty or tiny scoped results are valid results; do not broaden
unless the user asks or your `query_plan` explicitly marks a fallback.

### Plan Before Probe

For conclusion, broad history, failure investigation, or file evolution tasks,
prefer a retrieval script over interactive probing.

Good shape:

1. locate candidates with scope/artifact/semantic locators;
2. expand only selected hits;
3. dedupe and group in the script;
4. return compact evidence rows plus counts and limits.

If a second detail pass is needed, derive filters or facets from the first pass:
candidate sessions, discovered vocabulary, files, tools, timestamps, or
decisions. Prefer a learned faceted detail pass over `LIMIT 25` session windows.
If vocabulary is still unclear, use a small filtered window and say so in
`query_plan`.

### Structure Before Text

Use the database shape before asking the model to read text. This means
structured helpers and compact JS shaping first; raw SQL only when it expresses
the needed join, grouping, or exact schema-level check better than helpers.

- Count and aggregate in SQL or JS (`GROUP BY`, `COUNT`, `MAX`, `ORDER BY`, `LIMIT`).
- Join metadata from the owner table instead of inventing fields.
- Project compact rows; do not return whole sessions, complete workflow trees, full raw messages, or entire tool results.
- Keep synthesis runtime JSON around 10k-12k chars when possible.
- For recent failures, aggregate by session/task and return sparse examples.
- For file evolution, filter `fileHistory()` to `Edit`/`Write`, group by session or phase, and return short deltas.

Ordering and context are semantic:

- `sessions()`, `summaries()`, `workflows()`, and `failures()` are newest first.
- `memories()` without `query` is newest first; `memories({ query })` is FTS-ranked over memory `summary`/`path`, with lower rank sorting earlier.
- `fileHistory()` is oldest first.
- `search().context` is temporal neighbors in one session, not causal context.
- `context(uuid)` and `trace(uuid)` are for parent-chain/causal expansion.

### Evidence Before Conclusion

Obelisk's raw session layer stores original structure, not precompiled claims:
sessions, messages, summaries, tool calls/results, files, subagents, workflows,
parent chains, and raw JSONL windows. The memory layer can store
human-approved markdown conclusions, but treat them as prior notes to compare
against raw evidence when correctness matters.

For semantic questions, build a task-local evidence view:

```js
{
  query_plan: { mode, scope, facets, limits },
  prior_memories: [
    { id, path, anchors, session_id, created_at, summary }
  ],
  evidence: [
    { type, id, session_id, timestamp, content_type, is_meta, facet, snippet }
  ],
  omitted: 0
}
```

For message evidence, preserve `content_type` when projecting snippets.
`text` can support user-visible claims; `thinking` is only trace/debug context;
`tool_use` means follow `tool_calls` for structured details; `tool_result`
means follow `tool_results` for structured output. Mixed or unfamiliar message
surfaces remain `unknown`.

Preserve `is_meta` separately from `content_type`. Default message evidence
should exclude `is_meta=1` rows because they are transcript control-plane
content, not ordinary user intent or assistant conclusions. Include them only
when investigating injected caveats, command envelopes, or transcript structure.
When writing raw SQL for ordinary conversation evidence, add
`COALESCE(m.is_meta,0)=0` to message filters unless meta rows are the subject of
the investigation.

Memory recall is English-indexed: translate non-English user requests into
concise English query terms before calling `memories({ query })`. Memory
summaries registered with `remember()` are also English, regardless of the
conversation language.
`memories({ query })` uses safe FTS5 tokenization over memory `summary` and
`path`, so hyphens and punctuation do not need raw `MATCH` escaping.
`memories()` returns active memories only. For raw SQL memory recall, include
`deleted_at IS NULL`; archived memory records are management/audit data.

The agent may decide whether to use, ignore, or verify a recalled memory for the
current answer without user approval because no persistent state changes. If a
user explicitly says a memory is wrong, outdated, should be forgotten, or should
be replaced, that request is approval to mutate the exact matching memory. If
the agent discovers the conflict without an explicit user request, it should
answer from current evidence and ask before archiving or replacing the memory.

Then synthesize the conclusion in the final answer. Do not pretend the raw
evidence view is itself a stored Obelisk entity.

After synthesis, check whether the conclusion should become a memory. Offer to
write one when the result is durable, likely to help future sessions, and not
already covered by `prior_memories`. Good candidates include design decisions,
project conventions, abandoned alternatives, repeated failure causes, workflow
patterns, and conclusions synthesized across multiple raw evidence points. Do
not propose memory for one-off lookups, uncertain findings, or duplicate
coverage. The offer is only a proposal: write the markdown file and run
`--attune` only after user approval.

Memory updates are archive-plus-write, not in-place edits: run `forget()` on the
old record and `remember()` the replacement markdown file under the same user
approval.

## Text Search Semantics

`search(text)` passes text to SQLite FTS5 `MATCH`.

- Hyphens tokenize: for `workflow-script`, use `"workflow script"` or SQL `LIKE` for literal punctuation.
- Special characters may produce FTS syntax errors; simplify or quote the FTS query under the same scope.
- Exact phrase, token search, and literal punctuation are different semantics.
- Results are ordered by `ORDER BY rank`; lower rank sorts earlier. Prefer returned order over "closer to zero" comparisons.