feat: evolve retrieval layer with content_type, memory FTS, forget(), and recap references

Extract schema DDL into scripts/schema.sql. Add content_type and is_meta
  columns to messages for transcript control-plane filtering. Introduce
  FTS5-backed memory recall with safe tokenization, memory soft-delete via
  forget() through the renamed --attune runtime, and anchors on memory
  records. Expand query helpers (includeMeta, thread opts, overview
  project-path awareness). Add per-card recap retrieval and writing
  references under references/recap/.
This commit is contained in:
tommy0103
2026-06-15 02:25:21 +08:00
parent b52f57b538
commit 3822bda89e
24 changed files with 1290 additions and 157 deletions
+59 -5
View File
@@ -1,7 +1,7 @@
# Obelisk Query Patterns
These are copyable CodeAct patterns for `runtime.mjs --query` scripts, plus one
`--remember` registration pattern. They are not new APIs. Adapt them to the
These are copyable CodeAct patterns for `runtime.mjs --query` scripts plus
`--attune` memory mutation patterns. They are not new APIs. Adapt them to the
user's scope and return compact evidence.
Read this before the first query for broad synthesis, progress summaries,
@@ -43,14 +43,17 @@ return {
memories: map.current_project.memories.map(m => ({
id: m.id,
path: m.path,
anchors: m.anchors,
summary: m.summary?.slice(0, 240),
})),
},
prior_memories: memories({ ...scoped, query: topic, limit: 5 }).map(m => ({
id: m.id,
path: m.path,
anchors: m.anchors,
session_id: m.session_id,
created_at: m.created_at,
rank: m.rank,
summary: m.summary?.slice(0, 260),
})),
session_evidence: search(topic.replace(/[-_]/g, ' '), { ...scoped, limit: 8 })
@@ -88,6 +91,7 @@ return {
memories: map.current_project.memories.map(m => ({
id: m.id,
path: m.path,
anchors: m.anchors,
summary: m.summary?.slice(0, 240),
})),
},
@@ -138,11 +142,13 @@ const prior_memories = memories({
}).map(m => ({
id: m.id,
path: m.path,
anchors: m.anchors,
session_id: m.session_id,
message_start: m.message_start,
message_end: m.message_end,
created_at: m.created_at,
summary: m.summary?.slice(0, 260),
rank: m.rank,
}));
const session_evidence = search(ftsTopic, { project, limit: 8 })
@@ -167,14 +173,14 @@ return {
};
```
## Register Approved Memory
## Attune Approved Memory
Use this only after the user approves writing memory and the markdown file
already exists. `remember()` validates the file and stores a normalized absolute
path, so keep the script small and return the registered record.
Run this script with `runtime.mjs --remember <script>`. The `--remember` runtime
exposes only `remember()`, not retrieval helpers.
Run this script with `runtime.mjs --attune <script>`. The `--attune` runtime
exposes only `remember()` and `forget()`, not retrieval helpers.
```js
return remember({
@@ -182,6 +188,7 @@ return remember({
session_id: 'source-session-id',
message_start: 'first-message-uuid',
message_end: 'last-message-uuid',
anchors: [{ kind: 'file', path: 'SKILL.md' }],
summary: [
'Decision: Obelisk uses one user-facing entry that queries both memory and raw sessions.',
'Memory records are prior notes and must be identified naturally when they influence an answer.',
@@ -190,6 +197,53 @@ return remember({
});
```
## Forget Approved Memory
Use this only after the user asks to archive an outdated or wrong memory. Identify
the exact memory ID in a normal `--query` script first. If one candidate clearly
matches the user's request, that request is approval to archive it; if several
candidates match, ask which one to forget.
Run the mutation with `runtime.mjs --attune <script>`:
```js
return forget({
id: 'mem-id-to-delete',
reason: 'Outdated by newer project guidance.',
});
```
`forget()` archives the record. Active recall through `memories()` will omit it,
and the markdown file at `path` is left in place.
## Update Approved Memory
Use this when the user explicitly corrects an existing memory, or after the
agent proposes a replacement and the user approves. An update is one combined
operation: archive the old record and register the replacement markdown file.
The new markdown file must already exist before running `--attune`.
```js
const archived = forget({
id: 'old-memory-id',
reason: 'Replaced by updated memory from the current session.',
});
const created = remember({
path: '.obelisk/memories/updated-memory.md',
session_id: 'current-session-id',
message_start: 'first-message-uuid',
message_end: 'last-message-uuid',
anchors: [{ kind: 'file', path: 'src/path/to/file.ts' }],
summary: 'Updated summary: concise English retrieval surface for the replacement memory.',
});
return { archived, created };
```
If the agent only suspects a memory is stale, do not run this pattern yet.
Answer from current evidence and ask whether to archive or replace the memory.
## One-Shot Retrieval For Synthesis
Use this for conclusion, broad history, failure investigation, or file evolution
+9
View File
@@ -0,0 +1,9 @@
# Obelisk Recap Retrieval Patterns
Compatibility pointer.
The `/obelisk recap` flow now starts at `references/recap/overview.md`.
Do not use this as an all-in-one retrieval document. The current flow is
card-by-card: read the overview, then for each card read its `patternN-*.md`,
retrieve that card's evidence, read its `writingN-*.md`, and update the JSON
before moving on.
+8
View File
@@ -0,0 +1,8 @@
# Obelisk Recap Writing
Compatibility pointer.
The `/obelisk recap` writing contract now lives in the per-card writing files
under `references/recap/`, coordinated by `references/recap/overview.md`.
Read the overview first. Then use each per-card writing file immediately after
that card's retrieval pattern, rather than loading one large writing prompt.
+74
View File
@@ -0,0 +1,74 @@
# Obelisk Recap Overview
Use this only when the first word after `/obelisk` is `recap`. Everything after
`recap` is the target period or style hint.
## Highest Priority: Phase Loop
This workflow is sequential. Do not preload all recap files. Do not gather all
evidence first and write all cards at the end.
Follow this loop exactly:
1. Resolve the target period from the user's phrase.
2. Run only a tiny orientation pass such as `overview({ limit: 6 })`.
3. For Card 1, read `pattern1-cover.md`.
4. Retrieve only Card 1 evidence.
5. Read `writing1-cover.md`.
6. Update/write the JSON for Card 1 now.
7. Only after the JSON is updated, move to Card 2 and repeat.
Card order:
| card | retrieve | write |
|---|---|---|
| 1 cover | `pattern1-cover.md` | `writing1-cover.md` |
| 2 thinking | `pattern2-thinking.md` | `writing2-thinking.md` |
| 3 vibe | `pattern3-vibe.md` | `writing3-vibe.md` |
| 4 workflow | `pattern4-workflow.md` | `writing4-workflow.md` |
| 5 closing | `pattern5-closing.md` | `writing5-closing.md` |
The per-card files own retrieval details, JSON field duties, and card-specific
taste. Do not move those concerns back into this file.
## Period Targets
- `this week`, `last week`: calendar week in the user's runtime timezone.
- `this month`, `last month`: calendar month in the user's runtime timezone.
Do not infer timezone from examples, UTC suffixes, or file timestamps when
runtime/session timezone is available.
## Overall Deck Taste
This is a Spotify Wrapped-like set of personal share cards: concise, designed,
slightly playful, and built to make the user's work feel seen.
Do not criticize the user. Do not scold, diagnose, rank their personality, or
turn friction into a performance review.
Use designed English chrome where it feels like card UI: week/month labels,
archetype labels, compact stats, verdict seals, and signoffs. Preserve the
user's own language for prompts, quotes, catchphrases, and reactions. This is
not a translation task.
The deck should feel like a small artifact from someone who noticed the week,
not a report generated from a database.
## Archetypes
Choose one dominant archetype from the period's dominant attention, not from the
current recap-generation session.
| archetype | when it fits | tone baseline |
|---|---|---|
| `architect` | structure, boundaries, schema, systems | matter-of-fact structural pride |
| `debugger` | symptoms, false positives, root-cause loops | wry and bug-comfortable |
| `shipper` | dense implementation cadence | energetic but not breathless |
| `curator` | organization, memory, docs, refinement | reflective and precise |
| `director` | workflows, subagents, orchestration | observant from a slight remove |
| `cartographer` | moving boundaries and redrawing maps | patient and surveyor-like |
| `wanderer` | many projects without one center | gentle, exploratory |
If two fit, pick the one that describes what the user spent more thinking time
on, not what shipped.
+36
View File
@@ -0,0 +1,36 @@
# Card 1 Cover Retrieval
Goal: choose the recap's dominant claim, persona, activity shape, and compact
footer. The cover is not a topic inventory; it is one glanceable claim about
what the period felt like.
Use the period from `overview.md`. Start from `overview({ limit: 6 })`, then
look at in-period sessions, summaries, memories, and any obvious project scope.
If the user asked for a project, keep that scope; otherwise prefer the current
project only when the evidence makes it the clear center.
Prefer helpers first. If you need custom SQL for activity, token/message counts,
or source-session scope, read `references/schema.md` before writing the SQL.
Retrieve:
- dominant claim: one thing that defined the period, supported by raw evidence;
- persona: which archetype best matches the user's attention;
- source sessions and memories used by this cover claim;
- activity: weekly day intensities or monthly day intensities when supported;
- footer: compact public metric such as sessions, messages, or tokens.
Avoid:
- a claim that lists three topics;
- an archetype chosen from the recap-generation session itself;
- footer caveats like excluded projects, exact SQL filters, or long session names;
- making the cover a workflow metric when the week was really about a decision.
Read this card's writing file immediately after the cover evidence is stable:
`references/recap/writing1-cover.md`. Then update the JSON fields
`period`, `source`, `metrics`, `persona`, and the first `cards[]` entry.
Do not read `pattern2-thinking.md` until this JSON update is done.
Stop when the cover has one evidence-backed dominant claim, one chosen persona,
one metric scope, and at least one `evidence` anchor.
+33
View File
@@ -0,0 +1,33 @@
# Card 2 Thinking Retrieval
Goal: find turning points. This card is not a project timeline and not an implementation log. It is the record of what changed in the user's mind.
Retrieve 3-6 turns. A turn needs both sides:
- the user question, friction, doubt, or request that started the turn;
- the later decision, reframing, finding, or constraint that replaced the earlier
state.
Useful searches:
- user questions in the period: "为什么", "是不是", "怎么", "我觉得", "不应该";
- places where the user corrected the direction and then approved a new frame;
- summaries that name decisions, followed by `context()` or `thread()` for the
user's actual words;
- memory records only as hints; raw session evidence must provide the prompt and
turn.
Prefer helpers first. If you need custom SQL for message windows or user-turn
counts, read `references/schema.md` before writing the SQL.
Do not use workflow names, feature names, or agent task labels as prompts when
the user had their own wording. Do not use counts, "5 rounds", "13 agents", or
implementation effort as turns unless that count is the turn itself.
Read this card's writing file immediately after the turns are chosen:
`references/recap/writing2-thinking.md`. Then update the JSON `thinking_path`
card and add evidence for each item.
Do not read `pattern3-vibe.md` until this JSON update is done.
Stop when each item has a source-language prompt label, a short changed-state
prompt, turn, and an `evidence` anchor.
+30
View File
@@ -0,0 +1,30 @@
# Card 3 Vibe Retrieval
Goal: find small human signals in visible user messages. Vibe is not a correction log, not bracketed runtime text, and not a psychological profile.
Look for:
- catchphrases and repeated tiny reactions;
- unusually blunt praise or rejection;
- late-night disbelief, jokes, or rituals;
- one quotable sentence that captures the period's character.
Only count visible user messages. Helper APIs omit meta by default, but custom
SQL for phrase counts must filter user text with `COALESCE(m.is_meta,0)=0` and
`m.content_type='text'`. Do not count tool results, injected command envelopes,
UI labels, or bracketed runtime strings.
Useful retrieval:
- targeted phrase counts after you notice a likely catchphrase;
- `thread(sessionId)` around high-energy moments;
- `search()` for exact phrases, then `context()` for timing;
- a bounded SQL count only after reading `references/schema.md`.
Read this card's writing file immediately after you have the small user signals:
`references/recap/writing3-vibe.md`. Then update the JSON `vibe` card and add
evidence for every quote, count, and timestamp.
Do not read `pattern4-workflow.md` until this JSON update is done.
Stop when every observation is either exact user words or a tiny label backed by
exact user words.
+41
View File
@@ -0,0 +1,41 @@
# Card 4 Workflow Retrieval
Goal: find actual workflow runs and how the user received them. Card 4 is about
orchestration as experienced by the user, not an agent performance table.
Workflow rows have their own `workflows.timestamp`. During this card's retrieval,
call `workflows({ project: projectLike, after, before })` before concluding the
period had zero workflows.
Prefer helpers first. If you need custom SQL for workflow joins, timestamps, or
message reactions, read `references/schema.md` before writing the SQL.
Do not derive workflow counts only from `sessions({ after, before })`: long
sessions can start before the period and still contain workflow runs inside the
period. Do not scope workflow lookup by exact `project_path`; nested cwd values
can belong to the same Claude project slug.
For each candidate workflow:
- get the actual workflow_name from `workflows()` or `workflowTree()`;
- collect run id, timestamp, project, agent count, and compact result for stats
and evidence only;
- search the parent session for the user message immediately following the workflow completion;
- use that user reaction as `items[].reaction`.
Rank rows by the strength of the user reaction, not by agent count, workflow
size, duration, or implementation importance. A small workflow with "完美" is a
better row than a large workflow with no visible response.
Do not use architecture topics, memory-system milestones, app modules, or recap
feature work as workflow rows unless they are actual workflow_name values.
Do not make a row for a workflow with no visible user reaction; keep it only in
`stats`, `metrics`, or `evidence`.
Read this card's writing file immediately after workflow evidence is stable:
`references/recap/writing4-workflow.md`. Then update the JSON `workflow` card,
top-level workflow metrics, and source session ids for workflow evidence.
Do not read `pattern5-closing.md` until this JSON update is done.
Stop when every displayed row has an actual workflow name and a visible user
reaction.
+35
View File
@@ -0,0 +1,35 @@
# Card 5 Closing Retrieval
Goal: close with a small personal receipt. Use the same period and source scope
as the recap, or explicitly record a wider metric in `evidence`.
Retrieve:
- one consistent metric that can stand alone, such as streak, active days,
sessions, messages, or workflows;
- one or two compact receipts;
- most said phrase, only if a real repeated user phrase is supported;
- signoff material from the period's mood, not a second summary.
For phrase counts, count only non-meta visible user text. For streaks and active
days, define whether the scope is all Obelisk data, the current project, or the
selected evidence sessions. Keep the scope consistent with the cover footer
unless the evidence explicitly says otherwise.
Prefer helpers first. If you need custom SQL for phrase counts, active days, or
streaks, read `references/schema.md` before writing the SQL.
Avoid:
- naked numbers without units;
- project report bullets;
- internal session names;
- token audits;
- slogans, advice, or next-action commands.
Read this card's writing file immediately after the closing receipt is chosen:
`references/recap/writing5-closing.md`. Then update the JSON `closing` card and
add evidence for counts and phrases.
This is the final card; save the completed JSON before replying.
Stop when the closing can end the deck without explaining the whole week again.
+83
View File
@@ -0,0 +1,83 @@
# Card 1 Cover Writing
The cover should be readable in one glance: badge, persona, one plain claim,
activity, footer. Before writing, say the claim to the user in a chat bubble.
If it sounds like a topic list or report heading, shrink it.
## Mock taste anchor
```json
{
"type": "cover",
"badge": "Week 24",
"title": "The Architect",
"claim": "从零设计了一个完整的 memory 系统。",
"activity": [0.85, 0.92, 0.72, 0.45, 0.88, 0.30, 0],
"footer": "12 sessions · 2.4M tokens"
}
```
This works because `从零设计了一个完整的 memory 系统。` is one plain claim, in
the user's language, and can be read in one breath. `The Architect` is English
chrome; it gives the card a designed surface without translating the user's
actual work.
## JSON Shape
```ts
type CoverCard = {
type: "cover";
badge: string;
title: string;
claim: string;
activity: number[];
footer: string;
evidence_refs?: string[];
};
```
Field duties:
- `badge`: compact period chrome, such as `Week 24`.
- `title`: persona label, usually `The Architect`, `The Debugger`, etc.
- `claim`: one plain claim; not a topic list, project inventory, colon-led
tagline, or clever English that hides the user's language.
- `activity`: period intensity values from retrieval.
- `footer`: public metric line with no internal filter notes.
After writing, check that `persona.claim` and `cover.claim` tell the same
story, and attach `evidence_refs` to the claim or metric if it is surprisingly
specific.
## First JSON Write
After Card 1, create or update the recap JSON file. Do this before reading
`pattern2-thinking.md`.
Use this top-level shape:
```ts
type Recap = {
schema_version: "obelisk.recap.v1";
kind: "weekly" | "monthly";
generated_at: string;
period: { label: string; start: string; end: string; timezone: string };
source: { project?: string; session_ids: string[]; memory_ids?: string[] };
metrics: {
sessions?: number;
messages?: number;
tokens?: number;
active_days?: number[];
streak_days?: number;
workflows?: number;
workflow_agents?: number;
corrections?: number;
};
persona: { archetype: string; title: string; claim: string; tone: string };
cards: [CoverCard, { type: "thinking_path" }, { type: "vibe" }, { type: "workflow" }, { type: "closing" }];
evidence?: Array<{ id: string; session_id?: string; message_uuid?: string; memory_id?: string; summary?: string }>;
};
```
For app handoff, write JSON under `~/.obelisk/recap/`. Weekly filenames are
`recap-{YYYY}-W{WW}.json`; monthly filenames are `recap-{YYYY}-{MM}.json`.
+49
View File
@@ -0,0 +1,49 @@
# Card 2 Thinking Writing
Thinking Path should feel like a few bends in the user's reasoning, not a
weekly changelog. Before writing, test each row by asking: "what changed here?"
## Mock taste anchor
```json
{
"type": "thinking_path",
"title": "Five questions, five turns.",
"items": [
{ "day": "Mon", "prompt": "为什么要把 session 编译成 wiki", "turn": "raw SQLite, no wiki" },
{ "day": "Tue", "prompt": "buildWhere 是什么", "turn": "unified filter opts, not DSL" },
{ "day": "Wed", "prompt": "failures() 90% 误报", "turn": "is_error in JSONL" },
{ "day": "Thu", "prompt": "memory 层需要清理机制吗", "turn": "soft-delete, human-only" },
{ "day": "Fri", "prompt": "热力图不选中默认显示本月", "turn": "GitHub-style activity timeline" }
]
}
```
The prompts stay close to the user's words. Each turn is a short decision
fragment, not a full explanation.
## JSON Shape
```ts
type ThinkingPathCard = {
type: "thinking_path";
title: string;
items: Array<{
day: string;
prompt: string;
turn: string;
evidence_refs?: string[];
}>;
};
```
Field duties:
- `title`: designed deck line, not `本周路径`, not a research-paper heading.
- `prompt`: user's compact question, friction, or task. Use source language.
- `turn`: short decision fragment, finding, or shift; usually under 10 words.
Compact English fragments are allowed when they work as designed chrome.
After writing, remove any row whose prompt is a workflow name or whose turn
describes implementation rather than changed thinking.
Update the JSON now before reading `pattern3-vibe.md`.
+73
View File
@@ -0,0 +1,73 @@
# Card 3 Vibe Writing
Vibe is affectionate observation. It should make the user recognize themselves
without feeling evaluated. Before writing, remove anything that reads like a
correction audit, behavior label, diagnosis, or complaint ledger.
## Mock taste anchor
```json
{
"type": "vibe",
"title": "A short character study.",
"voice_lines": [
{ "label": "catchphrase", "text": "这太丑了", "count": 4 },
{ "label": "highest praise", "text": "可以" },
{ "label": "late night", "text": "你在干什么", "time": "02:47 AM" }
],
"meter": {
"label": "patience",
"value": 0.78,
"caption": "saint"
},
"quote": {
"text": "若无必要,勿增实体。",
"caption": "your most philosophical moment"
}
}
```
The humor comes from exact small lines. `可以` is funnier and truer than
"approval signal".
## JSON Shape
```ts
type VibeCard = {
type: "vibe";
title: string;
voice_lines: Array<{
label: string;
text: string;
count?: number;
time?: string;
evidence_refs?: string[];
}>;
meter?: {
label: string;
value: number;
caption: string;
};
quote?: {
text: string;
caption?: string;
evidence_refs?: string[];
};
};
```
Field duties:
- `title`: light character-study line, not a scorecard.
- `voice_lines[].text`: exact user words; no paraphrase, translation,
ellipsized half-quote, meta text, or correction log.
- `voice_lines[].label`: designed chrome can be English; the quoted user text
stays in source language.
- `meter`: meter is not a diagnosis. Keep the caption one or two words and
affectionate, never punitive.
- `quote.text`: one exact user phrase or sentence.
Do not use `[Request interrupted by user]`, tool output, injected context, or
UI status text as vibe. After writing, verify every `voice_lines[].text` and
`quote.text` can be traced to a non-meta user message.
Update the JSON now before reading `pattern4-workflow.md`.
+68
View File
@@ -0,0 +1,68 @@
# Card 4 Workflow Writing
Workflow is the orchestration card. It should show the strongest few workflow
runs and the user's reaction to them. Before writing, remove any row whose
reaction is not traceable to a visible user reaction.
## Mock taste anchor
```json
{
"type": "workflow",
"title": "Three workflows. Forty-two agents.",
"deck": "你召唤了机器军团。结果各有不同。",
"stats": "3 workflows · 42 agents",
"items": [
{ "name": "hono-plugin-review", "reaction": "完美" },
{ "name": "vue-migration", "reaction": "你这页面完全和之前的不一样…" },
{ "name": "split-render-js", "reaction": "可以" }
],
"verdict": "Mostly tolerated."
}
```
The row reactions are user reactions. The title carries the metric; the verdict
is a small English seal.
## JSON Shape
```ts
type WorkflowCard = {
type: "workflow";
title: string;
deck?: string;
stats?: string;
items: Array<{
name: string;
reaction: string;
evidence_refs?: string[];
}>;
verdict: string;
};
```
Field duties:
- `title`: human story line or compact metric line.
- `deck`: optional second line; do not repeat stats mechanically.
- `stats`: compact count line.
- `items[].name`: actual workflow name, command name, or run-id prefix.
- `items[].reaction`: exact or lightly trimmed user reaction. Preserve source
language. No feature description, implementation summary, agent count,
duration, "framework switch", "modularization", "theming landed", or other
internal progress label.
- `verdict`: compact seal based on the row reactions, often 3-6 words.
Agent counts belong only in `title` or `stats`, never in `items[].reaction`.
These row values are invalid because they are implementation labels, not user
reactions:
- `13 agents, the big build` is invalid.
- `9 agents, framework switch` is invalid.
- `6 agents, modularization` is invalid.
- `theming landed` is invalid.
If no user reaction exists, omit the row rather than write an implementation result.
After writing, check that every row name maps to retrieval evidence and every
reaction can be read as quoted user verdict text.
Update the JSON now before reading `pattern5-closing.md`.
+54
View File
@@ -0,0 +1,54 @@
# Card 5 Closing Writing
Closing is a receipt, not a second summary. Before writing, read the headline
alone. If it does not mean anything without the rest of the card, add the unit
or choose a better line.
## Mock taste anchor
```json
{
"type": "closing",
"headline": "19 days",
"receipts": ["847 messages exchanged", "12 corrections · 47 approvals"],
"most_said_phrase": "好的开始做吧",
"signoff": "See you next week."
}
```
This works because `19 days` has a unit, the receipts feel like a small receipt,
and `See you next week.` is a quiet goodbye instead of a slogan.
## JSON Shape
```ts
type ClosingCard = {
type: "closing";
headline: string;
receipts: string[];
most_said_phrase?: string;
signoff: string;
evidence_refs?: string[];
};
```
Field duties:
- `headline`: compact stat or phrase with its unit; not a naked number.
- `receipts`: at most two `receipts`, compact and personal.
- `most_said_phrase`: complete phrase the user actually said, or omit it.
- `signoff`: short and earned; quiet goodbye, not advice or a brand slogan.
English signoff chrome such as `See you next week.` is allowed.
After writing, remove internal scope notes from visible fields and put them in
`evidence`. The final card should feel like the deck ending, not the report
continuing.
Final save rules:
- The file contains only the JSON object: no Markdown fence, no prose.
- Keep exactly five cards in this order: cover, thinking_path, vibe, workflow,
closing.
- Keep private SQL, raw tool output, secrets, long paths, and source caveats out
of visible card text; put traceability in `evidence`.
- After saving, reply briefly with the saved path and important evidence caveats.
+34 -4
View File
@@ -78,7 +78,8 @@ the needed join, grouping, or exact schema-level check better than helpers.
Ordering and context are semantic:
- `sessions()`, `memories()`, `summaries()`, `workflows()`, and `failures()` are newest first.
- `sessions()`, `summaries()`, `workflows()`, and `failures()` are newest first.
- `memories()` without `query` is newest first; `memories({ query })` is FTS-ranked over memory `summary`/`path`, with lower rank sorting earlier.
- `fileHistory()` is oldest first.
- `search().context` is temporal neighbors in one session, not causal context.
- `context(uuid)` and `trace(uuid)` are for parent-chain/causal expansion.
@@ -97,19 +98,44 @@ For semantic questions, build a task-local evidence view:
{
query_plan: { mode, scope, facets, limits },
prior_memories: [
{ id, path, session_id, created_at, summary }
{ id, path, anchors, session_id, created_at, summary }
],
evidence: [
{ type, id, session_id, timestamp, facet, snippet }
{ type, id, session_id, timestamp, content_type, is_meta, facet, snippet }
],
omitted: 0
}
```
For message evidence, preserve `content_type` when projecting snippets.
`text` can support user-visible claims; `thinking` is only trace/debug context;
`tool_use` means follow `tool_calls` for structured details; `tool_result`
means follow `tool_results` for structured output. Mixed or unfamiliar message
surfaces remain `unknown`.
Preserve `is_meta` separately from `content_type`. Default message evidence
should exclude `is_meta=1` rows because they are transcript control-plane
content, not ordinary user intent or assistant conclusions. Include them only
when investigating injected caveats, command envelopes, or transcript structure.
When writing raw SQL for ordinary conversation evidence, add
`COALESCE(m.is_meta,0)=0` to message filters unless meta rows are the subject of
the investigation.
Memory recall is English-indexed: translate non-English user requests into
concise English query terms before calling `memories({ query })`. Memory
summaries registered with `remember()` are also English, regardless of the
conversation language.
`memories({ query })` uses safe FTS5 tokenization over memory `summary` and
`path`, so hyphens and punctuation do not need raw `MATCH` escaping.
`memories()` returns active memories only. For raw SQL memory recall, include
`deleted_at IS NULL`; archived memory records are management/audit data.
The agent may decide whether to use, ignore, or verify a recalled memory for the
current answer without user approval because no persistent state changes. If a
user explicitly says a memory is wrong, outdated, should be forgotten, or should
be replaced, that request is approval to mutate the exact matching memory. If
the agent discovers the conflict without an explicit user request, it should
answer from current evidence and ask before archiving or replacing the memory.
Then synthesize the conclusion in the final answer. Do not pretend the raw
evidence view is itself a stored Obelisk entity.
@@ -121,7 +147,11 @@ project conventions, abandoned alternatives, repeated failure causes, workflow
patterns, and conclusions synthesized across multiple raw evidence points. Do
not propose memory for one-off lookups, uncertain findings, or duplicate
coverage. The offer is only a proposal: write the markdown file and run
`--remember` only after user approval.
`--attune` only after user approval.
Memory updates are archive-plus-write, not in-place edits: run `forget()` on the
old record and `remember()` the replacement markdown file under the same user
approval.
## Text Search Semantics
+199 -27
View File
@@ -3,6 +3,9 @@
Advanced reference for the obelisk database.
Read this when `search()`, `context()`, or `sql()` are not enough.
Executable schema source: `scripts/schema.sql`. This document explains that
contract for agents and humans; it is not the runtime source of truth.
---
## 1. Database Schema
@@ -41,6 +44,8 @@ CREATE TABLE messages (
timestamp TEXT, -- ISO 8601
role TEXT, -- "user" or "assistant" (from message payload)
text TEXT, -- extracted text content (thinking + text blocks, truncated to 10k chars)
content_type TEXT, -- "text", "thinking", "tool_use", "tool_result", or "unknown"
is_meta INTEGER DEFAULT 0, -- 1 for injected/control-plane transcript messages
model TEXT, -- model name (e.g. "claude-opus-4-6-20250529"), NULL for user messages
is_sidechain INTEGER DEFAULT 0, -- 1 if this message is on a sidechain (retry/branch)
agent_id TEXT, -- subagent or workflow agent UUID (NULL for main conversation)
@@ -54,6 +59,22 @@ CREATE TABLE messages (
Indexes: `idx_messages_session(session_id)`, `idx_messages_agent(agent_id)`, `idx_messages_ts(session_id, timestamp)`.
`content_type` preserves the top-level Claude Code content block shape for the
message row. Treat `text` as user/assistant visible language, `thinking` as
trace/debug material, and `tool_use` as a marker that the assistant message
contains tool calls. `tool_result` marks a tool-result message, but the
structured payload remains in `tool_results`. Tool-call details remain in
`tool_calls`. Messages whose top-level content is not one of these four raw
message surfaces are `unknown`. Real user input is represented by `type='user'`
and `content_type='text'`, not by a separate `user_message` content type.
`is_meta` marks transcript control-plane content: injected caveats, command
envelopes such as `<command-name>/exit</command-name>`, and similar messages
that may appear as user-role text but are not ordinary user intent. It is
separate from `type`, `role`, and `content_type`. Default helpers hide meta
messages from ordinary recall; use `includeMeta: true` or explicit SQL when
investigating injected context, command messages, or transcript structure.
### messages_fts
FTS5 virtual table for full-text search over message text.
@@ -68,7 +89,28 @@ CREATE VIRTUAL TABLE messages_fts USING fts5(
);
```
Queried via `MATCH` syntax. Rebuilt on each index pass.
Queried via `MATCH` syntax. The table is kept in sync by the `messages_fts_*`
triggers above; rebuild it manually only when repairing FTS state.
### memories_fts
FTS5 virtual table for ranked memory recall over registered memory summaries
and paths.
```sql
CREATE VIRTUAL TABLE memories_fts USING fts5(
id UNINDEXED, -- memory record ID, carried for inspection
path, -- searchable memory file path
summary, -- searchable compact memory summary
content=memories,
content_rowid=rowid,
tokenize='unicode61 remove_diacritics 1'
);
```
`memories({ query })` queries this table with safe tokenization and joins back to
`memories`, omitting archived rows. It is rebuilt during index finalization;
`remember()` also inserts the new memory row into FTS immediately.
### tool_calls
@@ -177,6 +219,14 @@ CREATE TABLE index_state (
);
```
Sentinel rows use synthetic `jsonl_path` keys:
`__last_build__` stores the last completed build time, `__app_heartbeat__`
stores the optional app indexer's liveness heartbeat,
`__app_last_successful_build__` stores the app's last successful index build,
`__indexer_owner_app__` marks app ownership, and `__last_source_mtime__` records
the newest indexed source mtime. Skill-side lazy builds skip work only while the
app heartbeat and app successful-build marker are both fresh.
### memories
Human-approved markdown memory records registered in Obelisk. The markdown
@@ -191,14 +241,22 @@ CREATE TABLE memories (
message_start TEXT, -- first relevant message UUID, if known
message_end TEXT, -- last relevant message UUID, if known
path TEXT, -- normalized absolute markdown memory file path
anchors TEXT, -- optional JSON array of recall anchors
summary TEXT, -- retrieval summary of the memory
created_at TEXT -- ISO 8601 registration time
created_at TEXT, -- ISO 8601 registration time
deleted_at TEXT, -- ISO 8601 archive time, if forgotten
deleted_reason TEXT -- human/agent deletion reason, if forgotten
);
```
Indexes: `idx_memories_project(project)`,
`idx_memories_session(session_id)`, `idx_memories_created(created_at)`.
Active memory means `deleted_at IS NULL`. Recall helpers return active memories
only. Archived memories are management/audit data, not recall data. Query recall
uses `memories_fts` joined back to `memories`; when using raw SQL for memory
recall, include `deleted_at IS NULL`.
### Key Relationships
```
@@ -221,8 +279,8 @@ workflows.run_id <-- workflow_agents.run_id
## 2. Query API Reference
Read helpers are available as globals inside `--query` scripts. Memory write
helpers are available only inside `--remember` scripts. Scripts run in an async
Read helpers are available as globals inside `--query` scripts. Memory mutation
helpers are available only inside `--attune` scripts. Scripts run in an async
IIFE with a 30-second timeout.
### Simple Layer
@@ -240,6 +298,7 @@ Full-text search across all message text using FTS5.
| `opts.after` | `string` | ISO 8601 lower bound on timestamp |
| `opts.before` | `string` | ISO 8601 upper bound on timestamp |
| `opts.cwd` | `string` | Filter by working directory (supports LIKE) |
| `opts.includeMeta` | `boolean` | Include injected/control-plane messages (default `false`) |
**Scope note:** `sessions.project` is the stored Claude Code project slug,
`sessions.project_path` is the absolute session path derived from message `cwd`
@@ -248,15 +307,22 @@ Helper `project` filters are fuzzy `LIKE` filters over `sessions.project`. For
exact project membership, use `sql()` with `s.project = ?` or
`s.project_path = ?`.
**Returns:** `Array<{ message, session, rank, context }>` where `context` is the
6 nearest messages by timestamp in the same session. It is temporal neighbor
context, not a parent chain. `rank` is the FTS5 relevance score used by
`ORDER BY rank`; lower values sort earlier, so treat the returned order as the
relevance order unless you are deliberately using FTS5 ranking details.
**Returns:** `Array<{ message, session, rank, context }>` where `message`
includes `{ uuid, text, content_type, is_meta, role, timestamp, model, cwd }`
and `context` is the 6 nearest non-meta messages by timestamp in the same
session unless `includeMeta: true` is passed. It is temporal neighbor context,
not a parent chain. `rank` is the FTS5 relevance score used by `ORDER BY rank`;
lower values sort earlier, so treat the returned order as the relevance order
unless you are deliberately using FTS5 ranking details.
```js
const hits = search('MCTS exploration');
return hits.map(h => ({ title: h.session.title, text: h.message.text?.slice(0, 200) }));
return hits.map(h => ({
title: h.session.title,
content_type: h.message.content_type,
is_meta: h.message.is_meta,
text: h.message.text?.slice(0, 200),
}));
```
#### `context(uuid)`
@@ -285,8 +351,8 @@ Read-only SQL with parameterized bindings. Returns an array of row objects.
**Returns:** `Array<Object>` -- each row as `{ column: value }`.
Write statements are rejected. Use `--remember` and `remember()` for memory
registration after user approval.
Write statements are rejected. Use `--attune` with `remember()` or `forget()`
for memory mutation after user approval.
```js
const rows = sql('SELECT id, title FROM sessions WHERE project = ? ORDER BY ended_at DESC LIMIT 5', 'Users-tomiya-Code-quiet-zero');
@@ -306,9 +372,11 @@ const chain = trace('some-uuid');
return chain.map(m => ({ role: m.role, text: m.text?.slice(0, 100) }));
```
#### `thread(sessionId)`
#### `thread(sessionId, opts?)`
All messages in a session, ordered by timestamp.
Session messages ordered by timestamp. Meta messages are omitted by default;
pass `{ includeMeta: true }` to include injected caveats, command envelopes, and
other control-plane transcript rows.
**Returns:** `Array<message>`.
@@ -317,6 +385,31 @@ const msgs = thread('session-uuid');
return { count: msgs.length, first: msgs[0]?.text?.slice(0, 100) };
```
#### `raw(uuid, opts?)`
Windowed access to the original JSONL line for a message. Use this when indexed
text, tool inputs, or tool results were truncated and you need the raw source.
It resolves main-session, subagent, and workflow-agent JSONL paths from the
indexed message metadata.
| Param | Type | Description |
|-------|------|-------------|
| `uuid` | `string` | Message UUID |
| `opts.offset` | `number` | Character offset into the JSONL line (default 0) |
| `opts.limit` | `number` | Max characters to return (default 10000) |
**Returns:** `{ text, totalLength, offset, limit, hasMore }` or `null` if the
source line cannot be found.
```js
const line = raw('message-uuid', { offset: 0, limit: 4000 });
return {
preview: line?.text,
has_more: line?.hasMore,
total: line?.totalLength,
};
```
#### `subagents(opts?)`
All subagent spawns, with message counts. For backward compatibility, passing a string is treated as `sessionId`.
@@ -417,6 +510,34 @@ const last5 = recent(5);
return last5.map(s => ({ title: s.title, project: s.project_path, ended: s.ended_at }));
```
#### `summaries(opts?)`
Session summary rows, newest first. For backward compatibility, passing a
string is treated as `sessionId`, and passing a number is treated as `limit`.
| Param | Type | Description |
|-------|------|-------------|
| `opts.sessionId` | `string` | Restrict to one session |
| `opts.sessions` | `string[]` | Restrict to a set of session IDs |
| `opts.project` | `string` | SQL `LIKE` pattern over `sessions.project` |
| `opts.after` | `string` | ISO 8601 lower bound on summary timestamp |
| `opts.before` | `string` | ISO 8601 upper bound on summary timestamp |
| `opts.branch` | `string` | Filter by source session git branch (exact match) |
| `opts.limit` | `number` | Max results (default 100) |
**Returns:** `Array<summary_row & { session_title, project }>` ordered by
`timestamp` descending.
```js
const rows = summaries({ project: '%quiet-zero%', limit: 5 });
return rows.map(s => ({
session: s.session_title,
source: s.source,
summary: s.content?.slice(0, 240),
timestamp: s.timestamp,
}));
```
#### `overview(opts?)`
Compact orientation map for choosing the next retrieval scope. It is not an
@@ -457,7 +578,7 @@ from `process.cwd()` against `sessions.project_path`, then from exact
],
memory_total,
memories: [
{ id, path, summary, session_id, project, created_at }
{ id, path, anchors, summary, session_id, project, created_at }
]
} | null,
projects: [
@@ -491,6 +612,7 @@ return {
memories: map.current_project?.memories.map(m => ({
id: m.id,
path: m.path,
anchors: m.anchors,
summary: m.summary,
})),
};
@@ -522,12 +644,12 @@ return qz.map(s => ({ title: s.title, branch: s.git_branch, ended: s.ended_at })
#### `memories(opts?)`
Registered markdown memory records. Like other list helpers, passing a string
is treated as `sessionId`, and passing a number is treated as `limit`.
Active registered markdown memory records. Like other list helpers, passing a
string is treated as `sessionId`, and passing a number is treated as `limit`.
| Param | Type | Description |
|-------|------|-------------|
| `opts.query` | `string` | English term filter over `summary` and `path`; hyphens/underscores are treated as spaces |
| `opts.query` | `string` | English FTS recall query over `summary` and `path`; hyphens/underscores/punctuation are safely tokenized |
| `opts.project` | `string` | SQL `LIKE` pattern over `memories.project` |
| `opts.sessionId` | `string` | Restrict to one source session |
| `opts.sessions` | `string[]` | Restrict to a set of source session IDs |
@@ -536,9 +658,13 @@ is treated as `sessionId`, and passing a number is treated as `limit`.
| `opts.branch` | `string` | Filter by source session git branch (exact match) |
| `opts.limit` | `number` | Max results (default 50) |
**Returns:** `Array<memory_row>` ordered by `created_at` descending.
**Returns:** `Array<memory_row & { rank?: number }>` with archived memories
omitted. Without `query`, results are ordered by `created_at` descending. With
`query`, results are ordered by FTS rank first, then `created_at` descending;
lower rank sorts earlier.
`query` is a lightweight English term filter, not FTS5 ranking. Translate
`query` uses safe FTS5 tokenization rather than raw `MATCH`, so punctuation-only
queries return no rows instead of broadening into all memories. Translate
non-English user requests into concise English query terms before calling
`memories()`. Use it to avoid pulling all recent memories, then read the
markdown file at `path` when a memory looks relevant. The runtime rejects
@@ -553,6 +679,7 @@ const prior = memories({
return prior.map(m => ({
id: m.id,
path: m.path,
anchors: m.anchors,
session_id: m.session_id,
summary: m.summary?.slice(0, 240),
}));
@@ -562,11 +689,11 @@ return prior.map(m => ({
Register a human-approved markdown memory file. This is a write helper, not a
recall helper; use it only after the user has approved writing memory. It is
available only in scripts run with `runtime.mjs --remember`.
available only in scripts run with `runtime.mjs --attune`.
`--remember` exposes only `remember()`, not `search()`, `sql()`, `memories()`,
or other retrieval helpers. If source IDs are unknown, find them first with a
normal `--query` script.
`--attune` exposes only `remember()` and `forget()`, not `search()`, `sql()`,
`memories()`, or other retrieval helpers. If source IDs or memory IDs are
unknown, find them first with a normal `--query` script.
| Param | Type | Description |
|-------|------|-------------|
@@ -576,12 +703,14 @@ normal `--query` script.
| `record.message_start` | `string` | First relevant source message UUID, if known |
| `record.message_end` | `string` | Last relevant source message UUID, if known |
| `record.project` | `string` | Project slug override. Defaults from `sessions.project` for `session_id` |
| `record.anchors` | `array` or JSON `string` | Optional recall anchors stored as JSON text. Expected shape is an array of objects, such as `{ kind: 'file', path: 'src/index/builder.ts' }` |
`remember()` validates that `path` exists and is a regular file, and rejects
obvious CJK text in `summary`. It stores the normalized absolute path in
`memories.path`.
`memories.path`. `anchors` is nullable; omit it or pass an empty array when the
memory has no explicit file or object anchors.
**Returns:** `{ id, path, project, created_at }`.
**Returns:** `{ id, path, project, anchors, created_at }`.
```js
return remember({
@@ -589,10 +718,53 @@ return remember({
session_id: 'source-session-id',
message_start: 'first-message-uuid',
message_end: 'last-message-uuid',
anchors: [{ kind: 'file', path: 'src/index/builder.ts' }],
summary: 'Decision: keep Obelisk as one user-facing entry that queries both memory and raw session evidence. Memory is prior notes, not final authority.',
});
```
#### `forget(record)`
Archive a human-approved memory record. Use it when the user says a memory is
outdated, wrong, or should be forgotten. It is available only in scripts run
with `runtime.mjs --attune`.
`forget()` requires a precise memory ID. Do not pass a query string and let the
helper choose. If the ID is unknown, first use a normal `--query` script with
`memories()` to identify candidates. If exactly one candidate clearly matches
the user's request, the request is approval to archive it. If multiple memories
could match, ask the user which one to forget.
| Param | Type | Description |
|-------|------|-------------|
| `record.id` | `string` | Memory record ID to archive |
| `record.reason` | `string` | Required reason for audit and future management views |
`forget()` sets `deleted_at` and `deleted_reason`. It does not delete the
markdown file at `path`. Active recall helpers omit archived memories.
**Returns:** `{ id, deleted_at, deleted_reason }`, or the same fields plus
`already_deleted: true` if the record had already been forgotten.
```js
return forget({
id: 'mem-20260610-example',
reason: 'Outdated by newer project guidance.',
});
```
#### Memory Mutation Approval
Agents may decide whether to use, ignore, or verify memory in a single answer
without approval. Approval is required only for persistent memory mutations.
When the user explicitly says a memory is wrong, outdated, should be forgotten,
or should say something else, that utterance is approval to mutate the exact
matching memory. If multiple memories could match, ask the user to choose.
Updating is not an in-place edit. Archive the old record with `forget()`, then
write and register a replacement markdown file with `remember()` under the same
approval.
---
## 3. Common Query Patterns
@@ -667,7 +839,7 @@ const wfs = workflows();
for (const wf of wfs.slice(0, 3)) {
const tree = workflowTree(wf.run_id);
wf.agent_details = tree?.agents.map(a => ({
type: a.agent_type, desc: a.description, msgs: a.messages.length,
type: a.agent_type, desc: a.description, msgs: a.messageCount,
}));
}
return wfs.slice(0, 3);