Move all pure parse/discover helpers (message extraction, project-path, codex
helpers, discovery) out of db.mjs/indexer.mjs into scripts/parsing.mjs, which
imports only node:fs/path/os. Providers now import from parsing.mjs, so the
provider import graph no longer transitively loads node:sqlite — a prerequisite
for the app (Electron/Node without node:sqlite) to consume the compiled core.
Verbatim move, no behavior change; indexer.mjs 840→209 lines. 119/119.
Complete the skill-side provider migration: codex now goes through a pure adapter
and the shared persist layer, and the two original monolithic indexers are gone.
New:
- scripts/providers/codex.ts — pure codex adapter. Full-reparse (buffers the whole
file) because the event_msg↔response_item dedup needs whole-file, bidirectional
knowledge; emits SessionRecord with countMode 'total'. Handles guardian threads
(→ delete-session), agent spawns/tool calls (→ tool_call/subagent), token_count
(patched onto the message record) and task_complete (→ message-turn-duration).
Contract:
- SessionRecord.countMode ('total' | 'delta') tells persist whether to replace or
accumulate message_count — claude is line-incremental (delta), codex full-reparse
(total). SubagentRecord non-key fields are optional; persist merges them
column-wise with COALESCE. MessageTurnDurationRecord.turn_duration_ms is nullable.
Orchestration:
- buildIndex's codex branch parses via the adapter and writes via persist. An
unchanged file is skipped but still swept for stale guardian rows (routed through
persist as a delete-session), preserving prior behavior.
Cleanup:
- Remove the now-unused indexJsonl, indexCodexJsonl, deleteCodexThreadRows and
upsertCodexSubagent — their semantics now live in the adapters + persist.
indexer.mjs drops from ~840 to 428 lines. Codex pure helpers stay exported for
codex.ts and the guardian sweep (physical move deferred to the app-side reorg).
- Migrate the upsert drift test off indexJsonl to the claude.parse + persist path,
keeping the rowid-stability and count-replace regression guards.
Tests: tests/codex-parse.test.mjs (record-stream golden: dedup, tools, token patch,
turn-duration, guardian→delete) and tests/codex-index.test.mjs (full buildIndex
path: fresh build + incremental full-reparse, total-count replace, no duplicates).
Verified equivalent on the real ~/.obelisk index: codex messages 82476 and
subagents 522 identical before/after, zero guardian leakage; real incremental
confirmed (touch a codex file → reparsed idempotently, unchanged files skipped).
lint + typecheck clean, 119/119.
buildIndex's claude branch now parses via providers/claude.ts and writes via
persist.ts instead of the inlined indexJsonl. Behavior is equivalent — the full
buildIndex integration suite (runtime.test.mjs) stays green, 116/116 — and a
force rebuild of the real ~/.obelisk index (327 sessions, 119k messages)
reproduced identical session counts with project_path fully populated.
codex, indexSubagentMeta, workflows, history and the project_path pass are
untouched. indexJsonl is now unused by buildIndex (kept for its drift test;
removed once codex is migrated).
Adds tests/incremental-index.test.mjs: verifies resume/accumulate through the
full buildIndex path (append new lines to an indexed session → incremental
build resumes from the cursor, message_count accumulates, no duplicates). The
30s shouldSkipBuild debounce is cleared in-test so the incremental run fires.
Introduce scripts/persist.ts — the single, provider- and binding-agnostic
layer that consumes an adapter's IndexRecord stream and writes rows into an
injected SQLite handle (node:sqlite for skill/CLI, better-sqlite3 for the app).
It is the only layer that touches the database.
Write semantics are the canonical ones reconciled from the earlier drift:
- messages upsert via ON CONFLICT (turn_duration_ms not in the column list, so
it is never clobbered)
- sessions merge with the existing row: started_at MIN, ended_at MAX,
message_count reset-or-accumulate by resume state, fill-if-null for the rest;
project_path is preserved and left to refreshSessionProjectPaths
- message-turn-duration applies as a targeted UPDATE
- delete-session cascades across all tables
- the generator's return cursor is written back to index_state (mtime:lines →
the two existing columns; no schema migration yet)
Purely additive — buildIndex still uses the old indexJsonl path, so existing
behavior is unchanged. Rewiring happens in 5b-2b.
Adds tests/persist.test.mjs: all record kinds written, resume does not
double-count message_count, fresh re-scan resets it, delete-session cascades.
Full suite 115/115, lint + typecheck green.
claude.parse mirrors indexJsonl line-for-line but yields IndexRecords (no db).
Adds MessageTurnDurationRecord op. Purely additive — buildIndex still uses the
old path. 111/111 green.
Provider adapters own discovery/change-detection/cursoring and emit records;
one shared binding-agnostic persist consumes them. Records map 1:1 to schema
tables. Revises ADR-0001 to the provider-registry model.
scripts/indexer.mjs indexJsonl used INSERT OR REPLACE (rowid/FTS churn) and
carried message_count forward on full re-scan; align to app's ON CONFLICT upsert
+ count reset so the two indexers no longer silently diverge. Regression test added.
Core exposes buildIndex/searchText/executeQuery/executeAttune as the single shared
implementation for all transports. First TypeScript module, run via Node type
stripping in dev. Adds typescript-eslint. lint/typecheck green, test 107/107.
Root package.json (type: module), strict tsconfig with allowJs/checkJs:false for
gradual migration, flat ESLint config scoped to scripts/ + tests/. App untouched.
All three green: lint, typecheck, test 107/107.
search() falls back to safe per-token quoting on malformed FTS input instead of
crashing (documented in api-reference.md). Adds raw() to the doc-synced shape
contract. Full suite 107/107.
Extract helper API documentation into a standalone api-reference.md and
reduce schema.md to a compact field/join/table map for raw SQL work.
Add a Reference Map table to SKILL.md for quick routing by task type.
Extract helper API documentation into a standalone api-reference.md and
reduce schema.md to a compact field/join/table map for raw SQL work.
Add a Reference Map table to SKILL.md for quick routing by task type.
Obelisk now indexes ~/.codex/ sessions alongside Claude Code, storing
both under a unified schema with a column for provenance.
Includes DB migration to ~/.obelisk/, guardian thread filtering, and
source-aware query helpers.
Add `source` column to sessions and messages ('claude' | 'codex').
Discover and parse Codex JSONL files from ~/.codex/sessions/, mapping
Codex thread/item structures to the same schema (messages, tool_calls,
tool_results, subagents). Move DB to ~/.obelisk/ with legacy migration.
Add rebuild-to-temp-then-swap for safe full rebuilds. On the app side:
source filter toggle, collapsible untitled session fold, configurable
codexDir in Settings, and a dev script. Update SKILL.md and query
helpers to expose source fields and accept source filter opt.
Add a clickable session link in MemoryList detail panel that navigates
to the source session and scrolls to the originating message. Refactor
SessionDetail focus logic into a reusable focusPendingMessage() that
reads ?focus=<uuid> from the route query, scrolls the target into view,
and applies a pulse animation that fades out.
Restructure the README around the dual nature of Obelisk: agent-first
skill for querying session history, plus Electron desktop app for
browsing sessions, memories, activity, and recap cards. Trim verbose
implementation details and add app screenshot.
Extract schema DDL into scripts/schema.sql. Add content_type and is_meta
columns to messages for transcript control-plane filtering. Introduce
FTS5-backed memory recall with safe tokenization, memory soft-delete via
forget() through the renamed --attune runtime, and anchors on memory
records. Expand query helpers (includeMeta, thread opts, overview
project-path awareness). Add per-card recap retrieval and writing
references under references/recap/.
Add a flag set after initial data fetch completes so
SessionList and RecapList don't briefly show their empty/onboarding
states before content arrives.
Introduce a Settings view for configuring the Claude data directory
(with WSL auto-detection on Windows), sidebar project grouping module,
and an empty-state onboarding screen for SessionList. Refactor
recap-patterns.md into per-card reference files under references/recap/
with separate retrieval and writing guides. Remove the legacy panel.html.
On the data layer: incremental indexing via changedPaths, per-session
live-update IPC (obelisk:session-updated), and session dirty-tracking
in the renderer.
Introduce a Spotify-Wrapped-style recap feature: five themed cards
(Cover, Path, Vibe, Workflow, Closing) rendered per archetype palette,
with keyboard/swipe navigation and image export via capture IPC. Add
RecapList, RecapDetail, RecapExport views and recap component library.
Wire recap:list/read/updated IPC channels through preload, document the
retrieval-to-card contract in references/recap-patterns.md, and bundle
dist-renderer for production use.
Extract schema DDL into scripts/schema.sql shared between CLI and app.
Add an in-process chokidar-based indexer-service that watches ~/.claude/projects
for JSONL changes, debounces, and triggers background rebuilds via a worker
thread. Rename Usage view to Activity, flesh out MemoryDetail and SubagentDetail
views, and refine App.vue layout/routing. The main process now starts/stops the
indexer lifecycle and notifies renderer windows on index updates.
Introduce an Electron app with session browser, memory list, and usage
views (vanilla JS + Vue scaffolding). On the data layer: add content_type
and is_meta to messages for transcript control-plane filtering, introduce
FTS5-backed memory recall with safe tokenization, support memory archival
via forget() through the renamed --attune runtime, and expose anchors on
memory records.
Memory layer is now English-indexed: memories() query terms and
remember() summaries must be English. Adds a runtime assertion that
rejects CJK text in both paths, guiding the agent to translate
non-English user requests before querying or writing memories.
Ensures consistent retrieval regardless of conversation language.
Make overview() + memories() + search() the standard first pass for
broad retrieval tasks, with sql() positioned as an escalation path for
exact joins/aggregations. Add a Default First Pass section to SKILL.md,
a copyable first-pass pattern to query-patterns.md, and update
retrieval-semantics.md to reinforce the helper-before-sql principle.
The old slug-to-path conversion (replace hyphens with slashes) was
lossy and wrong for paths containing hyphens. Now infers project_path
from the most-frequent observed cwd across session messages, falling
back to slug decoding only when no cwd data exists. Adds
refreshSessionProjectPaths() to backfill existing sessions on rebuild.
Resolves current project from cwd, lists all known projects with session
and memory counts, and returns the current project's recent sessions and
memories in one call. Enables the agent to orient itself at the start of
a retrieval without multiple exploratory queries.
Add a memories table (survives index rebuilds) for agent-written
conclusions with provenance (session, message range, project). The agent
writes markdown files via Write tool (user-approved), then registers
them via a --remember CodeAct script with remember(). Recall via
memories() in --query scripts, filtered by project/session/time.
Separates query (read-only, assertReadOnlySql) from remember (write)
execution contexts in runtime.mjs.
semantics (context types, ordering, project scopes) out of pitfalls.md
into a new retrieval-semantics.md. Pitfalls.md becomes a compact debug
checklist: missing columns, FTS errors, over-large output, empty results.
SKILL.md query routing updated to point at the three reference tiers.
Teach the agent to derive second-pass filters from first-pass evidence
instead of pulling large message windows. Add the pattern and a pitfall
warning against defaulting to LIMIT 25 transcript browsing.
Teach the agent to keep intermediate retrieval inside the query script
and return compact evidence in a single turn, instead of spending
multiple conversation rounds showing raw results. Add the pattern to
query-patterns.md and the rationale to pitfalls.md.
Split the monolithic skill prompt into three tiers:
- Core API (search/context/sql) stays in the first prompt
- Structured helpers listed as one-liners with filter signatures
- Detailed patterns and pitfalls extracted to references/
Add references/query-patterns.md (copyable CodeAct recipes) and
references/pitfalls.md (scope, FTS, ordering, compactness traps).
Clarify project scope semantics (slug vs path vs cwd) throughout.
Add ORDER BY timestamp DESC to failures() for newest-first default.
lightweight workflowTree, build debounce, minor fixes
- Index phase, label, model, state, duration, tokens per workflow agent
- Index duration, total_tokens, status, name per workflow run
- workflowTree returns parsed result + agent summaries instead of
dumping all messages
- 30s debounce on buildIndex to avoid repeated directory scans
- Fix broken BASH_EXIT_PAT (SQLite LIKE has no character classes)
- Fix SKILL.md step numbering, document FTS5 hyphen limitation
Index the is_error boolean from JSONL tool_result blocks into a new
column, replacing the old ERROR_PATS text-matching approach that
produced ~90% false positives. Bash exit code pattern kept as fallback.
Add sessions() as first-class entry point for session discovery with
project/time/branch filters. Extend summaries, workflows, failures,
subagents, fileHistory with the same opts vocabulary that search()
already had — no more pulling full datasets to filter client-side.
Prepares for adding Codex JSONL support — new indexers can be added
without touching the query layer.
- db.mjs: schema, openDb, truncation helpers, shared utilities
- indexer.mjs: file discovery, JSONL parsing, SQLite writes
- query.mjs: all query API functions (search, context, raw, etc.)
- runtime.mjs: thin CLI entry point + VM script executor
- Replace readFileSync+split with chunked line reader (readLines) to
avoid loading entire JSONL files into memory at once
- Wrap each file's indexing in its own transaction so one corrupt file
doesn't roll back the entire index
- Log warnings to stderr instead of silently swallowing errors in
indexSubagentMeta, indexWorkflows, and indexHistory
- Add try/catch around readdirSync calls in discoverJsonlFiles and
indexWorkflows to survive unreadable directories
- Push error-pattern filtering in failures() into SQL LIKE clauses
instead of scanning all tool_results in JS
- Don't truncate workflow result_json and script (low-volume, must
stay parseable)
- Replace blind string truncation on tool_calls.input_json with
structure-aware truncJson that shortens individual string values
while keeping the JSON skeleton valid
- Add raw(uuid, {offset, limit}) API for windowed access to the
original JSONL line, so truncated content can be recovered on demand
Previously the session metadata accumulator started from zero on every
reindex pass, so an incremental update would overwrite started_at and
message_count with values derived only from the new lines.
Now reads the existing session row first and merges new data on top.
tommy0103