fix: coordinate sqlite index writers

Implement the full ADR-0006 plan: three-layer separation of transaction
correctness, retry policy, and cross-process writer coordination.

Layer 1 — scripts/tx.ts (transaction correctness):
- runWriteTransaction executes work exactly once; no internal retry.
- BEGIN IMMEDIATE takes the write lock up front (avoids SQLITE_BUSY_SNAPSHOT).
- Guarded rollback: checks inTransaction() via adapter before attempting
  ROLLBACK; never masks the primary exception.
- WriteTxDiagnostics attached to errors: phase, code, label,
  rollbackSucceeded, rollbackError, transactionActive.
- Binding adapters (betterSqliteTransactionAdapter, nodeSqliteTransactionAdapter)
  mapping better-sqlite3's `.inTransaction` and node:sqlite's `.isTransaction`.
- configureConnection centralizes WAL + synchronous + busy_timeout.

Layer 2 — scripts/write-coordinator.ts (retry policy):
- runRetryableWriteTransaction: bounded retry with total time budget.
- Only retries when the transaction confirmed ended (transactionActive=false)
  and the error is SQLITE_BUSY during work/commit phase.
- BEGIN-phase BUSY = abort entire build (isBeginBusyFailure); the caller
  returns `{ deferred: true, reason: 'writer_busy' }` instead of waiting.
- hasUnusableTransaction detects a still-active transaction after failure;
  aborts the build immediately, never retries.

Layer 3 — scripts/writer-lease.ts (cross-process coordination):
- acquireWriterLease: dedicated writer.lock.sqlite with busy_timeout=0 +
  BEGIN IMMEDIATE. Non-blocking attempt; bounded wait with retryDelayMs.
- writerLockPathFor derives lock path from the target DB path.
- Lease held for the entire build; released on completion or failure.
- Lock DB uses DELETE journal (not WAL); crash/close auto-releases.
- All consumers obey: skill acquires at build start (returns deferred if
  unavailable); app daemon (via worker) acquires for its build cycle.

Build semantics changes:
- affectedSessionIds updated only after successful commit.
- BuildIndexResult gains skipped/skippedFiles for observability.
- Skill finalize failure now fails the build (was silently warned).
- Checkpoint changed to PASSIVE (TRUNCATE reserved for maintenance/exit).
- Skill buildIndex returns { deferred, reason } on lease contention;
  indexer-service reschedules the build (deferredRetryMs) without publishing
  a heartbeat (so the build-deferred state is visible to cross-process
  arbitration).
- Service publishes heartbeat immediately on start() for correct arbitration.

Tests:
- tests/write-transaction.test.mjs: single-shot execution, diagnostics
  propagation, auto-rolled-back transaction detected, rollback failure
  captured as metadata, BEGIN IMMEDIATE semantics.
- tests/writer-lease.test.mjs: acquire/release, contention returns null,
  bounded wait with release during budget.
- tests/app-writer-lease.test.mjs: better-sqlite3 adapter integration.
- tests/app-rollback-guard.test.mjs: rewritten — transient BUSY recovered
  by coordinator, persistent BUSY skips file, begin-busy aborts build,
  live-transaction aborts build, phantom affectedSessionIds prevented.
- tests/daemon-arbitration.test.mjs: skill defers to fresh app heartbeat,
  builds when heartbeat is stale.
- tests/app-indexer-service.test.mjs: new cases for deferred-retry
  scheduling and immediate heartbeat on start.
- app/tests/electron-concurrency.mjs + child: dual-child IPC structure for
  real better-sqlite3 contention (holder acquires lock → build child starts
  → delayed release → result collected; persistent contention bounded).

ADR-0006 updated to reflect the implemented design.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
tommy0103
2026-07-11 00:42:43 +08:00
co-authored by Claude Opus 4.8
parent 44029676d4
commit e3e61cc7ab
25 changed files with 2173 additions and 471 deletions
+12 -4
View File
@@ -67,10 +67,18 @@ incremental indexing), plus heartbeat/last-build markers used for daemon
arbitration. arbitration.
**Daemon arbitration**: **Daemon arbitration**:
The mechanism by which the passive pull mode detects a fresh daemon (via The policy by which the passive pull mode detects a fresh daemon from the
`__app_heartbeat__` and `__app_last_successful_build__` markers in `index_state`) `__app_heartbeat__` marker and skips every skill-side mutation, including schema
and skips its own indexing. Because a fresh daemon owns writes, the two persist setup, indexing, checkpointing, and `attune`. The heartbeat alone means “the
layers never write concurrently. daemon should write”; `__app_last_successful_build__` records coverage/freshness,
not ownership. Both indexing modes use the same persist layer.
**Writer lease**:
The hard cross-process safety mutex behind daemon arbitration. A writer holds
`BEGIN IMMEDIATE` on `.obelisk/writer.lock.sqlite` for the complete mutation;
manual rebuild holds it through build, target-database replacement, and reopen.
The heartbeat expresses policy, while the writer lease prevents overlapping
writes during races, stale heartbeats, or processes from different versions.
## Memory ## Memory
+4 -3
View File
@@ -191,9 +191,10 @@ Full-text search via FTS5 covers message text across every session layer and ran
The index rebuilds incrementally — only new or modified JSONL files are re-parsed. The index rebuilds incrementally — only new or modified JSONL files are re-parsed.
When the optional app is running, it is the active indexer: it watches Claude When the optional app is running, it is the active indexer: it watches Claude
project files, builds in a worker thread, writes `__app_heartbeat__` plus project files and builds in a worker thread. A fresh `__app_heartbeat__` alone
`__app_last_successful_build__` into `index_state`, and the skill-side lazy means the daemon owns writes, so the skill remains read-only; a separate SQLite
build skips work only while both markers are fresh. writer lease prevents cross-process writes from overlapping. The
`__app_last_successful_build__` marker records index freshness, not ownership.
Zero npm dependencies. Uses Node 22's built-in node:sqlite with FTS5. The entire runtime is ~400 lines. Zero npm dependencies. Uses Node 22's built-in node:sqlite with FTS5. The entire runtime is ~400 lines.
+2 -1
View File
@@ -11,7 +11,8 @@
"dist": "electron-vite build && electron-builder --mac --win --linux", "dist": "electron-vite build && electron-builder --mac --win --linux",
"dist:mac": "electron-vite build && electron-builder --mac", "dist:mac": "electron-vite build && electron-builder --mac",
"dist:win": "electron-vite build && electron-builder --win", "dist:win": "electron-vite build && electron-builder --win",
"dist:linux": "electron-vite build && electron-builder --linux" "dist:linux": "electron-vite build && electron-builder --linux",
"test:electron": "electron-vite build && electron --no-sandbox tests/electron-concurrency.mjs"
}, },
"build": { "build": {
"appId": "com.obelisk.app", "appId": "com.obelisk.app",
+89 -9
View File
@@ -9,6 +9,7 @@ import { writeHeartbeat } from './indexer.ts';
import { createIndexerService } from './indexer-service.ts'; import { createIndexerService } from './indexer-service.ts';
import { createWorkerBuildIndex } from './indexer-worker-client.ts'; import { createWorkerBuildIndex } from './indexer-worker-client.ts';
import { buildRecapExportQuery } from './recap-capture-query.ts'; import { buildRecapExportQuery } from './recap-capture-query.ts';
import { acquireWriterLease, writerLockPathFor } from '../../../scripts/writer-lease.ts';
const __dirname = path.dirname(fileURLToPath(import.meta.url)); const __dirname = path.dirname(fileURLToPath(import.meta.url));
@@ -41,6 +42,16 @@ let db;
let indexerService; let indexerService;
let indexerWorker; let indexerWorker;
type WriterLeaseMode = 'acquire' | 'caller-held';
function acquireAppWriterLease(dbPath: string, waitMs = 0) {
return acquireWriterLease({
lockPath: writerLockPathFor(dbPath),
openDb: lockPath => new Database(lockPath),
waitMs,
});
}
function getConfiguredClaudeDir() { function getConfiguredClaudeDir() {
const persisted = loadPersistedSettings(); const persisted = loadPersistedSettings();
return persisted.claudeDir || DEFAULT_CLAUDE_DIR; return persisted.claudeDir || DEFAULT_CLAUDE_DIR;
@@ -60,15 +71,25 @@ function getPathsForClaudeDir(claudeDir = getConfiguredClaudeDir(), codexDir = g
}; };
} }
function migrateLegacyDbIfNeeded(paths = getPathsForClaudeDir()) { function migrateLegacyDbIfNeeded(
paths = getPathsForClaudeDir(),
{ writerLeaseMode = 'acquire' }: { writerLeaseMode?: WriterLeaseMode } = {},
) {
if (fs.existsSync(paths.dbPath)) return; if (fs.existsSync(paths.dbPath)) return;
const legacyDbPath = path.join(paths.claudeDir, 'obelisk.sqlite'); const legacyDbPath = path.join(paths.claudeDir, 'obelisk.sqlite');
if (!fs.existsSync(legacyDbPath)) return; if (!fs.existsSync(legacyDbPath)) return;
const lease = writerLeaseMode === 'acquire' ? acquireAppWriterLease(paths.dbPath) : null;
if (writerLeaseMode === 'acquire' && !lease) return false;
try { try {
if (fs.existsSync(paths.dbPath)) return true;
fs.mkdirSync(path.dirname(paths.dbPath), { recursive: true }); fs.mkdirSync(path.dirname(paths.dbPath), { recursive: true });
fs.copyFileSync(legacyDbPath, paths.dbPath); fs.copyFileSync(legacyDbPath, paths.dbPath);
return true;
} catch (error) { } catch (error) {
console.warn?.(`Obelisk legacy DB migration skipped: ${(error as Error).message}`); console.warn?.(`Obelisk legacy DB migration skipped: ${(error as Error).message}`);
return false;
} finally {
lease?.release();
} }
} }
@@ -151,15 +172,40 @@ function closeDb() {
db = null; db = null;
} }
function openDb(dbPath = getPathsForClaudeDir().dbPath) { function openDb(
dbPath = getPathsForClaudeDir().dbPath,
{ writerLeaseMode = 'acquire' }: { writerLeaseMode?: WriterLeaseMode } = {},
) {
closeDb(); closeDb();
if (!fs.existsSync(dbPath)) return null; if (!fs.existsSync(dbPath)) return null;
db = new Database(dbPath, { readonly: false }); db = new Database(dbPath, { readonly: false });
db.pragma('busy_timeout = 5000');
const lease = writerLeaseMode === 'acquire' ? acquireAppWriterLease(dbPath) : null;
if (writerLeaseMode === 'caller-held' || lease) {
try {
db.pragma('journal_mode = WAL'); db.pragma('journal_mode = WAL');
migrateDb(db); migrateDb(db);
} finally {
lease?.release();
}
}
return db; return db;
} }
function runAppDbWrite(work: () => void): boolean {
if (!db) return false;
const lease = acquireAppWriterLease(getPathsForClaudeDir().dbPath, 250);
if (!lease) {
throw new Error('Obelisk index writer is busy; memory change was not applied');
}
try {
work();
return true;
} finally {
lease.release();
}
}
function notifyIndexUpdated(result: { affectedSessionIds?: unknown } = {}) { function notifyIndexUpdated(result: { affectedSessionIds?: unknown } = {}) {
const affectedSessionIds = Array.isArray(result.affectedSessionIds) const affectedSessionIds = Array.isArray(result.affectedSessionIds)
? [...new Set(result.affectedSessionIds.filter(Boolean))] ? [...new Set(result.affectedSessionIds.filter(Boolean))]
@@ -200,8 +246,14 @@ function startIndexerService({ buildOnStart = false } = {}) {
projectsDir: paths.projectsDir, projectsDir: paths.projectsDir,
dbPath: paths.dbPath, dbPath: paths.dbPath,
}); });
if (result?.deferred) {
if (Array.isArray(result.affectedSessionIds) && result.affectedSessionIds.length) {
notifyIndexUpdated(result);
}
} else {
openDb(paths.dbPath); openDb(paths.dbPath);
notifyIndexUpdated(result); notifyIndexUpdated(result);
}
return result; return result;
}, },
writeHeartbeat: () => writeHeartbeat({ dbPath: paths.dbPath }), writeHeartbeat: () => writeHeartbeat({ dbPath: paths.dbPath }),
@@ -520,16 +572,16 @@ ipcMain.handle('db:readMemoryFile', (_, filePath) => {
}); });
ipcMain.handle('db:archiveMemory', (_, id, reason) => { ipcMain.handle('db:archiveMemory', (_, id, reason) => {
if (!db) return false; return runAppDbWrite(() => {
db.prepare(`UPDATE memories SET deleted_at = ?, deleted_reason = ? WHERE id = ?`) db.prepare(`UPDATE memories SET deleted_at = ?, deleted_reason = ? WHERE id = ?`)
.run(new Date().toISOString(), reason || 'Archived via panel', id); .run(new Date().toISOString(), reason || 'Archived via panel', id);
return true; });
}); });
ipcMain.handle('db:restoreMemory', (_, id) => { ipcMain.handle('db:restoreMemory', (_, id) => {
if (!db) return false; return runAppDbWrite(() => {
db.prepare(`UPDATE memories SET deleted_at = NULL, deleted_reason = NULL WHERE id = ?`).run(id); db.prepare(`UPDATE memories SET deleted_at = NULL, deleted_reason = NULL WHERE id = ?`).run(id);
return true; });
}); });
ipcMain.handle('db:getProjects', (_, opts = {}) => { ipcMain.handle('db:getProjects', (_, opts = {}) => {
@@ -830,8 +882,27 @@ ipcMain.handle('settings:rebuildIndex', async () => {
indexerWorker = createWorkerBuildIndex(); indexerWorker = createWorkerBuildIndex();
} }
cleanupDbFiles(tempDbPath); cleanupDbFiles(tempDbPath);
let writerLease: ReturnType<typeof acquireWriterLease> = null;
try { try {
migrateLegacyDbIfNeeded(paths); const writerLeasePath = writerLockPathFor(paths.dbPath);
writerLease = acquireWriterLease({
lockPath: writerLeasePath,
openDb: lockPath => new Database(lockPath),
waitMs: 2000,
});
if (!writerLease) {
return {
files: 0,
latestSourceMtime: 0,
affectedSessionIds: [],
ftsRebuilt: false,
skipped: 0,
skippedFiles: [],
deferred: true,
reason: 'writer_busy',
};
}
migrateLegacyDbIfNeeded(paths, { writerLeaseMode: 'caller-held' });
const result = await indexerWorker.buildIndex({ const result = await indexerWorker.buildIndex({
reason: 'manual-rebuild', reason: 'manual-rebuild',
force: true, force: true,
@@ -840,21 +911,30 @@ ipcMain.handle('settings:rebuildIndex', async () => {
projectsDir: paths.projectsDir, projectsDir: paths.projectsDir,
dbPath: tempDbPath, dbPath: tempDbPath,
preserveDbPath: fs.existsSync(paths.dbPath) ? paths.dbPath : null, preserveDbPath: fs.existsSync(paths.dbPath) ? paths.dbPath : null,
writerLeasePath,
writerLeaseMode: 'caller-held',
}); });
if (result?.deferred) return result;
closeDb(); closeDb();
replaceDbWithTemp(tempDbPath, paths.dbPath); replaceDbWithTemp(tempDbPath, paths.dbPath);
openDb(paths.dbPath); openDb(paths.dbPath, { writerLeaseMode: 'caller-held' });
notifyIndexUpdated(result); notifyIndexUpdated(result);
return result; return result;
} finally { } finally {
try {
cleanupDbFiles(tempDbPath); cleanupDbFiles(tempDbPath);
if (!db) { if (!db) {
try { try {
openDb(paths.dbPath); openDb(paths.dbPath, {
writerLeaseMode: writerLease ? 'caller-held' : 'acquire',
});
} catch (error) { } catch (error) {
console.warn?.(`Obelisk DB reopen after rebuild failed: ${(error as Error).message}`); console.warn?.(`Obelisk DB reopen after rebuild failed: ${(error as Error).message}`);
} }
} }
} finally {
writerLease?.release();
if (shouldRestartWatcher) startIndexerService({ buildOnStart: false }); if (shouldRestartWatcher) startIndexerService({ buildOnStart: false });
} }
}
}); });
+42 -9
View File
@@ -8,6 +8,7 @@ const DEFAULT_DEBOUNCE_MS = 2000;
const DEFAULT_STABILITY_MS = 500; const DEFAULT_STABILITY_MS = 500;
const DEFAULT_HEARTBEAT_MS = 30000; const DEFAULT_HEARTBEAT_MS = 30000;
const DEFAULT_WATCH_RETRY_MS = 5000; const DEFAULT_WATCH_RETRY_MS = 5000;
const DEFAULT_DEFERRED_RETRY_MS = 250;
type TimerHandle = ReturnType<typeof setTimeout>; type TimerHandle = ReturnType<typeof setTimeout>;
@@ -22,6 +23,15 @@ interface Watcher {
close(): unknown; close(): unknown;
} }
interface IndexerBuildResult {
deferred?: boolean;
}
type IndexerBuild = (args: {
reason?: string;
changedPaths?: string[];
}) => IndexerBuildResult | void | Promise<IndexerBuildResult | void>;
interface IndexerServiceOptions { interface IndexerServiceOptions {
projectsDir?: string; projectsDir?: string;
watchDirs?: string | string[]; watchDirs?: string | string[];
@@ -29,8 +39,9 @@ interface IndexerServiceOptions {
stabilityMs?: number; stabilityMs?: number;
heartbeatMs?: number; heartbeatMs?: number;
watchRetryMs?: number; watchRetryMs?: number;
buildIndex?: (args: { reason?: string; changedPaths?: string[] }) => unknown; deferredRetryMs?: number;
writeHeartbeat?: () => void; buildIndex?: IndexerBuild;
writeHeartbeat?: () => unknown;
watchProjects?: (onChange: (changedPath: string) => void) => Watcher | null; watchProjects?: (onChange: (changedPath: string) => void) => Watcher | null;
chokidar?: any; chokidar?: any;
timers?: Timers; timers?: Timers;
@@ -44,6 +55,7 @@ function createIndexerService({
stabilityMs = DEFAULT_STABILITY_MS, stabilityMs = DEFAULT_STABILITY_MS,
heartbeatMs = DEFAULT_HEARTBEAT_MS, heartbeatMs = DEFAULT_HEARTBEAT_MS,
watchRetryMs = DEFAULT_WATCH_RETRY_MS, watchRetryMs = DEFAULT_WATCH_RETRY_MS,
deferredRetryMs = DEFAULT_DEFERRED_RETRY_MS,
buildIndex, buildIndex,
writeHeartbeat = () => {}, writeHeartbeat = () => {},
watchProjects, watchProjects,
@@ -100,6 +112,7 @@ function createIndexerService({
let stabilityTimer: TimerHandle | null = null; let stabilityTimer: TimerHandle | null = null;
let heartbeatTimer: TimerHandle | null = null; let heartbeatTimer: TimerHandle | null = null;
let watchRetryTimer: TimerHandle | null = null; let watchRetryTimer: TimerHandle | null = null;
let deferredRetryTimer: TimerHandle | null = null;
let watcher: Watcher | null = null; let watcher: Watcher | null = null;
let stopped = false; let stopped = false;
let running = false; let running = false;
@@ -124,6 +137,15 @@ function createIndexerService({
return paths; return paths;
}; };
const publishHeartbeat = () => {
try {
return writeHeartbeat();
} catch (error) {
logger.warn?.(`Obelisk heartbeat failed: ${(error as Error).message}`);
return false;
}
};
const runBuildNow = (reason = "manual", paths: string[] | undefined = undefined) => { const runBuildNow = (reason = "manual", paths: string[] | undefined = undefined) => {
addChangedPath(paths); addChangedPath(paths);
if (stopped) return idlePromise; if (stopped) return idlePromise;
@@ -135,8 +157,18 @@ function createIndexerService({
pending = false; pending = false;
const buildChangedPaths = takeChangedPaths(); const buildChangedPaths = takeChangedPaths();
idlePromise = (async () => { idlePromise = (async () => {
await buildIndex({ reason, changedPaths: buildChangedPaths }); const result = await buildIndex({ reason, changedPaths: buildChangedPaths });
writeHeartbeat(); if (result?.deferred) {
addChangedPath(buildChangedPaths);
if (!stopped && !deferredRetryTimer) {
deferredRetryTimer = timers.setTimeout(() => {
deferredRetryTimer = null;
runBuildNow('writer-lease');
}, deferredRetryMs);
}
return;
}
publishHeartbeat();
})() })()
.catch((error) => { .catch((error) => {
// A build in flight when the service is stopped (e.g. a manual rebuild // A build in flight when the service is stopped (e.g. a manual rebuild
@@ -158,6 +190,8 @@ function createIndexerService({
addChangedPath(changedPath); addChangedPath(changedPath);
lastReason = reason; lastReason = reason;
if (running) pending = true; if (running) pending = true;
if (deferredRetryTimer) timers.clearTimeout(deferredRetryTimer);
deferredRetryTimer = null;
if (buildTimer) timers.clearTimeout(buildTimer); if (buildTimer) timers.clearTimeout(buildTimer);
if (stabilityTimer) timers.clearTimeout(stabilityTimer); if (stabilityTimer) timers.clearTimeout(stabilityTimer);
buildTimer = timers.setTimeout(() => { buildTimer = timers.setTimeout(() => {
@@ -186,15 +220,12 @@ function createIndexerService({
const start = ({ buildOnStart = true } = {}) => { const start = ({ buildOnStart = true } = {}) => {
stopped = false; stopped = false;
publishHeartbeat();
if (buildOnStart) scheduleBuild('startup'); if (buildOnStart) scheduleBuild('startup');
startWatching(); startWatching();
if (typeof timers.setInterval === 'function') { if (typeof timers.setInterval === 'function') {
heartbeatTimer = timers.setInterval(() => { heartbeatTimer = timers.setInterval(() => {
try { publishHeartbeat();
writeHeartbeat();
} catch (error) {
logger.warn?.(`Obelisk heartbeat failed: ${(error as Error).message}`);
}
}, heartbeatMs); }, heartbeatMs);
} }
}; };
@@ -208,6 +239,8 @@ function createIndexerService({
stabilityTimer = null; stabilityTimer = null;
if (watchRetryTimer) timers.clearTimeout(watchRetryTimer); if (watchRetryTimer) timers.clearTimeout(watchRetryTimer);
watchRetryTimer = null; watchRetryTimer = null;
if (deferredRetryTimer) timers.clearTimeout(deferredRetryTimer);
deferredRetryTimer = null;
if (heartbeatTimer && typeof timers.clearInterval === 'function') timers.clearInterval(heartbeatTimer); if (heartbeatTimer && typeof timers.clearInterval === 'function') timers.clearInterval(heartbeatTimer);
heartbeatTimer = null; heartbeatTimer = null;
if (watcher?.close) watcher.close(); if (watcher?.close) watcher.close();
+187 -45
View File
@@ -6,6 +6,9 @@ import Database from 'better-sqlite3';
import { parse as claudeParse } from '../../../scripts/providers/claude.ts'; import { parse as claudeParse } from '../../../scripts/providers/claude.ts';
import { parse as codexParse } from '../../../scripts/providers/codex.ts'; import { parse as codexParse } from '../../../scripts/providers/codex.ts';
import { persist } from '../../../scripts/persist.ts'; import { persist } from '../../../scripts/persist.ts';
import { runWriteTransaction, configureConnection, betterSqliteTransactionAdapter } from '../../../scripts/tx.ts';
import { acquireWriterLease, writerLockPathFor } from '../../../scripts/writer-lease.ts';
import { runRetryableWriteTransaction, isBeginBusyFailure, hasUnusableTransaction } from '../../../scripts/write-coordinator.ts';
import { import {
inferProjectPath, inferProjectPath,
isDir, isDir,
@@ -35,15 +38,6 @@ interface FileInfo {
source?: string; source?: string;
} }
// A rollback inside a catch must never throw over the real error. SQLite
// auto-rolls back certain failures (SQLITE_BUSY, disk full, ...); a following
// explicit ROLLBACK then throws "cannot rollback - no transaction is active",
// which would both mask the true cause and turn a skippable per-file error into
// a whole-build failure. Swallow only the rollback's own error.
function safeRollback(db: { exec: (sql: string) => unknown }) {
try { db.exec('ROLLBACK'); } catch { /* no active transaction */ }
}
function resolveSchemaPath() { function resolveSchemaPath() {
const candidates = [ const candidates = [
path.join(__dirname, 'schema.sql'), path.join(__dirname, 'schema.sql'),
@@ -63,8 +57,7 @@ function installSchema(db, schemaPath = resolveSchemaPath()) {
function openIndexDb({ dbPath = DEFAULT_DB_PATH, schemaPath = resolveSchemaPath(), DatabaseImpl = Database }: { dbPath?: string; schemaPath?: string; DatabaseImpl?: new (dbPath: string) => any } = {}) { function openIndexDb({ dbPath = DEFAULT_DB_PATH, schemaPath = resolveSchemaPath(), DatabaseImpl = Database }: { dbPath?: string; schemaPath?: string; DatabaseImpl?: new (dbPath: string) => any } = {}) {
fs.mkdirSync(path.dirname(dbPath), { recursive: true }); fs.mkdirSync(path.dirname(dbPath), { recursive: true });
const db = new DatabaseImpl(dbPath); const db = new DatabaseImpl(dbPath);
db.pragma('journal_mode = WAL'); configureConnection(db, { busyTimeoutMs: 250 });
db.pragma('synchronous = NORMAL');
installSchema(db, schemaPath); installSchema(db, schemaPath);
return db; return db;
} }
@@ -138,7 +131,10 @@ function normalizeChangedPath(projectsDir, changedPath) {
} }
function jsonlFileInfoFromPath(projectsDir, changedPath) { function jsonlFileInfoFromPath(projectsDir, changedPath) {
const fp = normalizeChangedPath(projectsDir, changedPath); let fp = normalizeChangedPath(projectsDir, changedPath);
if (fp?.toLowerCase().endsWith('.meta.json')) {
fp = fp.slice(0, -'.meta.json'.length) + '.jsonl';
}
if (!fp || !fp.endsWith('.jsonl')) return null; if (!fp || !fp.endsWith('.jsonl')) return null;
if (!fs.existsSync(fp)) return null; if (!fs.existsSync(fp)) return null;
const rel = path.relative(projectsDir, fp); const rel = path.relative(projectsDir, fp);
@@ -365,14 +361,16 @@ function indexCodexSessionIndex(db, { codexDir = DEFAULT_CODEX_DIR } = {}) {
const indexPath = path.join(codexDir, 'session_index.jsonl'); const indexPath = path.join(codexDir, 'session_index.jsonl');
if (!fs.existsSync(indexPath)) return; if (!fs.existsSync(indexPath)) return;
readLines(indexPath, (line) => { readLines(indexPath, (line) => {
let item;
try { try {
const item = JSON.parse(line); item = JSON.parse(line);
} catch (error) {
console.warn(`Warning: malformed Codex session index line: ${(error as Error).message}`);
return;
}
if (!item.id || !item.thread_name) return; if (!item.id || !item.thread_name) return;
db.prepare('UPDATE sessions SET title=COALESCE(title, ?), ended_at=COALESCE(ended_at, ?) WHERE id=? AND source=?') db.prepare('UPDATE sessions SET title=COALESCE(title, ?), ended_at=COALESCE(ended_at, ?) WHERE id=? AND source=?')
.run(item.thread_name, item.updated_at || null, codexDbId(item.id), 'codex'); .run(item.thread_name, item.updated_at || null, codexDbId(item.id), 'codex');
} catch (error) {
console.warn(`Warning: malformed Codex session index line: ${(error as Error).message}`);
}
}); });
} }
@@ -392,11 +390,16 @@ function refreshSessionProjectPaths(db) {
} }
function indexSubagentMeta(db, fi) { function indexSubagentMeta(db, fi) {
if (!fi.isSubagent) return; if (!fi.isSubagent) return false;
const mp = fi.path.replace('.jsonl', '.meta.json'); const mp = fi.path.replace('.jsonl', '.meta.json');
if (!fs.existsSync(mp)) return; if (!fs.existsSync(mp)) return false;
let meta;
try { try {
const meta = JSON.parse(fs.readFileSync(mp, 'utf8')); meta = JSON.parse(fs.readFileSync(mp, 'utf8'));
} catch (error) {
console.warn(`Warning: failed to read subagent meta ${mp}: ${(error as Error).message}`);
return false;
}
const tok = db.prepare('SELECT COALESCE(SUM(input_tokens),0)+COALESCE(SUM(output_tokens),0) as t FROM messages WHERE agent_id=?').get(fi.agentId); const tok = db.prepare('SELECT COALESCE(SUM(input_tokens),0)+COALESCE(SUM(output_tokens),0) as t FROM messages WHERE agent_id=?').get(fi.agentId);
const ts = db.prepare('SELECT MIN(timestamp) as t0, MAX(timestamp) as t1 FROM messages WHERE agent_id=?').get(fi.agentId); const ts = db.prepare('SELECT MIN(timestamp) as t0, MAX(timestamp) as t1 FROM messages WHERE agent_id=?').get(fi.agentId);
const dur = ts?.t0 && ts?.t1 ? new Date(ts.t1).getTime() - new Date(ts.t0).getTime() : null; const dur = ts?.t0 && ts?.t1 ? new Date(ts.t1).getTime() - new Date(ts.t0).getTime() : null;
@@ -405,9 +408,7 @@ function indexSubagentMeta(db, fi) {
} else { } else {
db.prepare('INSERT OR REPLACE INTO subagents VALUES(?,?,?,?,?,?,?)').run(fi.agentId, fi.sessionId, meta.toolUseId||null, meta.agentType||null, meta.description||null, dur, tok?.t||0); db.prepare('INSERT OR REPLACE INTO subagents VALUES(?,?,?,?,?,?,?)').run(fi.agentId, fi.sessionId, meta.toolUseId||null, meta.agentType||null, meta.description||null, dur, tok?.t||0);
} }
} catch (error) { return true;
console.warn(`Warning: failed to read subagent meta ${mp}: ${(error as Error).message}`);
}
} }
function indexWorkflows(db, { projectsDir = DEFAULT_PROJECTS_DIR } = {}) { function indexWorkflows(db, { projectsDir = DEFAULT_PROJECTS_DIR } = {}) {
@@ -426,8 +427,13 @@ function indexWorkflows(db, { projectsDir = DEFAULT_PROJECTS_DIR } = {}) {
try { wfFiles = fs.readdirSync(wd); } catch { continue; } try { wfFiles = fs.readdirSync(wd); } catch { continue; }
for (const f of wfFiles) { for (const f of wfFiles) {
if (!f.endsWith('.json')) continue; if (!f.endsWith('.json')) continue;
let wf;
try { try {
const wf = JSON.parse(fs.readFileSync(path.join(wd, f), 'utf8')); wf = JSON.parse(fs.readFileSync(path.join(wd, f), 'utf8'));
} catch (error) {
console.warn(`Warning: failed to read workflow ${f}: ${(error as Error).message}`);
continue;
}
if (!wf.runId) continue; if (!wf.runId) continue;
const ac = db.prepare('SELECT COUNT(*) as c FROM workflow_agents WHERE run_id=?').get(wf.runId); const ac = db.prepare('SELECT COUNT(*) as c FROM workflow_agents WHERE run_id=?').get(wf.runId);
db.prepare('INSERT OR REPLACE INTO workflows (run_id,session_id,task_id,script,result_json,timestamp,agent_count,duration_ms,total_tokens,status,workflow_name) VALUES(?,?,?,?,?,?,?,?,?,?,?)').run( db.prepare('INSERT OR REPLACE INTO workflows (run_id,session_id,task_id,script,result_json,timestamp,agent_count,duration_ms,total_tokens,status,workflow_name) VALUES(?,?,?,?,?,?,?,?,?,?,?)').run(
@@ -441,9 +447,6 @@ function indexWorkflows(db, { projectsDir = DEFAULT_PROJECTS_DIR } = {}) {
item.phaseTitle||null, item.label||null, item.model||null, item.state||null, item.phaseTitle||null, item.label||null, item.model||null, item.state||null,
item.durationMs||null, item.tokens||null, item.toolCalls||null, 'agent-' + item.agentId); item.durationMs||null, item.tokens||null, item.toolCalls||null, 'agent-' + item.agentId);
} }
} catch (error) {
console.warn(`Warning: failed to index workflow ${f}: ${(error as Error).message}`);
}
} }
} }
} }
@@ -452,12 +455,14 @@ function indexWorkflows(db, { projectsDir = DEFAULT_PROJECTS_DIR } = {}) {
function indexHistory(db, { historyPath = DEFAULT_HISTORY_PATH } = {}) { function indexHistory(db, { historyPath = DEFAULT_HISTORY_PATH } = {}) {
if (!fs.existsSync(historyPath)) return; if (!fs.existsSync(historyPath)) return;
readLines(historyPath, (line) => { readLines(historyPath, (line) => {
let item;
try { try {
const o = JSON.parse(line); item = JSON.parse(line);
if (o.sessionId && o.title) db.prepare('UPDATE sessions SET title=? WHERE id=? AND title IS NULL').run(o.title, o.sessionId);
} catch (error) { } catch (error) {
console.warn(`Warning: malformed history line: ${(error as Error).message}`); console.warn(`Warning: malformed history line: ${(error as Error).message}`);
return;
} }
if (item.sessionId && item.title) db.prepare('UPDATE sessions SET title=? WHERE id=? AND title IS NULL').run(item.title, item.sessionId);
}); });
} }
@@ -466,9 +471,13 @@ function rebuildFts(db) {
db.exec("INSERT INTO memories_fts(memories_fts) VALUES('rebuild')"); db.exec("INSERT INTO memories_fts(memories_fts) VALUES('rebuild')");
} }
function checkpointDb(db) { // PASSIVE by default: it checkpoints what it can without blocking concurrent
// readers/writers, so it is safe to run after every build. A blocking TRUNCATE
// (which reclaims the -wal file but needs exclusive access and can contend with
// the daemon + queries) is reserved for maintenance/exit — pass mode explicitly.
function checkpointDb(db, mode = 'PASSIVE') {
try { try {
db.pragma('wal_checkpoint(TRUNCATE)'); db.pragma(`wal_checkpoint(${mode})`);
} catch {} } catch {}
} }
@@ -497,14 +506,31 @@ function writeIndexMarker(db, key, value = Date.now()) {
db.prepare('INSERT OR REPLACE INTO index_state (jsonl_path, mtime, lines_processed) VALUES (?, ?, 0)').run(key, value); db.prepare('INSERT OR REPLACE INTO index_state (jsonl_path, mtime, lines_processed) VALUES (?, ?, 0)').run(key, value);
} }
function writeHeartbeat({ dbPath = DEFAULT_DB_PATH, DatabaseImpl = Database } = {}) { function writeHeartbeat({
dbPath = DEFAULT_DB_PATH,
writerLeasePath = writerLockPathFor(dbPath),
DatabaseImpl = Database,
LockDatabaseImpl = DatabaseImpl,
} = {}) {
if (!fs.existsSync(dbPath)) return; if (!fs.existsSync(dbPath)) return;
const db = new DatabaseImpl(dbPath); const lease = acquireWriterLease({
lockPath: writerLeasePath,
openDb: lockPath => new LockDatabaseImpl(lockPath),
});
if (!lease) return false;
try { try {
writeIndexMarker(db, '__app_heartbeat__'); const db = new DatabaseImpl(dbPath);
configureConnection(db, { busyTimeoutMs: 0 });
const txDb = betterSqliteTransactionAdapter(db);
try {
runWriteTransaction(txDb, () => writeIndexMarker(db, '__app_heartbeat__'), { label: 'heartbeat' });
return true;
} finally { } finally {
db.close(); db.close();
} }
} finally {
lease.release();
}
} }
interface BuildIndexOptions { interface BuildIndexOptions {
@@ -515,9 +541,19 @@ interface BuildIndexOptions {
dbPath?: string; dbPath?: string;
schemaPath?: string; schemaPath?: string;
DatabaseImpl?: new (dbPath: string) => any; DatabaseImpl?: new (dbPath: string) => any;
LockDatabaseImpl?: new (dbPath: string) => any;
force?: boolean; force?: boolean;
changedPaths?: string[]; changedPaths?: string[];
preserveDbPath?: string | null; preserveDbPath?: string | null;
writerLeasePath?: string;
writerLeaseWaitMs?: number;
writerLeaseMode?: 'acquire' | 'caller-held';
}
interface SkippedFile {
path: string;
error: string;
diagnostics?: unknown;
} }
interface BuildIndexResult { interface BuildIndexResult {
@@ -525,6 +561,27 @@ interface BuildIndexResult {
latestSourceMtime: number; latestSourceMtime: number;
affectedSessionIds: string[]; affectedSessionIds: string[];
ftsRebuilt: boolean; ftsRebuilt: boolean;
skipped: number;
skippedFiles: SkippedFile[];
deferred: boolean;
reason?: string;
}
function deferredBuildResult(
reason: string,
overrides: Partial<Omit<BuildIndexResult, 'deferred' | 'reason'>> = {},
): BuildIndexResult {
return {
files: 0,
latestSourceMtime: 0,
affectedSessionIds: [],
ftsRebuilt: false,
skipped: 0,
skippedFiles: [],
...overrides,
deferred: true,
reason,
};
} }
function buildIndex({ function buildIndex({
@@ -535,12 +592,33 @@ function buildIndex({
dbPath = DEFAULT_DB_PATH, dbPath = DEFAULT_DB_PATH,
schemaPath = resolveSchemaPath(), schemaPath = resolveSchemaPath(),
DatabaseImpl = Database, DatabaseImpl = Database,
LockDatabaseImpl = DatabaseImpl,
force = false, force = false,
changedPaths = undefined, changedPaths = undefined,
preserveDbPath = null, preserveDbPath = null,
writerLeasePath = writerLockPathFor(dbPath),
writerLeaseWaitMs = 2000,
writerLeaseMode = 'acquire',
}: BuildIndexOptions = {}): BuildIndexResult { }: BuildIndexOptions = {}): BuildIndexResult {
if (writerLeaseMode !== 'acquire' && writerLeaseMode !== 'caller-held') {
throw new Error(`Unknown writer lease mode: ${writerLeaseMode}`);
}
let lease: ReturnType<typeof acquireWriterLease> = null;
if (writerLeaseMode === 'acquire') {
lease = acquireWriterLease({
lockPath: writerLeasePath,
openDb: lockPath => new LockDatabaseImpl(lockPath),
waitMs: writerLeaseWaitMs,
});
if (!lease) {
return deferredBuildResult('writer_busy');
}
}
try {
const db = openIndexDb({ dbPath, schemaPath, DatabaseImpl }); const db = openIndexDb({ dbPath, schemaPath, DatabaseImpl });
const txDb = betterSqliteTransactionAdapter(db);
let messageFtsTriggersDropped = false; let messageFtsTriggersDropped = false;
try {
if (preserveDbPath && path.resolve(preserveDbPath) !== path.resolve(dbPath)) { if (preserveDbPath && path.resolve(preserveDbPath) !== path.resolve(dbPath)) {
copyMemoriesFromDb(db, preserveDbPath); copyMemoriesFromDb(db, preserveDbPath);
} }
@@ -558,8 +636,8 @@ function buildIndex({
try { try {
if (force) { if (force) {
runRetryableWriteTransaction(txDb, () => {
dropMessageFtsTriggers(db); dropMessageFtsTriggers(db);
messageFtsTriggersDropped = true;
db.prepare("DELETE FROM index_state WHERE substr(jsonl_path, 1, 2) != '__'").run(); db.prepare("DELETE FROM index_state WHERE substr(jsonl_path, 1, 2) != '__'").run();
db.prepare("DELETE FROM messages").run(); db.prepare("DELETE FROM messages").run();
db.prepare("DELETE FROM tool_calls").run(); db.prepare("DELETE FROM tool_calls").run();
@@ -569,29 +647,74 @@ function buildIndex({
db.prepare("DELETE FROM subagents").run(); db.prepare("DELETE FROM subagents").run();
db.prepare("DELETE FROM workflows").run(); db.prepare("DELETE FROM workflows").run();
db.prepare("DELETE FROM workflow_agents").run(); db.prepare("DELETE FROM workflow_agents").run();
}, { label: 'force-cleanup' });
messageFtsTriggersDropped = true;
}
} catch (error) {
if (isBeginBusyFailure(error)) {
return deferredBuildResult('database_busy', {
files: files.length,
latestSourceMtime,
});
}
throw error;
} }
const affectedSessionIds = new Set<string>(); const affectedSessionIds = new Set<string>();
const finalizeAffectedSessionIds = new Set<string>();
const changedMetaJsonlPaths = new Set<string>();
if (Array.isArray(changedPaths)) { if (Array.isArray(changedPaths)) {
for (const changedPath of changedPaths) { for (const changedPath of changedPaths) {
const sessionId = sessionIdFromChangedPath(projectsDir, changedPath); const sessionId = sessionIdFromChangedPath(projectsDir, changedPath);
if (sessionId) affectedSessionIds.add(sessionId); const normalizedChangedPath = normalizeChangedPath(projectsDir, changedPath);
const isMetaChange = normalizedChangedPath?.toLowerCase().endsWith('.meta.json');
if (isMetaChange && normalizedChangedPath) {
changedMetaJsonlPaths.add(
normalizedChangedPath.slice(0, -'.meta.json'.length) + '.jsonl',
);
}
// Transcript files report their session only after their own transaction
// commits. Workflow changes are applied during finalize, so stage those
// IDs until the finalize transaction commits. Meta files map back to their
// transcript transaction and are reported only after that commit.
if (sessionId && !changedPath.toLowerCase().endsWith('.jsonl') && !isMetaChange) {
finalizeAffectedSessionIds.add(sessionId);
} }
} }
}
const skipped: SkippedFile[] = [];
for (const file of files) { for (const file of files) {
db.exec('BEGIN');
try { try {
const indexed = file.source === 'codex' ? indexCodexFile(db, file) : indexClaudeFile(db, file); // The write is committed before affectedSessionIds is updated, so a
// failed/rolled-back file never reports a phantom updated session.
const indexed = runRetryableWriteTransaction(txDb, () => {
const result = file.source === 'codex' ? indexCodexFile(db, file) : indexClaudeFile(db, file);
const metaIndexed = file.source !== 'codex' && indexSubagentMeta(db, file);
if (!result?.sessionId && metaIndexed && changedMetaJsonlPaths.has(file.path)) {
return { sessionId: file.sessionId, path: file.path };
}
return result;
}, { label: `file:${file.path}` });
if (indexed?.sessionId) affectedSessionIds.add(indexed.sessionId); if (indexed?.sessionId) affectedSessionIds.add(indexed.sessionId);
if (file.source !== 'codex') indexSubagentMeta(db, file);
db.exec('COMMIT');
} catch (error) { } catch (error) {
safeRollback(db); if (isBeginBusyFailure(error)) {
return deferredBuildResult('database_busy', {
files: files.length,
latestSourceMtime,
affectedSessionIds: [...affectedSessionIds],
skipped: skipped.length,
skippedFiles: skipped,
});
}
if (hasUnusableTransaction(error)) throw error;
skipped.push({ path: file.path, error: (error as Error).message, diagnostics: (error as { obelisk?: unknown }).obelisk });
console.warn(`Warning: failed to index ${file.path}: ${(error as Error).message}`); console.warn(`Warning: failed to index ${file.path}: ${(error as Error).message}`);
} }
} }
db.exec('BEGIN');
let ftsRebuilt = false; let ftsRebuilt = false;
// Finalize is one transaction; a failure here fails the whole build (the
// index would otherwise be left inconsistent).
try { try {
runRetryableWriteTransaction(txDb, () => {
indexWorkflows(db, { projectsDir }); indexWorkflows(db, { projectsDir });
refreshSessionProjectPaths(db); refreshSessionProjectPaths(db);
indexHistory(db, { historyPath }); indexHistory(db, { historyPath });
@@ -599,16 +722,32 @@ function buildIndex({
if (messageFtsTriggersDropped) installSchema(db, schemaPath); if (messageFtsTriggersDropped) installSchema(db, schemaPath);
ftsRebuilt = ensureFtsReady(db, { force }); ftsRebuilt = ensureFtsReady(db, { force });
writeIndexMarker(db, '__last_build__'); writeIndexMarker(db, '__last_build__');
writeIndexMarker(db, '__app_heartbeat__');
writeIndexMarker(db, '__app_last_successful_build__'); writeIndexMarker(db, '__app_last_successful_build__');
writeIndexMarker(db, '__indexer_owner_app__'); writeIndexMarker(db, '__indexer_owner_app__');
if (latestSourceMtime) writeIndexMarker(db, '__last_source_mtime__', latestSourceMtime); if (latestSourceMtime) writeIndexMarker(db, '__last_source_mtime__', latestSourceMtime);
db.exec('COMMIT'); }, { label: 'finalize' });
} catch (error) { } catch (error) {
safeRollback(db); if (isBeginBusyFailure(error)) {
return deferredBuildResult('database_busy', {
files: files.length,
latestSourceMtime,
affectedSessionIds: [...affectedSessionIds],
skipped: skipped.length,
skippedFiles: skipped,
});
}
throw error; throw error;
} }
return { files: files.length, latestSourceMtime, affectedSessionIds: [...affectedSessionIds], ftsRebuilt }; for (const sessionId of finalizeAffectedSessionIds) affectedSessionIds.add(sessionId);
return {
files: files.length,
latestSourceMtime,
affectedSessionIds: [...affectedSessionIds],
ftsRebuilt,
skipped: skipped.length,
skippedFiles: skipped,
deferred: false,
};
} finally { } finally {
if (messageFtsTriggersDropped) { if (messageFtsTriggersDropped) {
try { try {
@@ -620,6 +759,9 @@ function buildIndex({
checkpointDb(db); checkpointDb(db);
db.close(); db.close();
} }
} finally {
lease?.release();
}
} }
export { export {
+27
View File
@@ -0,0 +1,27 @@
import { createRequire } from 'node:module';
import { createInterface } from 'node:readline';
const require = createRequire(import.meta.url);
const Database = require('better-sqlite3');
const [mode, payloadJson] = process.argv.slice(2);
const payload = JSON.parse(payloadJson || '{}');
if (mode === 'holder') {
const db = new Database(payload.lockPath);
db.pragma('busy_timeout = 0');
db.exec('BEGIN IMMEDIATE');
process.stdout.write('READY\n');
const input = createInterface({ input: process.stdin });
await new Promise(resolve => input.once('line', resolve));
db.exec('ROLLBACK');
db.close();
input.close();
} else if (mode === 'build') {
const { buildIndex } = await import('../out/main/indexer.js');
process.stdout.write('STARTING\n');
const result = buildIndex(payload.options);
process.stdout.write(`RESULT ${JSON.stringify(result)}\n`);
} else {
throw new Error(`Unknown concurrency child mode: ${mode}`);
}
+158
View File
@@ -0,0 +1,158 @@
// Real Electron/better-sqlite3 concurrency test (docs/adr/0006 Phase 2).
// Run: cd app && npx electron tests/electron-concurrency.mjs
//
// Exercises actual dual-connection contention against a WAL database using the
// Electron-ABI better-sqlite3 that the app uses in production.
import { app } from 'electron';
import { spawn } from 'node:child_process';
import { mkdirSync, writeFileSync, rmSync, mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { createInterface } from 'node:readline';
import { once } from 'node:events';
import { setTimeout as delay } from 'node:timers/promises';
import Database from 'better-sqlite3';
import { buildIndex } from '../out/main/indexer.js';
let failures = 0;
const childScript = join(dirname(fileURLToPath(import.meta.url)), 'electron-concurrency-child.mjs');
function assert(condition, msg) {
if (!condition) { console.error('FAIL:', msg); failures++; }
else console.log('PASS:', msg);
}
function spawnChild(mode, payload) {
return spawn(process.execPath, [childScript, mode, JSON.stringify(payload)], {
env: { ...process.env, ELECTRON_RUN_AS_NODE: '1' },
stdio: ['pipe', 'pipe', 'inherit'],
});
}
function lineReader(child) {
const lines = [];
const waiters = [];
createInterface({ input: child.stdout }).on('line', line => {
lines.push(line);
for (const waiter of [...waiters]) {
if (!line.startsWith(waiter.prefix)) continue;
waiters.splice(waiters.indexOf(waiter), 1);
waiter.resolve(line);
}
});
return {
waitFor(prefix) {
const existing = lines.find(line => line.startsWith(prefix));
if (existing) return Promise.resolve(existing);
return new Promise(resolve => waiters.push({ prefix, resolve }));
},
};
}
async function waitForSuccess(child) {
let code = child.exitCode;
if (code === null) [code] = await once(child, 'exit');
assert(code === 0, `child exited successfully, code=${code}`);
}
async function run() {
const home = mkdtempSync(join(tmpdir(), 'obelisk-electron-concurrency-'));
const dbPath = join(home, '.obelisk', 'obelisk.sqlite');
const projectsDir = join(home, '.claude', 'projects');
const projDir = join(projectsDir, '-proj');
mkdirSync(join(home, '.obelisk'), { recursive: true });
mkdirSync(projDir, { recursive: true });
function msg(uuid) {
return JSON.stringify({
uuid, type: 'user', timestamp: '2026-06-10T10:00:00Z', cwd: '/tmp',
message: { role: 'user', content: `concurrent ${uuid}` },
}) + '\n';
}
for (let i = 0; i < 20; i++) {
writeFileSync(join(projDir, `s${i}.jsonl`), msg(`m${i}`));
}
console.log('--- Test 1: buildIndex with real better-sqlite3 ---');
const result = buildIndex({
force: true,
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
DatabaseImpl: Database,
});
assert(result.files === 20, `indexed 20 files, got ${result.files}`);
assert(result.skipped === 0, `no files skipped, got ${result.skipped}`);
console.log('--- Test 2: concurrent reader during write ---');
const reader = new Database(dbPath, { readonly: true });
reader.pragma('journal_mode = WAL');
const readStmt = reader.prepare('SELECT COUNT(*) AS c FROM sessions');
const beforeCount = readStmt.get().c;
assert(beforeCount === 20, `reader sees 20 sessions, got ${beforeCount}`);
// Incremental build concurrent with open reader
const result2 = buildIndex({
force: false,
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
DatabaseImpl: Database,
});
assert(result2.skipped === 0, `concurrent build no skips, got ${result2.skipped}`);
const duringCount = readStmt.get().c;
assert(duringCount === 20, `reader snapshot stable, got ${duringCount}`);
reader.close();
console.log('--- Test 3: a real concurrent writer releases within the lease budget ---');
const lockPath = join(dirname(dbPath), 'writer.lock.sqlite');
const buildOptions = {
force: false,
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
writerLeaseWaitMs: 1500,
};
const holder = spawnChild('holder', { lockPath });
const holderLines = lineReader(holder);
await holderLines.waitFor('READY');
const contendedBuild = spawnChild('build', { options: buildOptions });
const buildLines = lineReader(contendedBuild);
await buildLines.waitFor('STARTING');
const startedAt = Date.now();
await delay(200);
holder.stdin.write('release\n');
const resultLine = await buildLines.waitFor('RESULT ');
const result3 = JSON.parse(resultLine.slice('RESULT '.length));
const waitedMs = Date.now() - startedAt;
assert(result3.deferred === false, `contended build completed, reason=${result3.reason}`);
assert(result3.skipped === 0, `contended build skipped no files, got ${result3.skipped}`);
assert(waitedMs >= 150, `build overlapped the held lease for ${waitedMs}ms`);
await Promise.all([waitForSuccess(holder), waitForSuccess(contendedBuild)]);
console.log('--- Test 4: persistent writer contention is bounded ---');
const persistentHolder = spawnChild('holder', { lockPath });
const persistentHolderLines = lineReader(persistentHolder);
await persistentHolderLines.waitFor('READY');
const boundedBuild = spawnChild('build', { options: { ...buildOptions, writerLeaseWaitMs: 200 } });
const boundedLines = lineReader(boundedBuild);
await boundedLines.waitFor('STARTING');
const boundedStartedAt = Date.now();
const boundedResultLine = await boundedLines.waitFor('RESULT ');
const boundedResult = JSON.parse(boundedResultLine.slice('RESULT '.length));
const boundedMs = Date.now() - boundedStartedAt;
assert(boundedResult.deferred === true, 'persistent contention returns deferred');
assert(boundedResult.reason === 'writer_busy', `persistent contention reason=${boundedResult.reason}`);
assert(boundedMs < 1000, `persistent contention returned within budget (${boundedMs}ms)`);
persistentHolder.stdin.write('release\n');
await Promise.all([waitForSuccess(persistentHolder), waitForSuccess(boundedBuild)]);
rmSync(home, { recursive: true, force: true });
console.log('---');
console.log(failures ? `${failures} TEST(S) FAILED` : 'ALL TESTS PASSED');
process.exitCode = failures ? 1 : 0;
}
app.whenReady().then(run).finally(() => app.quit());
@@ -1,76 +1,75 @@
# Write-transaction rollback safety and SQLite concurrency # Write-transaction rollback safety and SQLite concurrency
**Context.** The app surfaced `Obelisk index build failed: cannot rollback - no **Context.** The app surfaced `Obelisk index build failed: cannot rollback - no
transaction is active`. That message is a *secondary* error: SQLite auto-rolls transaction is active`. That text was a secondary cleanup failure. SQLite had
back certain failures (notably `SQLITE_BUSY` / `SQLITE_BUSY_SNAPSHOT`, also disk already ended the transaction, then the catch block's unguarded `ROLLBACK`
full), after which the per-file build loop's unguarded `db.exec('ROLLBACK')` in threw over the primary exception and turned a skippable per-file failure into a
its `catch` threw over the real error and aborted the whole build instead of whole-build failure. The masked exception was not preserved, so contention
skipping just the offending file. The underlying trigger is concurrency: the app (`SQLITE_BUSY` / `SQLITE_BUSY_SNAPSHOT`) is the leading explanation rather than
runs a daemon indexer, manual rebuilds, and read queries against one WAL a proven historical fact. It is plausible because daemon builds, manual
database, and the skill's passive-pull build can write the same database from a rebuilds, skill passive-pull indexing, heartbeat writes, and reads share one WAL
separate process. database.
`busy_timeout` is **not** the root-cause fix and must not be treated as one. `busy_timeout` alone is not a correctness fix. In particular,
better-sqlite3's constructor already defaults `timeout` to 5000ms, so the app hit `SQLITE_BUSY_SNAPSHOT` is not made safe by waiting longer, and retrying only the
`SQLITE_BUSY` *despite* a 5s wait — which points at `SQLITE_BUSY_SNAPSHOT` from failed statement can replay part of a transaction.
deferred (read-then-write) transactions, a snapshot conflict that `busy_timeout`
does not wait on. Only `BEGIN IMMEDIATE` plus whole-transaction retry addresses
that.
**Decision.** Split the work into a cheap stopgap now and a correctness/ **Decision.** Use one transaction primitive plus two explicit coordination
concurrency fix later. layers.
*Stopgap (done).* Both indexers use a guarded rollback (`safeRollback`): a - `scripts/tx.ts` owns the binding-agnostic `runWriteTransaction(db, work)`.
cleanup rollback swallows only its own error and never masks the primary one. Adapters expose transaction state from better-sqlite3's `inTransaction` and
Per-file failures are logged and the build continues; the finalize failure still node:sqlite's `isTransaction`. The primitive performs `BEGIN IMMEDIATE`, runs
propagates. The skill's connection (`node:sqlite`, which has **no** default busy `work` exactly once, commits, and attempts rollback only when the binding says
timeout) gets an explicit `PRAGMA busy_timeout = 5000`; the app adds no such a transaction is active or its state is unknown. Cleanup never masks the
pragma because better-sqlite3 already defaults to 5000ms — adding it there was primary exception. Diagnostics record phase, SQLite code, rollback outcome,
redundant and removed, precisely so nobody reads it as "the fix". transaction state, label, and attempts.
- Retry is an upper-layer policy in `scripts/write-coordinator.ts`, never hidden
inside the transaction primitive. Only an idempotent whole transaction that
failed during work/commit with `SQLITE_BUSY*` and is confirmed inactive may be
retried. The default is three attempts within a one-second budget with short
backoff. BEGIN contention is deferred to the build scheduler; an active or
unknown post-error transaction aborts the build.
- Per-file failures remain warnings and are reported in `skippedFiles`; finalize
failures propagate. `affectedSessionIds` is updated only after the relevant
commit. Force cleanup is one atomic, retryable transaction, and finalize is
likewise retried as a complete idempotent transaction.
- A fresh `__app_heartbeat__` is policy ownership: while it is fresh, the skill
opens no write connection and performs no migration, schema setup, checkpoint,
index build, or `attune`. `__app_last_successful_build__` remains an
observability/freshness marker and is not required for ownership. The skill
checks ownership again after acquiring the hard lease to close the TOCTOU
window. Search/query connections are read-only.
- A dedicated `.obelisk/writer.lock.sqlite` provides the cross-process safety
mutex on every platform. Acquisition is `BEGIN IMMEDIATE` with non-blocking or
bounded waiting; release is idempotent. App builds and heartbeats, skill builds
and attune, app schema/legacy migrations and memory mutations, and manual
rebuild all participate. Manual rebuild's main process owns the lease across
worker build, atomic target replacement, and database reopen; the worker uses
the explicit `caller-held` mode.
- The app's in-process indexer service permits one build at a time. A lease
deferral retains changed paths and schedules a short retry without announcing
a successful build. Service start publishes the ownership heartbeat
immediately, then refreshes it periodically.
- Index-writer and skill read connections use an explicit 250 ms SQLite busy
timeout inside the larger bounded coordination budget. The long-lived app
query connection retains a 5 s timeout; heartbeat is deliberately non-blocking
(`0 ms`) so it never stalls the Electron main thread. Builds use
`BEGIN IMMEDIATE`. Routine checkpointing is `PASSIVE`; blocking `TRUNCATE` is
reserved for explicit maintenance.
*Planned full fix (deferred, two phases).* **Verification.** Fast tests inject auto-rollback and BUSY failures to prove the
primary error is preserved, retry replays the whole transaction, persistent
per-file failure is skipped, force cleanup is atomic, and affected-session state
is commit-aware. The Electron harness uses real Electron-ABI better-sqlite3 and
two child processes: one holds the SQLite writer lease until signalled, while
the other runs synchronous `buildIndex`. It verifies both release-within-budget
success and bounded `writer_busy` deferral. Separate arbitration tests prove a
heartbeat-only daemon marker keeps query and attune paths read-only.
Phase 1 — transaction semantics: **Consequences.** Heartbeat and lease have deliberately different jobs: the
- A shared, binding-agnostic `runWriteTransaction(db, work)` (same injection heartbeat decides who should write, while the lease guarantees writers cannot
model as `persist`): BEGIN → work → COMMIT, guarded rollback on failure, overlap when policy information races or is stale. A single bad transcript can
original exception always preserved. Both app and skill call it, so their still be skipped so the index self-heals on a later build; structural/finalize
behaviour is identical. failures remain visible. Longer timeouts must not replace the transaction and
- Per-file callers catch and continue; finalize does **not** swallow — a finalize ownership rules recorded here.
failure fails the build (the skill currently only warns; that changes).
- In-memory state such as `affectedSessionIds` is updated **only after** a
successful COMMIT (today the app adds the id before COMMIT, so a failed commit
can report a wrong affected set).
- Structured diagnostics: `phase` (begin/file-write/commit/finalize/checkpoint),
SQLite `code`, file path, whether rollback succeeded, whether a txn is still
active; surface the skipped-file count in the build result rather than only
`console.warn` (no silent coverage gaps).
Phase 2 — concurrency:
- A stable concurrency test against real Electron `better-sqlite3` (daemon +
rebuild + skill writer), not an injected fake BUSY. Note this cannot run under
standalone `node --test` (better-sqlite3 is Electron-ABI); it needs an
Electron-hosted harness. The fast injected-BUSY unit test is kept as a
lower-level guard for `runWriteTransaction`, alongside it.
- Serialize all index writes through a single writer: an in-process
`BuildCoordinator`, plus the existing cross-process daemon arbitration
(`__app_heartbeat__` markers) so the skill defers to a live daemon. Single
writer = both layers together.
- `BEGIN IMMEDIATE` to take the write lock up front and avoid
`SQLITE_BUSY_SNAPSHOT` on read-then-write.
- Bounded, short-backoff retry of the **whole** transaction (not the single
failed statement) on `SQLITE_BUSY` / `SQLITE_BUSY_SNAPSHOT`. This relies on the
per-file work being idempotent (upsert/replace + delete-session cascade), which
it is — an invariant the retry depends on.
- Stop forcing `wal_checkpoint(TRUNCATE)` after every build; prefer a `PASSIVE`
checkpoint on idle, with `TRUNCATE` reserved for maintenance/exit.
- Centralized connection configuration (explicit 5000ms for `node:sqlite` — a
real behaviour change; explicit for better-sqlite3 too, acknowledging its
default is already 5000ms).
**Consequences.** With only the stopgap in place, contention no longer crashes a
build, but a conflicting file is *skipped* (non-fatal) and its data is briefly
missing until the next build — acceptable as a stopgap because the guard keeps
the index consistent and self-healing. The true fix is tracked as the two-phase
plan above. A future contributor should not "fix" the concurrency by bumping or
re-adding `busy_timeout`; the direction is a shared transaction module, single
writer, `BEGIN IMMEDIATE`, and whole-transaction retry.
+32 -5
View File
@@ -11,9 +11,10 @@
import { createContext, runInNewContext } from 'node:vm'; import { createContext, runInNewContext } from 'node:vm';
import { DB_PATH, openDb } from './db.mjs'; import { DB_PATH, openDb, openReadDb, openWriterLeaseDb } from './db.mjs';
import { buildIndex } from './indexer.mjs'; import { buildIndex, shouldSkipBuild } from './indexer.mjs';
import { createQueryApi, createAttuneApi } from './query.mjs'; import { createQueryApi, createAttuneApi } from './query.mjs';
import { acquireWriterLease, writerLockPathFor } from './writer-lease.ts';
export { buildIndex, DB_PATH }; export { buildIndex, DB_PATH };
@@ -33,7 +34,7 @@ function runInSandbox(api: SandboxApi, scriptContent: string): Promise<unknown>
// FTS search over indexed message text. Refreshes the index, then queries. // FTS search over indexed message text. Refreshes the index, then queries.
export function searchText(text: string, opts?: Record<string, unknown>): unknown { export function searchText(text: string, opts?: Record<string, unknown>): unknown {
buildIndex(); buildIndex();
const db = openDb(); const db = openReadDb();
try { try {
return createQueryApi(db).search(text, opts); return createQueryApi(db).search(text, opts);
} finally { } finally {
@@ -44,7 +45,7 @@ export function searchText(text: string, opts?: Record<string, unknown>): unknow
// Execute a read-only CodeAct query script and resolve its returned value. // Execute a read-only CodeAct query script and resolve its returned value.
export async function executeQuery(scriptContent: string): Promise<unknown> { export async function executeQuery(scriptContent: string): Promise<unknown> {
buildIndex(); buildIndex();
const db = openDb(); const db = openReadDb();
try { try {
return await runInSandbox(createQueryApi(db), scriptContent); return await runInSandbox(createQueryApi(db), scriptContent);
} finally { } finally {
@@ -54,11 +55,37 @@ export async function executeQuery(scriptContent: string): Promise<unknown> {
// Execute a memory-mutation CodeAct script (remember/forget only). // Execute a memory-mutation CodeAct script (remember/forget only).
export async function executeAttune(scriptContent: string): Promise<unknown> { export async function executeAttune(scriptContent: string): Promise<unknown> {
buildIndex(); const build = buildIndex() as { reason?: string } | undefined;
if (build?.reason === 'daemon_active') {
throw new Error('Obelisk daemon owns index writes; attune is read-only until the daemon stops');
}
if (build?.reason === 'writer_busy' || build?.reason === 'database_busy') {
throw new Error('Obelisk index writer is busy; attune was not applied');
}
const lease = acquireWriterLease({
lockPath: writerLockPathFor(DB_PATH),
openDb: openWriterLeaseDb,
waitMs: 1000,
});
if (!lease) throw new Error('Obelisk index writer is busy; attune was not applied');
try {
// Close the heartbeat TOCTOU window after acquiring the hard lease.
const ownershipDb = openReadDb();
try {
const ownership = shouldSkipBuild(ownershipDb, { ignoreRecentBuild: true });
if (ownership.reason === 'daemon_active') {
throw new Error('Obelisk daemon owns index writes; attune is read-only until the daemon stops');
}
} finally {
ownershipDb.close();
}
const db = openDb(); const db = openDb();
try { try {
return await runInSandbox(createAttuneApi(db), scriptContent); return await runInSandbox(createAttuneApi(db), scriptContent);
} finally { } finally {
db.close(); db.close();
} }
} finally {
lease.release();
}
} }
+15 -6
View File
@@ -1,5 +1,6 @@
import { createRequire } from 'node:module'; import { createRequire } from 'node:module';
import { CLAUDE_DIR, CODEX_DIR, TEXT_LIMIT, trunc, truncJson, extractText, extractContentType, extractMessageIsMeta, filePath, isDir, readLines } from './parsing.mjs'; import { CLAUDE_DIR, CODEX_DIR, TEXT_LIMIT, trunc, truncJson, extractText, extractContentType, extractMessageIsMeta, filePath, isDir, readLines } from './parsing.mjs';
import { configureConnection } from './tx.ts';
const require = createRequire(import.meta.url); const require = createRequire(import.meta.url);
const fs = require('node:fs'); const fs = require('node:fs');
const path = require('node:path'); const path = require('node:path');
@@ -22,17 +23,25 @@ function openDb() {
migrateLegacyDbIfNeeded(); migrateLegacyDbIfNeeded();
fs.mkdirSync(path.dirname(DB_PATH), { recursive: true }); fs.mkdirSync(path.dirname(DB_PATH), { recursive: true });
const db = new DatabaseSync(DB_PATH); const db = new DatabaseSync(DB_PATH);
db.exec('PRAGMA journal_mode=WAL'); configureConnection(db, { busyTimeoutMs: 250 });
db.exec('PRAGMA synchronous=NORMAL');
// Wait for a contended lock instead of erroring with SQLITE_BUSY: a running
// app daemon may be writing the same database while the skill reads/indexes.
db.exec('PRAGMA busy_timeout=5000');
migrateExistingColumns(db); migrateExistingColumns(db);
db.exec(SCHEMA); db.exec(SCHEMA);
migrateDb(db); migrateDb(db);
return db; return db;
} }
// Queries and daemon-arbitration checks must never migrate/configure the index.
// The caller is responsible for ensuring the database exists first.
function openReadDb() {
const db = new DatabaseSync(DB_PATH, { readOnly: true });
db.exec('PRAGMA busy_timeout=250');
return db;
}
function openWriterLeaseDb(lockPath) {
return new DatabaseSync(lockPath);
}
function ensureColumn(db, table, column, definition) { function ensureColumn(db, table, column, definition) {
const columns = db.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name); const columns = db.prepare(`PRAGMA table_info(${table})`).all().map(c => c.name);
if (!columns.includes(column)) db.exec(`ALTER TABLE ${table} ADD COLUMN ${column} ${definition}`); if (!columns.includes(column)) db.exec(`ALTER TABLE ${table} ADD COLUMN ${column} ${definition}`);
@@ -65,4 +74,4 @@ function rebuildMemoryFts(db) {
} }
export { CLAUDE_DIR, CODEX_DIR, OBELISK_DIR, DB_PATH, TEXT_LIMIT, openDb, rebuildMemoryFts, trunc, truncJson, extractText, extractContentType, extractMessageIsMeta, filePath, isDir, readLines, fs, path, os }; export { CLAUDE_DIR, CODEX_DIR, OBELISK_DIR, DB_PATH, TEXT_LIMIT, openDb, openReadDb, openWriterLeaseDb, rebuildMemoryFts, trunc, truncJson, extractText, extractContentType, extractMessageIsMeta, filePath, isDir, readLines, fs, path, os };
+101 -41
View File
@@ -1,23 +1,17 @@
import { openDb, rebuildMemoryFts } from './db.mjs'; import { DB_PATH, openDb, openReadDb, openWriterLeaseDb, rebuildMemoryFts } from './db.mjs';
import { import {
CLAUDE_DIR, CODEX_DIR, PROJECTS_DIR, fs, path, isDir, readLines, CLAUDE_DIR, CODEX_DIR, PROJECTS_DIR, fs, path, isDir, readLines,
inferProjectPath, discoverJsonlFiles, discoverCodexJsonlFiles, codexDbId, readCodexGuardianThreadInfo, inferProjectPath, discoverJsonlFiles, discoverCodexJsonlFiles, codexDbId, readCodexGuardianThreadInfo,
} from './parsing.mjs'; } from './parsing.mjs';
import { persist } from './persist.ts'; import { persist } from './persist.ts';
import { nodeSqliteTransactionAdapter } from './tx.ts';
import { acquireWriterLease, writerLockPathFor } from './writer-lease.ts';
import { runRetryableWriteTransaction, isBeginBusyFailure, hasUnusableTransaction } from './write-coordinator.ts';
import { parse as claudeParse } from './providers/claude.ts'; import { parse as claudeParse } from './providers/claude.ts';
import { parse as codexParse } from './providers/codex.ts'; import { parse as codexParse } from './providers/codex.ts';
const HISTORY_PATH = path.join(CLAUDE_DIR, 'history.jsonl'); const HISTORY_PATH = path.join(CLAUDE_DIR, 'history.jsonl');
// A rollback inside a catch must never throw over the real error. SQLite
// auto-rolls back certain failures (SQLITE_BUSY, disk full, ...); a following
// explicit ROLLBACK then throws "cannot rollback - no transaction is active",
// which would both mask the true cause and turn a skippable per-file error into
// a whole-build failure. Swallow only the rollback's own error.
function safeRollback(db) {
try { db.exec('ROLLBACK'); } catch { /* no active transaction */ }
}
function needsReindex(db, fp) { function needsReindex(db, fp) {
const mt = fs.statSync(fp).mtimeMs; const mt = fs.statSync(fp).mtimeMs;
@@ -31,14 +25,16 @@ function indexCodexSessionIndex(db) {
const indexPath = path.join(CODEX_DIR, 'session_index.jsonl'); const indexPath = path.join(CODEX_DIR, 'session_index.jsonl');
if (!fs.existsSync(indexPath)) return; if (!fs.existsSync(indexPath)) return;
readLines(indexPath, (line) => { readLines(indexPath, (line) => {
let item;
try { try {
const item = JSON.parse(line); item = JSON.parse(line);
} catch (e) {
process.stderr.write(`Warning: malformed Codex session index line: ${e.message}\n`);
return;
}
if (!item.id || !item.thread_name) return; if (!item.id || !item.thread_name) return;
db.prepare('UPDATE sessions SET title=COALESCE(title, ?), ended_at=COALESCE(ended_at, ?) WHERE id=? AND source=?') db.prepare('UPDATE sessions SET title=COALESCE(title, ?), ended_at=COALESCE(ended_at, ?) WHERE id=? AND source=?')
.run(item.thread_name, item.updated_at || null, codexDbId(item.id), 'codex'); .run(item.thread_name, item.updated_at || null, codexDbId(item.id), 'codex');
} catch (e) {
process.stderr.write(`Warning: malformed Codex session index line: ${e.message}\n`);
}
}); });
} }
@@ -62,8 +58,13 @@ function indexSubagentMeta(db, fi) {
if (!fi.isSubagent) return; if (!fi.isSubagent) return;
const mp = fi.path.replace('.jsonl', '.meta.json'); const mp = fi.path.replace('.jsonl', '.meta.json');
if (!fs.existsSync(mp)) return; if (!fs.existsSync(mp)) return;
let meta;
try { try {
const meta = JSON.parse(fs.readFileSync(mp, 'utf8')); meta = JSON.parse(fs.readFileSync(mp, 'utf8'));
} catch (e) {
process.stderr.write(`Warning: failed to read subagent meta ${mp}: ${e.message}\n`);
return;
}
const tok = db.prepare('SELECT COALESCE(SUM(input_tokens),0)+COALESCE(SUM(output_tokens),0) as t FROM messages WHERE agent_id=?').get(fi.agentId); const tok = db.prepare('SELECT COALESCE(SUM(input_tokens),0)+COALESCE(SUM(output_tokens),0) as t FROM messages WHERE agent_id=?').get(fi.agentId);
const ts = db.prepare('SELECT MIN(timestamp) as t0, MAX(timestamp) as t1 FROM messages WHERE agent_id=?').get(fi.agentId); const ts = db.prepare('SELECT MIN(timestamp) as t0, MAX(timestamp) as t1 FROM messages WHERE agent_id=?').get(fi.agentId);
const dur = ts?.t0 && ts?.t1 ? new Date(ts.t1).getTime() - new Date(ts.t0).getTime() : null; const dur = ts?.t0 && ts?.t1 ? new Date(ts.t1).getTime() - new Date(ts.t0).getTime() : null;
@@ -72,7 +73,6 @@ function indexSubagentMeta(db, fi) {
} else { } else {
db.prepare('INSERT OR REPLACE INTO subagents VALUES(?,?,?,?,?,?,?)').run(fi.agentId, fi.sessionId, meta.toolUseId||null, meta.agentType||null, meta.description||null, dur, tok?.t||0); db.prepare('INSERT OR REPLACE INTO subagents VALUES(?,?,?,?,?,?,?)').run(fi.agentId, fi.sessionId, meta.toolUseId||null, meta.agentType||null, meta.description||null, dur, tok?.t||0);
} }
} catch (e) { process.stderr.write(`Warning: failed to read subagent meta ${mp}: ${e.message}\n`); }
} }
function indexWorkflows(db) { function indexWorkflows(db) {
@@ -91,8 +91,13 @@ function indexWorkflows(db) {
try { wfFiles = fs.readdirSync(wd); } catch { continue; } try { wfFiles = fs.readdirSync(wd); } catch { continue; }
for (const f of wfFiles) { for (const f of wfFiles) {
if (!f.endsWith('.json')) continue; if (!f.endsWith('.json')) continue;
let wf;
try { try {
const wf = JSON.parse(fs.readFileSync(path.join(wd, f), 'utf8')); wf = JSON.parse(fs.readFileSync(path.join(wd, f), 'utf8'));
} catch (e) {
process.stderr.write(`Warning: failed to read workflow ${f}: ${e.message}\n`);
continue;
}
if (!wf.runId) continue; if (!wf.runId) continue;
const ac = db.prepare('SELECT COUNT(*) as c FROM workflow_agents WHERE run_id=?').get(wf.runId); const ac = db.prepare('SELECT COUNT(*) as c FROM workflow_agents WHERE run_id=?').get(wf.runId);
db.prepare('INSERT OR REPLACE INTO workflows (run_id,session_id,task_id,script,result_json,timestamp,agent_count,duration_ms,total_tokens,status,workflow_name) VALUES(?,?,?,?,?,?,?,?,?,?,?)').run( db.prepare('INSERT OR REPLACE INTO workflows (run_id,session_id,task_id,script,result_json,timestamp,agent_count,duration_ms,total_tokens,status,workflow_name) VALUES(?,?,?,?,?,?,?,?,?,?,?)').run(
@@ -106,7 +111,6 @@ function indexWorkflows(db) {
item.phaseTitle||null, item.label||null, item.model||null, item.state||null, item.phaseTitle||null, item.label||null, item.model||null, item.state||null,
item.durationMs||null, item.tokens||null, item.toolCalls||null, 'agent-' + item.agentId); item.durationMs||null, item.tokens||null, item.toolCalls||null, 'agent-' + item.agentId);
} }
} catch (e) { process.stderr.write(`Warning: failed to index workflow ${f}: ${e.message}\n`); }
} }
} }
} }
@@ -115,32 +119,54 @@ function indexWorkflows(db) {
function indexHistory(db) { function indexHistory(db) {
if (!fs.existsSync(HISTORY_PATH)) return; if (!fs.existsSync(HISTORY_PATH)) return;
readLines(HISTORY_PATH, (line) => { readLines(HISTORY_PATH, (line) => {
let item;
try { try {
const o = JSON.parse(line); item = JSON.parse(line);
if (o.sessionId && o.title) db.prepare('UPDATE sessions SET title=? WHERE id=? AND title IS NULL').run(o.title, o.sessionId); } catch (e) {
} catch (e) { process.stderr.write(`Warning: malformed history line: ${e.message}\n`); } process.stderr.write(`Warning: malformed history line: ${e.message}\n`);
return;
}
if (item.sessionId && item.title) db.prepare('UPDATE sessions SET title=? WHERE id=? AND title IS NULL').run(item.title, item.sessionId);
}); });
} }
const BUILD_DEBOUNCE_MS = 30000; const BUILD_DEBOUNCE_MS = 30000;
const APP_HEARTBEAT_FRESH_MS = 60000; const APP_HEARTBEAT_FRESH_MS = 60000;
function shouldSkipBuild(db, { now = Date.now() } = {}) { function shouldSkipBuild(db, { now = Date.now(), ignoreRecentBuild = false } = {}) {
const appHeartbeat = db.prepare("SELECT mtime FROM index_state WHERE jsonl_path='__app_heartbeat__'").get(); const appHeartbeat = db.prepare("SELECT mtime FROM index_state WHERE jsonl_path='__app_heartbeat__'").get();
const appSuccessfulBuild = db.prepare("SELECT mtime FROM index_state WHERE jsonl_path='__app_last_successful_build__'").get(); if (appHeartbeat && now - appHeartbeat.mtime < APP_HEARTBEAT_FRESH_MS) {
if ( return { skip: true, reason: 'daemon_active' };
appHeartbeat && now - appHeartbeat.mtime < APP_HEARTBEAT_FRESH_MS &&
appSuccessfulBuild && now - appSuccessfulBuild.mtime < APP_HEARTBEAT_FRESH_MS
) {
return { skip: true, reason: 'app_successful_build' };
} }
if (!ignoreRecentBuild) {
const last = db.prepare("SELECT mtime FROM index_state WHERE jsonl_path='__last_build__'").get(); const last = db.prepare("SELECT mtime FROM index_state WHERE jsonl_path='__last_build__'").get();
if (last && now - last.mtime < BUILD_DEBOUNCE_MS) { if (last && now - last.mtime < BUILD_DEBOUNCE_MS) {
return { skip: true, reason: 'recent_build' }; return { skip: true, reason: 'recent_build' };
} }
}
return { skip: false }; return { skip: false };
} }
function isMissingIndexStateTable(error) {
const message = error instanceof Error ? error.message : String(error);
return /no such table:\s*(?:main\.)?index_state\b/i.test(message);
}
function inspectBuildOwnership({ force = false } = {}) {
if (!fs.existsSync(DB_PATH)) return { skip: false };
const db = openReadDb();
try {
return shouldSkipBuild(db, { ignoreRecentBuild: force });
} catch (error) {
// A missing table means the write path must initialize a new/legacy index.
// Any other read failure leaves daemon ownership unknown, so fail closed.
if (isMissingIndexStateTable(error)) return { skip: false };
throw error;
} finally {
db.close();
}
}
// A one-shot record stream that retracts a session, for routing guardian sweeps // A one-shot record stream that retracts a session, for routing guardian sweeps
// through persist (the single db writer) instead of deleting rows directly. // through persist (the single db writer) instead of deleting rows directly.
function* guardianDelete(sessionId) { function* guardianDelete(sessionId) {
@@ -149,13 +175,25 @@ function* guardianDelete(sessionId) {
} }
function buildIndex({ force = false } = {}) { function buildIndex({ force = false } = {}) {
const db = openDb(); const ownership = inspectBuildOwnership({ force });
if (!force) { if (ownership.skip) return ownership;
const skip = shouldSkipBuild(db); const lease = acquireWriterLease({
if (skip.skip) { db.close(); return; } lockPath: writerLockPathFor(DB_PATH),
} openDb: openWriterLeaseDb,
});
if (!lease) return { skip: true, reason: 'writer_busy' };
try {
// Ownership may change between the first read and lease acquisition.
const ownershipAfterLease = inspectBuildOwnership({ force });
if (ownershipAfterLease.skip) return ownershipAfterLease;
const db = openDb();
const txDb = nodeSqliteTransactionAdapter(db);
const skippedFiles = [];
try {
try {
if (force) { if (force) {
runRetryableWriteTransaction(txDb, () => {
db.prepare("DELETE FROM index_state WHERE jsonl_path != '__last_build__'").run(); db.prepare("DELETE FROM index_state WHERE jsonl_path != '__last_build__'").run();
// Clearing index_state alone re-indexes existing files but leaves rows for // Clearing index_state alone re-indexes existing files but leaves rows for
// files that no longer exist on disk (stale sessions accumulate). A force // files that no longer exist on disk (stale sessions accumulate). A force
@@ -165,6 +203,13 @@ function buildIndex({ force = false } = {}) {
for (const table of ['messages', 'tool_calls', 'tool_results', 'sessions', 'summaries', 'subagents', 'workflows', 'workflow_agents']) { for (const table of ['messages', 'tool_calls', 'tool_results', 'sessions', 'summaries', 'subagents', 'workflows', 'workflow_agents']) {
db.prepare(`DELETE FROM ${table}`).run(); db.prepare(`DELETE FROM ${table}`).run();
} }
}, { label: 'force-cleanup' });
}
} catch (error) {
if (isBeginBusyFailure(error)) {
return { skip: true, reason: 'database_busy', skipped: skippedFiles.length, skippedFiles };
}
throw error;
} }
const files = [ const files = [
@@ -172,8 +217,8 @@ function buildIndex({ force = false } = {}) {
...discoverCodexJsonlFiles(), ...discoverCodexJsonlFiles(),
]; ];
for (const f of files) { for (const f of files) {
db.exec('BEGIN');
try { try {
runRetryableWriteTransaction(txDb, () => {
if (f.source === 'codex') { if (f.source === 'codex') {
// Codex goes through the pure adapter + shared persist (docs/adr/0001), // Codex goes through the pure adapter + shared persist (docs/adr/0001),
// full-reparse (countMode 'total') when the file changed. An unchanged // full-reparse (countMode 'total') when the file changed. An unchanged
@@ -200,14 +245,21 @@ function buildIndex({ force = false } = {}) {
} }
indexSubagentMeta(db, f); indexSubagentMeta(db, f);
} }
db.exec('COMMIT'); }, { label: `file:${f.path}` });
} catch (e) { } catch (e) {
safeRollback(db); if (isBeginBusyFailure(e)) {
return { skip: true, reason: 'database_busy', skipped: skippedFiles.length, skippedFiles };
}
if (hasUnusableTransaction(e)) throw e;
// A per-file failure is skippable: log and move on.
skippedFiles.push({ path: f.path, error: e.message, diagnostics: e.obelisk });
process.stderr.write(`Warning: failed to index ${f.path}: ${e.message}\n`); process.stderr.write(`Warning: failed to index ${f.path}: ${e.message}\n`);
} }
} }
db.exec('BEGIN'); // Finalize is one transaction and is NOT swallowed: a finalize failure fails
// the build (a half-finalized index would be inconsistent).
try { try {
runRetryableWriteTransaction(txDb, () => {
indexWorkflows(db); indexWorkflows(db);
refreshSessionProjectPaths(db); refreshSessionProjectPaths(db);
indexHistory(db); indexHistory(db);
@@ -215,12 +267,20 @@ function buildIndex({ force = false } = {}) {
db.exec("INSERT INTO messages_fts(messages_fts) VALUES('rebuild')"); db.exec("INSERT INTO messages_fts(messages_fts) VALUES('rebuild')");
rebuildMemoryFts(db); rebuildMemoryFts(db);
db.prepare("INSERT OR REPLACE INTO index_state (jsonl_path, mtime, lines_processed) VALUES ('__last_build__', ?, 0)").run(Date.now()); db.prepare("INSERT OR REPLACE INTO index_state (jsonl_path, mtime, lines_processed) VALUES ('__last_build__', ?, 0)").run(Date.now());
db.exec('COMMIT'); }, { label: 'finalize' });
} catch (e) { } catch (error) {
safeRollback(db); if (isBeginBusyFailure(error)) {
process.stderr.write(`Warning: failed to finalize index: ${e.message}\n`); return { skip: true, reason: 'database_busy', skipped: skippedFiles.length, skippedFiles };
} }
throw error;
}
return { skip: false, skipped: skippedFiles.length, skippedFiles };
} finally {
db.close(); db.close();
}
} finally {
lease.release();
}
} }
export { buildIndex, inferProjectPath, refreshSessionProjectPaths, shouldSkipBuild }; export { buildIndex, inferProjectPath, refreshSessionProjectPaths, shouldSkipBuild };
+138
View File
@@ -0,0 +1,138 @@
// Binding-agnostic SQLite write plumbing shared by the skill and app indexers
// (docs/adr/0006). The injected db must expose `exec(sql)`; this works for both
// node:sqlite (skill/CLI) and better-sqlite3 (app), same injection model as
// `persist`.
export interface WriteTxDb {
exec(sql: string): unknown;
inTransaction(): boolean;
}
export interface SqliteConnection {
exec(sql: string): unknown;
}
type Phase = 'begin' | 'work' | 'commit' | 'rollback';
export interface WriteTxDiagnostics {
phase: Phase;
code: string | null;
label?: string;
rollbackSucceeded: boolean | null;
rollbackError: string | null;
transactionActive: boolean | null;
attempts: number;
}
export interface WriteTxOptions {
// Diagnostic label for this transaction (e.g. a file path or 'finalize').
label?: string;
}
const BUSY_MESSAGE = /SQLITE_BUSY|database is locked|database is busy/i;
function busyCode(error: unknown): string | null {
const raw = error as { code?: unknown; errcode?: unknown; message?: unknown } | null;
const code = (raw?.code ?? raw?.errcode);
if (typeof code === 'string' && code.startsWith('SQLITE_BUSY')) return code;
if (typeof raw?.message === 'string' && BUSY_MESSAGE.test(raw.message)) return 'SQLITE_BUSY';
return null;
}
function errorCode(error: unknown): string | null {
const raw = error as { code?: unknown } | null;
return typeof raw?.code === 'string' ? raw.code : null;
}
interface BetterSqliteHandle {
exec(sql: string): unknown;
readonly inTransaction: boolean;
}
interface NodeSqliteHandle {
exec(sql: string): unknown;
readonly isTransaction: boolean;
}
export function betterSqliteTransactionAdapter(db: BetterSqliteHandle): WriteTxDb {
return {
exec: sql => db.exec(sql),
inTransaction: () => db.inTransaction,
};
}
export function nodeSqliteTransactionAdapter(db: NodeSqliteHandle): WriteTxDb {
return {
exec: sql => db.exec(sql),
inTransaction: () => db.isTransaction,
};
}
function transactionState(db: WriteTxDb): boolean | null {
try {
return db.inTransaction();
} catch {
return null;
}
}
function attachDiagnostics(error: unknown, diagnostics: WriteTxDiagnostics): void {
if (!error || typeof error !== 'object') return;
try {
(error as { obelisk?: WriteTxDiagnostics }).obelisk = diagnostics;
} catch {
// Frozen/native errors must still be rethrown unchanged.
}
}
// Runs `work` exactly once inside a transaction and returns its value. Retry and
// scheduling policy belongs to the build coordinator, which knows the operation's
// idempotency and total time budget. Cleanup never masks the primary exception.
export function runWriteTransaction<T>(db: WriteTxDb, work: () => T, options: WriteTxOptions = {}): T {
const { label } = options;
let phase: Phase = 'begin';
try {
db.exec('BEGIN IMMEDIATE');
phase = 'work';
const value = work();
phase = 'commit';
db.exec('COMMIT');
return value;
} catch (error) {
let rollbackSucceeded: boolean | null = null;
let rollbackError: string | null = null;
const activeBeforeRollback = transactionState(db);
if (activeBeforeRollback !== false) {
try {
db.exec('ROLLBACK');
rollbackSucceeded = true;
} catch (rollbackFailure) {
rollbackSucceeded = false;
rollbackError = rollbackFailure instanceof Error ? rollbackFailure.message : String(rollbackFailure);
}
}
const busy = busyCode(error);
const diagnostics: WriteTxDiagnostics = {
phase,
code: busy ?? errorCode(error),
label,
rollbackSucceeded,
rollbackError,
transactionActive: transactionState(db),
attempts: 1,
};
attachDiagnostics(error, diagnostics);
throw error;
}
}
// Applies the connection-level pragmas used by every Obelisk writer/reader. Uses
// exec (not better-sqlite3's .pragma) so one implementation covers both bindings.
// busy_timeout is a real behavior change for node:sqlite (no default); it is set
// explicitly for better-sqlite3 too, whose own default already happens to be
// 5000ms. It is NOT the concurrency fix — see docs/adr/0006.
export function configureConnection(db: SqliteConnection, { busyTimeoutMs = 5000 } = {}): void {
db.exec(`PRAGMA busy_timeout=${busyTimeoutMs}`);
db.exec('PRAGMA journal_mode=WAL');
db.exec('PRAGMA synchronous=NORMAL');
}
+95
View File
@@ -0,0 +1,95 @@
// Bounded retry policy above the transaction primitive. Callers opt in only for
// idempotent work; BEGIN contention and an uncertain/live transaction are never
// retried here.
import { runWriteTransaction, type WriteTxDb, type WriteTxOptions } from './tx.ts';
interface TransactionDiagnostics {
phase?: string;
code?: string | null;
transactionActive?: boolean | null;
attempts?: number;
}
export interface WriteRetryOptions {
maxAttempts?: number;
budgetMs?: number;
retryDelayMs?: number;
now?: () => number;
sleep?: (ms: number) => void;
}
function diagnostics(error: unknown): TransactionDiagnostics | null {
if (!error || typeof error !== 'object') return null;
return (error as { obelisk?: TransactionDiagnostics }).obelisk ?? null;
}
function isBusyCode(code: unknown): boolean {
return typeof code === 'string' && code.startsWith('SQLITE_BUSY');
}
function syncSleep(ms: number): void {
if (ms <= 0) return;
try {
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
} catch {
// Bounded attempts still prevent an infinite retry loop.
}
}
export function isBeginBusyFailure(error: unknown): boolean {
const info = diagnostics(error);
return (
info?.phase === 'begin' &&
isBusyCode(info.code) &&
info.transactionActive === false
);
}
export function hasUnusableTransaction(error: unknown): boolean {
const info = diagnostics(error);
return Boolean(info && info.transactionActive !== false);
}
export function isRetryableWriteFailure(error: unknown): boolean {
const info = diagnostics(error);
return (
(info?.phase === 'work' || info?.phase === 'commit') &&
isBusyCode(info.code) &&
info.transactionActive === false
);
}
export function runWithWriteRetry<T>(operation: () => T, {
maxAttempts = 3,
budgetMs = 1000,
retryDelayMs = 25,
now = Date.now,
sleep = syncSleep,
}: WriteRetryOptions = {}): T {
const startedAt = now();
for (let attempt = 1; ; attempt += 1) {
try {
return operation();
} catch (error) {
const info = diagnostics(error);
if (info) info.attempts = attempt;
if (!isRetryableWriteFailure(error) || attempt >= maxAttempts) throw error;
const remaining = budgetMs - (now() - startedAt);
if (remaining <= 0) throw error;
sleep(Math.min(retryDelayMs * attempt, remaining));
}
}
}
export function runRetryableWriteTransaction<T>(
db: WriteTxDb,
work: () => T,
transactionOptions: WriteTxOptions = {},
retryOptions: WriteRetryOptions = {},
): T {
return runWithWriteRetry(
() => runWriteTransaction(db, work, transactionOptions),
retryOptions,
);
}
+91
View File
@@ -0,0 +1,91 @@
// Cross-process single-writer lease for a complete Obelisk index build. The
// lock lives in a dedicated SQLite database so node:sqlite and better-sqlite3
// share identical locking semantics on every supported platform.
import { mkdirSync } from 'node:fs';
import { dirname, join } from 'node:path';
export interface WriterLeaseDb {
exec(sql: string): unknown;
close(): void;
}
export interface WriterLease {
release(): void;
}
export interface AcquireWriterLeaseOptions {
lockPath: string;
openDb: (path: string) => WriterLeaseDb;
waitMs?: number;
retryDelayMs?: number;
now?: () => number;
sleep?: (ms: number) => void;
}
const BUSY_MESSAGE = /SQLITE_BUSY|database is locked|database is busy/i;
function isBusy(error: unknown): boolean {
const raw = error as { code?: unknown; errcode?: unknown; message?: unknown } | null;
const code = raw?.code ?? raw?.errcode;
return (
(typeof code === 'string' && code.startsWith('SQLITE_BUSY')) ||
(typeof raw?.message === 'string' && BUSY_MESSAGE.test(raw.message))
);
}
function syncSleep(ms: number): void {
if (ms <= 0) return;
try {
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
} catch {
// If synchronous sleeping is unavailable, the bounded attempt count below
// still prevents an infinite acquisition loop.
}
}
export function writerLockPathFor(dbPath: string): string {
return join(dirname(dbPath), 'writer.lock.sqlite');
}
export function acquireWriterLease({
lockPath,
openDb,
waitMs = 0,
retryDelayMs = 25,
now = Date.now,
sleep = syncSleep,
}: AcquireWriterLeaseOptions): WriterLease | null {
mkdirSync(dirname(lockPath), { recursive: true });
const startedAt = now();
const maxAttempts = waitMs > 0 ? Math.ceil(waitMs / Math.max(1, retryDelayMs)) + 1 : 1;
for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
const db = openDb(lockPath);
try {
db.exec('PRAGMA busy_timeout=0');
db.exec('BEGIN IMMEDIATE');
let released = false;
return {
release() {
if (released) return;
released = true;
try {
db.exec('ROLLBACK');
} catch {
// Closing the connection releases any remaining SQLite lock.
} finally {
db.close();
}
},
};
} catch (error) {
db.close();
if (!isBusy(error)) throw error;
const remaining = waitMs - (now() - startedAt);
if (remaining <= 0 || attempt + 1 >= maxAttempts) return null;
sleep(Math.min(retryDelayMs, remaining));
}
}
return null;
}
+44
View File
@@ -73,6 +73,34 @@ test('indexer service runs one pending build after an in-flight build finishes',
assert.deepEqual(calls, ['first', 'pending']); assert.deepEqual(calls, ['first', 'pending']);
}); });
test('indexer service reschedules a writer-lease deferral without publishing a heartbeat', async () => {
const timers = manualTimers();
const calls = [];
let heartbeats = 0;
const service = createIndexerService({
buildIndex: async ({ reason, changedPaths }) => {
calls.push({ reason, changedPaths });
return calls.length === 1 ? { deferred: true, reason: 'writer_busy' } : { deferred: false };
},
watchProjects: () => null,
writeHeartbeat: () => { heartbeats += 1; },
timers,
stabilityMs: 0,
});
await service.runBuildNow('watch', ['project/session.jsonl']);
assert.equal(heartbeats, 0);
assert.equal(calls.length, 1);
timers.flush();
await service.idle();
assert.deepEqual(calls, [
{ reason: 'watch', changedPaths: ['project/session.jsonl'] },
{ reason: 'writer-lease', changedPaths: ['project/session.jsonl'] },
]);
assert.equal(heartbeats, 1);
});
test('indexer service does not log a build cancelled by a service stop', async () => { test('indexer service does not log a build cancelled by a service stop', async () => {
const timers = manualTimers(); const timers = manualTimers();
const warnings = []; const warnings = [];
@@ -160,6 +188,22 @@ test('indexer service retries watcher setup when the projects directory is missi
assert.equal(attempts, 2); assert.equal(attempts, 2);
}); });
test('indexer service publishes daemon ownership as soon as it starts', () => {
const timers = manualTimers();
let heartbeats = 0;
const service = createIndexerService({
buildIndex: async () => ({ deferred: false }),
watchProjects: () => null,
writeHeartbeat: () => { heartbeats += 1; },
timers,
stabilityMs: 0,
});
service.start({ buildOnStart: false });
assert.equal(heartbeats, 1);
service.stop();
});
test('indexer service watches Claude JSON files through chokidar', async () => { test('indexer service watches Claude JSON files through chokidar', async () => {
const projectsDir = mkdtempSync(join(tmpdir(), 'obelisk-chokidar-projects-')); const projectsDir = mkdtempSync(join(tmpdir(), 'obelisk-chokidar-projects-'));
const timers = manualTimers(); const timers = manualTimers();
+2 -2
View File
@@ -31,7 +31,7 @@ class TestDatabase {
} }
} }
test('app indexer builds the Obelisk database from Claude JSONL and records app heartbeat', () => { test('app indexer records build success without claiming daemon ownership', () => {
const home = mkdtempSync(join(tmpdir(), 'obelisk-app-indexer-')); const home = mkdtempSync(join(tmpdir(), 'obelisk-app-indexer-'));
const claudeDir = join(home, '.claude'); const claudeDir = join(home, '.claude');
const projectDir = join(claudeDir, 'projects', '-tmp-obelisk-app'); const projectDir = join(claudeDir, 'projects', '-tmp-obelisk-app');
@@ -57,7 +57,7 @@ test('app indexer builds the Obelisk database from Claude JSONL and records app
assert.deepEqual(firstBuild.affectedSessionIds, [sessionId]); assert.deepEqual(firstBuild.affectedSessionIds, [sessionId]);
assert.equal(firstBuild.ftsRebuilt, true); assert.equal(firstBuild.ftsRebuilt, true);
assert.equal(db.prepare("SELECT uuid FROM messages_fts WHERE messages_fts MATCH 'hello'").get().uuid, 'msg-app-1'); assert.equal(db.prepare("SELECT uuid FROM messages_fts WHERE messages_fts MATCH 'hello'").get().uuid, 'msg-app-1');
assert.equal(db.prepare("SELECT jsonl_path FROM index_state WHERE jsonl_path='__app_heartbeat__'").get().jsonl_path, '__app_heartbeat__'); assert.equal(db.prepare("SELECT jsonl_path FROM index_state WHERE jsonl_path='__app_heartbeat__'").get(), undefined);
assert.equal(db.prepare("SELECT jsonl_path FROM index_state WHERE jsonl_path='__app_last_successful_build__'").get().jsonl_path, '__app_last_successful_build__'); assert.equal(db.prepare("SELECT jsonl_path FROM index_state WHERE jsonl_path='__app_last_successful_build__'").get().jsonl_path, '__app_last_successful_build__');
assert.equal(db.prepare('SELECT project_path FROM sessions WHERE id=?').get(sessionId).project_path, '/tmp/obelisk-app'); assert.equal(db.prepare('SELECT project_path FROM sessions WHERE id=?').get(sessionId).project_path, '/tmp/obelisk-app');
db.close(); db.close();
+190 -29
View File
@@ -7,7 +7,27 @@ import { mkdirSync, rmSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os'; import { tmpdir } from 'node:os';
import { join } from 'node:path'; import { join } from 'node:path';
import { acquireWriterLease } from '../scripts/writer-lease.ts';
const require = createRequire(import.meta.url); const require = createRequire(import.meta.url);
const { DatabaseSync } = require('node:sqlite');
class SqliteCompatDatabase {
constructor(dbFile) {
this.db = new DatabaseSync(dbFile);
}
pragma(statement) { this.db.exec(`PRAGMA ${statement}`); }
exec(sql) { return this.db.exec(sql); }
close() { return this.db.close(); }
prepare(sql) {
const stmt = this.db.prepare(sql);
return {
all: (...params) => stmt.all(...params),
get: (...params) => stmt.get(...params),
run: (...params) => stmt.run(...params),
};
}
}
// The app main process is now an ES module. It runs side-effectfully on import // The app main process is now an ES module. It runs side-effectfully on import
// (registers ipcMain handlers, opens windows, etc.) and has no exports, so we // (registers ipcMain handlers, opens windows, etc.) and has no exports, so we
@@ -110,6 +130,7 @@ async function loadMainForWindowFlags(flags) {
class FakeDatabase { class FakeDatabase {
pragma() {} pragma() {}
exec() {}
close() {} close() {}
prepare() { prepare() {
return { get: () => null, all: () => [], run: () => ({}) }; return { get: () => null, all: () => [], run: () => ({}) };
@@ -191,6 +212,7 @@ test('main process watches Codex sessions directory instead of Codex root', asyn
class FakeDatabase { class FakeDatabase {
pragma() {} pragma() {}
exec() {}
close() {} close() {}
prepare() { prepare() {
return { get: () => null, all: () => [], run: () => ({}) }; return { get: () => null, all: () => [], run: () => ({}) };
@@ -245,6 +267,77 @@ test('main process watches Codex sessions directory instead of Codex root', asyn
} }
}); });
test('main process forwards committed IDs without reopening after a deferred build', async () => {
const originalHome = process.env.HOME;
const home = join(tmpdir(), `obelisk-main-deferred-build-${Date.now()}`);
mkdirSync(join(home, '.claude', 'projects'), { recursive: true });
mkdirSync(join(home, '.codex', 'sessions'), { recursive: true });
mkdirSync(join(home, '.obelisk'), { recursive: true });
writeFileSync(join(home, '.obelisk', 'obelisk.sqlite'), '');
process.env.HOME = home;
let databaseOpens = 0;
let serviceOptions;
let notifications = 0;
class FakeDatabase {
constructor() { databaseOpens += 1; }
pragma() {}
exec() {}
close() {}
prepare() { return { get: () => null, all: () => [], run: () => ({}) }; }
}
class FakeBrowserWindow {
constructor() {
this.webContents = { on() {}, setZoomLevel() {}, openDevTools() {}, send() { notifications += 1; } };
}
loadFile() {}
loadURL() {}
close() {}
static getAllWindows() { return [{ webContents: { send() { notifications += 1; } } }]; }
static fromWebContents() { return null; }
}
const restore = registerMocks([
[ELECTRON_URL, { namedExports: electronNamespace({ BrowserWindow: FakeBrowserWindow }) }],
[DATABASE_URL, { defaultExport: FakeDatabase }],
[CHOKIDAR_URL, { defaultExport: noopChokidar() }],
[INDEXER_URL, { namedExports: { writeHeartbeat() {} } }],
[INDEXER_SERVICE_URL, {
namedExports: {
createIndexerService: (options) => {
serviceOptions = options;
return { start() {}, stop() {}, idle: async () => {}, runBuildNow() { return Promise.resolve(); } };
},
},
}],
[INDEXER_WORKER_URL, {
namedExports: {
createWorkerBuildIndex: () => ({
buildIndex: async () => ({ deferred: true, reason: 'database_busy', affectedSessionIds: ['session-1'] }),
stop() {},
}),
},
}],
]);
try {
await importMain();
const opensBeforeBuild = databaseOpens;
const notificationsBeforeBuild = notifications;
const result = await serviceOptions.buildIndex({ reason: 'writer-lease' });
assert.equal(result.deferred, true);
assert.equal(databaseOpens, opensBeforeBuild);
assert.equal(notifications, notificationsBeforeBuild + 2);
} finally {
restore();
process.env.HOME = originalHome;
rmSync(home, { recursive: true, force: true });
}
});
test('session IPC hides Codex rows by default and supports explicit source opt-in', async () => { test('session IPC hides Codex rows by default and supports explicit source opt-in', async () => {
const originalHome = process.env.HOME; const originalHome = process.env.HOME;
const home = join(tmpdir(), `obelisk-main-source-filter-${Date.now()}`); const home = join(tmpdir(), `obelisk-main-source-filter-${Date.now()}`);
@@ -257,6 +350,7 @@ test('session IPC hides Codex rows by default and supports explicit source opt-i
class FakeDatabase { class FakeDatabase {
pragma() {} pragma() {}
exec() {}
close() {} close() {}
prepare(sql) { prepare(sql) {
return { return {
@@ -369,31 +463,6 @@ test('main process migrates an existing app database before source-filtered IPC
const ipcHandlers = new Map(); const ipcHandlers = new Map();
// better-sqlite3-compatible adapter over node:sqlite so the real migration
// logic (ALTER TABLE ADD COLUMN source, etc.) runs against a real database.
class SqliteCompatDatabase {
constructor(dbFile) {
this.db = new DatabaseSync(dbFile);
}
pragma(statement) {
this.db.exec(`PRAGMA ${statement}`);
}
exec(sql) {
return this.db.exec(sql);
}
close() {
return this.db.close();
}
prepare(sql) {
const stmt = this.db.prepare(sql);
return {
all: (...params) => stmt.all(...params),
get: (...params) => stmt.get(...params),
run: (...params) => stmt.run(...params),
};
}
}
class FakeBrowserWindow { class FakeBrowserWindow {
constructor() { constructor() {
this.webContents = { on() {}, setZoomLevel() {}, openDevTools() {}, send() {} }; this.webContents = { on() {}, setZoomLevel() {}, openDevTools() {}, send() {} };
@@ -441,6 +510,70 @@ test('main process migrates an existing app database before source-filtered IPC
} }
}); });
test('main process keeps schema and memory mutations behind the writer lease', async () => {
const originalHome = process.env.HOME;
const home = join(tmpdir(), `obelisk-main-migration-lease-${Date.now()}`);
const obeliskDir = join(home, '.obelisk');
const dbPath = join(obeliskDir, 'obelisk.sqlite');
mkdirSync(obeliskDir, { recursive: true });
process.env.HOME = home;
const legacy = new DatabaseSync(dbPath);
legacy.exec('CREATE TABLE sessions (id TEXT PRIMARY KEY)');
legacy.close();
const holder = acquireWriterLease({
lockPath: join(obeliskDir, 'writer.lock.sqlite'),
openDb: lockPath => new DatabaseSync(lockPath),
});
assert.ok(holder);
const ipcHandlers = new Map();
class FakeBrowserWindow {
constructor() {
this.webContents = { on() {}, setZoomLevel() {}, openDevTools() {}, send() {} };
}
loadFile() {}
loadURL() {}
close() {}
static getAllWindows() { return []; }
static fromWebContents() { return null; }
}
const restore = registerMocks([
[ELECTRON_URL, {
namedExports: electronNamespace({
BrowserWindow: FakeBrowserWindow,
ipcMain: {
handle(channel, handler) { ipcHandlers.set(channel, handler); },
},
}),
}],
[DATABASE_URL, { defaultExport: SqliteCompatDatabase }],
[CHOKIDAR_URL, { defaultExport: noopChokidar() }],
[INDEXER_URL, { namedExports: { writeHeartbeat() {} } }],
[INDEXER_SERVICE_URL, { namedExports: defaultIndexerService() }],
[INDEXER_WORKER_URL, { namedExports: defaultIndexerWorkerClient() }],
]);
try {
await importMain();
const check = new DatabaseSync(dbPath, { readOnly: true });
const columns = check.prepare('PRAGMA table_info(sessions)').all().map(column => column.name);
check.close();
assert.deepEqual(columns, ['id']);
assert.throws(
() => ipcHandlers.get('db:archiveMemory')(null, 'memory-1', 'test'),
/writer is busy/i,
);
} finally {
restore();
holder.release();
process.env.HOME = originalHome;
rmSync(home, { recursive: true, force: true });
}
});
test('closing the last macOS window releases background resources until activation', async () => { test('closing the last macOS window releases background resources until activation', async () => {
const originalPlatform = Object.getOwnPropertyDescriptor(process, 'platform'); const originalPlatform = Object.getOwnPropertyDescriptor(process, 'platform');
const originalHome = process.env.HOME; const originalHome = process.env.HOME;
@@ -459,6 +592,7 @@ test('closing the last macOS window releases background resources until activati
class FakeDatabase { class FakeDatabase {
pragma() {} pragma() {}
exec() {}
close() { serviceEvents.push('db-close'); } close() { serviceEvents.push('db-close'); }
prepare() { prepare() {
return { get: () => null, all: () => [], run: () => ({}) }; return { get: () => null, all: () => [], run: () => ({}) };
@@ -575,13 +709,16 @@ test('settings rebuild reopens the database from the configured Claude path', as
const openedDbPaths = []; const openedDbPaths = [];
const buildCalls = []; const buildCalls = [];
const serviceEvents = []; const serviceEvents = [];
let competingLeaseDuringBuild;
class FakeDatabase { class FakeDatabase {
constructor(dbPath) { constructor(dbPath) {
this.lockDb = dbPath.endsWith('writer.lock.sqlite') ? new DatabaseSync(dbPath) : null;
openedDbPaths.push(dbPath); openedDbPaths.push(dbPath);
} }
pragma() {} pragma() {}
close() {} exec(sql) { return this.lockDb?.exec(sql); }
close() { this.lockDb?.close(); }
prepare() { prepare() {
return { get: () => null, all: () => [], run: () => ({}) }; return { get: () => null, all: () => [], run: () => ({}) };
} }
@@ -628,6 +765,12 @@ test('settings rebuild reopens the database from the configured Claude path', as
buildIndex: async (args) => { buildIndex: async (args) => {
serviceEvents.push('build'); serviceEvents.push('build');
buildCalls.push(args); buildCalls.push(args);
const competingLease = acquireWriterLease({
lockPath: args.writerLeasePath,
openDb: lockPath => new DatabaseSync(lockPath),
});
competingLeaseDuringBuild = Boolean(competingLease);
competingLease?.release();
writeFileSync(args.dbPath, 'rebuilt temp db'); writeFileSync(args.dbPath, 'rebuilt temp db');
return { files: 2, affectedSessionIds: ['session-1', 'session-2'] }; return { files: 2, affectedSessionIds: ['session-1', 'session-2'] };
}, },
@@ -649,12 +792,21 @@ test('settings rebuild reopens the database from the configured Claude path', as
assert.equal(buildCalls.at(-1).codexDir, customCodexDir); assert.equal(buildCalls.at(-1).codexDir, customCodexDir);
assert.notEqual(buildCalls.at(-1).dbPath, join(home, '.obelisk', 'obelisk.sqlite')); assert.notEqual(buildCalls.at(-1).dbPath, join(home, '.obelisk', 'obelisk.sqlite'));
assert.equal(buildCalls.at(-1).preserveDbPath, join(home, '.obelisk', 'obelisk.sqlite')); assert.equal(buildCalls.at(-1).preserveDbPath, join(home, '.obelisk', 'obelisk.sqlite'));
assert.equal(buildCalls.at(-1).writerLeasePath, join(home, '.obelisk', 'writer.lock.sqlite'));
assert.equal(buildCalls.at(-1).writerLeaseMode, 'caller-held');
assert.equal(competingLeaseDuringBuild, false);
assert.equal(openedDbPaths.at(-1), join(home, '.obelisk', 'obelisk.sqlite')); assert.equal(openedDbPaths.at(-1), join(home, '.obelisk', 'obelisk.sqlite'));
assert.equal( assert.equal(
require('node:fs').readFileSync(join(home, '.obelisk', 'obelisk.sqlite'), 'utf8'), require('node:fs').readFileSync(join(home, '.obelisk', 'obelisk.sqlite'), 'utf8'),
'rebuilt temp db', 'rebuilt temp db',
); );
assert.ok(serviceEvents.indexOf('build') > serviceEvents.indexOf('stop')); assert.ok(serviceEvents.indexOf('build') > serviceEvents.indexOf('stop'));
const postRebuildLease = acquireWriterLease({
lockPath: join(home, '.obelisk', 'writer.lock.sqlite'),
openDb: lockPath => new DatabaseSync(lockPath),
});
assert.ok(postRebuildLease);
postRebuildLease.release();
} finally { } finally {
restore(); restore();
process.env.HOME = originalHome; process.env.HOME = originalHome;
@@ -686,10 +838,15 @@ test('settings rebuild keeps the existing database after a worker failure', asyn
class FakeDatabase { class FakeDatabase {
constructor(dbPath) { constructor(dbPath) {
this.dbPath = dbPath; this.dbPath = dbPath;
openedDbPaths.push(dbPath); this.lockDb = dbPath.endsWith('writer.lock.sqlite') ? new DatabaseSync(dbPath) : null;
if (!this.lockDb) openedDbPaths.push(dbPath);
} }
pragma() {} pragma() {}
close() { closedDbPaths.push(this.dbPath); } exec(sql) { return this.lockDb?.exec(sql); }
close() {
if (this.lockDb) this.lockDb.close();
else closedDbPaths.push(this.dbPath);
}
prepare() { prepare() {
return { get: () => null, all: () => [], run: () => ({}) }; return { get: () => null, all: () => [], run: () => ({}) };
} }
@@ -788,8 +945,12 @@ test('settings rebuild cancels an in-flight background build instead of waiting
let buildIndexCalls = 0; let buildIndexCalls = 0;
class FakeDatabase { class FakeDatabase {
constructor(dbPath) {
this.lockDb = dbPath.endsWith('writer.lock.sqlite') ? new DatabaseSync(dbPath) : null;
}
pragma() {} pragma() {}
close() {} exec(sql) { return this.lockDb?.exec(sql); }
close() { this.lockDb?.close(); }
prepare() { prepare() {
return { get: () => null, all: () => [], run: () => ({}) }; return { get: () => null, all: () => [], run: () => ({}) };
} }
+237 -33
View File
@@ -1,9 +1,7 @@
// Regression test for the "cannot rollback - no transaction is active" crash. // Regression tests for the write-transaction runner (docs/adr/0006):
// SQLite auto-rolls back certain failures (SQLITE_BUSY, disk full, ...). The // - a transient BUSY (auto-rolled-back txn) is retried and recovers;
// per-file build loop then ran an explicit ROLLBACK in its catch, which threw a // - a persistent BUSY exhausts retries, and that file is SKIPPED, not fatal;
// SECOND error over the real one and aborted the whole build instead of skipping // - the guarded rollback never masks the real error ("cannot rollback ...").
// just the bad file. buildIndex now uses a guarded rollback; this test injects a
// DB that faithfully reproduces the condition and asserts the build survives.
import { test } from 'node:test'; import { test } from 'node:test';
import assert from 'node:assert/strict'; import assert from 'node:assert/strict';
import { createRequire } from 'node:module'; import { createRequire } from 'node:module';
@@ -15,15 +13,17 @@ const require = createRequire(import.meta.url);
import { buildIndex } from '../app/src/main/indexer.ts'; import { buildIndex } from '../app/src/main/indexer.ts';
const { DatabaseSync } = require('node:sqlite'); const { DatabaseSync } = require('node:sqlite');
// Wraps node:sqlite and simulates SQLite's auto-rollback-on-error: the first // Wraps node:sqlite and simulates SQLite auto-rollback-on-error: a poisoned write
// write inside a transaction throws a BUSY-like error AND ends the real // throws a BUSY-like error AND ends the real transaction, so a following explicit
// transaction, so a following explicit ROLLBACK errors with "no transaction". // ROLLBACK errors with "no transaction is active". `shouldPoison(args)` decides
class AutoRollbackOnceDatabase { // which writes are poisoned.
function makeDbClass(shouldPoison) {
return class PoisonDatabase {
constructor(dbPath) { constructor(dbPath) {
this.db = new DatabaseSync(dbPath); this.db = new DatabaseSync(dbPath);
this.inTxn = false; this.inTxn = false;
this.fired = false;
} }
get inTransaction() { return this.inTxn; }
pragma(statement) { this.db.exec(`PRAGMA ${statement}`); } pragma(statement) { this.db.exec(`PRAGMA ${statement}`); }
exec(sql) { exec(sql) {
const head = sql.trim().slice(0, 8).toUpperCase(); const head = sql.trim().slice(0, 8).toUpperCase();
@@ -43,8 +43,7 @@ class AutoRollbackOnceDatabase {
get: (...args) => stmt.get(...args), get: (...args) => stmt.get(...args),
all: (...args) => stmt.all(...args), all: (...args) => stmt.all(...args),
run: (...args) => { run: (...args) => {
if (!self.fired && self.inTxn) { if (self.inTxn && shouldPoison(args)) {
self.fired = true;
self.db.exec('ROLLBACK'); // SQLite auto-rolled the txn back on the error self.db.exec('ROLLBACK'); // SQLite auto-rolled the txn back on the error
self.inTxn = false; self.inTxn = false;
throw new Error('SQLITE_BUSY: database is locked'); throw new Error('SQLITE_BUSY: database is locked');
@@ -54,39 +53,244 @@ class AutoRollbackOnceDatabase {
}; };
} }
close() { return this.db.close(); } close() { return this.db.close(); }
};
} }
test('a per-file write that auto-rolls-back the transaction is skipped, not fatal', () => { function twoFileHome(alphaContent, betaContent) {
const home = mkdtempSync(join(tmpdir(), 'obelisk-rollback-guard-')); const home = mkdtempSync(join(tmpdir(), 'obelisk-tx-'));
const projectDir = join(home, '.claude', 'projects', '-tmp-proj'); const projectDir = join(home, '.claude', 'projects', '-tmp-proj');
mkdirSync(projectDir, { recursive: true }); mkdirSync(projectDir, { recursive: true });
const msg = (uuid) => JSON.stringify({
uuid, type: 'user', timestamp: '2026-06-10T10:00:00Z', cwd: '/tmp/proj',
message: { role: 'user', content: `hello ${uuid}` },
}) + '\n';
writeFileSync(join(projectDir, 'alpha.jsonl'), msg('a1'));
writeFileSync(join(projectDir, 'beta.jsonl'), msg('b1'));
const dbPath = join(home, '.obelisk', 'obelisk.sqlite');
mkdirSync(join(home, '.obelisk'), { recursive: true }); mkdirSync(join(home, '.obelisk'), { recursive: true });
const msg = (uuid, content) => JSON.stringify({
uuid, type: 'user', timestamp: '2026-06-10T10:00:00Z', cwd: '/tmp/proj',
message: { role: 'user', content },
}) + '\n';
writeFileSync(join(projectDir, 'alpha.jsonl'), msg('a1', alphaContent));
writeFileSync(join(projectDir, 'beta.jsonl'), msg('b1', betaContent));
return { home, dbPath: join(home, '.obelisk', 'obelisk.sqlite'), projectsDir: join(home, '.claude', 'projects') };
}
// The unguarded ROLLBACK used to make this throw the masking rollback error. function subagentHome(description = 'first description') {
let result; const home = mkdtempSync(join(tmpdir(), 'obelisk-meta-tx-'));
assert.doesNotThrow(() => { const projectDir = join(home, '.claude', 'projects', '-tmp-proj');
result = buildIndex({ const subagentDir = join(projectDir, 'session', 'subagents');
const dbPath = join(home, '.obelisk', 'obelisk.sqlite');
mkdirSync(subagentDir, { recursive: true });
mkdirSync(join(home, '.obelisk'), { recursive: true });
const message = uuid => JSON.stringify({
uuid,
type: 'user',
timestamp: '2026-06-10T10:00:00Z',
cwd: '/tmp/proj',
message: { role: 'user', content: `message ${uuid}` },
}) + '\n';
writeFileSync(join(projectDir, 'session.jsonl'), message('main-message'));
writeFileSync(join(subagentDir, 'agent.jsonl'), message('agent-message'));
const metaPath = join(subagentDir, 'agent.meta.json');
writeFileSync(metaPath, JSON.stringify({ agentType: 'Explore', description }));
return {
home,
dbPath,
projectsDir: join(home, '.claude', 'projects'),
metaPath,
changedMetaPath: join('-tmp-proj', 'session', 'subagents', 'agent.meta.json'),
};
}
function run(home, dbPath, projectsDir, DatabaseImpl) {
return buildIndex({
force: true, force: true,
claudeDir: join(home, '.claude'), claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'), codexDir: join(home, '.codex'),
projectsDir: join(home, '.claude', 'projects'), projectsDir,
dbPath, dbPath,
DatabaseImpl: AutoRollbackOnceDatabase, DatabaseImpl,
}); });
}, 'build must not abort on a transaction-aborting per-file error'); }
function makeBeginBusyDbClass(shouldFail) {
const Base = makeDbClass(() => false);
return class BeginBusyDatabase extends Base {
constructor(dbPath) {
super(dbPath);
this.isWriterLease = dbPath.endsWith('writer.lock.sqlite');
this.beginCalls = 0;
}
exec(sql) {
if (!this.isWriterLease && sql.trim().toUpperCase().startsWith('BEGIN')) {
this.beginCalls += 1;
if (shouldFail(this.beginCalls)) {
throw Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' });
}
}
return super.exec(sql);
}
};
}
test('a failed changed file is not reported as an affected session', () => {
const { home, dbPath, projectsDir } = twoFileHome('POISON alpha', 'hello beta');
const Db = makeDbClass((args) => args.some(a => typeof a === 'string' && a.includes('POISON')));
const result = buildIndex({
force: false,
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
DatabaseImpl: Db,
changedPaths: ['-tmp-proj/alpha.jsonl'],
});
assert.deepEqual(result.affectedSessionIds, []);
assert.equal(result.skipped, 1);
});
test('a transient BUSY during force cleanup is retried and recovers', () => {
const { home, dbPath, projectsDir } = twoFileHome('hello alpha', 'hello beta');
// Fire exactly once, on the first write inside a transaction, then never again.
let fired = false;
const Db = makeDbClass(() => (fired ? false : (fired = true)));
let result;
assert.doesNotThrow(() => { result = run(home, dbPath, projectsDir, Db); });
// Exactly one file's write was poisoned; the other indexed cleanly.
const check = new DatabaseSync(dbPath); const check = new DatabaseSync(dbPath);
const sessions = check.prepare('SELECT COUNT(*) AS c FROM sessions').get().c; const sessions = check.prepare('SELECT COUNT(*) AS c FROM sessions').get().c;
check.close(); check.close();
assert.equal(sessions, 1, 'the surviving file is indexed; the poisoned one is skipped'); assert.equal(sessions, 2, 'the retried file recovered; both files indexed');
assert.ok(result.files >= 2, 'both files were discovered'); assert.equal(result.skipped, 0, 'nothing was skipped');
});
test('a transient BUSY during a file transaction is retried and recovers', () => {
const { home, dbPath, projectsDir } = twoFileHome('hello alpha', 'hello beta');
let fired = false;
const Db = makeDbClass((args) => {
const isAlphaWrite = args.some(arg => typeof arg === 'string' && arg.includes('hello alpha'));
if (!isAlphaWrite || fired) return false;
fired = true;
return true;
});
const result = run(home, dbPath, projectsDir, Db);
const check = new DatabaseSync(dbPath);
const sessions = check.prepare('SELECT COUNT(*) AS c FROM sessions').get().c;
check.close();
assert.equal(sessions, 2);
assert.equal(result.skipped, 0);
});
test('a persistent BUSY exhausts retries and skips just that file, not the build', () => {
const { home, dbPath, projectsDir } = twoFileHome('POISON alpha', 'hello beta');
// Always poison writes that carry alpha's marker text; beta is untouched.
const Db = makeDbClass((args) => args.some(a => typeof a === 'string' && a.includes('POISON')));
let result;
assert.doesNotThrow(() => { result = run(home, dbPath, projectsDir, Db); });
const check = new DatabaseSync(dbPath);
const sessions = check.prepare('SELECT id FROM sessions ORDER BY id').all().map(r => r.id);
check.close();
assert.deepEqual(sessions, ['beta'], 'the persistently-failing file is skipped; the other indexes');
assert.equal(result.skipped, 1, 'the skipped file is reported in the build result');
assert.equal(result.skippedFiles[0].diagnostics?.phase, 'work', 'diagnostics record the failing phase');
});
test('BEGIN contention during force cleanup defers the build', () => {
const { home, dbPath, projectsDir } = twoFileHome('hello alpha', 'hello beta');
const Db = makeBeginBusyDbClass(beginCall => beginCall === 1);
const result = run(home, dbPath, projectsDir, Db);
assert.equal(result.deferred, true);
assert.equal(result.reason, 'database_busy');
});
test('BEGIN contention during finalize defers the build', () => {
const { home, dbPath, projectsDir } = twoFileHome('hello alpha', 'hello beta');
const Db = makeBeginBusyDbClass(beginCall => beginCall === 3);
const result = buildIndex({
force: false,
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
DatabaseImpl: Db,
});
assert.equal(result.deferred, true);
assert.equal(result.reason, 'database_busy');
});
test('a finalize database error is propagated instead of swallowed as malformed input', () => {
const { home, dbPath, projectsDir } = twoFileHome('hello alpha', 'hello beta');
const workflowDir = join(projectsDir, '-tmp-proj', 'alpha', 'workflows');
mkdirSync(workflowDir, { recursive: true });
writeFileSync(join(workflowDir, 'run.json'), JSON.stringify({
runId: 'workflow-1',
workflowName: 'POISON WORKFLOW',
}));
const Db = makeDbClass(args => args.some(arg => typeof arg === 'string' && arg.includes('POISON WORKFLOW')));
assert.throws(() => buildIndex({
force: false,
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
DatabaseImpl: Db,
}), /SQLITE_BUSY/);
});
test('a changed subagent meta file is applied and reported only after its file transaction commits', () => {
const { home, dbPath, projectsDir, metaPath, changedMetaPath } = subagentHome();
const Db = makeDbClass(() => false);
buildIndex({
force: true,
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
DatabaseImpl: Db,
});
writeFileSync(metaPath, JSON.stringify({ agentType: 'Explore', description: 'updated description' }));
const result = buildIndex({
force: false,
changedPaths: [changedMetaPath],
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
DatabaseImpl: Db,
});
const check = new DatabaseSync(dbPath, { readOnly: true });
assert.equal(check.prepare('SELECT description FROM subagents WHERE agent_id=?').get('agent').description, 'updated description');
check.close();
assert.deepEqual(result.affectedSessionIds, ['session']);
});
test('a failed subagent meta transaction does not report its session as affected', () => {
const { home, dbPath, projectsDir, metaPath, changedMetaPath } = subagentHome();
buildIndex({
force: true,
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
DatabaseImpl: makeDbClass(() => false),
});
writeFileSync(metaPath, JSON.stringify({ agentType: 'Explore', description: 'POISON META' }));
const FailingDb = makeDbClass(args => args.some(arg => typeof arg === 'string' && arg.includes('POISON META')));
const result = buildIndex({
force: false,
changedPaths: [changedMetaPath],
claudeDir: join(home, '.claude'),
codexDir: join(home, '.codex'),
projectsDir,
dbPath,
DatabaseImpl: FailingDb,
});
assert.deepEqual(result.affectedSessionIds, []);
assert.equal(result.skipped, 1);
}); });
+97
View File
@@ -0,0 +1,97 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createRequire } from 'node:module';
import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { buildIndex } from '../app/src/main/indexer.ts';
import { acquireWriterLease, writerLockPathFor } from '../scripts/writer-lease.ts';
const require = createRequire(import.meta.url);
const { DatabaseSync } = require('node:sqlite');
class TestDatabase {
constructor(dbPath) {
this.db = new DatabaseSync(dbPath);
}
get inTransaction() { return this.db.isTransaction; }
exec(sql) { return this.db.exec(sql); }
pragma(statement) { return this.db.exec(`PRAGMA ${statement}`); }
prepare(sql) { return this.db.prepare(sql); }
close() { return this.db.close(); }
}
test('an app build defers without opening the target database when another writer owns the lease', () => {
const home = mkdtempSync(join(tmpdir(), 'obelisk-app-writer-lease-'));
const claudeDir = join(home, '.claude');
const projectsDir = join(claudeDir, 'projects');
const projectDir = join(projectsDir, '-tmp-project');
const dbPath = join(home, '.obelisk', 'obelisk.sqlite');
mkdirSync(projectDir, { recursive: true });
mkdirSync(join(home, '.obelisk'), { recursive: true });
writeFileSync(join(projectDir, 'session.jsonl'), JSON.stringify({
uuid: 'message-1',
type: 'user',
timestamp: '2026-07-10T00:00:00Z',
message: { role: 'user', content: 'hello' },
}) + '\n');
const lease = acquireWriterLease({
lockPath: writerLockPathFor(dbPath),
openDb: path => new DatabaseSync(path),
});
assert.ok(lease);
try {
const result = buildIndex({
claudeDir,
projectsDir,
dbPath,
DatabaseImpl: TestDatabase,
writerLeaseWaitMs: 0,
});
assert.equal(result.deferred, true);
assert.equal(result.reason, 'writer_busy');
assert.equal(existsSync(dbPath), false);
} finally {
lease.release();
}
});
test('a failed force cleanup leaves the existing index intact', () => {
const home = mkdtempSync(join(tmpdir(), 'obelisk-app-force-atomic-'));
const claudeDir = join(home, '.claude');
const projectsDir = join(claudeDir, 'projects');
const projectDir = join(projectsDir, '-tmp-project');
const dbPath = join(home, '.obelisk', 'obelisk.sqlite');
mkdirSync(projectDir, { recursive: true });
mkdirSync(join(home, '.obelisk'), { recursive: true });
writeFileSync(join(projectDir, 'session.jsonl'), JSON.stringify({
uuid: 'message-1',
type: 'user',
timestamp: '2026-07-10T00:00:00Z',
message: { role: 'user', content: 'hello' },
}) + '\n');
buildIndex({ claudeDir, projectsDir, dbPath, DatabaseImpl: TestDatabase });
class FailingCleanupDatabase extends TestDatabase {
prepare(sql) {
const statement = super.prepare(sql);
if (!sql.includes('DELETE FROM sessions')) return statement;
return {
get: (...args) => statement.get(...args),
all: (...args) => statement.all(...args),
run: () => { throw new Error('cleanup interrupted'); },
};
}
}
assert.throws(
() => buildIndex({ claudeDir, projectsDir, dbPath, DatabaseImpl: FailingCleanupDatabase, force: true }),
/cleanup interrupted/,
);
const check = new DatabaseSync(dbPath, { readOnly: true });
assert.equal(check.prepare('SELECT COUNT(*) AS count FROM sessions').get().count, 1);
assert.equal(check.prepare('SELECT COUNT(*) AS count FROM messages').get().count, 1);
check.close();
});
+133
View File
@@ -0,0 +1,133 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createRequire } from 'node:module';
import { mkdirSync, mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { spawnSync } from 'node:child_process';
import { acquireWriterLease, writerLockPathFor } from '../scripts/writer-lease.ts';
const require = createRequire(import.meta.url);
const { DatabaseSync } = require('node:sqlite');
const repoRoot = resolve(new URL('..', import.meta.url).pathname);
test('a passive query does not mutate the index while a fresh daemon owns writes', () => {
const home = mkdtempSync(join(tmpdir(), 'obelisk-daemon-arbitration-'));
const obeliskDir = join(home, '.obelisk');
const dbPath = join(obeliskDir, 'obelisk.sqlite');
mkdirSync(obeliskDir, { recursive: true });
const db = new DatabaseSync(dbPath);
db.exec('CREATE TABLE index_state (jsonl_path TEXT PRIMARY KEY, mtime REAL, lines_processed INTEGER)');
const marker = db.prepare('INSERT INTO index_state (jsonl_path, mtime, lines_processed) VALUES (?, ?, 0)');
const now = Date.now();
marker.run('__app_heartbeat__', now);
db.close();
const queryPath = join(home, 'query.mjs');
writeFileSync(queryPath, "return 'read-only';");
const result = spawnSync(process.execPath, ['scripts/runtime.mjs', '--query', queryPath], {
cwd: repoRoot,
env: { ...process.env, HOME: home },
encoding: 'utf8',
});
assert.equal(result.status, 0, result.stderr || result.stdout);
assert.equal(JSON.parse(result.stdout), 'read-only');
const check = new DatabaseSync(dbPath, { readOnly: true });
const tables = check.prepare("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name").all().map(row => row.name);
check.close();
assert.deepEqual(tables, ['index_state']);
});
test('attune refuses to mutate the index while a fresh daemon owns writes', () => {
const home = mkdtempSync(join(tmpdir(), 'obelisk-daemon-attune-'));
const obeliskDir = join(home, '.obelisk');
const dbPath = join(obeliskDir, 'obelisk.sqlite');
mkdirSync(obeliskDir, { recursive: true });
const db = new DatabaseSync(dbPath);
db.exec('CREATE TABLE index_state (jsonl_path TEXT PRIMARY KEY, mtime REAL, lines_processed INTEGER)');
const marker = db.prepare('INSERT INTO index_state (jsonl_path, mtime, lines_processed) VALUES (?, ?, 0)');
const now = Date.now();
marker.run('__app_heartbeat__', now);
db.close();
const attunePath = join(home, 'attune.mjs');
writeFileSync(attunePath, 'return true;');
const result = spawnSync(process.execPath, ['scripts/runtime.mjs', '--attune', attunePath], {
cwd: repoRoot,
env: { ...process.env, HOME: home },
encoding: 'utf8',
});
assert.equal(result.status, 1);
assert.match(JSON.parse(result.stdout).error, /daemon owns index writes/i);
const check = new DatabaseSync(dbPath, { readOnly: true });
const tables = check.prepare("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name").all().map(row => row.name);
check.close();
assert.deepEqual(tables, ['index_state']);
});
test('a passive query stays read-only when another process holds the writer lease', () => {
const home = mkdtempSync(join(tmpdir(), 'obelisk-writer-owned-'));
const obeliskDir = join(home, '.obelisk');
const dbPath = join(obeliskDir, 'obelisk.sqlite');
mkdirSync(obeliskDir, { recursive: true });
const db = new DatabaseSync(dbPath);
db.exec('CREATE TABLE index_state (jsonl_path TEXT PRIMARY KEY, mtime REAL, lines_processed INTEGER)');
db.close();
const lease = acquireWriterLease({
lockPath: writerLockPathFor(dbPath),
openDb: path => new DatabaseSync(path),
});
assert.ok(lease);
try {
const queryPath = join(home, 'query.mjs');
writeFileSync(queryPath, "return 'writer-busy';");
const result = spawnSync(process.execPath, ['scripts/runtime.mjs', '--query', queryPath], {
cwd: repoRoot,
env: { ...process.env, HOME: home },
encoding: 'utf8',
});
assert.equal(result.status, 0, result.stderr || result.stdout);
assert.equal(JSON.parse(result.stdout), 'writer-busy');
} finally {
lease.release();
}
const check = new DatabaseSync(dbPath, { readOnly: true });
const tables = check.prepare("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name").all().map(row => row.name);
check.close();
assert.deepEqual(tables, ['index_state']);
});
test('a passive query fails closed when daemon ownership cannot be read', () => {
const home = mkdtempSync(join(tmpdir(), 'obelisk-daemon-ownership-error-'));
const obeliskDir = join(home, '.obelisk');
const dbPath = join(obeliskDir, 'obelisk.sqlite');
mkdirSync(obeliskDir, { recursive: true });
const db = new DatabaseSync(dbPath);
db.exec('CREATE TABLE index_state (jsonl_path TEXT PRIMARY KEY)');
db.close();
const lease = acquireWriterLease({
lockPath: writerLockPathFor(dbPath),
openDb: path => new DatabaseSync(path),
});
assert.ok(lease);
try {
const queryPath = join(home, 'query.mjs');
writeFileSync(queryPath, "return 'ownership-unknown';");
const result = spawnSync(process.execPath, ['scripts/runtime.mjs', '--query', queryPath], {
cwd: repoRoot,
env: { ...process.env, HOME: home },
encoding: 'utf8',
});
assert.equal(result.status, 1, result.stderr || result.stdout);
assert.match(JSON.parse(result.stdout).error, /no such column: mtime/i);
} finally {
lease.release();
}
});
+8 -6
View File
@@ -56,7 +56,7 @@ test('refreshSessionProjectPaths repairs indexed sessions from message cwd', ()
db.close(); db.close();
}); });
test('shouldSkipBuild requires both fresh app heartbeat and successful app build', () => { test('shouldSkipBuild treats a fresh heartbeat alone as daemon write ownership', () => {
const db = new DatabaseSync(':memory:'); const db = new DatabaseSync(':memory:');
db.exec(` db.exec(`
CREATE TABLE index_state ( CREATE TABLE index_state (
@@ -68,17 +68,19 @@ test('shouldSkipBuild requires both fresh app heartbeat and successful app build
100000, 100000,
); );
assert.equal(shouldSkipBuild(db, { now: 110000 }).skip, false); assert.deepEqual(
shouldSkipBuild(db, { now: 110000 }),
{ skip: true, reason: 'daemon_active' },
);
db.prepare('DELETE FROM index_state').run();
db.prepare('INSERT INTO index_state (jsonl_path, mtime, lines_processed) VALUES (?, ?, 0)').run( db.prepare('INSERT INTO index_state (jsonl_path, mtime, lines_processed) VALUES (?, ?, 0)').run(
'__app_last_successful_build__', '__app_last_successful_build__',
100000, 100000,
); );
assert.deepEqual( assert.equal(shouldSkipBuild(db, { now: 110000 }).skip, false);
shouldSkipBuild(db, { now: 110000 }),
{ skip: true, reason: 'app_successful_build' },
);
assert.equal(shouldSkipBuild(db, { now: 200000 }).skip, false); assert.equal(shouldSkipBuild(db, { now: 200000 }).skip, false);
db.close(); db.close();
}); });
+166
View File
@@ -0,0 +1,166 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createRequire } from 'node:module';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { nodeSqliteTransactionAdapter, runWriteTransaction } from '../scripts/tx.ts';
import { hasUnusableTransaction, isBeginBusyFailure, isRetryableWriteFailure } from '../scripts/write-coordinator.ts';
const require = createRequire(import.meta.url);
const { DatabaseSync } = require('node:sqlite');
test('a failed rollback with an active transaction never retries or masks the primary error', () => {
const primary = Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' });
let active = false;
let workCalls = 0;
const db = {
exec(sql) {
if (sql.startsWith('BEGIN')) {
if (active) throw new Error('cannot start a transaction within a transaction');
active = true;
} else if (sql === 'ROLLBACK') {
throw new Error('rollback failed with I/O error');
} else if (sql === 'COMMIT') {
active = false;
}
},
inTransaction() {
return active;
},
};
assert.throws(
() => runWriteTransaction(db, () => {
workCalls += 1;
throw primary;
}),
error => error === primary,
);
assert.equal(workCalls, 1);
assert.equal(active, true);
});
test('an automatic rollback rethrows the primary error without issuing another rollback', () => {
const primary = Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' });
let active = false;
let rollbackCalls = 0;
const db = {
exec(sql) {
if (sql.startsWith('BEGIN')) active = true;
if (sql === 'ROLLBACK') rollbackCalls += 1;
},
inTransaction() {
return active;
},
};
assert.throws(
() => runWriteTransaction(db, () => {
active = false;
throw primary;
}),
error => error === primary,
);
assert.equal(rollbackCalls, 0);
assert.equal(primary.obelisk.transactionActive, false);
});
test('an active transaction is rolled back once before the primary error is rethrown', () => {
const primary = new Error('persist failed');
let active = false;
let rollbackCalls = 0;
const db = {
exec(sql) {
if (sql.startsWith('BEGIN')) active = true;
if (sql === 'ROLLBACK') {
rollbackCalls += 1;
active = false;
}
},
inTransaction() {
return active;
},
};
assert.throws(() => runWriteTransaction(db, () => { throw primary; }), error => error === primary);
assert.equal(rollbackCalls, 1);
assert.equal(primary.obelisk.rollbackSucceeded, true);
assert.equal(primary.obelisk.transactionActive, false);
});
test('an unknown post-error transaction state is unsafe for the next file', () => {
const primary = new Error('transaction state unavailable');
const db = {
exec() {},
inTransaction() {
throw new Error('binding cannot report transaction state');
},
};
assert.throws(() => runWriteTransaction(db, () => { throw primary; }), error => error === primary);
assert.equal(primary.obelisk.transactionActive, null);
assert.equal(hasUnusableTransaction(primary), true);
});
test('node:sqlite generic error codes preserve BUSY classification from the message', () => {
const primary = Object.assign(new Error('database is locked'), { code: 'ERR_SQLITE_ERROR' });
let active = false;
const db = {
exec(sql) {
if (sql === 'BEGIN IMMEDIATE') active = true;
},
inTransaction() { return active; },
};
assert.throws(() => runWriteTransaction(db, () => {
active = false;
throw primary;
}), error => error === primary);
assert.equal(primary.obelisk.code, 'SQLITE_BUSY');
assert.equal(isRetryableWriteFailure(primary), true);
});
test('a real node:sqlite BEGIN lock is classified as a deferrable BUSY', () => {
const dbPath = join(mkdtempSync(join(tmpdir(), 'obelisk-node-sqlite-busy-')), 'index.sqlite');
const holder = new DatabaseSync(dbPath);
const contender = new DatabaseSync(dbPath);
holder.exec('PRAGMA busy_timeout=0; CREATE TABLE test (value TEXT); BEGIN IMMEDIATE');
contender.exec('PRAGMA busy_timeout=0');
try {
assert.throws(
() => runWriteTransaction(nodeSqliteTransactionAdapter(contender), () => {}),
error => isBeginBusyFailure(error) && error.obelisk.code === 'SQLITE_BUSY',
);
} finally {
holder.exec('ROLLBACK');
holder.close();
contender.close();
}
});
test('a BEGIN failure with an active transaction is not deferrable', () => {
const primary = Object.assign(new Error('database is locked'), { code: 'SQLITE_BUSY' });
let rollbackCalls = 0;
const db = {
exec(sql) {
if (sql === 'BEGIN IMMEDIATE') throw primary;
if (sql === 'ROLLBACK') {
rollbackCalls += 1;
throw new Error('rollback failed with I/O error');
}
},
inTransaction() {
return true;
},
};
assert.throws(() => runWriteTransaction(db, () => {}), error => error === primary);
assert.equal(rollbackCalls, 1);
assert.equal(primary.obelisk.transactionActive, true);
assert.equal(hasUnusableTransaction(primary), true);
assert.equal(isBeginBusyFailure(primary), false);
});
+25
View File
@@ -0,0 +1,25 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { createRequire } from 'node:module';
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { acquireWriterLease } from '../scripts/writer-lease.ts';
const require = createRequire(import.meta.url);
const { DatabaseSync } = require('node:sqlite');
test('a writer lease excludes another writer until it is released', () => {
const lockPath = join(mkdtempSync(join(tmpdir(), 'obelisk-writer-lease-')), 'writer.lock.sqlite');
const openDb = path => new DatabaseSync(path);
const first = acquireWriterLease({ lockPath, openDb });
assert.ok(first);
assert.equal(acquireWriterLease({ lockPath, openDb }), null);
first.release();
const afterRelease = acquireWriterLease({ lockPath, openDb });
assert.ok(afterRelease);
afterRelease.release();
});
+1
View File
@@ -7,6 +7,7 @@
"types": ["node"], "types": ["node"],
"strict": true, "strict": true,
"noEmit": true, "noEmit": true,
"allowImportingTsExtensions": true,
"allowJs": true, "allowJs": true,
"checkJs": false, "checkJs": false,
"erasableSyntaxOnly": true, "erasableSyntaxOnly": true,