mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* perf(extract_atoms): batch idempotency check via atomsExistingForHashes Replaces the per-hash transcript loop (7K SQL roundtrips on big brains) with one batch query using `frontmatter->>'source_hash' = ANY($2::text[])`. Migration v104 adds the partial expression index that keeps the new query O(log n) at scale (mirrors v97 pattern: CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain CREATE INDEX on PGLite). Helper exported so test/cycle/extract-atoms-batch.test.ts can drive it directly without orchestrating the full phase. Fail-open posture preserved from the prior per-hash helper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cycle): shorter lock TTL + active in-phase refresh + progress wiring Issue 3 + Issue 2 of the v0.41.20.0 ops-fix-wave. Codex caught during plan review that yieldBetweenPhases (the existing external hook) does NOT refresh the cycle DB lock — it's just a setImmediate() from jobs.ts:1405 / autopilot.ts:632, and lock.refresh() was never called from inside runCycle. Combined with the 30min TTL, crashed cycles wedged the lock for the full window before another worker could take over. Three coordinated changes: 1. LOCK_TTL_MINUTES 30 → 5 (src/core/cycle.ts). Crash recovers in ≤5 min instead of ≤30 min. 2. buildYieldDuringPhase(lock, outer) — exported closure that calls lock.refresh() AND the existing yieldBetweenPhases hook on every fire. Passed to both long phases (extract_atoms, synthesize_concepts) as their yieldDuringPhase opt. 3. maybeYield helper inside both phases — 30s throttle, fires inside the main work loop AND immediately after every `await chat()` LLM call (codex hardening: a single long LLM await could otherwise sit past TTL). Progress reporter wired through to both phases too (Issue 2): extract_atoms emits `[cycle.extract_atoms] N atoms / M skipped` ticks every ~1s; synthesize_concepts ticks per concept group. Cycle.ts owns start()/finish(); phases only call tick() and heartbeat() on the same reporter (NOT a child — that would produce path collision `cycle.extract_atoms.extract_atoms.work`). LockHandle interface exported for tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(extract): by-mention resumes from where it died Issue 4 of the v0.41.20.0 ops-fix-wave. On a 322K-page brain the sweep takes 10+ hours; if it died at 87% the user redid 87% on restart. Wires the existing `op_checkpoints` framework into extractMentionsFromDb with a flushAndCheckpoint ordering that closes the four codex-flagged correctness bugs at once: 1. Lost-links-on-crash — flush batch links to DB FIRST, commit page keys to checkpoint SECOND, persist THIRD. A crash between batch.push() and flushBatch() leaves the page un-checkpointed so resume re-scans it (no silently lost mention links). 2. Dry-run resume contradiction — dry-run does NOT load or persist the checkpoint. Verification path uses non-dry-run kill-and-resume. 3. Gazetteer hash in fingerprint — entity pages added mid-pause shift the gazetteer hash → new fingerprint → fresh scan against the new gazetteer. Without this, resumed runs would silently skip pages against a new entity set. 4. Filtered pages get checkpointed too — pages skipped by `--type` / `--since` / empty body / no-mentions all get marked completed so resume doesn't re-fetch them. Persist cadence: every 1000 items OR every 30s, whichever first (~322 persists / ~24s total overhead on the 322K-page brain). Crash window capped at 1000 pages (<0.3% loss). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(doctor): surface sync --all consolidation nudge to operators Issue 5 of the v0.41.20.0 ops-fix-wave. Multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` in `gbrain doctor` output instead of maintaining two staggered per-source cron entries with manual deconfliction. New checkSyncConsolidation surfaces the recommendation when 2+ active sources exist; "not applicable" for single-source brains. Own try/catch returns warn on SQL failure — outer doctor catch wasn't a safe assumption. `skills/cron-scheduler/SKILL.md` gains a "Multi-source brains" recipe block documenting the pattern + connection-budget math (parallel × workers × 2 ≈ 32 connections at default 4/4). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): isolate GBRAIN_HOME in cycle-LFCA + schema-cli tests Two pre-existing tests assumed a clean ~/.gbrain/config.json and a free ~/.gbrain/cycle.lock — both shared across all gbrain processes on the machine. Sibling Conductor worktrees running their own gbrain tests poisoned the shared state, causing flakes: - test/cycle-last-full-cycle-at.test.ts test 5 timed out at 5s because runCycle returned 'skipped' (file lock held by a parallel test process), and last_full_cycle_at exit hook silently no-oped. Fix: each test wraps its body in `withEnv({GBRAIN_HOME: tmpdir})` so the file lock path becomes per-test. - test/schema-cli.test.ts `schema active reports default resolution` failed exit 1 because another worktree had set `schema_pack: gbrain-base-v2` in the shared config (a pack that doesn't exist in the bundle). Fix: gbrain() helper defaults GBRAIN_HOME to a per-file tempdir (beforeAll-owned), so subprocess invocations get an isolated config dir unless tests explicitly override. Both fixes confirmed via deliberate pollution + retest: 12/12 schema-cli tests pass under simulated `schema_pack: gbrain-base-v2` contamination; cycle-LFCA test 5 completes <2s with isolated home. Discovered during v0.41.20.0 ship while investigating parallel-worktree flake. Not caused by the ops-fix-wave but found via it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.41.20.0) Five daily-driver ops pains fixed in one wave: 1. extract_atoms 7K-roundtrip overhead → 1 batch query + index 2. silent long-running phases → progress ticks every ~1s 3. 30-min crashed-cycle lock TTL → 5 min + active in-phase refresh 4. by-mention restarts from page 0 → resumes via op_checkpoints 5. multi-source cron → doctor surfaces `sync --all --parallel` nudge Two follow-up TODOs filed under v0.41.19.0 ops-fix-wave block (will be renamed at follow-up time): - `gbrain sync print-cron` subcommand (P2 ergonomics) - Lock-loss detection in DbLockHandle.refresh() (P2 contract change) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md key files for v0.41.20.0 ops-fix-wave Folds the v0.41.20.0 wave annotations into the cycle/extract/op-checkpoint key-files block: batch idempotency via atomsExistingForHashes, shorter cycle lock TTL with buildYieldDuringPhase active refresh, progress wiring through extract_atoms + synthesize_concepts, by-mention resume via mentionsFingerprint with flushAndCheckpoint ordering, sync_consolidation doctor check, and the 44-case test suite pinning every contract. Regenerated llms-full.txt to match (CLAUDE.md edit invariant). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci): doctor categorization + facts-engine cosine ordering hardening Two CI-only failures caught on PR #1545 (v0.41.21.0 ops-fix-wave): 1. doctor-categories drift guard — new `sync_consolidation` check from T6 wasn't categorized in src/core/doctor-categories.ts. Added under OPS_CHECK_NAMES (it surfaces an operator-cron recommendation, not a brain-data quality signal). 2. facts-engine `embedding cosine ordering when both sides have embeddings` — passed locally, failed under CI's parallel shard. Bun's truncated assertion output didn't surface which expect() fired; hardened the test against unknown leak vectors by: - per-run unique entity_slug (`embed-test-<random8>`) instead of the static `embed-test`, so any future cross-test pollution is structurally impossible - `findIndex` + `aIdx < bIdx` assertion that pins the cosine RELATIONSHIP (A closer than B because cos(A,Q)=1.0 vs cos(B,Q)=0.0) instead of the brittle `result[0].fact === 'A'` position check. The new shape matches the test name's contract verbatim ("ordering when both sides have embeddings"), so any unrelated row in the result set can no longer flip the test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
130 lines
4.7 KiB
TypeScript
130 lines
4.7 KiB
TypeScript
// v0.41.19.0 — T4 of ops-fix-wave.
|
|
//
|
|
// Pins that extract_atoms wires its progress reporter inside the work
|
|
// loop (one tick per processed item) and emits a heartbeat before the
|
|
// batch idempotency check. Codex caught that cycle.ts must NOT pass a
|
|
// child reporter — phases receive the SAME reporter and only call tick
|
|
// / heartbeat (cycle.ts owns start / finish).
|
|
|
|
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
|
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
|
import { runPhaseExtractAtoms } from '../../src/core/cycle/extract-atoms.ts';
|
|
import { resetPgliteState } from '../helpers/reset-pglite.ts';
|
|
import type { ProgressReporter } from '../../src/core/progress.ts';
|
|
import type { ChatResult, ChatOpts } from '../../src/core/ai/gateway.ts';
|
|
|
|
let engine: PGLiteEngine;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({});
|
|
await engine.initSchema();
|
|
}, 60000);
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
await resetPgliteState(engine);
|
|
});
|
|
|
|
function makeMockReporter(): {
|
|
reporter: ProgressReporter;
|
|
events: Array<{ kind: 'start' | 'tick' | 'heartbeat' | 'finish' | 'child'; phase?: string; note?: string; n?: number }>;
|
|
} {
|
|
const events: Array<{ kind: 'start' | 'tick' | 'heartbeat' | 'finish' | 'child'; phase?: string; note?: string; n?: number }> = [];
|
|
const reporter: ProgressReporter = {
|
|
start: (phase, _total) => { events.push({ kind: 'start', phase }); },
|
|
tick: (n, note) => { events.push({ kind: 'tick', n, note }); },
|
|
heartbeat: (note) => { events.push({ kind: 'heartbeat', note }); },
|
|
finish: (note) => { events.push({ kind: 'finish', note }); },
|
|
child: (phase) => {
|
|
events.push({ kind: 'child', phase });
|
|
return reporter; // return self for simplicity
|
|
},
|
|
};
|
|
return { reporter, events };
|
|
}
|
|
|
|
function stubChat(text: string): (o: ChatOpts) => Promise<ChatResult> {
|
|
return async (_o: ChatOpts) => ({
|
|
text,
|
|
blocks: [{ type: 'text', text }],
|
|
stopReason: 'end',
|
|
usage: { input_tokens: 100, output_tokens: 50, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
|
model: 'anthropic:claude-haiku-4-5',
|
|
providerId: 'anthropic',
|
|
});
|
|
}
|
|
|
|
describe('extract_atoms progress wiring (T4)', () => {
|
|
test('phase does NOT call start or finish — cycle.ts owns those', async () => {
|
|
const { reporter, events } = makeMockReporter();
|
|
const validAtomJson = JSON.stringify([
|
|
{ title: 'A', atom_type: 'insight', body: 'body a' },
|
|
]);
|
|
await runPhaseExtractAtoms(engine, {
|
|
sourceId: 'default',
|
|
_transcripts: [
|
|
{ filePath: '/tmp/t1.txt', content: 'transcript 1 body', contentHash: 'h1'.repeat(8) },
|
|
],
|
|
_pages: [],
|
|
_chat: stubChat(validAtomJson),
|
|
progress: reporter,
|
|
});
|
|
const startEvents = events.filter(e => e.kind === 'start');
|
|
const finishEvents = events.filter(e => e.kind === 'finish');
|
|
expect(startEvents.length).toBe(0);
|
|
expect(finishEvents.length).toBe(0);
|
|
});
|
|
|
|
test('emits a heartbeat before the batch idempotency check', async () => {
|
|
const { reporter, events } = makeMockReporter();
|
|
await runPhaseExtractAtoms(engine, {
|
|
sourceId: 'default',
|
|
_transcripts: [
|
|
{ filePath: '/tmp/t1.txt', content: 'transcript', contentHash: 'h1'.repeat(8) },
|
|
],
|
|
_pages: [],
|
|
_chat: stubChat('[]'),
|
|
progress: reporter,
|
|
});
|
|
const heartbeats = events.filter(e => e.kind === 'heartbeat');
|
|
expect(heartbeats.length).toBeGreaterThanOrEqual(1);
|
|
// Note mentions the count
|
|
expect(heartbeats[0].note).toMatch(/checking existing atoms/);
|
|
});
|
|
|
|
test('one tick per processed work item with running count note', async () => {
|
|
const { reporter, events } = makeMockReporter();
|
|
const validAtomJson = JSON.stringify([
|
|
{ title: 'A', atom_type: 'insight', body: 'body a' },
|
|
]);
|
|
await runPhaseExtractAtoms(engine, {
|
|
sourceId: 'default',
|
|
_transcripts: [
|
|
{ filePath: '/tmp/t1.txt', content: 'a', contentHash: 'h1'.repeat(8) },
|
|
{ filePath: '/tmp/t2.txt', content: 'b', contentHash: 'h2'.repeat(8) },
|
|
{ filePath: '/tmp/t3.txt', content: 'c', contentHash: 'h3'.repeat(8) },
|
|
],
|
|
_pages: [],
|
|
_chat: stubChat(validAtomJson),
|
|
progress: reporter,
|
|
});
|
|
const ticks = events.filter(e => e.kind === 'tick');
|
|
expect(ticks.length).toBe(3);
|
|
expect(ticks[0].note).toMatch(/atoms.*skipped/);
|
|
});
|
|
|
|
test('no progress wiring required — opts.progress is optional', async () => {
|
|
// Sanity: phase works without a reporter.
|
|
const result = await runPhaseExtractAtoms(engine, {
|
|
sourceId: 'default',
|
|
_transcripts: [],
|
|
_pages: [],
|
|
});
|
|
expect(result.phase).toBe('extract_atoms');
|
|
});
|
|
});
|