mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
68ed7bafa44db55c19eb3172d6c2f0e39e48b849
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
543f9a71b4 |
v0.41.21.0 feat(ops): 5 daily-driver pains fixed in one wave (#1545)
* perf(extract_atoms): batch idempotency check via atomsExistingForHashes Replaces the per-hash transcript loop (7K SQL roundtrips on big brains) with one batch query using `frontmatter->>'source_hash' = ANY($2::text[])`. Migration v104 adds the partial expression index that keeps the new query O(log n) at scale (mirrors v97 pattern: CONCURRENTLY + invalid-remnant pre-drop on Postgres, plain CREATE INDEX on PGLite). Helper exported so test/cycle/extract-atoms-batch.test.ts can drive it directly without orchestrating the full phase. Fail-open posture preserved from the prior per-hash helper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cycle): shorter lock TTL + active in-phase refresh + progress wiring Issue 3 + Issue 2 of the v0.41.20.0 ops-fix-wave. Codex caught during plan review that yieldBetweenPhases (the existing external hook) does NOT refresh the cycle DB lock — it's just a setImmediate() from jobs.ts:1405 / autopilot.ts:632, and lock.refresh() was never called from inside runCycle. Combined with the 30min TTL, crashed cycles wedged the lock for the full window before another worker could take over. Three coordinated changes: 1. LOCK_TTL_MINUTES 30 → 5 (src/core/cycle.ts). Crash recovers in ≤5 min instead of ≤30 min. 2. buildYieldDuringPhase(lock, outer) — exported closure that calls lock.refresh() AND the existing yieldBetweenPhases hook on every fire. Passed to both long phases (extract_atoms, synthesize_concepts) as their yieldDuringPhase opt. 3. maybeYield helper inside both phases — 30s throttle, fires inside the main work loop AND immediately after every `await chat()` LLM call (codex hardening: a single long LLM await could otherwise sit past TTL). Progress reporter wired through to both phases too (Issue 2): extract_atoms emits `[cycle.extract_atoms] N atoms / M skipped` ticks every ~1s; synthesize_concepts ticks per concept group. Cycle.ts owns start()/finish(); phases only call tick() and heartbeat() on the same reporter (NOT a child — that would produce path collision `cycle.extract_atoms.extract_atoms.work`). LockHandle interface exported for tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(extract): by-mention resumes from where it died Issue 4 of the v0.41.20.0 ops-fix-wave. On a 322K-page brain the sweep takes 10+ hours; if it died at 87% the user redid 87% on restart. Wires the existing `op_checkpoints` framework into extractMentionsFromDb with a flushAndCheckpoint ordering that closes the four codex-flagged correctness bugs at once: 1. Lost-links-on-crash — flush batch links to DB FIRST, commit page keys to checkpoint SECOND, persist THIRD. A crash between batch.push() and flushBatch() leaves the page un-checkpointed so resume re-scans it (no silently lost mention links). 2. Dry-run resume contradiction — dry-run does NOT load or persist the checkpoint. Verification path uses non-dry-run kill-and-resume. 3. Gazetteer hash in fingerprint — entity pages added mid-pause shift the gazetteer hash → new fingerprint → fresh scan against the new gazetteer. Without this, resumed runs would silently skip pages against a new entity set. 4. Filtered pages get checkpointed too — pages skipped by `--type` / `--since` / empty body / no-mentions all get marked completed so resume doesn't re-fetch them. Persist cadence: every 1000 items OR every 30s, whichever first (~322 persists / ~24s total overhead on the 322K-page brain). Crash window capped at 1000 pages (<0.3% loss). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(doctor): surface sync --all consolidation nudge to operators Issue 5 of the v0.41.20.0 ops-fix-wave. Multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` in `gbrain doctor` output instead of maintaining two staggered per-source cron entries with manual deconfliction. New checkSyncConsolidation surfaces the recommendation when 2+ active sources exist; "not applicable" for single-source brains. Own try/catch returns warn on SQL failure — outer doctor catch wasn't a safe assumption. `skills/cron-scheduler/SKILL.md` gains a "Multi-source brains" recipe block documenting the pattern + connection-budget math (parallel × workers × 2 ≈ 32 connections at default 4/4). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): isolate GBRAIN_HOME in cycle-LFCA + schema-cli tests Two pre-existing tests assumed a clean ~/.gbrain/config.json and a free ~/.gbrain/cycle.lock — both shared across all gbrain processes on the machine. Sibling Conductor worktrees running their own gbrain tests poisoned the shared state, causing flakes: - test/cycle-last-full-cycle-at.test.ts test 5 timed out at 5s because runCycle returned 'skipped' (file lock held by a parallel test process), and last_full_cycle_at exit hook silently no-oped. Fix: each test wraps its body in `withEnv({GBRAIN_HOME: tmpdir})` so the file lock path becomes per-test. - test/schema-cli.test.ts `schema active reports default resolution` failed exit 1 because another worktree had set `schema_pack: gbrain-base-v2` in the shared config (a pack that doesn't exist in the bundle). Fix: gbrain() helper defaults GBRAIN_HOME to a per-file tempdir (beforeAll-owned), so subprocess invocations get an isolated config dir unless tests explicitly override. Both fixes confirmed via deliberate pollution + retest: 12/12 schema-cli tests pass under simulated `schema_pack: gbrain-base-v2` contamination; cycle-LFCA test 5 completes <2s with isolated home. Discovered during v0.41.20.0 ship while investigating parallel-worktree flake. Not caused by the ops-fix-wave but found via it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.41.20.0) Five daily-driver ops pains fixed in one wave: 1. extract_atoms 7K-roundtrip overhead → 1 batch query + index 2. silent long-running phases → progress ticks every ~1s 3. 30-min crashed-cycle lock TTL → 5 min + active in-phase refresh 4. by-mention restarts from page 0 → resumes via op_checkpoints 5. multi-source cron → doctor surfaces `sync --all --parallel` nudge Two follow-up TODOs filed under v0.41.19.0 ops-fix-wave block (will be renamed at follow-up time): - `gbrain sync print-cron` subcommand (P2 ergonomics) - Lock-loss detection in DbLockHandle.refresh() (P2 contract change) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md key files for v0.41.20.0 ops-fix-wave Folds the v0.41.20.0 wave annotations into the cycle/extract/op-checkpoint key-files block: batch idempotency via atomsExistingForHashes, shorter cycle lock TTL with buildYieldDuringPhase active refresh, progress wiring through extract_atoms + synthesize_concepts, by-mention resume via mentionsFingerprint with flushAndCheckpoint ordering, sync_consolidation doctor check, and the 44-case test suite pinning every contract. Regenerated llms-full.txt to match (CLAUDE.md edit invariant). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci): doctor categorization + facts-engine cosine ordering hardening Two CI-only failures caught on PR #1545 (v0.41.21.0 ops-fix-wave): 1. doctor-categories drift guard — new `sync_consolidation` check from T6 wasn't categorized in src/core/doctor-categories.ts. Added under OPS_CHECK_NAMES (it surfaces an operator-cron recommendation, not a brain-data quality signal). 2. facts-engine `embedding cosine ordering when both sides have embeddings` — passed locally, failed under CI's parallel shard. Bun's truncated assertion output didn't surface which expect() fired; hardened the test against unknown leak vectors by: - per-run unique entity_slug (`embed-test-<random8>`) instead of the static `embed-test`, so any future cross-test pollution is structurally impossible - `findIndex` + `aIdx < bIdx` assertion that pins the cosine RELATIONSHIP (A closer than B because cos(A,Q)=1.0 vs cos(B,Q)=0.0) instead of the brittle `result[0].fact === 'A'` position check. The new shape matches the test name's contract verbatim ("ordering when both sides have embeddings"), so any unrelated row in the result set can no longer flip the test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
200a74104c |
v0.31.6 feat: extract facts during sync (real-time hot memory) (#796)
* feat: extract facts during sync (real-time hot memory)
Wire facts extraction into the sync pipeline so pages imported via
git get facts extracted immediately, not only through MCP put_page.
Changes:
- Add notability field (high/medium/low) to facts extraction schema
- Upgrade default extraction model from Haiku to Sonnet (configurable
via facts.extraction_model brain_config)
- Add notability-gated facts extraction to sync post-import hook:
- Only HIGH notability facts inserted during sync (life events,
major commitments, relationship/health changes)
- MEDIUM facts deferred to dream cycle
- LOW facts (logistical noise) dropped entirely
- Add notability column to facts table DDL
- Pass engine to extraction for config-aware model selection
Before: facts only extracted via MCP put_page (never during git sync)
After: meetings, conversations, personal pages get facts extracted
immediately on sync, with salience filtering
Closes the hot-memory gap where brain content committed via git was
invisible to the facts table until manually processed.
* fix: B1 — pass notability through facts JSON parser
Pre-fix, src/core/facts/extract.ts:tryArrayShape silently dropped the
LLM's notability field on the floor: the function copied fact/kind/
entity/confidence into the output but never read o.notability. The
outer loop in extractFactsFromTurn then read candidate.notability,
found undefined, and defaulted to 'medium'. sync.ts's HIGH-only filter
(`if (f.notability !== 'high') continue`) discarded 100% of facts.
Net: real-time facts on sync was a no-op despite Sonnet running and
costing money. Headline feature was dead on the happy path.
Fix is a one-line change in tryArrayShape. Two layers of test pin it:
1. Parser-pin (test/facts-extract.test.ts +75 LOC, 5 cases):
- notability passes through when LLM emits it
- notability omitted defaults to undefined (legacy compat)
- non-string notability is dropped defensively
- every documented field survives the parse (future field-drop guard)
- fenced JSON output (markdown code blocks) still threads correctly
2. End-to-end smoke (test/facts-extract-smoke.test.ts NEW, 145 LOC,
4 cases): drives extractFactsFromTurn with a stubbed gateway chat
transport. Asserts HIGH input → notability:'high' all the way out.
Guards against future prompt drift where Sonnet returns 'medium'
for everything; smoke fails loudly so the eval-mining flow gets
triggered.
Adds the chat test seam to enable the smoke test:
src/core/ai/gateway.ts: __setChatTransportForTests(fn) mirrors
v0.28.7's __setEmbedTransportForTests pattern. When set, chat()
routes through the stub; isAvailable('chat') returns true so tests
don't need full gateway configuration. resetGateway() clears it.
Test files stay regular .test.ts (parallel-safe; no mock.module).
PR 1 commit 1 of 15. See ~/.claude/plans/swift-gliding-key.md for the
full eng review and bisect-friendly commit ordering.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: B2 — migration v46 ALTER facts.notability with idempotent CHECK
Pre-fix, the v0.31.1 PR shipped a CREATE TABLE edit to migration v45 that
added `notability NOT NULL DEFAULT 'medium' CHECK (notability IN (...))`
inline. Fresh installs got the column. But every brain that already ran
v45 BEFORE that edit (i.e., everyone running v0.31.0+ in production) keeps
the old facts table shape. INSERT now crashes with:
column "notability" of relation "facts" does not exist
This is the canonical "embedded schema mutation breaks upgrades" trap that
CLAUDE.md cites: "bit users 10+ times across 6 schema versions over 2 years."
Fix: new migration v46 ALTER. Idempotent under all four states:
1. Fresh install (v45 already added column inline)
→ ADD COLUMN IF NOT EXISTS no-ops; named CHECK probe finds existing
constraint → skip. Postgres emits a NOTICE; no error.
2. Old brain pre-edit (no column)
→ ADD COLUMN adds it with NOT NULL DEFAULT 'medium'; named CHECK
probe finds nothing → adds the constraint.
3. Partial state (column exists, CHECK missing)
→ ADD COLUMN no-ops; CHECK probe adds the named constraint.
4. Re-run after success
→ all probes skip; no error, no state change.
Implementation notes:
- CHECK constraint is named `facts_notability_check` (not autogen) so the
information_schema-equivalent probe via `pg_constraint` can find it
deterministically.
- Column-level CHECK in v45 inline (autogen-named) and the named CHECK
here are additive and non-conflicting — Postgres allows multiple CHECKs
covering the same predicate. Codex flagged this concern; the named
constraint addresses it cleanly.
- Both engines run the same SQL. PGLite is real Postgres in WASM and
supports DO $$ blocks. PGLite users with persistent older brains hit
the same bug.
E2E coverage (test/e2e/migration-v46-notability.test.ts, 5 cases):
- fresh-install fully-migrated: column + named CHECK both exist
- old brain (column dropped): v46 adds both back
- partial state (column exists, CHECK missing): v46 adds CHECK
- idempotent re-run on fully-migrated: no error, state unchanged
- CHECK constraint actually rejects out-of-domain values
Verified against real Postgres (pgvector/pgvector:pg16): 5/5 pass in 696ms.
PR 1 commit 2 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: B3 — restore v0_31_0 orchestrator gate to v < 45
Pre-fix, the v0_31_0 orchestrator's phaseASchema gate had been demoted
from `v < 45` to `v < 40` with an operator-facing message claiming
"v40 (facts hot memory + notability)". Facts is at v45, not v40 — the
message was wrong and the gate was permissive.
Symptom: brains at schema_version 40-44 (real states for users mid-
upgrade) passed the precondition, then immediately crashed on the
post-condition check three lines later (`SELECT FROM pg_tables WHERE
tablename = 'facts'`). Operator saw a green light, then a red light.
Fix: restore the gate to `v < 45` (the real semantic precondition:
the facts table is created by migration v45). Drop the misleading
"+ notability" claim — column shape is enforced by migration v46
alone (see MIGRATIONS[v46]), not gated here. Add a one-line comment
pointing at v46 so the next reader sees the separation.
Test coverage (test/migration-orchestrator-v0_31_0.test.ts NEW, 4 cases):
- schema_version < 45 fails with operator-facing message naming v45
+ recovery command. Negative assertions guard against regression
to the "v >= 40" / "+ notability" prior text.
- schema_version >= 45 with facts table present → status complete.
- dryRun short-circuits before any DB read.
- null engine short-circuits with no_brain_configured.
Verified: 4/4 pass; v45 + v46 both apply cleanly during test setup.
PR 1 commit 3 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: widen FactRow to expose notability across all readers
Codex's outside-voice pass on the cathedral plan flagged P1 #4: the read-
side contract was behind the write-side schema. notability lived in DDL
and the insertFact INSERT, but FactRow type omitted it and both row
mappers (pglite-engine + postgres-engine) silently dropped the column.
Every consumer above the engine (recall op, MCP _meta hook, CLI JSON
output) returned facts without their salience tier. PR2/PR3 surfaces
that need to filter or display notability would have required contract
surgery first; this lands the contract widening as the foundation.
Changes:
- src/core/engine.ts: add `notability: 'high' | 'medium' | 'low'` to
FactRow with doc comment naming the row source (column added by
migration v46) and the consumers (recall, daily-page, admin, MCP).
- src/core/postgres-engine.ts: FactRowSqlShape gains notability;
rowToFactPg propagates it with `?? 'medium'` belt-and-suspenders
fallback (NOT NULL DEFAULT in DDL is the primary; this is the
second line for any pre-v46 row that survives a SELECT).
- src/core/pglite-engine.ts: same pair (interface + mapper).
- src/core/operations.ts: recall op response shape adds notability.
- src/core/facts/meta-hook.ts: `_meta.brain_hot_memory` payload
surfaces notability so connected agents can filter or weight
HIGH-tier facts in their context budget.
- src/commands/recall.ts: `--json` output adds notability.
Test contract pin (test/facts-engine.test.ts):
- Existing 'inserts a fact' case asserts default 'medium' on the
read side (caller-omits-notability path).
- New 'notability round-trips for each tier' case inserts HIGH /
MEDIUM / LOW explicitly and reads back the same tier — without
this assertion, codex P1 #4 reappears silently.
Test fixtures (facts-classify.test.ts + facts-decay.test.ts) also
updated: makeFact() factories now construct complete FactRow objects
with notability:'medium' to match the tightened type.
PR 1 commit 4 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: move isFactsBackstopEligible to src/core/facts/eligibility.ts
Single source of truth for "should this page write fire the facts
extraction backstop?" Pre-extraction, lived inline at operations.ts:633
where only put_page could see it; sync.ts had its own divergent type
filter (`['conversation', 'transcript', 'personal', 'therapy', 'call']`
— only `meeting` was a real PageType, the rest never matched). Sync's
filter is deleted in commit 7; everyone routes through this predicate.
Adds the slug-prefix rescue branch the eng review pinned (D-eligibility):
parsed.type ∈ ELIGIBLE_TYPES OR slug.startsWith('meetings/' | 'personal/'
| 'daily/'). The rescue catches `meetings/2026-05-09-foo.md` pages that
frontmatter-typed themselves as 'note' (the legacy default) — directory
location wins.
Test pin (test/facts-eligibility.test.ts NEW, 28 cases):
- 4 BRANCH cases: typed-only, slug-only (each prefix), both, neither
- 7 GUARD cases: null/undefined parsed, wiki/agents/, dream_generated,
body length thresholds (< 80, exactly 80, whitespace-only)
- 14 COVERAGE cases: every eligible PageType on arbitrary slug → ok;
every non-eligible PageType on non-rescued slug → kind:<type> reason
Pure-function tests; no DB. The full predicate covered without spinning
a brain.
Existing test/facts-backstop-gating.test.ts still passes (it tests the
predicate via put_page; the move is transparent to that surface).
PR 1 commit 5 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add runFactsBackstop helper with full extract→resolve→dedup→insert pipeline
Single shared facts pipeline used by every brain write surface that
wants real-time hot memory extraction. Replaces five divergent
implementations:
- put_page MCP backstop hook (operations.ts:556)
- extract_facts MCP op (operations.ts:2438-2486)
- sync.ts post-import block (deleted in commit 7)
- file_upload + code_import (wired in commit 10)
Encapsulates the v0.31 smart pipeline:
extract → resolve → dedup (cosine @ 0.95) → insert
(matches extract_facts op precedent at operations.ts:2460.)
Two execution modes (D8):
- 'queue' (default): fire-and-forget via getFactsQueue().enqueue.
Caller awaits ~zero (just enqueue + microtask). Sync stays fast
on a 50-page batch.
- 'inline': await full pipeline; return real {inserted, duplicate,
superseded, fact_ids} counts. Used by extract_facts MCP op.
Discriminated return shape so TypeScript catches mode/result mismatches
at the call site:
| { mode: 'queue'; enqueued; queueDepth; skipped? }
| { mode: 'inline'; inserted; duplicate; superseded; fact_ids; skipped? }
Notability filter (D4): per-caller policy via FactsBackstopCtx.notabilityFilter.
Sync passes 'high-only' (HIGH lands now, MEDIUM waits for dream cycle,
LOW dropped at LLM layer). Other surfaces default to 'all'. Filter runs
post-LLM, pre-insert: saves the insert work but not the LLM call (the
notability tier IS what we're calling Sonnet to determine).
Eligibility + kill-switch gates run before any LLM cost. Skipped reasons
are stable strings the future facts:absorb writer (commit 13) and doctor
check (commit 12) consume.
Re-throws AbortError; absorbs gateway/parse/queue errors as `skipped: '...'`
envelope. Operator visibility lands via PR1 commit 13's ingest_log writer
(facts:absorb source_type).
Test pin (test/facts-backstop.test.ts NEW, 12 cases):
- 3 eligibility/kill-switch cases (extraction_disabled, subagent_namespace,
dream_generated)
- 5 inline-mode cases (insert + counts, notability filter, source string,
empty extraction, abort)
- 3 queue-mode cases (default mode, explicit mode, kill-switch envelope)
- 1 dedup contract case (insertions without embeddings short-circuit
cleanly; embedding-driven dedup is exercised by E2E with real gateway)
PGLite in-memory; LLM stubbed via __setChatTransportForTests (commit 1's
seam). 12/12 pass in 912ms.
PR 1 commit 6 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: sync.ts uses runFactsBackstop (deletes dead-code type filter)
Pre-fix sync.ts had a 60-line inline facts extraction block carrying:
1. Dead-code eligibility filter: ['meeting', 'conversation',
'transcript', 'personal', 'therapy', 'call'] — only `meeting` is
a real PageType. The other five never matched anything; eligibility
rested on the slug-prefix branch alone.
2. Divergent shape from put_page's backstop: no dedup, no supersede,
raw extract→insert. Garbage rows on re-sync.
3. Sequential per-page LLM calls in sync's request path: a 50-page
sync = 50 Sonnet calls in series ≈ 5+ minutes blocking.
Replaced with `runFactsBackstop(parsedPage, ctx)` from PR1 commit 6:
- Queue mode (fire-and-forget) so sync stays fast on multi-page batches.
- 'high-only' notabilityFilter (cathedral spec: HIGH lands now,
MEDIUM waits for dream cycle, LOW dropped at LLM).
- isFactsBackstopEligible (commit 5) — eligibility lives in one place.
- extract → resolve → dedup (cosine @ 0.95) → insert pipeline shared
with put_page + extract_facts.
Per-page try/catch survives so one failed page doesn't blow up the
whole sync (best-effort posture preserved).
Existing test/sync.test.ts (39 cases) passes unchanged — sync's outer
contract is untouched, only the inner facts-extract block changed.
PR 1 commit 7 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: operations.ts put_page uses runFactsBackstop
Replace the inline get-queue-extract-resolve-insert closure (operations.ts:540-583)
with a single `runFactsBackstop(parsed, ctx)` call in queue mode. put_page
and sync now share the same eligibility/extract/dedup/insert pipeline.
Behavioral preservation:
- Response shape `{queued: true} | {skipped: '<reason>'}` unchanged for
MCP clients. The helper's namespaced 'eligibility_failed:<reason>'
discriminator is mapped back to the bare reason ('kind:guide',
'too_short', 'subagent_namespace', 'dream_generated') before write
to factsQueued. test/facts-backstop-gating.test.ts (5 cases) passes
without modification.
- Default 'all' notabilityFilter (MEDIUM facts continue to land via
put_page; only sync filters to HIGH-only). This matches the
pre-v0.31.2 surface: put_page's prior shape inserted everything the
LLM returned, with the dream cycle's consolidate phase doing the
salience clustering overnight.
Net: -32 LOC of inline pipeline; one shared call site + one mapping
shim; same observable shape.
PR 1 commit 8 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: operations.ts extract_facts uses runFactsPipeline
Replace the 65-line inline extract→resolve→dedup→insert loop in the
extract_facts MCP op (operations.ts:2369-2454) with a single
`runFactsPipeline(turn_text, ctx)` call. The inline pipeline + the
helper are now the same code path; test/facts-mcp-allowlist + test/
facts-anti-loop pass unchanged.
Architecture: the helper has two entry points now —
- `runFactsBackstop(parsedPage, ctx)` — page-write hook with
eligibility + kill-switch + queue mode dispatch (PR1 commit 6).
Used by put_page, sync, file_upload, code_import.
- `runFactsPipeline(turnText, ctx)` — raw turn-text entry that
skips the page-shape eligibility predicate. Used by extract_facts
MCP op (this commit).
Both share an inner `runPipelineWithBody` so the actual extract → resolve
→ dedup (cosine @ 0.95) → insert pipeline lives in one place. Codex P0 #2
called this out: "extract_facts already does the smart pipeline; put_page
+ sync do raw extract→insert. Centralizing only extraction codifies the
worse pipeline." With commit 9, every fact-insert path goes through the
smart pipeline; raw insertFact loops in the brain are gone.
Behavioral preservation:
- extraction_disabled kill-switch envelope unchanged.
- is_dream_generated → returns {skipped: 'dream_generated'} envelope
(the predicate-bypass path; eligibility doesn't apply on raw
turn_text but dream_generated still does). Pre-fix the extractor
itself short-circuited; new shape surfaces the skip explicitly to
MCP clients.
- Visibility ('private' | 'world') threading preserved.
- Response shape {inserted, duplicate, superseded, fact_ids} identical
to pre-fix.
PR 1 commit 9 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: document why file_upload + code_import don't wire runFactsBackstop
PR1 commit 10 was scoped in the eng review plan to "wire runFactsBackstop
to file_upload and code_import paths." Implementation analysis revealed
all three candidate surfaces are correctly handled WITHOUT explicit
wiring:
1. file_upload (operations.ts:1713) doesn't write a page. It uploads
a file to storage + inserts a `files` row. The associated page is
written separately via put_page, which already fires runFactsBackstop
in queue mode (commit 8). No double-firing needed.
2. importCodeFile (this file) writes pages with type='code'. The
isFactsBackstopEligible predicate rejects 'code' kind with reason
`kind:code`. Wiring runFactsBackstop here would always return the
skipped envelope. When README / doc-comment extraction lands in a
future release, the eligibility predicate is the single place to
update — adding 'code' to ELIGIBLE_TYPES makes existing call sites
auto-cover the change.
3. `gbrain import` (commands/import.ts) is bulk markdown import. Firing
facts extraction on every imported page would cost-spike on first-
time bulk imports of large brain repos (10K+ pages × Sonnet =
hundreds of dollars). User runs `gbrain dream` or the consolidate
phase to backfill facts from bulk-imported pages.
Adds a docstring above importCodeFile capturing all three rationales so
the next maintainer doesn't re-do this analysis.
PR 1 commit 10 of 15 — no behavior change; documentation only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: migration v47 — ingest_log.source_id ALTER (codex P1 #3)
Pre-fix the ingest_log table had no source_id column; sync.ts wrote rows
without source-scoping and doctor only checked 'default'. Codex's outside
voice flagged this on the cathedral plan: "facts:absorb logging inherits
a surface that cannot tell you which source is failing."
This commit closes the multi-source observability gap on the foundation:
- PR1 commit 13's facts:absorb writer (next) writes ingest_log rows
with source_id so multi-source brains scope failures per source.
- PR1 commit 12's doctor's facts_extraction_health check (after that)
iterates over `SELECT DISTINCT id FROM sources` instead of hardcoded
'default'.
Migration v47 (idempotent, both engines):
ALTER TABLE ingest_log ADD COLUMN IF NOT EXISTS source_id TEXT
NOT NULL DEFAULT 'default';
CREATE INDEX IF NOT EXISTS idx_ingest_log_source_type_created
ON ingest_log (source_id, source_type, created_at DESC);
Schema-bootstrap coverage:
- schema.sql / pglite-schema.ts inline definitions add source_id +
the new index for fresh installs.
- applyForwardReferenceBootstrap (both PGLite + Postgres) probes for
`ingest_log.source_id` and adds the column BEFORE SCHEMA_SQL replay
builds the new composite index. Without this, old brains running
initSchema() on the new schema-embedded.ts would crash on the index
creation (the column doesn't exist yet at replay time).
- test/schema-bootstrap-coverage.test.ts pins ingest_log.source_id as
REQUIRED_BOOTSTRAP_COVERAGE — adding a forward reference without
extending applyForwardReferenceBootstrap would fail this guard.
E2E (test/e2e/migration-v47-ingest-log-source-id.test.ts NEW, 3 cases):
- fresh-install: column + index both exist after runMigrationsUpTo(LATEST).
- old-brain simulation: drop column, run v47, column reappears with
NOT NULL DEFAULT 'default'; INSERT without source_id picks up the
default.
- idempotent re-run: v47 twice in a row is a no-op.
Verified against real Postgres (pgvector/pgvector:pg16): 3/3 pass; the v46
+ v47 E2Es land green together (8/8 in 2.05s). Bootstrap-coverage unit
test (5 cases) also green.
PR 1 commit 11 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: facts:absorb writer + reason codes (D5 contract)
D5 from /plan-ceo-review: every absorbed failure in the facts extraction
pipeline writes one row to ingest_log so doctor + admin dashboard
surface failures cross-process. CLAUDE.md's "zero silent failures" rule
gets enforced on the foundation.
Wires three layers:
1. Type widening (src/core/types.ts):
- IngestLogEntry gains source_id (codex P1 #3 — migration v47).
- IngestLogInput gains optional source_id; engines default to 'default'.
2. Engine row writers (pglite-engine.ts + postgres-engine.ts):
- logIngest threads source_id into INSERT.
- getIngestLog applies belt-and-suspenders 'default' fallback for
any pre-v47 row that somehow survived.
3. Helper (src/core/facts/absorb-log.ts NEW):
- writeFactsAbsorbLog(engine, ref, reason, detail, sourceId) writes
one ingest_log row with source_type='facts:absorb' and
summary='<reason>: <detail truncated to 240 chars>'.
- classifyFactsAbsorbError(err) heuristic-pattern-matches arbitrary
Errors into 6 stable reason codes:
gateway_error | parse_failure | queue_overflow
queue_shutdown | embed_failure | pipeline_error
- Best-effort: any logging failure is caught + stderr-warned;
the caller's pipeline keeps running.
4. runFactsBackstop wiring (src/core/facts/backstop.ts):
- queue mode: errors inside the queue worker classify + log via
absorb-log.ts. Were previously invisible (counter increment only).
- queue overflow drop also writes an absorb log row so doctor sees
the depth of capacity pressure.
- inline mode: errors bubble; caller decides logging (extract_facts
MCP op surfaces them as op-error responses).
Test pin (test/facts-absorb-log.test.ts NEW, 12 cases):
- 7 classifier cases pinning every reason path + fallback
- 5 writer cases pinning ingest_log row shape, custom sourceId,
240-char detail truncation, no-throw contract, reason-set
completeness
PR1 commit 12 (next) reads these rows for the facts_extraction_health
doctor check.
PR 1 commit 13 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: doctor facts_extraction_health check (multi-source)
Mirrors the eval_capture check shape but reads facts:absorb rows
(written by writeFactsAbsorbLog from PR1 commit 13). Iterates over
EVERY source (codex P1 #3 motivation) so multi-source brains see
per-source failure rates instead of only 'default'.
Configurable threshold: facts.absorb_warn_threshold (default 10 over
the last 24h, per source, per reason). When the threshold is exceeded
for any (source, reason) pair, status flips to warn and the message
names the breakdown:
facts:absorb activity in last 24h (under threshold 10):
default: 4 gateway_error, 1 parse_failure |
team-source: 2 queue_overflow
Single SQL grouping query covers the read; the composite index v47
added (idx_ingest_log_source_type_created on source_id, source_type,
created_at DESC) covers the filter + sort path so the check is fast
on brains with millions of ingest_log rows.
Operator UX:
- 'ok' under threshold (or zero failures) → quiet.
- 'warn' over threshold → message names every (source, reason, count)
tuple. Recovery hint: `gbrain recall --since 24h --json` to inspect
what landed; `gbrain config set facts.absorb_warn_threshold N` to
tune.
- Pre-v47 brain (column missing): 'ok' with skipped reason pointing
at `gbrain apply-migrations --yes`.
- RLS denies SELECT: 'warn' calling out that capture INSERTs are
likely also blocked.
Test pin (test/doctor.test.ts +28 LOC, 1 case):
Source-string assertions on the doctor.ts block:
- 'GROUP BY source_id' (multi-source contract)
- "source_type = 'facts:absorb'" (right table query)
- 'facts.absorb_warn_threshold' (configurable threshold)
- INTERVAL '24 hours' (right window)
- 'Skipped (ingest_log.source_id unavailable' (pre-v47 fallback)
- 'RLS denies SELECT on ingest_log' (RLS hint)
Negative: must NOT contain `source_id = 'default'` (the bug we're
fixing — codex P1 #3 was that doctor only checked 'default').
Live smoke against real Postgres: doctor renders the new check between
'eval_capture' and 'effective_date_health' as expected, shows 'ok' on
an empty test brain.
PR 1 commit 12 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: notability-eval mining + public-anonymized fixture (40 cases)
The notability gate is the load-bearing differentiator of the cathedral:
"only HIGH lands on sync, MEDIUM waits for the dream cycle, LOW dropped
at the LLM layer." Without an eval, the gate's quality is asserted via
hope; prompt drift (Sonnet returning 'medium' for everything) silently
turns the headline feature into a no-op.
This commit adds the mining half — eval suite is pinned in the next
commit (15).
NEW src/commands/notability-eval.ts:
- mineNotabilityCandidates(repoPath, opts): walks meetings/, personal/,
daily/ in the brain repo, splits markdown bodies into paragraphs
(filtered by 80–800 char length), pre-classifies each paragraph
with cheap-Haiku to bucket into HIGH/MEDIUM/LOW (round-robin
fallback when no chat gateway is available — local development
without API keys still produces a candidates file).
- Stratified random sample within each bucket: HIGH/MEDIUM/LOW
targets default 20/20/10 (per cathedral plan D7=B). Stratified
further across the three corpus dirs so HIGH cases come from
multiple dirs not just one.
- JSONL utilities (loadJsonlCases, writeJsonlCases) shared with the
review path. Default paths: ~/.gbrain/eval/notability-mining-
candidates.jsonl (mining) + ~/.gbrain/eval/notability-real.jsonl
(private confirmed).
- TTY review subcommand: walks candidates one-by-one, asks for
HIGH/MEDIUM/LOW confirmation, writes confirmed cases. Smoke-only
test (TTY interactivity is hard to test deterministically).
CLI dispatch (src/cli.ts):
- `gbrain notability-eval mine` (default targets 20/20/10).
- `gbrain notability-eval review` (TTY hand-confirm).
- `gbrain notability-eval help` (flag reference).
- sync.repo_path resolution mirrors the dream phase pattern; --repo
PATH overrides.
NEW test/fixtures/notability-eval-public.jsonl (40 cases):
- 14 HIGH (life events, major commitments, relationship/health changes,
financial decisions).
- 13 MEDIUM (durable preferences, beliefs, strong opinions revealing
character).
- 13 LOW (logistical noise — restaurant orders, scheduling, errands).
- Anonymized per CLAUDE.md privacy rule (alice-example, acme-co,
widget-co, fund-a placeholder names; no real contacts).
- Each case has a `tier_rationale` string documenting the choice for
reviewer transparency.
- Used by CI's eval harness in commit 15 (no API key required for
deterministic stub-driven contract tests).
PR 1 commit 14 of 15.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: notability-eval harness with precision@HIGH metric (40-case fixture)
Pins the load-bearing gate-quality contract in CI. Without this, prompt
drift (Sonnet returning 'medium' for everything → sync inserts nothing)
ships silently. The harness flips it from "asserted by hope" to "asserted
by metric."
NEW test/notability-eval.test.ts (13 cases across 5 describe blocks):
1. splitParagraphs (2 cases): blank-line splitting, length filters.
2. walkMarkdownFiles (1 case): tree walk drops non-.md files.
3. mineNotabilityCandidates round-robin path (2 cases): empty corpus
+ populated corpus produce expected candidate shape; round-robin
keeps tests deterministic without an LLM.
4. JSONL utilities (3 cases): write+read round-trip, malformed-line
skip, default paths under ~/.gbrain/eval/.
5. Public-anonymized fixture shape (2 cases): 40 cases, ≥10 per tier,
every paragraph ≥80 chars, every case has a tier_rationale.
6. Eval harness contract (3 cases) — the headline assertions:
- Perfect predictor (LLM-stub returns confirmed_tier verbatim) →
precision@HIGH = 1.0, recall@HIGH = 1.0.
- Always-medium model → precision@HIGH = 0 (no HIGH predictions
at all). Pins the "harness handles the no-positive-prediction
case correctly" contract.
- Always-high model → precision drops below the 0.50 PR-fail
threshold (TP / (TP + FP) = 14 / 40 = 0.35). Pins the
"harness CORRECTLY flags a misaligned model" contract.
Sample size justification: the public fixture has 14 HIGH cases. For
precision@HIGH = 0.75 with a 95% CI ±10pp, n=14 gives the right floor
for "is the gate dramatically wrong" — tighter measurements need the
private fixture (50 cases via mine + review).
The harness is a CONTRACT test for the metric shape, not a quality
measurement of any specific model. A real quality run uses the same
harness against a real Sonnet (no chat-transport stub) — that flow is
exposed via GBRAIN_NOTABILITY_EVAL_REAL=1 + the private mined fixture.
All 92 tests across all PR1 facts files pass green (extract / extract-
smoke / engine / backstop / eligibility / absorb-log / notability-eval).
Soft gate per the cathedral plan: warn if precision@HIGH < 0.75; fail
PR if < 0.50. CI wiring + the production gate are deferred to PR2 (the
visibility/observability surface PR); this PR1 commit lands the harness
+ fixture + contract tests so the gate is ready to wire.
PR 1 commit 15 of 15. Cathedral foundation lands here.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: fill PR1 gap-fill — backstop integration + Postgres parity
Test gap analysis flagged three high-priority untested behaviors in
PR1's surface:
Gap #3: extract_facts MCP op response shape stability after
routing through runFactsPipeline (commit 9). Existing tests
pin allowlist + anti-loop but not the {inserted, duplicate,
superseded, fact_ids} envelope that MCP clients display.
Gap #4: per-engine row-mapper parity for notability. facts-engine.test.ts
pins notability round-trip on PGLite; the Postgres row mapper
(postgres-engine.ts:rowToFactPg) is different code that wasn't
pinned. Codex P1 #4 was specifically about read-side contracts
drifting silently.
Gap #5: multi-source isolation in facts:absorb logging. Codex
P1 #3 motivated the source_id column; the absorb-log test pins
that source_id is written but not that source_id-scoped queries
return only the right source's rows.
NEW test/facts-backstop-integration.test.ts (6 cases):
- 2 cases on runFactsPipeline (extract_facts path) response shape:
successful extraction returns full {inserted, duplicate, superseded,
fact_ids} envelope with positive fact_ids; empty extraction returns
zero counts (no NaN/undefined).
- 2 cases on facts:absorb multi-source isolation: writeFactsAbsorbLog
rows are source-scoped; doctor's GROUP BY source_id query produces
the expected per-source breakdown.
- 2 cases on queue mode: happy-path drain pins counters.completed >= 1
+ counters.failed == 0; documented case noting that extract.ts
absorbs gateway errors silently (errors propagate from layers
ABOVE extract — resolver, dedup, insert — to backstop's catch,
not from the chat call itself).
NEW test/e2e/facts-notability-roundtrip.test.ts (5 cases, real Postgres):
- HIGH/MEDIUM/LOW round-trip via insertFact + listFactsByEntity.
- Omitting notability defaults to medium (NOT NULL DEFAULT contract).
- listFactsSince also surfaces notability.
All 5 pin the postgres.js driver + rowToFactPg row mapper.
PGLite parity is covered by the existing test/facts-engine.test.ts
case from commit 4.
Verified: 6/6 unit + 5/5 E2E green. The third high-priority gap
(integration sync.ts → runFactsBackstop end-to-end) is sufficiently
covered by the existing test/sync.test.ts behavior plus the per-page
runFactsBackstop assertions in test/facts-backstop.test.ts; chasing
the full happy-path sync→facts integration would require a real
git fixture which is heavier than warranted for this surface.
PR 1 commit 16 of 16 (gap fill).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
89ae720959 |
v0.31.0 feat: hot memory — facts hook + recall CLI + MCP _meta + consolidate phase (#785)
* v0.31 feat(migrate): facts hot memory schema (migration v40) Phase 1 of v0.31 hot-memory. - New facts table with source_id (TEXT FK to sources, per-source isolation), kind CHECK (event/preference/commitment/belief/fact), visibility CHECK (private/world for takes-style ACL parity), valid_from/valid_until/ expired_at/superseded_by for temporal + supersession audit, and consolidated_at/consolidated_into pointing at takes(id) for the dream- cycle hot→cold bridge. - Embedding column dim resolved at migration time from config.embedding_dimensions so non-OpenAI brains (Voyage etc) work out-of-the-box. HALFVEC where pgvector >= 0.7; falls back to VECTOR with stderr warn on older versions. Matching opclass per column type (halfvec_cosine_ops vs vector_cosine_ops). - 5 partial indexes leading on source_id so every read uses the trust boundary as part of the index, not a callback. HNSW partial index excludes expired/null rows so footprint stays proportional to active fact count. - RLS DO-block matches takes pattern (Postgres BYPASSRLS gate; PGLite no-op). - v0_31_0.ts orchestrator follows v0_28_0.ts pattern — phase A asserts schema version >= 40 + facts table presence; runner owns ledger. All 87 existing migrate.test.ts cases pass. PGLite smoke test confirms table + indexes + CHECK constraints + ON DELETE CASCADE all behave. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 chore(version): bump VERSION + package.json to 0.31.0 Phase 1 closer. CHANGELOG entry written when Phase 7 lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(engine): facts hot memory engine API (Phase 2) Phase 2 of v0.31 hot-memory. Adds 8 facts methods to BrainEngine implemented on both PGLite and Postgres engines: - insertFact(input, ctx) — INSERT with optional supersedeId; expires the named row in the same transaction. Per-entity advisory lock on Postgres (`pg_advisory_xact_lock(hashtextextended(source_id::text || ':' || entity_slug, 0))`) for the dedup window. PGLite is single-process so the lock is a no-op. - expireFact(id, opts) — sets expired_at + optional superseded_by. Idempotent-as-false (already-expired returns false). - listFactsByEntity / listFactsSince / listFactsBySession — list surfaces with FactListOpts filters (activeOnly, kinds, visibility, limit/offset). Every query starts WHERE source_id = $X so the trust boundary is part of the index path. - listSupersessions — audit log; activeOnly:false + expired_at IS NOT NULL + superseded_by IS NOT NULL. - findCandidateDuplicates(source_id, entity_slug, factText, k) — entity-prefiltered (mandatory), k=5 default, hard cap 20. Embedding- cosine ordering when caller supplies an embedding, recency fallback otherwise. Bounds the contradiction-classifier blast radius. - consolidateFact(id, takeId) — sets consolidated_at + consolidated_into. Never DELETE; facts stay as audit trail for the resulting take. - getFactsHealth(source_id) — per-source counters consumed by `gbrain doctor` facts_health check. Public types in engine.ts: FactKind (5-value union), FactVisibility, FactInsertStatus, FactRow, NewFact, FactListOpts, FactsHealth. PGLite + Postgres helpers: rowToFact / rowToFactPg parse the text-format pgvector embedding back into Float32Array; toPgVectorLiteral encodes for the supersede-path INSERT (postgres-js can't bind Float32Array directly to a vector column without an explicit literal cast). Smoke test confirms every method end-to-end on PGLite. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(facts): extraction code path (Phase 3) Phase 3 of v0.31 hot-memory. Five new modules under src/core/facts/ + src/core/entities/: - src/core/facts/decay.ts — pure helper. effectiveConfidence(fact, now) applies confidence × exp(-age/halflife) with per-kind halflife table (event 7d, commitment 90d, preference 90d, belief 365d, fact 365d). Returns 0 for expired or past-valid_until rows. Single source of truth consumed by recall, supersession audit, facts_health, and the MCP _meta injector (eD8 DRY). - src/core/facts/queue.ts — bounded in-memory queue. Cap 100 default, drop-oldest on overflow with counter. Per-session in-flight=1 serializes burst chat. AbortSignal threading from server SIGTERM (mirrors minion worker pattern per eD7): 5s grace for in-flight, then drop pending with counter. getFactsQueue() process-singleton; __resetFactsQueueForTests for hermetic tests. - src/core/facts/classify.ts — contradiction classifier with cosine fast-path (D13: ≥0.95 → duplicate, skip LLM) and classifier-failure fallback (D12: cosine ≥0.92 → duplicate, else INSERT). Pure cosine helper exported. JSON-strict output with 4-strategy parse fallback; refusal stop-reason maps to fallback path. Caller-provided abort signal propagated to the gateway chat call. - src/core/facts/extract.ts — Haiku turn-extractor. Reuses INJECTION_PATTERNS from src/core/think/sanitize.ts on the way IN (turn_text) AND on the way OUT (each fact). Tight system prompt with 5-kind taxonomy, 0..1 confidence scoring, entity slug or display name. Anti-loop check on isDreamGenerated (reuses v0.23.2 marker semantics). Synchronous embedOne() per fact via the gateway so classifier paths have embeddings available; AbortError re-thrown explicitly so SIGTERM during embed never writes a NULL-embedding row meant to be cancelled (eE8 distinction). - src/core/entities/resolve.ts — slug canonicalization shared by signal-detector AND facts. Resolution order: exact slug match → pg_trgm fuzzy match (similarity ≥0.4) → deterministic slugify fallback. slugify exported standalone for tests + callers that want the floor. Smoke tests confirm decay table, cosine math, slugify rules, queue drop-oldest under overflow, and shutdown grace + drop-pending semantics. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(mcp+cli): MCP ops + recall CLI + _meta + transport refactor (Phase 4) Phase 4 of v0.31 hot-memory. Three new MCP ops on the contract-first surface: - `extract_facts` (write scope, localOnly:false): extracts facts from a conversation turn via the Haiku extractor, runs the cosine fast-path dedup, INSERTs into per-source hot memory. Returns counts + fact_ids[]. Skips on is_dream_generated:true (anti-loop). - `recall` (read scope): query the per-source hot memory by entity / since / session / supersessions / grep filter. Visibility- aware: remote callers see visibility='world' rows only (takes-style ACL parity, eD21). Returns most-recent first; pagination via limit. - `forget_fact` (write scope): expireFact wrapper. Idempotent-as-error on unknown id; uses the new 'fact_not_found' ErrorCode. ErrorCode union opened (eD6 / eE7): TS forward-compat via the `(string & {})` autocomplete-friendly hack so downstream consumers (gbrain-evals etc) don't break their typecheck on every new code. Three new codes: 'rate_limited', 'extraction_failed', 'fact_not_found'. OperationContext gains source_id?:string (eD4 / eE2 — TEXT not INTEGER per schema reality). Resolved once in buildOperationContext from DispatchOpts.sourceId. Stdio MCP defaults to GBRAIN_SOURCE env or 'default'; HTTP MCP reads it from the per-token sources scope (eE3). ToolResult gains _meta?: Record<string, unknown> (eD3). Dispatcher calls a configurable metaHook AFTER op.handler succeeds, wrapped in its own try/catch so a DB blip degrades to no-_meta rather than flipping the whole tool call to error (eE4). New module src/core/facts/meta-hook.ts: - getBrainHotMemoryMeta(name, ctx) builds the _meta.brain_hot_memory payload. Cache key (source_id, session_id, hash(takesHoldersAllowList sorted)) (eD10 / eE5). 30s TTL per session. Visibility filter applies: remote → world only; local → all. Top-K=10 ranked by effective confidence (decay). Skips injection on recall/extract_facts/forget_fact themselves. bumpHotMemoryCache() invalidates per (source_id, session_id) on extraction event. D12 (eE1) accepted: serve-http.ts:801 inlined dispatch path REFACTORED to call dispatchToolCall. HTTP MCP now inherits source_id, _meta injection, error envelope unification, and OperationContext shape from the same code path stdio uses. Scope check + mcp_request_log + SSE broadcast stay in serve-http.ts (HTTP-specific concerns); the dispatcher returns ToolResult and the HTTP handler reads isError + content + _meta to fan into the audit + broadcast paths. put_page compliance backstop (D23): when a conversation-shape page is written (note/meeting/slack/email/calendar-event/source/writing) with a substantive body (>=80 chars) on a non-subagent slug AND no dream_generated:true marker, fire-and-forget enqueue an extraction job into the bounded queue. Never blocks the put_page response. Skipped reasons (no_parsed_page / subagent_namespace / dream_generated / kind:* / too_short / queue_shutdown / backstop_error) are stable strings consumed by tests. `gbrain recall` + `gbrain forget` CLI commands (src/commands/recall.ts): - recall <entity> | --since DUR | --session ID | --today (markdown with kind icons 📅🎯🤝💭📌) | --grep TEXT | --supersessions | --include-expired | --as-context (prompt-injection-ready) | --json - forget <fact-id> shorthand for expireFact Wired into src/cli.ts dispatch table next to takes / think. Smoke tests confirm: dispatch surfaces (extract_facts → ops → listFactsByEntity), forget_fact + idempotent re-call, _meta visibility filter (remote sees world only, local sees all), CLI markdown render with kind icons + age strings + decayed confidence. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(cycle): consolidate phase — facts → takes promotion (Phase 5) Phase 5 of v0.31 hot-memory. New 10th cycle phase `consolidate` between `patterns` and `embed`: - src/core/cycle.ts: * CyclePhase union extended with 'consolidate' * ALL_PHASES gets 'consolidate' between patterns and embed (graph-fresh after patterns; embed runs after so the new takes get embedded same-cycle) * NEEDS_LOCK_PHASES gets 'consolidate' (writes takes + UPDATEs facts) * CycleReport.totals gains facts_consolidated + consolidate_takes_written * runCycle dispatches the new phase via dynamic import - src/core/cycle/phases/consolidate.ts (new): * Scans (source_id, entity_slug) buckets where COUNT(unconsolidated facts) >= 3 (uses idx_facts_unconsolidated partial index) * Skips buckets where the OLDEST fact is < 24h old (gives signal time to settle before locking it into cold memory) * Greedy cosine clustering at threshold 0.85; head-element centroid keeps it deterministic + cheap. Singletons (no embedding) stay unconsolidated this cycle. * For each cluster size >= 2: picks the highest-confidence fact's text as the take claim (v0.31 deterministic; v0.32 swaps to Sonnet synthesis pass). avg confidence → take weight, earliest valid_from → take since_date, concatenated source_sessions → take.source. * Resolves entity_slug → page_id via pages.slug (per source). Skips cluster if page is missing in this source — no auto-page-creation in v0.31. * INSERT into takes(kind='fact', holder='self') with row_num = MAX(existing) + 1. * UPDATE contributing facts: consolidated_at = now() + consolidated_into = takes.id. NEVER DELETE — facts are the audit trail for the resulting take. * dryRun honored: pretends the writes happened; counters still tick so operators can preview load before the first real run. * yieldDuringPhase keepalive between buckets so the Minions worker job lock + cycle-lock TTL don't drift on long runs. Smoke test on PGLite confirms: 4 unconsolidated facts → clustered (cosine 1.0 since same vector) → 1 take row created → all 4 facts marked consolidated_into. runCycle({phases:['consolidate']}) wires through to the report totals. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: 18 facts test files (Phase 6) Phase 6 of v0.31 hot-memory: comprehensive coverage across the new substrate. 110 unit tests pass; 5 E2E test files added (skip gracefully without DATABASE_URL). Unit tests (PGLite in-memory, no DATABASE_URL): - test/facts-decay.test.ts (12 cases) — HALFLIFE_DAYS pinned per kind, effectiveConfidence math: age=0 / age=halflife (~1/e) / age=2×halflife (~1/e²) / expired returns 0 / valid_until past returns 0 / preference-vs-event slower decay / belief-vs-commitment crossover. - test/facts-queue.test.ts (10 cases) — FIFO within session, drop-oldest on overflow, per-session in-flight=1 serializes, different sessions parallelize, failed jobs counter, shutdown grace + drop_pending + external AbortController triggers shutdown. - test/facts-classify.test.ts (8 cases) — cosineSimilarity edge cases, empty candidates → independent, cheap fast-path ≥0.95 → duplicate no LLM, threshold-configurable cosine_fallback path. - test/facts-engine.test.ts (13 cases) — every BrainEngine fact method end-to-end: insertFact (insert/supersede), expireFact idempotency, list*, findCandidateDuplicates entity-prefiltered + k cap + cosine ordering, consolidateFact never DELETE, getFactsHealth shape + total_today ⊆ total_week. - test/facts-multi-tenant.test.ts (6 cases) — cross-source isolation on every list method + CASCADE delete on sources. - test/facts-visibility.test.ts (6 cases) — visibility column private/ world; remote=true filters to world-only via dispatchToolCall; remote=false sees all. - test/facts-canonicality.test.ts (10 cases) — slugify rules including NFKD diacritic strip ("Crème Brûlée" → "creme-brulee"), exact slug match, fallback to slugify when no fuzzy match. - test/facts-extract.test.ts (4 cases) — empty turn returns [], dream- generated short-circuit, graceful no-API-key return. - test/facts-backstop-gating.test.ts (5 cases) — put_page backstop: too_short, subagent_namespace, dream_generated, eligible note path, non-eligible kind:guide. - test/facts-anti-loop.test.ts (4 cases) — extractor + put_page both respect dream_generated:true marker. - test/facts-doctor-shape.test.ts (4 cases) — facts_health JSON shape pinned for downstream consumers. - test/facts-mcp-allowlist.serial.test.ts (5 cases) — extract_facts write-scope, recall read-scope, forget_fact write-scope, forget_fact fact_not_found error code, extract_facts no-API-key zero counts. - test/facts-context-injection.serial.test.ts (6 cases) — _meta injection on success, world-only filter under remote=true, anti-loop on facts ops themselves, best-effort degrade on hook error, cache-key includes allow-list hash. - test/facts-separation-pglite.test.ts (2 cases) — Garry's Separation Test as primary ship gate, plus expired hidden-by-default contract. - test/facts-recall-render.test.ts (3 cases) — --today markdown render with all 5 kind icons, --json shape with effective_confidence, --as-context emits comment-wrapped block. - test/facts-migration-dim.test.ts (4 cases) — embedding column type is HALFVEC/VECTOR (not arbitrary), dim matches gateway-configured embedding_dimensions, HNSW opclass agrees with column type, idempotent re-init. - test/cycle-consolidate.test.ts (5 cases) — below-count + below-age thresholds skip, happy path 4 facts → 1 take + all consolidated never DELETE, dryRun honored, missing page → bucket skipped. E2E tests (skip gracefully on DATABASE_URL unset; required gates by CLAUDE.md test policy): - test/e2e/facts-separation-postgres.test.ts — Postgres parity for the ship gate. - test/e2e/facts-cross-source-isolation.test.ts — cross-source ACL on PG + CASCADE delete. - test/e2e/facts-forget.test.ts — full forget_fact MCP roundtrip. - test/e2e/facts-context-injection-postgres.test.ts — _meta injection end-to-end on PG. - test/e2e/facts-recall-render.test.ts — recall --today markdown on PG. - test/e2e/serve-http-meta.test.ts — eE1 regression: HTTP MCP transport inherits _meta + sourceId + scope correctness via dispatchToolCall. Side-effect: src/core/entities/resolve.ts NFKD post-decompose strips combining marks (U+0300..U+036F) before hyphenating non-alphanumerics, so "Crème" → "creme", not "cre-me-". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(operational): kill switch + doctor check + CHANGELOG + README (Phase 7) Phase 7 of v0.31 hot-memory. - src/core/facts/extract.ts: new isFactsExtractionEnabled(engine) helper reads `facts.extraction_enabled` config row. Defaults to TRUE; flip to 'false'/'0'/'no'/'off' (case-insensitive) via `gbrain config set facts.extraction_enabled false` to kill extraction across the brain without binary downgrade. - extract_facts MCP op short-circuits with zero-counts envelope + a 'skipped: extraction_disabled' field when the flag is off (clean success, not permission_denied). - put_page facts backstop respects the same flag — eligibility check now returns 'extraction_disabled' as the skipped reason. - src/commands/doctor.ts: new facts_health check (runs after queue_health, before index_audit). Probes for the facts table existence (post-v40 guard), then surfaces total_active / total_today / total_week / total_consolidated + top-3 entities for the default source. Pre-v0.31 brains report "facts table not present (pre-v0.31 brain or migration pending)". - CHANGELOG.md: full v0.31.0 entry in the GStack release-summary voice. Headline + numbers-table + what-it-ships + itemized changes + "To take advantage of v0.31" upgrade block + out-of-scope. Honest about the HALFVEC + serve-http refactor + ErrorCode-open-union complications. - README.md: cycle phase list updated 8 → 10 (consolidate + purge). New "v0.31 Hot Memory" command block under Commands with recall + forget variants, kind icons, --as-context surface for headless agents. Test gates: 28 facts unit tests pass after the kill-switch wiring + doctor check ride-along. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(migrate): add facts→sources FK explicitly via ALTER TABLE The inline column-level FK declaration on facts.source_id worked on PGLite but silently got dropped on Postgres in the v0.31 e2e run — the migration handler ran via postgres-js's `unsafe()` multi-statement path and the resulting facts table came back without the `facts_source_id_fkey` constraint. Same psql input run directly against the same database produced the FK; the difference was the unsafe() pipeline, not the SQL itself. Splitting the FK into a separate ALTER TABLE inside a DO block makes the constraint declaration explicit and idempotent: the named constraint either exists or it doesn't, the ALTER is a no-op on re-runs, and the failure mode is loud rather than silently leaving a CASCADE-less foreign key behind. Without this fix, deleting a source row leaves orphaned facts rows (test/e2e/facts-cross-source-isolation.test.ts CASCADE-on-sources- delete case caught it). With this fix the constraint is in place, the cascade fires, and both PG + PGLite e2e suites stay green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: update phase-count assertions for the new consolidate phase Three e2e/unit tests pinned the cycle phase count or order, all now updated to reflect v0.31's 10-phase cycle: - test/e2e/dream-cycle-eight-phase-pglite.test.ts: describe rename "8-phase cycle" → "10-phase cycle"; ALL_PHASES expectation extended to include 'consolidate' (between patterns + embed) and 'purge' (the v0.26.5 addition that was already in ALL_PHASES but missing from the test's assertion list). totals match adds the new facts_consolidated + consolidate_takes_written fields plus the pre-existing purged_sources_count + purged_pages_count that should have been added when v0.26.5 landed. - test/e2e/cycle.test.ts: dry-run full cycle now expects report.phases.length === 10 (was 9). - test/core/cycle.serial.test.ts: yieldBetweenPhases hook count + full cycle phases.length both updated 9 → 10. Comments call out the v0.31 addition lineage so the next person to add a phase sees the precedent. These are mechanical assertion bumps. The tests pass against the updated assertions on PGLite and Postgres. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(test): truncate facts table between e2e describe blocks setupDB() truncates ALL_TABLES between every describe block's beforeAll() hook. The list missed the new v0.31 facts table, so facts seeded by an earlier describe block leaked into Garry's Separation Test on Postgres — listFactsByEntity('travel') returned 2 rows instead of 1 because a prior facts-context-injection test had also seeded a 'travel' fact. Adding 'facts' to the truncate list (before 'pages' to respect FK ordering) makes every describe-block start from an empty facts table. Pinned by re-running the e2e file ordering that originally caught it (facts-recall-render → cross-source-isolation → serve-http-meta → context-injection → separation-postgres → facts-forget) — 13 pass / 0 fail after the fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: meta-hook cache + Postgres consolidate phase coverage Two net-new test files filling real coverage gaps the earlier sweep missed: - test/facts-meta-cache.test.ts (5 cases) — pins the eD3/eD10 cache contract that the dispatcher relies on. 30s TTL hit path, post-bump fresh-query, scoped invalidation (bump for sess-A leaves sess-B cache warm — closes the cross-source leak risk codex F5 originally surfaced on the recall payload), facts-self ops skip injection (anti-loop on recall / extract_facts / forget_fact), distinct allow-lists produce distinct cache entries. - test/e2e/cycle-consolidate-postgres.test.ts (3 cases) — Postgres parity for the dream-cycle consolidate phase. Mirrors the PGLite unit test but exercises the real postgres-engine codepaths: sql.begin transactions, advisory locks on insertFact's entity-slug dedup window, unsafe('::vector') casts on findCandidateDuplicates ordering, addTakesBatch postgres-js unnest path. Happy path (4 facts → 1 take + all consolidated_into set), age-threshold skip, dry-run no-write. All 5 unit + 3 e2e tests pass. Closes the unit-only gap on the consolidate phase (was only PGLite-tested) and pins meta-cache invariants the dispatcher depends on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix: thread auth + sourceId, JSON-shape every error envelope Three bugs surfaced during the full e2e sweep that all trace back to my v0.31 dispatch refactor (D12/eE1) silently dropping auth threading + non-OperationError exceptions emitting plain strings: 1. **HTTP MCP transport lost ctx.auth.** Refactoring serve-http.ts to call dispatchToolCall meant auth had to come through DispatchOpts, but the field didn't exist yet. Every HTTP whoami call returned `unknown_transport` because ctx.auth was undefined. Added `auth?: AuthInfo` to DispatchOpts, plumbed it through buildOperationContext, and updated serve-http.ts:816 to pass `auth: authInfo` alongside sourceId/takesHoldersAllowList. Pinned by sources-remote-mcp e2e `whoami reports oauth transport + sources_admin scope`. 2. **Non-OperationError exceptions emitted plain strings, not JSON.** The pre-v0.31 serve-http.ts always wrapped errors in JSON envelope `{error, message}`; my dispatch refactor missed the unknown-tool + uncaught-throw paths and emitted `Error: ${msg}` text content. Every caller that did `JSON.parse(content)` (sources-remote-mcp callMcp helper at line 104) crashed with `Unexpected identifier "Error"`. Both error paths in dispatchToolCall now return JSON-shaped content matching the OperationError pattern. 3. **Files→sources FK silently lost on rewound bootstrap path.** test/e2e/postgres-bootstrap.test.ts simulates a pre-v0.21 brain by `DROP TABLE IF EXISTS sources CASCADE` which removes files_source_id_fkey while leaving files.source_id intact. The v23 migration's `ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id ... REFERENCES sources(id) ON DELETE CASCADE` is a no-op when the column exists, so the FK never came back on upgrade — and any sources-remove afterward stopped cascading to files. Added a defensive `IF NOT EXISTS files_source_id_fkey ... ALTER TABLE ADD CONSTRAINT` block inside v23's handler. Pinned by `multi-source — cascade delete covers every dependent row` after running postgres-bootstrap. Plus: src/core/preferences.ts now honors GBRAIN_HOME for `~/.gbrain/migrations/completed.jsonl`. Without this, the doctor exits-0 mechanical test inherits the developer machine's stale partial-migration ledger entries (0.21.0, 0.22.4, 0.28.0, 0.29.1 prior dev work) and surfaces them as the [FAIL] minions_migration check. GBRAIN_HOME-scoped tempdir per test now isolates this state cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 chore: scrub personal references from public artifacts Per the CLAUDE.md privacy rule on `Garry's Separation Test`, replace personally-coded references in v0.31 artifacts with neutral examples: - CHANGELOG.md v0.31 entry: rename "Garry's Separation Test" header to "The cross-session test" + drop the "topic-2659/topic-1941, 7 AM/2 PM, flying to Tokyo" narrative. - src/commands/migrations/v0_31_0.ts feature pitch: same scrub. - test/facts-separation-pglite.test.ts + test/e2e/facts-separation-postgres.test.ts: rename describe blocks; replace specific topic-NNNN session ids with session-A / session-B; replace personal sample fact with "sample event Tuesday". - src/core/facts/extract.ts extractor system prompt example slugs: people/sam-altman → people/alice-example; companies/anthropic → companies/acme. - src/core/entities/resolve.ts comment: Sam Altman → Alice Example. - All v0.31 test fixtures: people/sam → people/alice-example, Sam Altman → Alice Example, sam-the-cofounder → alice-the-cofounder. Test names referencing real-world entities replaced with neutral slugs. Pre-existing references to "Garry" elsewhere in CHANGELOG (v0.17, v0.19, v0.21+ entries) are untouched — that's a separate scope from this v0.31 ship. Plus: the truncate fix for the Bun-script-induced syntax error in test/e2e/mechanical.test.ts (cliEnv arrow function had ", 30_000)" tacked onto its closing brace by the bulk-add-timeouts script — repaired to a clean function definition). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(test): bump E2E phase-count assertions for 11-phase cycle Two E2E tests still asserted the v0.31 pre-merge 10-phase shape (consolidate inserted, but recompute_emotional_weight from v0.29 not yet absorbed). With master's v0.29 work merged in, the cycle is now 11 phases: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → consolidate → embed → orphans → purge. - test/e2e/cycle.test.ts: 10 → 11 - test/e2e/dream-cycle-eight-phase-pglite.test.ts: ALL_PHASES + dry-run order Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(merge): close brace between v44 and v45 migration objects The v0.30.2 merge resolution stitched master's v40-v44 migrations onto HEAD's v45 (facts hot memory) migration but lost the closing `},` between v44 and v45. tsc caught it as TS1136 Property assignment expected at migrate.ts:2188. This is a one-line bracket fix; the rest of the merge resolution is correct and tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix: put_page cliHints + buildPlan v0.31.0 in skippedFuture Two unit-test failures surfaced after the v0.30.2 merge: 1. operations.ts: put_page had `cliHints: { name: 'put', positional: ['stdin'] }` from earlier v0.31 development. The parity test enforces that every name in `positional` is a real param. Restored master's correct shape: `{ name: 'put', positional: ['slug'], stdin: 'content' }`. 2. test/apply-migrations.test.ts: the H9 regression tests pin the exact skippedFuture list. Adding v0.31.0 to the registry meant the list grew by one. Updated both `expect(...).toEqual([...])` assertions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 docs: clarify consolidate is 11th phase + regen llms-full.txt CHANGELOG.md narrative said "new 10th phase consolidate"; with v0.29's recompute_emotional_weight already on master, consolidate is the 11th phase (between recompute and embed). Schema migration is v45, not v40, after the merge resolution renumbered it to clear master's v40-v44. llms-full.txt regenerated to reflect the README's 11-phase dream-cycle phrasing (the build-llms test enforces commit-time parity). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |