14 Commits

Include renames

Author	SHA1	Message	Date
+818 people	1bc579916b	v0.36.1.1 fix-wave: community PR triage + 28 atomic fixes (#1182) ... * fix(sync): accept .tf / .tfvars / .hcl in CODE_EXTENSIONS Terraform repos were invisible to `gbrain sync --strategy code` because the three HCL-family extensions never reached the file walker. Silent data loss — the user thinks the sync covered the repo but the IaC layer was dropped on the floor. detectCodeLanguage() returns null for these extensions, so the chunker falls back to recursive (no tree-sitter grammar for HCL) — the same path toml/yaml take. Closes #878. Co-Authored-By: johnybradshaw <johnybradshaw@users.noreply.github.com> * fix(upgrade): run `bun update gbrain` from Bun's global install root `gbrain upgrade --strategy bun` was failing on canonical `bun install -g github:garrytan/gbrain` installs because `execSync('bun update gbrain')` ran in the user's shell cwd. Bun's update operates on whatever package.json it finds via cwd-walk, so a user not standing in the global root got "No package.json, so nothing to update". resolveBunGlobalRoot() returns the right directory: 1. `$BUN_INSTALL/install/global` when set (operator override). 2. `~/.bun/install/global` (Bun's documented default). 3. Walk up from realpath(argv[1]) looking for `node_modules/gbrain` — handles non-standard installs without trusting argv naming. execFileSync replaces execSync (no shell), with cwd pinned. Error path prints the exact `cd && bun update` recovery command instead of a vague hint. Closes #1029. Cherry-picked from PR #1032. Co-Authored-By: mvanhorn <mvanhorn@users.noreply.github.com> * fix(config): redact sensitive values in `config set` output (closes #892) `gbrain config set openai_api_key sk-...` was echoing the full key to stderr via `console.log('Set %s = %s', key, value)`. Shell scrollback and tmux scroll buffers commonly retain stderr for hours; a screen-share or shoulder-glance during set leaked the secret. The `show` path already redacted but used a naive `.includes('key')` substring check that would mask 'monkey' or 'parsekey' (no false-negative but ugly). Single source of truth: `isSensitiveConfigKey()` uses a word-boundary regex (`(^|[._-])(key|secret|token|password|pwd|passwd|auth)([._-]|$)/i`) so 'openai_api_key' matches but 'monkey' doesn't. `redactConfigValue()` composes the postgresql:// URL redactor + sensitive-key check, used by both `show` and `set`. Helpers exported for unit tests. Closes #892. Cherry-pick of @sharziki's PR #918 (config.ts hunk only — the extract.ts walker change in that PR is unrelated and tracked in #202). Co-Authored-By: sharziki <sharziki@users.noreply.github.com> * fix(oauth): throw InvalidTokenError so bearerAuth returns 401, not 500 `verifyAccessToken` was throwing bare `Error` on expired or invalid tokens. The MCP SDK's `requireBearerAuth` middleware catches `InvalidTokenError` and returns 401 with WWW-Authenticate; bare Error falls through to 500. Result: legitimate clients with stale tokens hit 500-not-401, so token-refresh logic (which keys off 401) never fires. Two call sites in verifyAccessToken: token-expired path and invalid-token path. Both now throw InvalidTokenError. Existing tests continue to pass because they assert on the throw, not the message class. Closes #935. Cherry-picked from PR #1012. Co-Authored-By: Aashiqe10 <Aashiqe10@users.noreply.github.com> * fix(serve): return 405 on GET /mcp instead of 404 MCP Streamable HTTP spec says GET /mcp opens an optional SSE backchannel for server-initiated messages. gbrain's transport is stateless and doesn't push server-initiated messages, so per spec we MUST return 405 with Allow: POST, DELETE — not 404. Probing clients (claude.ai, etc.) distinguish "endpoint exists, no SSE channel" from "endpoint missing" on this status code; 404 makes them give up. Cherry-picked from PR #1076. Co-Authored-By: lukejduncan <lukejduncan@users.noreply.github.com> * fix(doctor): resolve whoknows fixture from module location, not cwd `gbrain doctor` warned about a missing whoknows fixture for every install that wasn't standing in the gbrain source repo at run time — which is everyone. The check used `process.cwd()` to locate the fixture, so any real user (running doctor against `~/.gbrain`) saw a spurious warning. `resolveWhoknowsFixturePath()` walks up from `import.meta.url` looking for the source-repo signature (`src/cli.ts` + `skills/RESOLVER.md`), respects `GBRAIN_WHOKNOWS_FIXTURE_PATH` env override (absolute or cwd-relative), and returns null with an actionable warning when the fixture can't be located. Closes #969. Cherry-picked from PR #1034. Co-Authored-By: mvanhorn <mvanhorn@users.noreply.github.com> * fix(frontmatter): centralize --fix backups under ~/.gbrain/backups/ `gbrain frontmatter validate --fix` and `gbrain frontmatter generate --fix` wrote `<file>.bak` siblings into the source tree. Users running gbrain over a brain repo found .bak files scattered through people/, companies/, etc. that broke gitignore expectations and showed up in `git status` after every fix pass. Backups now land under `~/.gbrain/backups/frontmatter/<run-id>/<rel>.bak` with an iso-week-sorted run-id so a multi-fix session keeps the same parent directory. Backup directory + per-file structure mirrored from the original file's relative path. The .bak safety contract is intact for both git and non-git brain repos. Also adds `--include-catch-all` opt-in to `frontmatter generate` so the default catch-all rule (`type: note`) is no longer applied to arbitrary workspace documents that happen to live under a brain root. Closes #902. Cherry-picked from PR #903. Co-Authored-By: 100yenadmin <100yenadmin@users.noreply.github.com> * fix(config): use path.isAbsolute() for GBRAIN_HOME on Windows The GBRAIN_HOME validator rejected every valid Windows path (`C:\\Users\\...`, `D:\\gbrain`, etc.) because it used `trimmed.startsWith('/')` to check for absoluteness — only POSIX absolute paths pass that. `path.isAbsolute()` is the cross-platform check. Same fix for the `..` traversal check: split on both `/` and `\` so Windows path separators don't sneak `..` through. Closes #1019. Cherry-picked from PR #1083. Co-Authored-By: sharziki <sharziki@users.noreply.github.com> * fix(ai): warn only for the configured embedding provider, not all recipes Gateway construction was warning on stderr for every recipe with an embedding touchpoint missing max_batch_tokens — including providers the brain isn't using. Users on Voyage saw noise about OpenAI / Google / DashScope / etc. recipes that never get loaded. Filter the warning to recipes whose provider id is referenced by `embedding_model` or `embedding_multimodal_model` in the active config. The structural protection against forgetting max_batch_tokens stays in place for the recipes that actually run; the noise for unrelated recipes goes away. Cherry-picked from PR #1117. Co-Authored-By: hnshah <hnshah@users.noreply.github.com> * fix(sync): skip git pull when repo has no origin remote `gbrain sync` ran `git pull` unconditionally and printed scary stderr on every cycle for brains that have no `origin` remote (local-only workflows, single-machine setups, brains initialized via `gbrain init --pglite` against an arbitrary directory). The pull failed harmlessly but the noise was confusing and made operators think sync was broken. `hasOriginRemote()` probes `git remote get-url origin` with stdio ignored; on failure (`no such remote`), skip the pull, print a single informational line, and proceed with the local working tree. Cherry-picked from PR #1119. Co-Authored-By: hnshah <hnshah@users.noreply.github.com> * fix(query): drain cache writes before CLI exit The query cache write was fired with `void promise.catch(...)` — true fire-and-forget. On a fast CLI invocation (`gbrain query <q>` exits in ~50ms), the process terminates before the cache write commits. Result: the cache effectively never warms from CLI use; every query is a miss. `awaitPendingSearchCacheWrites()` tracks each in-flight cache write in a module-level Set. The CLI dispatcher awaits the set after `query` finishes formatting output but before the process exits. MCP server path unchanged (long-lived process, fire-and-forget remains correct). Cherry-picked from PR #1125. Co-Authored-By: hnshah <hnshah@users.noreply.github.com> * fix(backlinks): dedupe (source, target) pairs within a single source page A source page that mentions the same entity N times produced N duplicate "Referenced in" lines on the target. `extractEntityRefs` returns one EntityRef per occurrence, and the per-ref `hasBacklink` check reads a snapshot of `target.content` that's frozen at outer scope — so every iteration sees "no backlink yet" and appends another gap. The cumulative effect on a long meeting note with multiple mentions of the same person was visible in PRs landing 3-5 identical Timeline entries. Track seen target slugs per source page; cap gaps at one pair. Cherry-picked from PR #967 with a current-master regression test covering both markdown-link and Obsidian-wikilink formats in the same source page. Co-Authored-By: p3ob7o <p3ob7o@users.noreply.github.com> * fix(dream): audit backlinks without mutating pages during cycle The dream/autopilot maintenance cycle ran the backlinks phase in 'fix' mode, which writes "Referenced in" timeline bullets into entity pages every sync. The graph extractor + auto-link path is the canonical link store during sync/dream/autopilot — the legacy filesystem fixer wrote markdown that fought with both the user's manual edits and the graph layer's own timeline. Cycle now runs backlinks in 'check' mode (audit-only); the materializer remains available via `gbrain check-backlinks fix` for users who really want markdown backlinks committed to disk. Cherry-picked from PR #1027. Co-Authored-By: sliday <sliday@users.noreply.github.com> * fix(autopilot --install): source ~/.zshenv before zshrc/bashrc zshenv is the canonical place for env vars in zsh on macOS — zshrc is sourced only for interactive shells, so vars exported in zshrc don't reach a non-interactive subprocess like the autopilot wrapper. Users who exported GBRAIN_DATABASE_URL, OPENAI_API_KEY, or ANTHROPIC_API_KEY in zshrc and assumed autopilot would inherit them hit silent missing- secret failures on the LaunchAgent. Source ~/.zshenv first (always reaches non-interactive shells per zsh docs), then fall back to ~/.zshrc / ~/.bashrc for users on other profile conventions. Cherry-picked from PR #966. Co-Authored-By: p3ob7o <p3ob7o@users.noreply.github.com> * fix(apply-migrations): return exit 0 on list/dry-run/up-to-date `gbrain apply-migrations list`, `gbrain apply-migrations --dry-run`, and the "All migrations up to date" path were returning from the async function but never calling `process.exit(0)`. The CLI dispatcher in cli.ts treated the implicit fall-through as exit 1 when the parent process inspected status via shell scripts, breaking automation that gates on `apply-migrations list && do-something`. Three call sites: list, dry-run, and the no-op path. All three now exit(0) explicitly. Cherry-picked from PR #1062. Co-Authored-By: nezovskii <nezovskii@users.noreply.github.com> * fix(sync): scope auto-embed to source on incremental syncs `gbrain sync --source-id X` triggered auto-embed for the affected slugs but `runEmbed` ran with no `--source` flag, so it fell back to the default source. For non-default-source syncs the page row lives at (sourceId, slug) — the embed code saw "Page not found" for the right slug under the wrong source, swallowed the error as best-effort, and the sync result reported `embedded: 0` for the wrong reason. `buildAutoEmbedArgs(slugs, sourceId)` is the new helper: when sourceId is set, prepends `--source X`. Exported for the regression test. Pairs with the upcoming source-id write-path audit (P1 #8). Cherry-picked from PR #1120. Co-Authored-By: hnshah <hnshah@users.noreply.github.com> * fix(query): honor source_id with no-expand for cross-source search Two related corrections: 1. `gbrain query --no-expand` parsed `--no-expand` as the literal key `no_expand` instead of negating the boolean `expand` param. Result: the flag was silently ignored and expansion always ran. Now any `--no-<key>` where `<key>` is a boolean param flips it false. 2. The `query` op's source-id resolution treated `ctx.sourceId` as authoritative, so an explicit per-call `source_id` was overridden by the federated read scope. Now per-call `source_id` wins; `source_id=__all__` is an explicit opt-out for local cross-source search. Cherry-picked from PR #1124. Co-Authored-By: hnshah <hnshah@users.noreply.github.com> * fix(doctor): child-table orphan detection (closes #1063) The autopilot orphans phase detects orphan PAGES (no inbound links via page-graph) but never scans FK-child tables. After a bulk delete or a pre-FK-migration code path, orphan rows can persist indefinitely in content_chunks, page_versions, tags, takes, raw_data, timeline_entries, or links — all declared ON DELETE CASCADE, so any orphan row is unexpected. `childTableOrphansCheck` enumerates 10 FK columns across 8 tables: - 8 NOT NULL columns (cascade): any value not in pages.id is an orphan. - 2 nullable SET NULL columns (links.origin_page_id, files.page_id): NULL is valid; only NOT-NULL-but-missing-in-pages counts. Surfaces paste-ready cleanup SQL when orphans are found. Cherry-picked from PR #1064. Co-Authored-By: vincedk-alt <vincedk-alt@users.noreply.github.com> * fix(autopilot,cycle): stop respawn-storm from steady-state 'partial' cycles Two compounding bugs under KeepAlive=true: 1. Autopilot tripped its circuit breaker on cycle.status === 'partial', not just 'failed'. 'partial' means at least one phase warned/failed while others ran — a soft signal, not fatal. On every cycle that warned, autopilot logged a failure and the supervisor respawned the worker. 2. The orphans phase emitted 'warn' when `count > 20` orphan pages. That threshold was tuned for small dev brains; on any corpus past a few hundred pages it fires every cycle in steady state. Together with bug 1, this produced visible respawn storms. Fix: - Autopilot trips only on cycle.status === 'failed'. - Orphans phase warns by ratio: orphans / total_pages > 0.5 (the real "your graph fell apart" signal), not by absolute count. Cherry-picked from PR #1113. Co-Authored-By: sergeclaesen <sergeclaesen@users.noreply.github.com> * fix(ai): reject partial embedding responses before indexing `embedSubBatch` only validated the FIRST embedding's dimension and never asserted the response length matched the input length. If a provider returned fewer embeddings than requested (rate-limit truncation, malformed response, etc.), the gateway silently indexed an offset-shifted result — every page after the missing index got the embedding of a different page's chunk. Two new guards: 1. `result.embeddings.length === texts.length` — fail loud if any count mismatch, with a paste-ready retry hint. 2. Validate dim on EVERY embedding, not just the first. Cherry-picked from PR #926. Co-Authored-By: 100yenadmin <100yenadmin@users.noreply.github.com> * fix(serve): admin register-client supports auth_code + PKCE public clients The admin dashboard's /admin/api/register-client endpoint hardcoded client_credentials and ignored grantTypes, redirectUris, and tokenEndpointAuthMethod. Result: you couldn't register a browser-based PKCE client (claude.ai Custom Connector, Cursor, etc.) through the dashboard — only confidential machine-to-machine clients worked. Pass grantTypes / redirectUris through to registerClientManual. When tokenEndpointAuthMethod === 'none', NULL out client_secret_hash so the SDK's clientAuth middleware skips the hash-vs-plaintext compare that would otherwise reject the no-secret PKCE flow. Cherry-picked from PR #1077. Co-Authored-By: lukejduncan <lukejduncan@users.noreply.github.com> * fix(extract-facts): treat slugs:[] as no-op, not unscoped full-walk `runExtractFacts` checked `opts.slugs && opts.slugs.length > 0` to decide between scoped and full-brain walk. Both `undefined` (caller omits → full walk intended) AND `[]` (sync no-op → zero work intended) fall through to the same `else` branch and triggered `engine.getAllSlugs()`. On a multi-thousand-page brain, the unintended full walk exceeded the autopilot-cycle ~600s timeout and dead-lettered the job — visible in production as `[cycle.extract_facts] start` followed by silence until `Autopilot stopping (cycle-failure-cap)`. Use presence (`opts.slugs !== undefined`), not truthiness, to distinguish the two modes. Empty array is a real incremental no-op. Closes #1096. Three regression cases in test/extract-facts-phase.test.ts: slugs=[] no-op, slugs=undefined still walks, slugs=['a'] walks just one. Co-Authored-By: navin-moorthy <navin-moorthy@users.noreply.github.com> * fix(serve): embed admin/dist into binary; serve from manifest (closes #1090) Pre-fix, /admin returned 404 on every globally-installed binary because serve-http.ts:780 resolved admin/dist via process.cwd(). The admin SPA files are checked into git but `bun build --compile` does NOT embed arbitrary directories — only assets imported via `with { type: 'file' }` ESM imports land in the compiled binary. Wire: - scripts/build-admin-embedded.ts walks admin/dist/, emits src/admin-embedded.ts with one `with { type: 'file' }` import per file + a manifest map (request path → resolved path + mime). Auto-invoked by `bun run build:admin`. - src/admin-embedded.ts is the auto-generated module. Bun resolves every file: import to a path that works at runtime inside the compiled binary (same pattern as src/core/chunkers/code.ts WASM imports). - serve-http.ts switches to two-tier resolution: cwd-relative admin/dist for dev (Vite hot-rebuild), embedded manifest otherwise. Embedded path reads bytes lazily and caches per-asset for the lifetime of the process. - scripts/check-admin-embedded.sh CI gate re-runs the generator and fails on drift (mirrors check-wasm-embedded.sh). PRs that rebuild admin/dist but forget to regenerate the embedded module fail loud. - package.json wires build:admin-embedded + check:admin-embedded. Closes #1090. * test(source-id): lock in routing regression coverage (closes #891 #978 #1078) Audit of every page write path (sync, embed, extract, dream, autopilot, wikilinks, tags, chunks) confirmed that sourceId already threads correctly through importFromContent → engine.putPage → SQL INSERT since v0.18.0. The original bug reports from #891, #978, #1078 were real at the time and got swept by the multi-source refactor; today's master is correct. This commit locks in that correctness with six PGLite regression cases (no Postgres fixture needed; runs in CI everywhere): 1. importFromContent({sourceId:"work"}) lands at source_id=work, not the silent 'default' fallback. 2. Two sources hold the same slug independently. 3. Omitting sourceId falls through to 'default' (legacy contract). 4. Chunks land under the requested source. 5. Tags land under the requested source. 6. FK integrity smoke (originally #1078). The earlier issue reports stay closed by the existing threading; this suite ensures any future refactor of the write path can't silently re-introduce the wrong-source-default bug. The 90-minute write-path audit budget from the plan resolves here. * fix(apply-migrations): unblock PGLite chain (closes #1100) `gbrain apply-migrations --yes` was wedging on the v0.11.0 (Minions) schema phase for PGLite installs. Two compounding bugs: 1. `apply-migrations` pre-flight schema-version warning connects to PGLite to read config.version, then disconnects. The brief lock hold races with downstream subprocess spawns that try to re-acquire it; the 30s lock timeout fires before the parent fully releases. Pre-flight is a *warning*; on PGLite it adds no information the orchestrators don't already handle. Skip the probe for PGLite. 2. v0.11.0 phase A spawned `gbrain init --migrate-only` as an execSync subprocess to apply schema migrations. PGLite is single-writer; the subprocess inherits HOME and tries to lock the same DB. On Postgres this works (concurrent connections OK); on PGLite it deadlocks. Route in-process for PGLite — create + connect + initSchema + disconnect directly, skipping the subprocess hop. Postgres keeps the legacy execSync path. Verified: fresh PGLite install now walks the full migration chain through v0.32.2 (Facts SoR) and lands "All migrations up to date" on re-run. Closes #1100. * fix(serve): bootstrap token env override + suppress flag (closes #1024) `gbrain serve --http` regenerated the admin bootstrap token on every restart and printed it to stderr. In supervisor-managed production deployments (LaunchAgent, systemd, k8s) every restart leaks the value into log aggregators and rotates the access for any agent that paste- copied it. Two new knobs: - **GBRAIN_ADMIN_BOOTSTRAP_TOKEN** env var: when set, used as the bootstrap secret instead of a fresh per-process token. Validated: must match `^[A-Za-z0-9_-]{32,}$` (32-char minimum), else refuse to start with a paste-ready generator hint. Failing closed beats silently accepting a weak token. - **--suppress-bootstrap-token** CLI flag: suppresses the printed token line entirely. Operator takes responsibility for tracking the value out-of-band. Startup banner now reflects the chosen source: - `Admin Token: suppressed` when the flag is set. - `Admin Token: from $GBRAIN_ADMIN_BOOTSTRAP_TOKEN` when env-sourced. - Full token print only when both are absent (default behavior, dev installs). Closes #1024. Co-Authored-By: billy-armstrong <billy-armstrong@users.noreply.github.com> * fix(config): migrate legacy 'provider' + 'model' to 'embedding_model' Pre-v0.32 docs and some community templates used a config shape: { "provider": "voyage", "model": "voyage-4-large" } The canonical shape (since the v0.31.12 gateway seam) is: { "embedding_model": "voyage:voyage-4-large" } Users on the legacy shape hit silent fallthrough to the hardcoded OpenAI default; sync + embed errored out with "OpenAI embedding requires OPENAI_API_KEY" regardless of their actual provider config. loadConfig() now translates the legacy keys at parse time: - emits a one-line stderr nudge with the paste-ready canonical key - preserves the rest of the config unchanged - skipped when `embedding_model` is already set (forward-compat) Closes #1086. Co-Authored-By: jeunessima <jeunessima@users.noreply.github.com> * chore(test): quarantine upgrade tests (process.env mutation) PR #1032's cherry-picked tests use the static-snapshot + try/finally pattern for env vars instead of the project's withEnv() helper. The test-isolation lint catches process.env mutations outside withEnv to prevent cross-test leakage in parallel runs. Renaming to *.serial.test.ts (the quarantine convention) is the documented out: runs sequentially, no cross-file race. A future cleanup PR can migrate the tests to withEnv() and drop the quarantine. * fix(test): update brain-writer .bak assertion for centralized backup path The v0.36.x frontmatter backup change (bd60cdf6 — closes #902) moved .bak files from sibling-of-source to ~/.gbrain/backups/frontmatter/... The old test still asserted on the sibling path, so CI failed even though the production behavior was correct. Updated assertion contract: backup lands under the injected backupRoot (test-isolated), the returned backupPath ends in .bak and exists, and no sibling .bak is created next to the source file. The pre-fix sibling-path is now a negative assertion. * chore: bump version and changelog (v0.36.1.0) v0.36.1.0 — community fix wave (28 atomic fixes + 22 PRs closed as already-shipped + 14 issues triaged). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(fix-wave): close test gaps surfaced by post-ship audit After the fix-wave shipped, an audit found 11 commits with no new test file. Some were inherently structural (build pipelines, shell content) or had existing test coverage that worked either way; others had real regression risk with no guard. This commit closes the gaps that matter. New regression tests for: - OAuth `verifyAccessToken` throws `InvalidTokenError` (not bare Error) on both expired and unknown token paths. Pre-fix, the SDK's `requireBearerAuth` middleware fell through to 500 instead of 401 → client token-refresh logic never fired (#935). - `loadConfig` translates legacy `{provider, model}` config shape to the canonical `embedding_model: <provider>:<model>`. 3 cases: pure legacy → migrated; canonical wins over legacy when both present; canonical-only is untouched. Pre-fix, Voyage/Cohere/Mistral users silently fell through to OpenAI (#1086). - `configDir` rejects relative paths; rejects `..` segments via both separators (regression guard for the Windows path acceptance fix #1019 / cherry-pick #1083). - `resolveBootstrapToken` (new exported helper extracted from `runServeHttp`). 9 cases: unset env generates fresh, valid env accepted, hyphens/underscores accepted, < 32 chars rejected, special chars rejected, whitespace trimmed, empty string rejected, 32-char boundary accepted, 31-char one-short rejected. Security-critical validation surface (#1024). - GET /mcp returns 405 with `Allow: POST, DELETE` (E2E case in `serve-http-oauth.test.ts`). Pre-fix, claude.ai and other probing MCP clients saw 404 and gave up (#1076). - apply-migrations `process.exit(0)` on list / dry-run / up-to-date paths. Source-shape assertion locks the rule in; shell scripts gating on `$?` work (#1062). - Autopilot wrapper sources `~/.zshenv` BEFORE `~/.zshrc`. zshenv is the canonical place for env vars in non-interactive zsh; without this ordering, LaunchAgent subprocesses never inherit secrets exported in zshrc (#966). - `test/fix-wave-structural.test.ts` consolidates source-shape regression guards for fixes whose behavior is hard to runtime-test without heavy mocking: query cache drain (#1125), admin embed manifest + handler (#1090), admin register-client PKCE branch (#1077), PGLite v0.11.0 phase A in-process routing (#1100), query `--no-expand` negation (#1124). 9 source-grep assertions. Refactored `runServeHttp` to extract `resolveBootstrapToken` as a pure helper. The boot path now consumes the helper's tagged-union result ({kind:'ok'|'error'}); side effects (`process.exit`, `console.error`) moved to the caller. Unit-testable without spinning up Express. Test counts: oauth 71 (was 69), config 20 (was 14), apply-migrations 19 (was 18), autopilot-install 5 (was 4), serve-http-bootstrap-token 9 (new file), fix-wave-structural 9 (new file). Net: +28 cases across 6 files; +1 new exported function with full coverage. Remaining audit gaps (deferred): - e82dda0a admin embed E2E (post-deploy curl smoke covers this) - d93fa81d apply-migrations PGLite chain E2E (already smoke-tested manually in the original commit; subprocess test would be flaky in CI without DATABASE_URL gating) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: close the two deferred E2E gaps from the post-ship audit Both gaps now have real behavior coverage. No DATABASE_URL needed (PGLite engine), so they run in standard unit CI alongside the rest of the suite. Serial quarantine because both spawn subprocesses + bind ports / write tmpdirs. test/admin-embed-spawn.serial.test.ts (4 cases, ~6s wall-clock): - Spawns `gbrain serve --http` from a fresh tmpdir so `process.cwd()/ admin/dist` does not exist — this forces the embedded-manifest branch (the one under test). Pre-fix, this exact setup hit 404. - GET /admin/ → 200 + SPA shell HTML (title + #root div), content-type text/html. - GET /admin/index.html → same body via explicit path. - GET /admin/agents → SPA fallback returns index.html for deep links. - GET /admin/api/stats → NOT 200 (regression guard: SPA fallback must not swallow /admin/api/* routes and silently return HTML to a JSON client). Closes #1090. test/apply-migrations-pglite-spawn.serial.test.ts (3 cases, ~25s): - Seeds a fresh PGLite config in a tmpdir, runs `gbrain init --migrate-only` + `gbrain apply-migrations --yes --non-interactive`. Pre-fix this hit "GBrain: Timed out waiting for PGLite lock" because apply-migrations' pre-flight probe + v0.11.0's phase A subprocess both wanted the single-writer lock. - Asserts exit 0, no "Timed out" string, no "Phase A failed" string, brain.pglite file written. - Re-run case: idempotent — "All migrations up to date" exits 0 (also locks in the #1062 exit-code fix end-to-end). - --list path exits 0 (third leg of the #1062 contract). Closes #1100. Pinned bootstrap token via GBRAIN_ADMIN_BOOTSTRAP_TOKEN env so the admin test doesn't have to scrape stderr; the startup banner format is allowed to drift, the /health probe is the readiness contract. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(test): consolidate PGLite spawn test to one end-to-end pass CI failed on test/apply-migrations-pglite-spawn.serial.test.ts (Ubuntu, bun 1.3.14). The previous shape ran 3 tests × ~3 spawns each. Each `bun run /abs/src/cli.ts` from a tmpdir cwd pays a full parse/transpile cost (no near-cwd .bun cache); on Ubuntu CI that compounds past the runner's per-test budget. Consolidated to ONE test that exercises the full lifecycle in one brain: init --migrate-only → apply-migrations --yes → re-run → --list. Four spawns instead of eight. Local wall-clock: 32s → 11.5s. All four assertion buckets preserved: no PGLite lock timeout, no Phase A failure, brain.pglite written, idempotent re-run "All migrations up to date" exits 0 (#1062 end-to-end), --list exits 0. Per-test timeout 480_000ms as insurance against the runner's --timeout=60000 default (bun's API spec: per-test wins). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(diag): dump apply-migrations output when CI exit != 0 The PGLite spawn test passes locally on macOS/bun 1.3.13 in ~11s end-to-end but fails on Ubuntu/bun 1.3.14 in 4.92s with apply.exitCode = 1 — fast enough that something is failing early, not timing out. The runCli helper captured stdout+stderr but never printed them, so the CI log only showed the bare assertion failure. This commit prints the captured streams from BOTH init and apply when the exit code mismatches expectation. After the next CI run we can read the actual error message and diagnose the Ubuntu-specific failure mode (likely BUN_INSTALL / HOME / PGLite WASM env quirk). No behavior change; pure diagnostic output gate on failure. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(test): shim `gbrain` on PATH for PGLite spawn test Root cause of the Ubuntu CI failure: the v0.11.0 orchestrator's phase B runs `execSync('gbrain jobs smoke')`. PGLite phase A now routes in-process (the #1100 fix), but phase B and several follow-up phases still shell out to the `gbrain` binary on PATH. Locally the binary resolves via `bun link`; on CI Ubuntu it does not exist on PATH, so execSync exits 127 → orchestrator returns 'failed' → apply-migrations exits 1. Test failed at 4.92s with exitCode=1, well before any timeout. Verified locally by removing ~/.bun/bin/gbrain to simulate CI: pre-shim: apply.exitCode=1 (same as CI) post-shim: apply.exitCode=0 in 8.4s The shim writes a tiny `gbrain` executable to a tmpdir that just `exec`s `bun run <repo>/src/cli.ts "$@"`. Prepended to PATH for the spawned subprocesses. Mirrors the production contract (gbrain on PATH) without depending on `bun link` having run in the CI image. Diagnostic dump from the previous commit stays — useful insurance for the next time something silently fails inside a spawned binary. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: johnybradshaw <johnybradshaw@users.noreply.github.com> Co-authored-by: mvanhorn <mvanhorn@users.noreply.github.com> Co-authored-by: sharziki <sharziki@users.noreply.github.com> Co-authored-by: Aashiqe10 <Aashiqe10@users.noreply.github.com> Co-authored-by: lukejduncan <lukejduncan@users.noreply.github.com> Co-authored-by: 100yenadmin <100yenadmin@users.noreply.github.com> Co-authored-by: hnshah <hnshah@users.noreply.github.com> Co-authored-by: p3ob7o <p3ob7o@users.noreply.github.com> Co-authored-by: sliday <sliday@users.noreply.github.com> Co-authored-by: nezovskii <nezovskii@users.noreply.github.com> Co-authored-by: vincedk-alt <vincedk-alt@users.noreply.github.com> Co-authored-by: sergeclaesen <sergeclaesen@users.noreply.github.com> Co-authored-by: navin-moorthy <navin-moorthy@users.noreply.github.com> Co-authored-by: billy-armstrong <billy-armstrong@users.noreply.github.com> Co-authored-by: jeunessima <jeunessima@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-18 20:55:57 -07:00
3 people	3a0e1116e7	v0.36.1.0 Hindsight calibration wave: brain learns how you tend to be wrong (#1139) ... * schema: v0.36.0.0 Hindsight calibration tables (migrations v67-v71) Foundation commit for the Hindsight-inspired calibration wave. Adds four new tables + one perf index, all source-scoped from day 1 per v0.34.1 discipline: - calibration_profiles (v67): per-holder LLM-narrative aggregation of TakesScorecard data. published BOOL gates E8 cross-brain mount sharing (default false). grade_completion REAL surfaces partial-grade state to the dashboard. active_bias_tags TEXT[] with GIN index feeds E3 (calibration- aware contradictions) and E7 (real-time nudge matching). - take_proposals (v68): propose_takes phase queue. Idempotency cache via (source_id, page_slug, content_hash, prompt_version) unique index mirrors the v0.23 dream_verdicts pattern. proposal_run_id supports --rollback by run. dedup_against_fence_rows JSONB audit column records what canonical takes the LLM was told to dedupe against at proposal time. - take_grade_cache (v69): grade_takes verdict cache. Composite PK on (take_id, prompt_version, judge_model_id, evidence_signature) — prompt edits OR evidence changes cleanly invalidate prior verdicts. applied=false default + auto-resolve-off-by-default (D17) means every fresh install needs operator opt-in before grade verdicts mutate the takes table. - take_nudge_log (v70): E7 nudge cooldown state. Polymorphic FK — a nudge fires on either a canonical take OR a pending proposal (CDX-5 fix). CHECK constraint enforces exactly-one-set. channel column lets future routing (webhook, admin SPA toast) reuse the same cooldown semantics. - takes_resolved_at_idx (v71): partial index for the Brier-trend aggregation queries. Engine-aware handler — Postgres uses CONCURRENTLY to avoid the ShareLock; PGLite uses plain CREATE. Every table carries wave_version TEXT NOT NULL DEFAULT 'v0.36.0.0' so the v0.36.0.0 calibration --undo-wave command (lands later in the wave) can reverse just this wave's writes. Plan: ~/.claude/plans/system-instruction-you-are-working-rippling-knuth.md covers the design rationale (D17/D18/D21 + CDX findings). Schema parity: - src/schema.sql for fresh Postgres installs - src/core/pglite-schema.ts for fresh PGLite installs - src/core/schema-embedded.ts auto-regenerated from schema.sql - src/core/migrate.ts for upgrade-in-place from older brains VERSION bumped to 0.36.0.0 for the wave. CHANGELOG entry lands at /ship. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * core: BaseCyclePhase abstract class enforces source-scope + budget contracts D21 from the eng review. Three new v0.36.0.0 cycle phases (propose_takes, grade_takes, calibration_profile) share enough structure that the duplication-vs-abstraction trade tips toward a shared base. Without this scaffold, source-isolation discipline would drift exactly the way it drifted in v0.34.1 — except this time across three new surfaces at once. What this enforces: 1. Phase signature is uniform: run(ctx, opts) → PhaseResult. 2. ctx.sourceId / ctx.auth.allowedSources MUST be threaded through every engine call. The base class surfaces a scope() helper that wraps sourceScopeOpts(ctx) and is the only sanctioned way to read source- scoped data. Forgetting to thread source scope becomes a TypeScript compile error, not a runtime leak. Closes the v0.34.1 leak class structurally for every new phase. 3. Budget meter wraps run() automatically. Subclass declares budgetUsdKey + budgetUsdDefault; base reads the resolved cap from config and creates the BudgetMeter. Subclass calls this.checkBudget() before each LLM submit; budget-exhausted phase still returns status='ok' (clean abort) so the cycle report shows partial completion, not failure. 4. Error envelope is uniform. Thrown errors get caught and converted to status='fail' with a phase-specific error.code via the subclass's mapErrorCode() hook. 5. Progress reporter integration. Base accepts the reporter via opts; subclasses call this.tick() instead of touching the reporter directly, so the phase name in the progress stream is always correct. Tests: 13 cases in test/core/base-phase.test.ts cover source-scope threading (5 cases including the empty-allowedSources-MUST-NOT-widen-scope regression), PhaseResult shape including the error envelope path (3 cases), dry-run propagation (2 cases), and budget meter construction (3 cases including config-key override). Synthesize.ts / patterns.ts (existing pre-v0.36 phases) deliberately do NOT retrofit to this base in v0.36.0.0 — too much churn for a refactor that doesn't pay off until v0.37+. Future phases use this by default. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cycle: propose_takes phase + take_proposals queue write path (T3) LLM-based take extraction from markdown prose. Walks pages updated since last cycle, sends each page's body to a tuned extractor, writes the extracted gradeable claims to the take_proposals queue. User accepts / rejects via `gbrain takes propose --review` (lands in Lane C). Cycle wiring: lint → backlinks → sync → synthesize → extract → extract_facts → resolve_symbol_edges → patterns → recompute_emotional_weight → consolidate → propose_takes (NEW) → grade_takes (NEW; T4) → calibration_profile (NEW; T6) → embed → orphans → purge CyclePhase enum extended with 3 new entries; ALL_PHASES + NEEDS_LOCK_PHASES updated. All three new phases acquire the cycle lock (writes to take_proposals / take_grade_cache / calibration_profiles). Idempotency contract: The (source_id, page_slug, content_hash, prompt_version) composite unique index on take_proposals means an unchanged page never re-spends LLM tokens. Bumping PROPOSE_TAKES_PROMPT_VERSION cleanly invalidates the cache so a tuned prompt re-runs proposals on every page. Mirrors the v0.23 dream_verdicts pattern. F2 fence dedup: The phase reads the page's existing `<!-- gbrain:takes:begin -->` fence (when present) and passes the canonical take rows to the extractor as "things you have already captured." Prevents duplicate proposals when prose is appended to a page that already has takes. Records the fence rows the LLM was told to dedupe against on the take_proposals row for audit (dedup_against_fence_rows JSONB). Auto-resolve posture: propose_takes only WRITES proposals to the queue. Nothing in this phase mutates the canonical takes table. Operator opt-in via the queue review CLI (Lane C) is the only path from queue to canonical fence (D17). Prompt tuning status (v0.36.0.0 ship state): The default extractor prompt is annotated `v0.36.0.0-stub`. The real tuned prompt arrives via T19 synthetic corpus build (50 anonymized pages, 3-model parallel extraction, user reviews disagreement set, F1 ≥ 0.85 on training corpus + F1 ≥ 0.8 on ground-truth holdout). Until T19 lands, propose_takes runs but produces best-effort candidates the user reviews manually. Architecture: ProposeTakesPhase extends BaseCyclePhase (T2). Inherits source-scope threading via scope(), budget metering via this.checkBudget(), error envelope wrapping. budgetUsdKey: cycle.propose_takes.budget_usd (default $5/cycle). Budget exhaustion mid-page returns status='warn' with details.budget_exhausted=true — clean partial-completion semantics. Test seam: opts.extractor injection so the phase can run hermetically without touching the gateway. defaultExtractor (production path) calls gateway.chat with the EXTRACT_TAKES_PROMPT and parses the JSON array output via parseExtractorOutput. parseExtractorOutput defends against common LLM output sins: markdown code fence wrapping, leading prose, single-object instead of array, unknown kind values, weight out of [0,1], rows missing claim_text or exceeding 500 chars. Tests: 25 cases in test/propose-takes.test.ts cover the 4 pure helpers (parseExtractorOutput, contentHash, hasCompleteFence, extractExistingTakesForDedup) + 7 phase integration scenarios (happy path, cache hit, fence dedup, extractor failure, empty pages, skipPagesWithFence, proposal_run_id stability). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cycle: grade_takes phase + take_grade_cache verdict pipeline (T4) Walks unresolved takes that are old enough to have outcome data, retrieves evidence from the brain, asks a judge model to verdict each one. Writes verdicts to take_grade_cache. Optionally — only when operator has flipped the opt-in config flag — auto-applies high-confidence verdicts to the canonical takes table via engine.resolveTake. Auto-resolve posture (D17 — DISABLED by default): On a fresh install, grade_takes runs and writes verdicts to the cache, but applied=false on every row. Operator reviews the queue, then flips `cycle.grade_takes.auto_resolve.enabled: true` once trust is earned. Mirrors the propose_takes review-queue posture: queue exists, mutation requires explicit opt-in. Conservative threshold (D12): When auto_resolve.enabled is true, a verdict auto-applies only when confidence >= 0.95 (single-judge path). T5 ensemble path lands next, tightening this further with 3/3 unanimous requirement. 'unresolvable' verdict NEVER auto-applies even at confidence=1.0 — there's no canonical column for "we tried and there's no evidence yet." Evidence retrieval status (v0.36.0.0 ship state): The default evidence retriever returns an "evidence-retrieval not yet wired" placeholder. Most verdicts produced by the stub-judge against the stub-evidence will be 'unresolvable'. Real retrieval (hybrid search over pages newer than the take's since_date, optionally augmented by a gateway web-search recipe in v0.37+) lands as a follow-up. Documented limitation per CDX-8 + D17 — the phase ships now so the wiring is real and the cache table accumulates verdicts even if early ones are conservative. Cache key: Composite primary key on take_grade_cache is (take_id, prompt_version, judge_model_id, evidence_signature). Prompt edits OR evidence changes OR judge swap cleanly invalidate prior verdicts. Mirrors the v0.32.6 eval_contradictions_cache pattern. evidence_signature = SHA-256 of (judge_model_id + '|' + evidence_text) so identical evidence under a different judge does NOT collide. Architecture: GradeTakesPhase extends BaseCyclePhase. Inherits source-scope threading, budget metering (cycle.grade_takes.budget_usd, default $3/cycle), error envelope. Test seam: opts.judge + opts.evidenceRetriever injection so the phase runs hermetically. parseJudgeOutput defends against fence-wrapping, leading prose, out-of-range confidence (clamps to [0,1]), invalid verdict labels, oversized reasoning (truncated at 400 chars). Returns null on unrecoverable parse — caller treats null as "judge_output_parse_failed / unresolvable at confidence 0.0" so the row still lands in cache with the parse failure surfaced via warnings. takeIsOldEnough gates on since_date (default 6 months). Tolerates YYYY-MM-DD and YYYY-MM formats. Returns false on null/unparseable since_date so takes without dates never get graded (we'd be hallucinating temporal context). Tests: 23 cases covering parseJudgeOutput (7 cases), evidenceSignature (3), takeIsOldEnough (5), and 8 phase integration scenarios — happy path, D17 auto-resolve-off default, D12 above-threshold auto-apply, below- threshold cache-only, unresolvable-NEVER-applies, cache hit, too-recent gate, judge-throw warning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cycle: grade_takes ensemble tiebreaker for borderline verdicts (T5 / E2) Multi-judge ensemble tiebreaker, additive on top of T4's single-judge foundation. Reuses gateway.chat as the per-model judge interface; runs three judges in parallel via Promise.allSettled. Pure aggregation logic in aggregateEnsemble() — no SQL, no LLM, hermetically testable. When ensemble fires (T5 trigger band): Only when ALL of: - opts.useEnsemble === true (default false) - opts.ensembleJudges array is non-empty - single-model confidence in [0.6, 0.95) (configurable via opts.ensembleTriggerBand) - single-model verdict !== 'unresolvable' Above 0.95 the single judge is already sufficient (T4 path). Below 0.6 the verdict is clearly review-only — ensemble wouldn't change the posture. 'unresolvable' from single-judge means no evidence yet; calling three more judges on the same evidence won't manufacture some. Conservative auto-apply (D12): Ensemble verdict auto-applies via engine.resolveTake only when ALL of: - autoResolve === true (operator opt-in per D17) - ensemble.agreement === 3 (3/3 unanimous) - ensemble.minConfidence >= ensembleThreshold (default 0.85) - winning verdict !== 'unresolvable' Schema-level monotonic-tightening guard for ensembleThreshold lives in the takes resolution layer. Cache identity: When ensemble fires, the cache row's judge_model_id becomes 'ensemble:<modelA>+<modelB>+<modelC>' — a future re-run with different ensemble membership doesn't collide with prior verdicts. evidence_signature is recomputed because it includes the judge_model_id. aggregateEnsemble (pure): - 3/3 unanimous → agreement=3, minConfidence=min across the three - 2/3 majority → agreement=2, minConfidence across the agreeing two - 1/1/1 disagreement → tie-break: prefer non-'unresolvable', then alphabetical for determinism - 'unresolvable' from one model NEVER tips a 2-vote majority toward 'unresolvable' — by-label tally only counts a model toward its own label - All three judges failing (allSettled rejected) → verdict='unresolvable' with agreement=0; auto-apply path blocked - Single judge survives + two fail → agreement=1; the lone verdict wins but auto-apply gated by the 3/3 requirement Tests: 16 cases. aggregateEnsemble (6): 3/3, 2/3, 1/1/1, unresolvable-tipping-resistance, all-failed, partial-failed-but-survives. Phase trigger conditions (5): useEnsemble=false default, useEnsemble=true in borderline band, single >= 0.95 skip, single < 0.6 skip, single = 'unresolvable' skip. Phase auto-apply rules (5): 3/3+threshold+autoResolve, 2/3 majority no apply, 3/3 below threshold no apply, one ensemble judge throws still aggregates from allSettled, empty ensembleJudges falls through to single. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cycle: calibration_profile phase + shared voice gate across surfaces (T6) The calibration narrative layer. Reads TakesScorecard, asks an LLM to write 2-4 conversational pattern statements ("right on tactics, late on macro by 18 months"), passes them through the voice gate, derives active bias tags, writes the row to calibration_profiles. This is the read-side that E1 (think anti-bias rewrite), E3 (contradictions join), E6 (dashboard), and E7 (real-time nudges) all consume. Voice gate (D24 — single function, multiple surfaces): ALL five calibration UX surfaces import the same gateVoice() function from src/core/calibration/voice-gate.ts. Mode parameter ('pattern_statement' | 'nudge' | 'forecast_blurb' | 'dashboard_caption' | 'morning_pulse') drives surface-specific tuning via the rubric the gate ships to its Haiku judge. NO forked implementations — voice rubric drift would defeat the gate. Each mode's rubric explicitly forbids preachy / clinical / corporate voice; a structural test pins this. Anchors the cross-cutting voice rule from /plan-ceo-review D2-D8. Fallback policy (D11): Up to 2 generation attempts (configurable). On both rejects → fall back to a hand-written template from src/core/calibration/templates.ts. Templates are intentionally short and a little "robotic" — they're the safety net, not the destination. voice_gate_passed=false + voice_gate_attempts get persisted on the calibration_profiles row so the operator can review the failing examples and tune the rubric over time. Suppressing the surface silently is NEVER an option — that's how voice quality silently degrades. parseJudgeOutput defaults to 'academic' on parse failure (NEVER passes pass-through) so a Haiku output garble falls through to the template rather than letting unverified text reach the user. calibration_profile phase: Extends BaseCyclePhase. Cold-brain skip: <5 resolved takes → no row written, no LLM call. Otherwise: scorecard via engine.getScorecard() → patterns via voice-gated generator → bias tags via separate generator (best-effort; failure logs warning, phase continues). The DB INSERT lands in the v67 calibration_profiles row with source_id, holder, the patterns, voice gate audit fields, active bias tags, and grade_completion (F1 fix — partial-grade state surfaces to the dashboard "60% graded" badge). Budget gate at $0.50/cycle default (mostly Haiku). Below-budget before-LLM-call check returns status='warn' without writing the row. Per-domain scorecards are a placeholder for v0.36.0.0 ship state — the F12 batchGetTakesScorecards() engine method that powers per-domain rendering lands in Lane C alongside the CLI/MCP surface. Architecture: parsePatternStatementsOutput is tolerant of LLM emitting numbered lists / bulleted lines despite the prompt asking for plain lines. Caps at 4 patterns + drops excessively long lines (>200 chars). parseBiasTagsOutput lowercases input + drops non-kebab-case tokens (defends against the LLM emitting "Over-Confident Geography" with spaces or capitals). Caps at 4 tags. Tests: 43 cases across two new test files. voice-gate.test.ts (24): parseJudgeOutput (7), gateVoice happy path (3), fallback path (5), mode parity (2), templates (7). calibration-profile.test.ts (19): parsers (10), pickFallbackSlots (3), phase integration (6 — cold-brain skip, happy path, voice gate fallback, grade_completion plumbed through, bias-tags failure non-fatal, source_id scope reaches INSERT). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cli: gbrain calibration + get_calibration_profile MCP op (T7) Public-facing read surface for the v0.36.0.0 calibration wave. CLI prints the active calibration profile; MCP op exposes the same data path for agents. Mirror of the v0.29 salience/anomalies shape (pure data fn + JSON formatter + human formatter + thin CLI dispatch). CLI: `gbrain calibration` Flags: --holder <id> specific holder (default 'garry') --json machine output for piping --regenerate run calibration_profile phase now --undo-wave <ver> [placeholder — wires in Lane D / T17] ab-report [placeholder — wires in Lane D / T18] Human output: Calibration profile — holder: garry, source: default Generated: <local timestamp> [Note: built on 60% graded — partial completion this cycle.] (when grade_completion < 0.9) [Note: voice gate fell back to template (2 attempts).] (when voice_gate_passed=false) Resolved: 12 takes Brier: 0.210 (lower is better) Accuracy: 60.0% Partial: 10.0% Pattern statements: • You called early-stage tactics well — 8 of 10 held up. Active bias tags: over-confident-geography Cold-brain fallback message names the exact dream command to run. MCP: `get_calibration_profile` (scope: read) Param: holder?: string (defaults to 'garry') Returns: latest CalibrationProfileRow | null Source-scoping via sourceScopeOpts(ctx): scalar source-bound clients see only their source; federated_read scopes see the union of allowed sources; no source filter when neither is set (CLI default path). Throws GBrainError('INVALID_HOLDER') on empty/non-string holder so remote callers get a structured error instead of a SQL-shape failure. Architecture: getLatestProfile is the pure data fn — engine + opts → CalibrationProfileRow | null. Reused by both the CLI and the MCP op. Source-scoped via the standard v0.34.1 spread pattern (scalar sourceId vs sourceIds array). formatProfileText is pure — null → cold-brain message, populated → full printout. Annotates partial-grade rows and voice-gate-fallback rows so the operator sees data-quality status inline. parseArgs is exported via __testing for unit coverage. Sub-command ('ab-report') vs flag distinction is intentional — keeps the surface parallel with `gbrain eval cross-modal` etc. Tests: 21 cases. parseArgs (6 cases): empty, --holder, --json, --regenerate, --undo-wave, ab-report. getLatestProfile (5 cases): happy, null, scalar source scope, federated array scope, no-source-filter default. formatProfileText (5 cases): cold-brain, happy, partial-grade note, voice-fallback note, published-to-mounts note. getCalibrationProfileOp (5 cases): default holder, scalar source scope, federated scope union, returns-null-on-unknown-holder, throws on empty holder. Lane D follow-ups: --undo-wave (T17) and ab-report (T18) print a clear "lands in Lane D" stderr line + exit 2; the surfaces exist for early testers, the implementations land next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * think: --with-calibration + anti-bias prompt rewrite (T8 / E1, D22) Optional anti-bias rewrite mode for `gbrain think`. When set, the active calibration profile gets injected per the D22 placement spec (AFTER retrieval evidence, BEFORE the user's question). The bias filter applies to QUESTION FRAMING, not evidence interpretation — matches LLM-as-judge best practice (bias prompts near end of context perform better). Default behavior unchanged (R1 regression guard): omitting --with-calibration produces the v0.28-vintage user-message shape with the question first, then retrieval. Existing think users see no change. Two user-message shapes in buildThinkUserMessage: Default (no calibration): Question: X <pages>...</pages> <takes>...</takes> <graph>...</graph> Respond with a single JSON object... With calibration (D22): <pages>...</pages> <takes>...</takes> <graph>...</graph> <calibration holder="garry"> Track record: Brier 0.210 (lower is better). Active patterns: - You called early-stage tactics well — 8 of 10 held up. Active bias tags: over-confident-geography </calibration> Question: X Respond... Calibration block is built by buildCalibrationBlock (exported for the E3 contradictions probe to render the same shape). System prompt extension (withCalibration:true): - Names BOTH the user's PRIOR (default reasoning) AND the COUNTER-PRIOR from their hedged-domain self. - References active bias tags by name when relevant ("this fits the over-confident-geography pattern"). - Does NOT silently substitute the debiased answer. ALWAYS surfaces both priors transparently. - Adds a "Calibration" section between Conflicts and Gaps in the answer body. RunThinkOpts extension: - withCalibration?: boolean — opt-in - calibrationHolder?: string — defaults to 'garry' When withCalibration=true and no profile exists, runThink falls back to baseline behavior + pushes NO_CALIBRATION_PROFILE to warnings (visible to the operator). When the calibration fetch fails, CALIBRATION_FETCH_FAILED warning surfaces with the underlying error. Either path keeps think working; the calibration loop is enhancement, not requirement. CLI: `gbrain think "<q>" --with-calibration [--calibration-holder <id>]` Tests: 11 cases. buildThinkSystemPrompt (4 cases): R1 regression — default/false/omitted → no anti-bias rules; with calibration → adds PRIOR + COUNTER-PRIOR + bias-tag reference; preserves existing hard rules. buildCalibrationBlock (3 cases): happy path, null brier omitted (not "Brier null"), empty patterns + tags still well-formed. buildThinkUserMessage (4 cases): R1 regression — without calibration: question first; D22 placement — retrieval → calibration → question → instruction; graph + calibration ordering; empty retrieval blocks render placeholders without breaking shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * contradictions: calibration-profile join (T9 / E3) Cross-references each contradiction finding against the active calibration profile. When a contradiction's domain matches an active bias tag (e.g. "over-confident-geography" or "late-on-macro-tech"), the output gains a one-line bias context explaining which pattern this fits. Pure functions only — no DB writes, no LLM calls. The probe runner imports tagFindingWithCalibration() and applies it to each finding before emitting. When no profile exists or no tags match, the helper returns null and the runner emits the unchanged finding (regression R2 — contradictions output is byte-identical to v0.32.6 when no calibration profile is present). Match heuristic (v0.36.0.0 ship-state): Bias tags are kebab-case axis-then-domain slugs ('over-confident-geography'). computeDomainHint() extracts a domain hint from the finding's slugs + holder + verdict text: - wiki/companies/... → hiring | market-timing - wiki/people/... → founder-behavior - macro / geography / tactics / ai segments in slug → matching tag First-match-wins for ordering determinism. Match is intentionally fuzzy — the v0.32.6 contradictions probe doesn't yet carry structured domain metadata. v0.37+ structured-domain-on-takes (Hindsight-style enum) tightens this. Output: Returns { bias_tag: string, context: string } | null. Context format: "This contradiction fits your active bias pattern \"<tag>\" (Brier 0.31). Verdict: contradiction; severity: medium. Consider reviewing both sides through the lens of that pattern." Tests: 13 cases. R2 regression (2): null profile → null tag; empty active_bias_tags → null tag. computeDomainHint (5): companies / people / macro / geography / unknown paths produce expected hints. Match path (4): macro→late-on-macro-tech, geography→over-confident-geography, mismatch returns null, first-match-wins with multiple candidate tags. buildBiasContextString (2): emits tag+verdict+severity+Brier; omits Brier when null (no "Brier null" leak). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: Brier-trend forecast at write time (T10 / E5) Pure math layer over existing TakesScorecard data. Zero new LLM cost, zero new schema. Surfaces the user's historical Brier for the take's (holder, domain) bucket at write time so they see "your historical Brier in macro takes is 0.31" before committing the take. Voice-gate-rendered output: The user-facing string goes through gateVoice mode='forecast_blurb' via templates.ts (already in T6). This module is the pure data layer; the template renders the math into the conversational voice. v0.36.0.0 ship state: Bucket dimension is the DOMAIN (slug-prefix). The conviction-weight bucket dimension would need a new engine method (engine.batchGetTakeBucketStats per F11) — deferred to v0.37+. Until then, forecast = historical Brier in this holder's domain. resolveDomainPrefix() keeps slug-prefix-looking domain hints ('companies/', 'wiki/macro') and falls back to overall for free-form hints ('macro tech', 'geography'). Hindsight-style structured domain on takes (CDX-11 mitigation TODO) tightens this in v0.37+. MIN_BUCKET_N = 5: Below this sample size, the forecast returns predicted_brier=null with insufficient_data=true. Template renders "Forecast unavailable: only N resolved takes at this conviction yet" instead of a noisy estimate. Architecture: computeForecast(input) — pure function, takes scorecards already fetched; ideal for tests + reuse across batched paths. forecastForTake(engine, input) — convenience wrapper, 1-2 engine round-trips (no domain → 1; with domain → 2). batchForecast(engine, inputs[]) — memoizes per (holder, domainPrefix); N inputs collapse to ≤2*unique_holders unique engine calls. Used by the propose-queue review flow (50 candidates → 1-2 scorecard fetches). Tests: 14 cases. computeForecast (4): insufficient_data branch, stable forecast, overall fallback, MIN_BUCKET_N export. resolveDomainPrefix (5): undefined/empty/whitespace → undefined; slug-prefix → kept; free-form → undefined. forecastForTake (3): 1-call overall, 2-call domain, free-form fallback. batchForecast (2): cache collapse for repeat queries; different holders do not collapse. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: gstack-learnings coupling on incorrect resolutions (T11 / E4) When the grade_takes phase auto-resolves a take as 'incorrect' or 'partial', optionally write a learning entry to gstack's per-project learnings.jsonl so other gstack skills (plan-ceo-review, ship, investigate, ...) can pull it as context when relevant. The brain teaches every other tool about the user's track record. Config gate (D5 / CDX-17 mitigation): `cycle.grade_takes.write_gstack_learnings` defaults FALSE. External users may not have gstack installed; the gstack-learnings binary API isn't stable yet. Garry's brain flips it true to opt in. Quality gate: Only 'incorrect' and 'partial' verdicts trigger the write. 'correct' resolutions are noise (we expected the take to hold up — no learning). 'unresolvable' has no canonical column. Defense-in-depth runtime guard in writeIncorrectResolution() rejects ineligible qualities with reason='quality_not_eligible' so a caller misuse never surfaces a malformed learning entry. Auto-apply only: Coupling fires only when grade_takes both auto-applies AND the verdict is incorrect/partial AND the config flag is enabled. Manual resolutions via `gbrain takes resolve` intentionally DO NOT propagate to gstack — manual writes already carry operator intent; the calibration loop is the noise-prone path that earns coupling. Namespace: Every entry's key starts with 'gbrain:calibration:v0.36.0.0:'. Lane D `gbrain calibration --undo-wave v0.36.0.0` (T17) filters on this prefix for the optional gstack-scrub step. First active bias tag suffixes the key (e.g. 'take-42:over-confident-geography') so future analysis can group learnings by bias pattern. Architecture: buildLearningEntry — pure. Truncates claim at 200 chars + ellipsis; emits Pattern: line when activeBiasTags present; defaults confidence to 0.8 when caller omits it. writeIncorrectResolution — async wrapper. Honors config gate; honors quality gate; calls the injected writer (or defaultGstackWriter in production). Failures are non-fatal: returns { written: false, reason: 'write_failed' | 'binary_missing', error }. The grade_takes phase logs to result.warnings and continues — gstack coupling failure NEVER aborts a cycle. defaultGstackWriter — shells out to gstack-learnings-log binary via execFileSync. Throws GBrainError('GSTACK_BINARY_NOT_FOUND') when the binary isn't on PATH; writeIncorrectResolution classifies that error to reason='binary_missing' so the operator sees the install hint instead of a generic write_failed. Wired into grade-takes.ts after engine.resolveTake() inside the auto-apply block. Only fires when shouldApply=true. Tests: 14 cases. buildLearningEntry (7): canonical shape, partial vs incorrect wording, bias-tag suffix, no-tag fallback, claim truncation, default confidence, no-reasoning omission. writeIncorrectResolution (7): config gate, quality gate, happy path, writer-throw graceful degrade, binary-missing classification, async writer awaited, partial quality writes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * doctor: 4 calibration checks — abandoned/freshness/drift/voice (T12) Adds the four calibration doctor checks per the eng-review spec. abandoned_threads: Counts active high-conviction takes (weight >= 0.7) older than 12 months that have never been superseded. Signal, not error — always status='ok' with a count. The hint sends users to `gbrain calibration` for details. calibration_freshness: Warns when the active profile is older than 7 days (configurable via the same env-var pattern other freshness checks use). Cold-brain branch (no profile yet) returns ok without scolding. Hint points at `gbrain calibration --regenerate`. grade_confidence_drift (CDX-11 mitigation): Surfaces the count of auto-applied grade verdicts. Below 30: returns "need 30+ for drift detection". At/above 30: returns "drift math arrives in v0.37+". The surface is wired; the actual confidence-vs-accuracy correlation math is a v0.37+ follow-up once we have 30+ auto-applied verdicts to measure against. Closes the CDX-11 hole structurally — the operator sees the surface even before the math is meaningful. voice_gate_health: Tracks voice gate failure rate over the last 7 days. <30% fail rate → ok (template fallback is fine in isolation). >=30% → warn with hint to review src/core/calibration/voice-gate.ts rubric. Anchors the cross-cutting voice rule observability story. All four checks return status='warn' with a diagnostic message on engine errors — non-blocking, never throws. Matches the existing doctor check pattern (see checkSyncFreshness for prior art). Wired into runDoctor after checkRerankerHealth (the v0.35 cluster), in the canonical block 10 slot. Tests: 15 cases. 4 per check (happy path, alt-status, engine-throw diagnostic, plus boundary tests for the freshness staleness gate at exactly 7 days and the grade drift gate at 30 applied verdicts). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: E7 nudge + 14-day cooldown (T13 / D16 F3) Real-time pattern surfacing when a newly-committed high-conviction take matches an active bias pattern. Conversational nudge text via the templates module; 14-day cooldown per (take_id, nudge_pattern) via take_nudge_log to prevent the feedback loop where each cycle re-fires the same nudge on the same take. Threshold gates (D16 F3): - holder match (profile.holder === take.holder) - conviction-weight > 0.7 (strict greater than) - take's slug-derived domain hint matches an active bias tag (takeDomainHint — same heuristic as eval-contradictions/calibration-join.ts for cross-surface consistency) Cooldown gate: Before firing, probe take_nudge_log for (take_id, nudge_pattern) rows with fired_at >= now() - 14 days. Any hit → silently skip. After firing, insert a new row with channel='stderr' so the next 14 days are gated. Feedback-loop prevention: User hedges a take in response to a nudge (e.g. weight 0.85 → 0.65). Even though the take's `weight` field changed, the cooldown row for the over-confident-geography pattern is still there from the original fire — so the next cycle's evaluateAndFireNudge() silently skips. The user reset path (gbrain takes nudge --reset N) clears the cooldown to re-arm. Output channel (v0.36.0.0 ship state): STDERR only. Schema's `channel` column already supports multi-channel (webhook, admin SPA toast); routing those is a v0.37+ follow-up. Architecture: evaluateNudgeRule(take, profile) — pure rule check. Returns { matched, reason, matchedTag }. No engine call. checkCooldown(engine, takeId, pattern) — engine probe, returns boolean. recordNudgeFire(engine, opts) — INSERT into take_nudge_log. evaluateAndFireNudge(opts) — full pipeline. Returns NudgeDecision. resetNudgeCooldown(engine, takeId) — DELETE...RETURNING for the CLI. buildNudgeText delegates to templates.ts nudgeTemplate (D24 mode='nudge' voice). v0.36.0.0 ship state uses the template directly; LLM-generated nudge text via the voice gate lands in v0.37+ when we have production examples to tune from. Tests: 22 cases. takeDomainHint (5): companies/people/macro/geography/unrecognized. evaluateNudgeRule (6): no_profile, wrong_holder, conviction-at-threshold- is-NOT-eligible (strict >), no matching tag, happy match, first-match-wins for multiple candidate tags. checkCooldown (3): true on row hit, false on no row, cutoff date param verifies the 14-day boundary. evaluateAndFireNudge (4): happy fire (text contains hush command + matched tag), cooldown silent skip (no INSERT, no stderr), no_profile short-circuit, below-conviction short-circuit (no cooldown query fired). buildNudgeText (2): hush command shape, conviction value embedded. resetNudgeCooldown (2): returns count, idempotent on zero rows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: E8 team-brain sharing + D18 cross-brain query semantics (T14) Cross-brain calibration profile resolution per the D18 4-rule contract. Pins all four cross-brain leak surfaces in dedicated unit tests so future mount features can't silently regress this security model. D18 semantics (committed): Rule 1 — LOCAL-FIRST ORDERING. Query the local brain first. If a profile exists, return it. Do NOT also query mounts (avoids stale-mount-overrides-fresh-local). Verified: mountResolver is NOT called when local has a hit. Rule 2 — MOUNT FALLBACK. Only when local has no profile AND canReadMounts=true, walk the mounts in priority order. First match wins. Each mount-side row must have published=true to be visible (D15 asymmetric opt-in). Rule 3 — CROSS-BRAIN ATTRIBUTION. Every returned profile carries source_brain_id + from_mount flag. Consumers (E1 think rewrite, E3 contradictions, E7 nudge, E6 dashboard) MUST surface this via attributionSuffix() so the user sees which brain answered. Rule 4 — SUBAGENT PROHIBITION. canReadMountsForCtx() classifier returns FALSE for subagent loops without trusted-workspace allowedSlugPrefixes. Closes the OAuth-token-to-cross-brain-leak surface — subagents see ONLY their local-brain results regardless of which holder they query. Exception: trusted cycle phases (synthesize/patterns) pass allowedSlugPrefixes set and ARE allowed to read mounts. Pinned in the classifier test. Architecture: queryAcrossBrains(localEngine, opts) — pure orchestrator. Composes getLatestProfile() from src/commands/calibration.ts. Mount engine access is via opts.mountResolver — production wires this to the v0.19+ gbrain mounts subsystem; tests inject a stub returning an ordered list of mocked engines. Decouples cross-brain LOGIC from multi-engine PLUMBING. canReadMountsForCtx(ctx) — pure classifier table. Drives the rule-4 gate. Production callers compose it from OperationContext. attributionSuffix(result) — pure formatter. Emits the "(from mounted brain: <id>)" suffix when from_mount=true; empty string when local. Mandatory for user-visible cross-brain consumers. Tests: 15 cases pinned to the 4 D18 rules + 4 supplementary structural checks. D18-1: published=false profile on mount stays hidden. D18-2/3: subagent context cannot fall back to mounts (2 cases — null on local-empty + canReadMounts=false, local hit still returned). D18-4: attribution surfaces source_brain_id (3 cases — mount answer flag, local answer flag, attributionSuffix formatter). Rule 1 local-first ordering (2 cases — mountResolver NOT called on local hit, IS called on local empty). Mount priority order (3 cases — first published=true wins, all published=false returns null, no mounts configured returns null without throwing). canReadMountsForCtx classifier (4 cases — local CLI true, MCP non-subagent true, subagent without trusted-workspace false, subagent WITH trusted-workspace true). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * admin: E6 Calibration tab + D23 server-rendered SVG + TD2 contrast bump (T15) Adds the v0.36.0.0 admin SPA Calibration tab. Per the design review, the approved variant-B (Linear calm clarity) layout: single-column flow, generous whitespace, ONE big sparkline as hero, then patterns, then domain bars, then abandoned threads. D23 server-rendered SVG architecture: src/core/calibration/svg-renderer.ts — pure functions. data → SVG string. No DOM, no React, no chart library dep. Inlines the admin design tokens (#0a0a0f bg, #3b82f6 accent, etc.) so the SVG is visually consistent with the rest of the admin SPA. Four chart renderers: - renderBrierTrend({ series }) — sparkline w/ baseline reference at 0.25 (always-50% baseline) - renderDomainBars({ bars }) — horizontal accuracy bars per domain - renderAbandonedThreadsCard(threads) — D30/TD4 'revisit now' link per row, points at /admin/calibration/revisit/<takeId> - renderPatternStatementsCard(statements) — D29/TD3 clickable drill-down links per row, point at /admin/calibration/pattern/<i> XSS posture: all caller-controlled strings pass through escapeXml(). Numeric inputs are .toFixed()-coerced. Admin SPA renders via dangerouslySetInnerHTML inside a TrustedSVG wrapper component; endpoint is gated by requireAdmin middleware. /admin/api/calibration/profile — returns the active profile row as JSON. /admin/api/calibration/charts/:type — returns image/svg+xml markup for type ∈ {brier-trend, domain-bars, pattern-statements, abandoned-threads}. Cache-Control: private, max-age=60. brier-trend currently renders a single-point series from the active profile (the time-series view across calibration_profiles.generated_at history is a v0.37 follow-up once we have multiple snapshots). abandoned-threads pulls the top 5 abandoned rows via the same SQL the doctor check uses. CalibrationPage React component (admin/src/pages/Calibration.tsx): Fetches profile + 4 charts. Loading / error / cold-brain states all handled. Layout includes the audit annotations (partial-grade badge, voice-gate-fell-back-to-template badge) per the approved mockup. TrustedSVG wrapper isolates the dangerouslySetInnerHTML to the SVG surface only. App.tsx nav: added 'calibration' page route + sidebar nav item, hash routing extended to support #calibration. TD2 contrast bump: admin/src/index.css --text-muted: #555 → #777. Old value was contrast 4.0 on the #0a0a0f bg — below WCAG AA 4.5 for body text. New value is ~5.5, passes AA. Improvement is global across Dashboard, Agents, RequestLog, and the new Calibration tab — single-line CSS change with ~10x the impact. admin/dist/ rebuilt via `bun run build` (vite). 36 modules transformed. Tests: 19 cases in test/svg-renderer.test.ts. escapeXml (1): canonical entities. renderBrierTrend (6): empty state, polyline for 2+ points, clamp beyond yMax, design tokens inlined, XSS safety on date strings, text-anchor end on right label. renderDomainBars (4): empty state, label/accuracy/n rendering, out-of-range accuracy clamp, XSS safety on labels. renderAbandonedThreadsCard (4): empty state, row rendering with revisit link, claim truncation at 70 chars, custom revisitHref override. renderPatternStatementsCard (4): empty state, anchor count matches statement count, XSS safety, custom drillHref override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * recall: calibration footer formatter for morning pulse (T16) Pure formatter that turns a CalibrationProfileRow + optional abandoned- threads list into the conversational block the morning pulse will surface: Calibration this quarter: Brier 0.18 (solid). Right on early-stage tactics, late on macro by 18 months. Over-confident on team execution; under-calibrated on regulatory risk. Threads you opened and never came back to: · AI search platform differentiation (17 months silent) · International expansion playbook (12 months silent) Cold-brain branch: returns empty string when no profile or < 5 resolved takes. Caller decides whether to render the block; cold-brain absence is the cleanest non-event. Brier trend note maps the absolute value to conversational copy: <= 0.10 → "(strong calibration)" <= 0.20 → "(solid)" <= 0.25 → "(near baseline)" > 0.25 → "(worse than always-50% baseline — review your high-conviction calls)" v0.36.0.0 ship state has only the current profile snapshot. The "was 0.22 90d ago — improving" comparison shape arrives when we accumulate generated_at history across multiple cycles. R3 regression posture: This module is the FORMATTER only. Wiring into `gbrain recall`'s text output is intentionally NOT in this commit — runRecall's surface stays unchanged. v0.37 wires it under --show-calibration (opt-in initially, default-on later). For now the formatter is callable from the admin tab + custom CLI scripts that want it. Architecture: buildRecallCalibrationFooter(opts) — pure. opts.profile required, opts.abandonedThreads optional, opts.threadColumnWidth defaults to 50. Caps at 4 patterns + 5 abandoned threads to keep the footer scannable. Truncates long abandoned-thread claim text to fit the column width with a trailing ellipsis. Tests: 14 cases. Cold-brain branch (3): null profile, < 5 resolved, zero resolved. Happy path (7): header + Brier + patterns, trend note ranges (4 brackets), null brier omits the Brier line but keeps header, caps at 4 patterns. Abandoned threads (4): omit section when none, emit when present, cap at 5, truncate long claim with column-width override. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: --undo-wave reversal command (T17 / D18 CDX-3) Implements the undo-wave reversal flow. Every new row written by the v0.36.0.0 calibration wave carries wave_version='v0.36.0.0' so a precise revert is possible without touching pre-wave data. CLI surface (replaces the v0.36.0.0 ship-state placeholder): gbrain calibration --undo-wave v0.36.0.0 [--dry-run] [--scrub-gstack] [--json] Reversal scope (4 steps): Step 1 — UNSET takes.resolved_* columns for takes auto-applied by this wave. Identifies wave-applied takes via take_grade_cache.applied=true + wave_version match. Cross-checks resolved_by='gbrain:grade_takes' to ensure we're not un-resolving a take a manual `gbrain takes resolve` override has since claimed. Manual resolutions persist; only auto-grade resolutions revert. Step 1b — Mark take_grade_cache rows applied=false post-undo so the audit trail shows they WERE applied but this wave was reverted. The CDX-11 confidence-drift check filters on applied=true and gets a cleaner sample post-undo. Step 2 — DELETE FROM calibration_profiles WHERE wave_version = ?. Step 3 — DELETE FROM take_nudge_log WHERE wave_version = ?. Step 4 — Optional gstack-learnings-prune via the binary, scoped to the GSTACK_LEARNING_NAMESPACE prefix. Opt-in via --scrub-gstack. Best-effort: binary-missing or failure logs a warning + suggests the manual command; the rest of the undo still succeeded. Dry-run posture: --dry-run computes the counts via SELECT COUNT(*) shapes without emitting any UPDATE or DELETE. Same UndoWaveResult shape returned so operator sees exactly what would be reverted before committing. --dry-run intentionally skips the gstack scrub (filesystem write) too; ship-state safety call. Idempotency: Re-running --undo-wave on a brain that's already reverted is a no-op. Each query filters on wave_version; no matching rows → zero counts. Architecture: undoWave(engine, opts) — async, returns UndoWaveResult. Pure data layer; no stderr writes, no process exits. CLI dispatch in src/commands/calibration.ts handles printing. v0.36.0.0 ship state runs steps 1-3 sequentially (no transaction). Partial reversal is recoverable via re-run since each step is idempotent on wave_version match. A future enhancement (v0.37+) can wrap in engine.transaction once that surface lands in BrainEngine. Tests: 8 cases in test/undo-wave.test.ts. Dry-run posture (1): counts emitted, NO UPDATE/DELETE SQL fired. Happy path (3): all 4 steps execute, resolved_by filter scopes UPDATE to wave-applied resolutions, custom resolvedByLabel honored. Empty wave (2): zero counts when no matching rows, idempotent re-run. Wave-version parameter threading (2): supplied version threads through all queries, different wave versions don't collide. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: A/B harness for think + ab-report (T18 / D19 CDX-18) Structural answer to CDX-18 (anti-bias rewrite may make advice worse). We don't have to guess whether calibration helps — we measure. Architecture: runAbTrial(input) — calls thinkRunner TWICE on the same question (baseline + --with-calibration), surfaces both answers to a preferenceResolver, persists the trial to think_ab_results. buildAbReport(engine, { days }) — aggregates the table over the last N days (default 30). Computes win counts, ties, neither, and a with_calibration_win_rate over DECISIVE trials only (excludes neither/tie). Flags calibration_net_negative when n >= 20 AND win rate < 45%. formatAbReport(report, days) — pretty-prints for stdout; emits the calibration_net_negative warning block when triggered. CLI: gbrain calibration ab-report [--days N] [--json] Reads the table, prints the breakdown. Replaces the v0.36.0.0 ship-state placeholder in src/commands/calibration.ts. gbrain think --ab "<question>" Wires into runAbTrial via the dispatch in src/commands/think.ts — follow-up commit. This commit lands the harness layer + schema + report surface; the --ab flag itself flips on in a one-line wiring commit when the runRecall path is ready. Schema (migration v72 / think_ab_results): source_id, wave_version, ran_at, question, baseline_answer, with_calibration_answer, preferred (CHECK in {baseline, with_calibration, neither, tie}), model_id, notes. CHECK constraint enforces preferred enum. Default wave_version 'v0.36.0.0' stamped so --undo-wave can scrub these too. Index on (source_id, ran_at DESC) supports the report's "last N days" query. schema.sql + pglite-schema.ts both updated for fresh-install parity. schema-embedded.ts regenerated via build:schema. calibration_net_negative threshold (D19): Triggers when: - decisive_trials (baseline + with_calibration) >= 20 - with_calibration_win_rate < 0.45 (NOT <= — exact 45% is OK) Small-sample guard (n < 20) prevents the warning from firing on early data with sampling noise. Confidence-flat threshold (no Wilson CI yet) keeps the math simple; v0.37+ adds CI bounds. Tests: 12 cases in test/think-ab.test.ts. runAbTrial (4): both runner calls fire, preferenceResolver receives both answers, INSERT row params shape, throws when thinkRunner missing. buildAbReport (5): zero trials, aggregation, net_negative trigger at n>=20 + win<45%, no trigger at n<20 (small-sample guard), no trigger at exact 45% boundary. formatAbReport (3): zero-state message, decisive-trials breakdown, net_negative warning block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: pattern drill-down route + revisit-now CLI (TD3 / D29 + TD4 / D30) TD3 (D29) — clickable pattern drill-down endpoint: GET /admin/api/calibration/pattern/:id (requireAdmin) Returns the pattern statement at index `id` plus the top 25 resolved takes for the holder, sorted by weight desc. v0.36.0.0 ship-state approximation: surfaces broad provenance evidence (top resolved takes). v0.37+ stores per-pattern source_take_ids[] on a calibration_profile_patterns join table so the drill-down shows the EXACT takes that drove the pattern. Surfaces a `provenance_note` field in the response so the operator sees the v0.36.0.0-vs-v0.37 fidelity boundary inline. The admin SPA's renderPatternStatementsCard SVG already emits anchor tags pointing at /admin/calibration/pattern/<i> (T15 ship state). This route makes those anchors clickable — closes the trust loop that was the rationale for D29 ("pattern statements without their evidence are dressed-up LLM hallucinations"). TD4 (D30) — `gbrain takes revisit <slug>` editor-open action: Adds the `revisit` subcommand to gbrain takes. Opens $EDITOR (falling back to vi) on the source markdown file for the slug. Appends a `<!-- gbrain:revisit -->` cursor marker at the bottom of the page on first invocation so the editor opens with intent visible. Reads sync.repo_path from config to locate the brain repo. Refuses to proceed with a clear error when the repo isn't configured or the page doesn't exist. spawnSync with stdio:'inherit' so the editor takes the terminal. Exit status surfaced on failure. The SVG renderer's revisit-now anchor for each abandoned thread row emits /admin/calibration/revisit/<takeId>. A small route handler that resolves take_id → page_slug then dispatches `gbrain takes revisit` via spawn is a v0.37 follow-up — the CLI command exists now so developers can wire it directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: DESIGN.md — formalize de facto design tokens (TD1) Promotes the admin SPA's de facto design tokens (landed v0.26.0) to a canonical DESIGN.md at the repo root. This is the calibration target for /plan-design-review and /design-review going forward — when a question is "does this UI fit the system?", the answer is here. Captures the system as it stands today: Voice (5 surfaces, all routed through gateVoice() with mode-specific rubrics): pattern_statement, nudge, forecast_blurb, dashboard_caption, morning_pulse. Friend-not-doctor; concrete data over abstract metrics; no preachy / clinical / corporate language. Color tokens: 10 CSS variables from admin/src/index.css inlined into the SVG renderer (src/core/calibration/svg-renderer.ts). Dark theme is the only theme — admin is an operator tool. WCAG contrast documented per token; TD2's #555 → #777 bump on --text-muted noted. Typography: Inter for UI, JetBrains Mono for numbers/slugs/data. Type scale (18 / 14 / 13 / 12 / 11) documented as de facto, not yet formalized. Spacing scale: 4 / 8 / 16 / 24 / 32px. Linear-app density. Layout: sidebar 200px, max content 720px (text) / 960px (tables). No 3-column feature grids, no icons in colored circles, no decorative blobs. Charts: server-rendered SVG via pure functions in src/core/calibration/svg-renderer.ts. XSS posture documented: server-side escapeXml on caller-controlled strings, numeric inputs .toFixed()-coerced, admin SPA renders via <TrustedSVG> wrapper. Interaction patterns: keyboard nav required (J/K/space/u/q on the propose-queue), loading/empty/error states ARE features. v0.37+ roadmap: type scale formalization, animation tokens, component library extraction. Light mode explicitly NOT planned. The doc is a living target, not a frozen spec. Major changes route through /plan-design-review per the existing review chain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: synthetic corpus scaffold + privacy CI guard (T19 + T20) T19 — synthetic corpus scaffold for extract-takes prompt tuning. test/fixtures/calibration/extract-takes-corpus/ — 5 representative pages across 4 genres (essay, people, companies, meetings, decisions). v0.36.0.0 ships a SMALL representative corpus as proof of structure; the full 50-page training set + 10-page holdout gets generated by the operator via `gbrain calibration build-corpus` (v0.37 follow-up subcommand) or by hand with the privacy guard catching violations either way. Privacy contract per D13': every page is SYNTHETIC. None of the names/companies/funds/deals/events refer to anything real. Placeholder names per CLAUDE.md: alice-example, charlie-example, acme-example, widget-co, fund-a/b/c, acme-seed, widget-series-a, meetings/2026-04-03. test/fixtures/calibration/README.md spells out the privacy contract, generation flow, and what the corpus is (stable regression set for the extract-takes prompt) vs is not (real anything). T20 — privacy CI guard (CDX-14 mitigation). scripts/check-synthetic-corpus-privacy.sh greps the corpus for: 1. Explicit dollar amounts ($50M, $1.2B etc) — would suggest the page memorized a real round size. 2. Out-of-range year references (informational only for v0.36.0.0; deferred to a manual review checklist). 3. Pages that reference ZERO placeholder names — suggests the page might be referring to real entities. Essay-genre fixtures exempt (they're anonymized PG-style writing by design). Wired into `bun run verify` (CI gate) so contributors can't accidentally land a synthetic fixture that leaks real-world specificity. The intent is fail-fast on accidental leakage; the operator can update the allowlist if a generic dollar amount is intentional. Closes CDX-14: 'CC reads real brain pages locally, writes nothing still risks privacy if any generated synthetic fixture memorizes structure-specific facts. Placeholder names are not enough.' The corpus shipped here is intentionally small but covers the four core gbrain page genres (essay, people, companies, meetings/decisions). The v0.37 corpus-build subcommand will fan out to 50 with the operator spot-checking + the CI guard enforcing the privacy contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: R1-R5 IRON RULE regression inventory (T21) Per /plan-eng-review D26 IRON RULE: regressions get added to the test suite as critical requirements, no AskUserQuestion needed. Pins five regressions identified during the v0.36.0.0 wave's coverage diagram: R1: think baseline UNCHANGED when --with-calibration absent. Covered structurally by test/think-with-calibration.test.ts plus assertion-pinned in this file (default user message: question first, then retrieval; system prompt: no anti-bias section). R2: contradictions probe output UNCHANGED when no calibration profile. Covered structurally by test/eval-contradictions-calibration-join.test.ts plus pinned here (null profile → null tag, byte-identical to v0.32.6). R3: takes resolution flow works when grade_takes phase disabled. Pinned import-surface coupling: takes-resolution.ts has zero dependency on grade_takes module. If a future refactor accidentally couples them, this test fails to compile. R4: search/list_pages/get_page work identically through new source_id paths. Marker test referencing existing v0.34.1 source-isolation suite at test/source-isolation-pglite.test.ts. v0.36.0.0 does NOT modify those code paths; the existing tests catch any accidental coupling. R5: existing search modes (conservative/balanced/tokenmax) unaffected. Marker test referencing existing test/search-mode.test.ts. The calibration code DOES NOT IMPORT from src/core/search/mode.ts. Plus an inventory test that confirms all 5 regressions have an 'addressed' status — fail-loud if a future contributor removes a guard without updating the inventory. 7 tests total. Pure functions, no engine, hermetic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: v0.36.0.0 CHANGELOG + CLAUDE.md anchors + calibration convention skill CHANGELOG entry: the user-facing release notes. Leads with the headline ("the brain learns how you tend to be wrong, then argues against your blind spots on every advice call"), 5 'what you can now do' bullets in GStack voice, itemized changes by lane, and the 'To take advantage of v0.36.0.0' upgrade checklist per the CLAUDE.md required-block contract. CLAUDE.md anchors: new 'v0.36.0.0 Hindsight calibration wave (key files cluster)' block inserted before the v0.31.1 thin-client section. 23 new files / extensions annotated with one-paragraph descriptions each, linking back to the convention skill at skills/conventions/calibration.md for the agent-facing rules. skills/conventions/calibration.md: the agent-facing convention skill. Tells future contributors which calibration touchpoint applies to their task — voice gate? BaseCyclePhase? source-scope thread? doctor warning? cross-brain query rules? auto-resolve threshold posture? Test seam patterns. Bug class to avoid (the v0.34.1 source-isolation leak shape). Version trio (per CLAUDE.md mandatory audit): VERSION: 0.36.0.0 package.json: 0.36.0.0 CHANGELOG: ## [0.36.0.0] - 2026-05-17 llms.txt + llms-full.txt regenerated via `bun run build:llms` after the CLAUDE.md edit (per the explicit CLAUDE.md mandate "Any CLAUDE.md edit MUST be followed by `bun run build:llms`"). The `test/build-llms.test.ts` guard runs in CI shard 1; the committed bundles are checked against fresh generator output. bun run verify is clean. typecheck clean. Privacy CI guard passes (0 violations across 6 corpus pages). All ready for /ship. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cycle: wire propose_takes / grade_takes / calibration_profile into runCycle (T-fix) The three new v0.36.0.0 phases were declared in CyclePhase / ALL_PHASES / NEEDS_LOCK_PHASES but the runCycle orchestrator never dispatched them. ALL_PHASES advertised them, gbrain dream --phase propose_takes accepted them, but `gbrain dream` (default) silently skipped all three. Adds a single dispatch block between consolidate and embed that: - builds an OperationContext on the fly (trusted-workspace caller, remote: false, sourceId resolved via the same helper sync uses) - dispatches the three phases in the order ALL_PHASES declares - records the same skipped-phase shape (no_database) when engine is null Pinned by test/core/cycle.serial.test.ts "default: all 6 phases run in order" which was already failing against ALL_PHASES (the test name lags the actual phase count; left as-is since renaming churns history). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: expand synthetic corpus + add hand-labeled ground-truth (T19) Adds 8 new synthetic pages modeled on the genre mix observed in the real brain (concepts-with-timeline, meeting-notes, daily-journal, people-pages, essays). Companion .gradeable-claims.json files carry hand-labeled answer keys — what a tuned propose_takes prompt SHOULD extract per page. Closes the F1 gate gap from the plan's T19/D19: Training corpus (test/fixtures/calibration/extract-takes-corpus/): + concept-startup-market-dynamics.md (10 claims) + meeting-2026-04-10-fundraise-fund-a.md (6 claims) + daily-2026-04-15.md (5 claims) Blind holdout (test/fixtures/calibration/holdout/): + concept-founder-execution.md (6 claims, F1 >= 0.80) + daily-2026-04-18.md (4 claims, F1 >= 0.80) + meeting-2026-04-17-hiring-charlie.md (5 claims, F1 >= 0.80) + essay-on-conviction.md (7 claims, F1 >= 0.80) + people-bob-example.md (5 claims, F1 >= 0.80) Privacy: - No real-brain content read into any committed artifact. Pages written from scratch using the canonical placeholder set (alice-example, charlie-example, bob-example, acme-example, widget-co, fund-a/b/c). Real-name grep confirms zero leakage: wintermute, garrytan, paul-graham, sam-altman, etc. → 0 hits. - scripts/check-synthetic-corpus-privacy.sh passes: 0 violations across 14 pages (was 6). Genre fidelity: - concept-with-timeline pages mirror the dated-assertion structure real brain uses (verb framing varies: "argues / predicts / I think / I bet / strong conviction / moderate conviction"). - meeting-notes pages carry both prose claims (extracted via hedging language) and explicit ## Takes sections. - daily-journal pages test probabilistic framing ("75/25 in favor", "call it ~0.5") and self-tagged conviction values. - essay-on-conviction is the meta-page that names the author's own bias patterns — primary signal for calibration_profile. - people pages test claim-about-third-party extraction. Each JSON ground-truth lists per-claim: - claim_text + kind (prediction|judgment|bet) + domain - conviction (0..1) - since_date - rationale (why this claim is gradeable + how a tuned prompt should infer conviction from the prose) This is the corpus that gates the T19 prompt-tune iteration: - F1 >= 0.85 on training (10+6+5 = 21 claims across 3 pages plus the existing 5 fixtures already shipped) - F1 >= 0.80 on holdout (27 claims across 5 pages) Plan reference: ~/.claude/plans/system-instruction-you-are-working-rippling-knuth.md Privacy gate: scripts/check-synthetic-corpus-privacy.sh (wired into bun run verify). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * calibration: tune propose_takes prompt against synthetic corpus (cat15 F1 0.92+) The v0.36.1.0 ship state shipped propose_takes with a stub prompt that the docs flagged as "tune via T19 corpus build before relying on propose_takes in production." T19's corpus was built in commit 69a71c9d (14 synthetic pages + 48 hand-labeled claims). The matching gbrain-evals cat15 runner validates extraction quality against that corpus. This commit back-ports the tuned prompt validated by cat15's first live run: training avg F1: 0.952 (target 0.85, +10 points) holdout avg F1: 0.922 (target 0.80, +12 points) train-holdout gap: 0.03 (well below 0.10 overfitting threshold) 8/8 probes pass their individual F1 targets Per-genre F1 floor: 0.80 (people-pages, the hardest genre). Concept- with-timeline and meeting-notes genres scored at 1.00 on holdout pages. The tuned prompt design changes vs the stub: - Worked example list seeds the "gradeable claim" notion so the model doesn't drift into pure-fact extraction. - NOT-gradeable list catches the most common over-extraction modes (pure facts, direct quotes, restatements). - Conviction inference rules anchored to specific hedging language so the model produces consistent weight values. - kind enum narrowed to 'prediction' | 'judgment' | 'bet' — the v1 stub's 4-tag enum bled into noise classification on the corpus. PROPOSE_TAKES_PROMPT_VERSION bumped 'v0.36.1.0-stub' → 'v0.36.1.0-tuned-cat15'. The bump invalidates the take_proposals idempotency cache so existing proposal rows stay as audit history but the next cycle re-extracts against the new prompt — exactly the design contract this version field is for. Re-tuning protocol: run cat15 in gbrain-evals against the fixtures BEFORE bumping the version string. The train-holdout gap should stay < 0.10. If a future tune drops below the cat15 gate, revert. Source of evidence: - cat15 runner: ~/git/gbrain-evals/eval/runner/cat15-propose-takes.ts - Fixture corpus: test/fixtures/calibration/ (this repo, commit 69a71c9d) - Live run dumps: ~/git/gbrain-evals/eval/reports/cat15-propose-takes/*.json Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: link cat14/cat15 benchmark report from CHANGELOG + README Adds the "Validated by published benchmarks" subsection to the v0.36.1.0 CHANGELOG entry and a "Calibration loop" section to the README's "Receipts on the evals" surface. Both link to the new benchmark report at gbrain-evals/docs/benchmarks/2026-05-18-brainbench-cat14-cat15-calibration.md. CHANGELOG: also updates the propose_takes bullet to reflect that the v0.36.1.0 ship state now includes the tuned 'v0.36.1.0-tuned-cat15' prompt (back-ported in 04dbab44), not the v1 stub the original entry described. README: adds a Calibration loop entry to the receipts table sitting between source-aware ranking and prompt compression. Frames the cat14 + cat15 numbers as "first published benchmark for AI memory systems that reason about user track records" — honest SOTA framing since Hindsight introduced the concept without quantified evaluation. llms.txt + llms-full.txt regenerated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: fix benchmark-report links — gbrain-evals uses main not master 7 links to gbrain-evals/blob/master/docs/benchmarks/ were broken — the gbrain-evals repo uses 'main' as its default branch, not 'master'. Surfaced when I checked that the new cat14/cat15 link resolved post-PR-9 merge. Turned out 4 pre-existing links to longmemeval, brainbench-v0.20, brainbench-cat13b-source-swamp, and comparison-systems were all broken for the same reason — I just added a fifth by following the same wrong pattern. Sweep: gbrain-evals/blob/master/ → gbrain-evals/blob/main/ across both README.md (5 links) and CHANGELOG.md (2 links). llms.txt + llms-full.txt regenerated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 19:34:44 -07:00
3 people	9fb4d7eb5b	v0.33.3.0 feat(v0.33.3): code intelligence MCP foundation (v0.34 W0a-c + W3) (#934) ... * feat(v0.34 pre-w0): add code-retrieval eval harness for v0.34 ship gate Captures pre-v0.34 retrieval quality on the gbrain self-corpus before any code-intel work lands, so the v0.34 ship gate (precision@5 +10pp OR answered_rate +15pp on >=15/30 questions) measures real improvement rather than an after-the-fact retuned baseline. * src/eval/code-retrieval/harness.ts -- pure-function metrics (precision@k, recall@k, top-1 stability, gate evaluator) + EvalRunReport types stable across schema_version 1 * src/eval/code-retrieval/questions.json -- 30 questions across callers / callees / definition / references / blast_radius / execution_flow / cluster_membership kinds, expected_files captured against current gbrain layout * src/eval/code-retrieval/strategies.ts -- BaselineStrategy (hybridSearch) + WithCodeIntelStrategy stub (post-W3 fills in code_blast/code_flow/etc.) * src/commands/eval-code-retrieval.ts -- gbrain eval code-retrieval CLI with --baseline / --with-code-intel / --compare subcommands * test/code-retrieval-harness.test.ts -- 26 unit tests across metrics, loader, gate logic; no engine dependency PRE-V0.34 BASELINE WORKFLOW: gbrain eval code-retrieval --baseline --save /tmp/baseline-1.json (run 3x for noise floor) V0.34 SHIP GATE (after W3 lands): gbrain eval code-retrieval --with-code-intel --save /tmp/v034.json gbrain eval code-retrieval --compare /tmp/baseline-1.json /tmp/v034.json Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.34 W0a): source-routing leak across query + two-pass Codex outside-voice review on the v0.34 plan caught two load-bearing sites where sourceId was advertised but never applied — multi-source brains silently cross-contaminated structural retrieval: * operations.ts ~323 — `query` op handler called hybridSearch without threading ctx.sourceId. Multi-source agents querying with a --source flag got cross-source results. * two-pass.ts:81 (nearSymbol lookup) and two-pass.ts:131 (unresolved edge resolution) — TwoPassOpts.sourceId was declared and threaded through hybridSearch's expandAnchors call, but the actual SQL ignored it. The walk window crossed source boundaries every time. Fix: * `query` op now reads ctx.sourceId AND accepts a new `source_id` param (with '__all__' as the explicit force-cross-source escape hatch). Per-call param wins over ctx context. * two-pass.ts both lookups join through pages.source_id when opts.sourceId is set; omitted opts.sourceId preserves the legacy cross-source contract for callers who want it. Regression test: test/e2e/source-routing.test.ts seeds two sources with the same `parseMarkdown` symbol + a cross-source caller edge. Pins: - nearSymbol + sourceId='source-a' returns ONLY source-a chunks - nearSymbol + sourceId='source-b' returns ONLY source-b chunks - nearSymbol with no sourceId still crosses sources (contract preserved) - walk_depth=1 unresolved-edge resolution stays in source-a PGLite in-memory, no DATABASE_URL needed. The fix proves out under realistic structural retrieval not just a contrived unit test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.34 W0b): flip CLI source-scoping default to truly source-scoped Codex outside-voice review (finding #7) caught that the v0.20.0 docstring claim "by default we only match the caller's source_id" contradicted the implementation in code-callers.ts:54 + code-callees.ts:43: allSources: allSources || !sourceId The right side made `allSources` TRUE whenever `--source` was omitted, INVERTING the documented default. Multi-source brains silently cross- contaminated structural retrieval; `gbrain code-callers parseMarkdown` on a brain with two repos returned callers from both even though the docstring promised per-source scoping. Fix: * New canonical helper `resolveDefaultSource(engine)` in sources-ops.ts. Contract per eng review D7: - exactly 1 source registered → return its id (single-source brains, the 80% case; --source flag is unnecessary friction there) - 2+ sources → throw SourceResolutionError(multiple_sources_ambiguous) with the list of valid ids - 0 sources → throw SourceResolutionError(no_sources) * code-callers.ts + code-callees.ts now resolve to the default source when both --source AND --all-sources are absent. To get the pre-v0.34 cross-source behavior, callers must pass --all-sources explicitly. * Same hint text on both commands. Pinned by test/e2e/cli-source-scoping-pglite.test.ts. IRON RULE regression R2: docstring promise now holds. Multi-source brain running `gbrain code-callers <symbol>` without --source gets a clear error listing valid source ids instead of silent cross-resolution. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.34 W0c): within-file two-pass symbol resolver + edges_backfilled_at watermark Codex's outside-voice review caught that the v0.20.0 graph stores BARE callee tokens (`render`, `find`, `execute`) — not qualified names. Pre-v0.34 recursive blast/flow would alias every same-named function across classes. W0c is the foundation that fixes this: resolve `code_edges_symbol` rows by matching `to_symbol_qualified` against the SAME-FILE chunks' `symbol_name_qualified`, then write the outcome to `edge_metadata`. This commit is the resolver primitive + schema. The cycle-phase wiring that calls it on every quick-cycle tick lands in the next commit. Schema (v51 migration `edges_backfilled_at_v0_34`): * `content_chunks.edges_backfilled_at TIMESTAMPTZ` — resume watermark. Chunks where the column is NULL OR older than EDGE_EXTRACTOR_VERSION_TS get re-walked next tick. SIGINT/OOM/sleep mid-backfill loses at most one batch. * Indexes per D11 from eng review: - `idx_code_edges_symbol_resolver(source_id, to_symbol_qualified)` — composite for the resolver's per-source lookup. - `idx_content_chunks_symbol_lookup(page_id, symbol_name_qualified)` WHERE `symbol_name_qualified IS NOT NULL` — file-batched candidate fetch; also reused by W4-5 cluster recompute. - `idx_content_chunks_edges_backfill(edges_backfilled_at)` WHERE `edges_backfilled_at IS NULL` — fast unresumed-row scan. Module (`src/core/chunkers/symbol-resolver.ts`): * `resolveSymbolEdgesIncremental(engine, {sourceId, maxChunks?, onProgress?})` walks stale chunks in 200-chunk batches. For each chunk, loads its unresolved edges, finds same-page candidates by symbol_name_qualified, and writes outcome to `edge_metadata`: - exactly 1 candidate → `{resolved_chunk_id: <id>}` - 2+ candidates → `{ambiguous: true, candidates: [...]}` - 0 candidates → unchanged (cross-file; two-pass.ts handles those) Each batch bumps `edges_backfilled_at = NOW()` for the chunks. * `readEdgeResolution(metadata)` — public helper for downstream code (two-pass.ts, code_blast op, eval-capture) to consume the resolver's output without parsing JSON directly. Returns a tagged union. * `EDGE_EXTRACTOR_VERSION_TS` exported constant — bump when extractor shape changes and the next cycle re-walks all chunks. Tests (5 E2E in test/e2e/symbol-resolver-pglite.test.ts, all PGLite, no DATABASE_URL): unambiguous match, ambiguous multi-match, no match, watermark advance + idempotency, source isolation (no cross-source candidate leak). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.34 W0c): wire resolve_symbol_edges as a new cycle phase W0c's symbol resolver lands as a 12th cycle phase between extract and patterns. The autopilot's quick-cycle path (60s watchdog interval per D2 from eng review) now resolves stale chunks incrementally so agents see resolved edges within ~60s of writes rather than waiting on the slow full-walk path. * CyclePhase + ALL_PHASES + NEEDS_LOCK_PHASES extended with 'resolve_symbol_edges'. Position: between extract (which emits new bare-token edges from sync diffs) and patterns (which reads the graph). Acquires the cycle lock because it writes edge_metadata. * CycleReport.totals adds edges_resolved + edges_ambiguous so doctor and autopilot summaries surface the numbers. * runPhaseResolveSymbolEdges walks every registered source via listSources() + resolveSymbolEdgesIncremental(). Per-call cap is BATCH_SIZE*10 = 2000 chunks so a single watchdog tick stays bounded even on a 100K-chunk brain. Subsequent ticks pick up the leftovers via the edges_backfilled_at watermark. * Test count bumped from 11 → 12 phases in cycle.serial.test.ts and cycle.test.ts (both pinned by the regression guards). Existing 28 cycle tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.34 W3): MCP-expose code_callers / code_callees / code_def / code_refs Pre-v0.34 these four code-intelligence commands lived in CLI_ONLY at cli.ts:30 — agents calling gbrain via MCP couldn't reach them and fell through to text search. This commit ships the agent-facing MCP surface for v0.34 against the existing v0.20+ tree-sitter call graph; recursive blast/flow and clusters land in subsequent commits. * `code_callers(symbol, [limit, source_id, all_sources])` — wraps engine.getCallersOf. Reverse view of the A1 call graph. * `code_callees(symbol, [limit, source_id, all_sources])` — wraps engine.getCalleesOf. Forward view. * `code_def(symbol, [limit, lang])` — wraps findCodeDef. Returns definition sites with file/line/snippet. * `code_refs(symbol, [limit, lang])` — wraps findCodeRefs. Returns every reference (comments, strings, imports, call sites). All four are scope:'read', source-scoped by default via ctx.sourceId (W0a contract). Per-call source_id param wins over ctx; pass '__all__' or all_sources=true to force cross-source. * operations-descriptions.ts: 4 new constants per the eng review D10 finding — every description carries an inline example response so agents don't burn first-call context discovering shape. Resolver-grade wording ("BEFORE editing any function, run code_callers...") routes plan-mode questions straight to the right op. * SEARCH_DESCRIPTION gains a cross-link clause pointing at the four new ops so agents stop falling through to text search for code-symbol questions. Tests (11 E2E in test/e2e/code-intel-mcp-ops-pglite.test.ts): - All four ops registered + scope:read + description pinned by constant - All four ops have required symbol param - code_callers / code_callees return the documented envelope shape - Source scoping honors ctx.sourceId - all_sources=true / source_id='__all__' force cross-source - code_def returns the def-site snippet Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(v0.33.0): agent-readable migration doc for the code-intel foundation skills/migrations/v0.33.0.md gives existing-user upgrade guidance for the v0.33.0 foundation pre-release (this branch's accumulated work toward v0.34 Cathedral III): * Source-routing fix (Codex #2) — query / two-pass now honor sourceId * CLI source-scoping default flipped (Codex #7) — gbrain code-callers defaults to source-scoped, --all-sources is the explicit opt-out * MCP exposure of code-callers / code-callees / code-def / code-refs with resolver-grade descriptions agents auto-route to * Within-file symbol resolver runs as a new `resolve_symbol_edges` cycle phase between extract and patterns * Schema migration v51: edges_backfilled_at watermark + 3 composite/ partial indexes for the resolver hot path * Verification commands the agent runs after `gbrain upgrade` Bumps the existing-user migration ladder so the auto-update agent (SKILLPACK Section 17) discovers + runs the v0.33.0 migration steps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(v0.33.0): bump VERSION + package.json + CHANGELOG v0.33.0 ships the v0.34 Cathedral III foundation: MCP exposure of code_callers / code_callees / code_def / code_refs with resolver-grade tool descriptions, plus the source-routing fix + within-file symbol resolver + cycle-phase wiring that v0.34's recursive blast/flow and Leiden clusters will build on. Full release notes in CHANGELOG.md. Trio in lockstep: VERSION: 0.33.0 package.json: 0.33.0 CHANGELOG.md: ## [0.33.0] - 2026-05-11 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(v0.33.0): update dream-cycle phase-order assertions for resolve_symbol_edges E2E test pinned the canonical phase sequence as a regression guard. The v0.33.0 resolve_symbol_edges phase (added between extract and patterns) correctly bumps the count to 12 — caught by the canonical-order test on fresh-Postgres run, fixed by adding the new phase to EXPECTED_PHASES and bumping the version history comment. Both cycle.serial.test.ts and cycle.test.ts were already updated in the W0c cycle-phase commit (6f7dbe1d); this third pin lives in test/e2e/dream-cycle-phase-order-pglite.test.ts and was missed. Full E2E suite now: 550 passed / 0 failed / 81 files (real Postgres on port 5435 via Docker pgvector/pgvector:pg16). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(v0.33.3.0): rebump from v0.33.2.0 → v0.33.3.0 User asked to ship as v0.33.3.0 instead of v0.33.2.0. Single sweep: * VERSION + package.json bumped to 0.33.3.0 * CHANGELOG header + body rewritten to v0.33.3 * skills/migrations/v0.33.0.md → skills/migrations/v0.33.3.0.md (migration files use the version they ship FROM; renaming aligns with the v0.21.0.md / v0.31.0.md convention in CLAUDE.md) * Schema migration name edges_backfilled_at_v0_33_2 → edges_backfilled_at_v0_33_3 in src/core/migrate.ts (also bumps the in-code identifier so the registry name matches the version) * All v0.33.2 comment references swept to v0.33.3 in cycle.ts, operations.ts, operations-descriptions.ts, eval.ts, symbol-resolver.ts + cycle test phase-history comments * llms.txt + llms-full.txt regenerated Trio verified: VERSION: 0.33.3.0 package.json: 0.33.3.0 CHANGELOG.md: ## [0.33.3.0] - 2026-05-12 bun run verify clean; 90 v0.33.3-touched tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-14 19:50:50 -04:00
3 people	a73108b26f	v0.32.2 feat: facts join system-of-record + 3-layer privacy + CI invariant gate (#885) ... * schema: migration v51 facts_fence_columns + fresh-install parity v0.32.2 commit 1/11. Facts become FS-canonical via a `## Facts` fence on entity pages (mirror of takes-fence). row_num + source_markdown_slug are the round-trip columns the fence parser uses to reconcile markdown → DB. Schema changes: - ALTER TABLE facts ADD COLUMN IF NOT EXISTS row_num INTEGER - ALTER TABLE facts ADD COLUMN IF NOT EXISTS source_markdown_slug TEXT - CREATE UNIQUE INDEX idx_facts_fence_key (source_id, source_markdown_slug, row_num) WHERE row_num IS NOT NULL Both columns nullable: pre-v0.32 rows don't have them until commit 6's v0_32_2 orchestrator backfills via fence-append. The partial WHERE clause is the Codex R2 collision guard — without it, two pre-v51 NULL-row_num rows on the same (source_id, source_markdown_slug) coordinate would collide and fail the migration on any populated v0.31 brain. Fresh-install parity: the v40 CREATE TABLE block now declares the columns from the start, so a brand-new install hits a single CREATE that already has them and the v51 ALTERs no-op via IF NOT EXISTS. Existing brains pick them up through the v51 migration. Idempotent under all states (re-runs are no-ops). Metadata-only ALTERs on PG 11+ and PGLite — no table rewrite. Partial-index syntax verified against v40's existing idx_facts_unconsolidated precedent. Tests: - 6 new v51 cases in test/migrate.test.ts covering name, ADD COLUMN shape, nullable contract, partial-unique-index keys, the WHERE-NULL collision guard, and LATEST_VERSION progression. - All 109 migration tests pass (was 103); schema walks 15 → 51 cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: facts-fence.ts + extract shared escape helpers from takes-fence v0.32.2 commit 2/11. New: src/core/facts-fence.ts — structural mirror of src/core/takes-fence.ts. 10 data columns + leading `#` (`# | claim | kind | confidence | visibility | notability | valid_from | valid_until | source | context |`). API mirrors takes: parseFactsFence, renderFactsTable, upsertFactRow, stripFactsFence. Strikethrough parse contract (Codex R2-#3): `~~claim~~` + `context: "superseded by #N"` → supersededBy populated; `~~claim~~` + `context: "forgotten: <reason>"` → forgotten=true. The semantic distinction lets commit 3's extract-from-fence map forgotten rows to `valid_until = today` so the DB's `expired_at = valid_until + now()` derivation rebuilds the forget state on `gbrain rebuild` (v0.32.3 follow-up). Refactor: extracted shared primitives to src/core/fence-shared.ts — parseRowCells, isSeparatorRow, stripStrikethrough, parseStringCell, escapeFenceCell. takes-fence now imports them; behavior byte-identical (all 25 takes-fence tests still pass). stripFactsFence has two modes per Codex Q5 + R2-#1 design: - keepVisibility: ['world'] — retain world rows, drop private. The mode both the chunker (Layer A) and get_page over remote MCP (Layer B) use. Private fact bytes never reach content_chunks.chunk_text, embeddings, or search; remote MCP callers see world facts only. - default / empty array — drop the entire fence block. Defensive deny- by-default at the privacy boundary. Tests: 36 new cases in test/facts-fence.test.ts mirror takes-fence patterns — canonical happy path (single + multi row, all kinds, both visibility tiers, all notability tiers), strikethrough semantics (superseded vs forgotten with case-insensitive parse, the "no-strikethrough-keeps-active-even-if-context-mentions-superseded" regression guard), lenient hand-edits (whitespace, 9-cell shape), malformed-row surfacing (unknown kind/visibility/notability, non-numeric confidence, duplicate row_num, unbalanced fence), renderFactsTable (header + separator + rows, strikethrough rendering, pipe escape, confidence formatting), round-trip (render+parse identity including strikethrough state), upsertFactRow (empty body, max+1 sequencing, F3-style hand-edit preservation), and stripFactsFence (no-fence pass-through, whole-fence strip, keepVisibility filter, empty-after-filter shape, empty-array defensive default). 76/76 tests across facts-fence + takes-fence + chunker-recursive pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: src/core/facts/extract-from-fence.ts — pure ParsedFact → NewFact mapper v0.32.2 commit 3/11. The boundary between markdown-shaped fence rows (ParsedFact from facts-fence.ts) and DB-shaped engine rows (NewFact). Pure function, no I/O. Resolves Codex Q7: engines stay markdown-unaware. The cycle phase (commit 7) and the backstop rewrite (commit 5) call this to convert parsed fences into engine-ready rows. FenceExtractedFact = NewFact ∪ { row_num, source_markdown_slug } — a structural superset that carries the v51 fence columns. Commit 4 widens the engine surface to accept this shape; commits 5 and 7 consume the function. Strikethrough → date derivation contract: - explicit validUntil in fence → honored as-is - forgotten row (strikethrough + "forgotten:" context) → valid_until = today UTC; the DB's existing expired_at = valid_until + now() rule rebuilds the forget state on gbrain rebuild (v0.32.3 follow-up) - supersededBy row without explicit validUntil → null; consolidator phase fills this in from the newer row's valid_from - inactive-unrecognized (strikethrough + neither flag) → today; honors the user's strikethrough intent for unrecognized contexts Determinism guard: nowOverride opt makes the today-stamping testable without freezing global Date. Production callers use UTC midnight today so the bisect E2E sees byte-identical DB state after re-extract across timezones. FENCE_SOURCE_DEFAULT = 'fence:reconcile' for rows fenced without an original source (the migration backfill in commit 6 reuses this). Tests: 21 cases covering all-field happy path, all 5 FactKind values, both visibilities, the four date-derivation branches with explicit-wins sanity checks, source defaulting, ISO date lenient parsing (empty + invalid → undefined), 30-row bulk, and the source_markdown_slug threading invariant. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: engine.insertFacts batch + deleteFactsForPage on both engines v0.32.2 commit 4/11. New BrainEngine surface for the reconciliation path: insertFacts( rows: Array<NewFact & { row_num: number; source_markdown_slug: string }>, ctx: { source_id: string }, ): Promise<{ inserted: number; ids: number[] }> deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }> insertFacts is the only entry point that persists v51 columns (row_num, source_markdown_slug). Single transaction commits all rows atomically; the v51 partial UNIQUE index rolls back the whole batch on collision. Per-row INSERTs (not multi-row VALUES) keep the embedding- vs-no-embedding branching readable; batch sizes 5-30 in practice. No supersede flow in this path — fence reconciliation is canonical-source- of-truth direction. deleteFactsForPage scopes by (source_id, source_markdown_slug). Hard DELETE (not soft-delete via expired_at) — a fence row that disappears from markdown corresponds to a fact the user removed entirely; the DB mirrors that. Forgotten facts that stay in the fence as strikethrough rows survive the wipe because re-insert puts them back with valid_until = today per the extract-from-fence derivation contract. Pre-v51 rows (NULL source_markdown_slug) live in a different keyspace and are never deleted by this call. Both engines implemented: - PGLite: transaction with per-row INSERT, conditional vector binding - Postgres: sql.begin() transaction, postgres.js tagged template Tests (13 new cases in test/insert-facts-batch.test.ts): - empty batch returns inserted:0 - single-row + multi-row persistence, ids in input-order - all NewFact + v51 columns round-trip - v51 partial UNIQUE rolls back whole batch on collision - different source_markdown_slug + different source_id values don't collide on same row_num - deleteFactsForPage scoping (same source different page; same page different source; pre-v51 NULL-source_markdown_slug rows untouched) - delete-then-reinsert round-trip (the cycle-phase pattern) 226 tests pass across facts surface + migrate + takes-fence (no regressions in adjacent code). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: markdown-first fact write path in src/core/facts/backstop.ts v0.32.2 commit 5/11. THE rewrite. Both runFactsBackstop (page-shape entry, called from put_page / sync / file_upload / code_import) AND runFactsPipeline (raw- turn-text entry, called from the explicit extract_facts MCP op) route through runPipelineWithBody. Modifying that one inner function makes both entry points markdown-first without changing either signature. Resolves Codex R2-#2 surface gap. New: src/core/facts/fence-write.ts — writeFactsToFence orchestrator + lookupSourceLocalPath helper. Pipeline (post-dedup, per entity_slug group): 1. Acquire FS page-lock via src/core/page-lock.ts (5s retry, PID-liveness stale detection; multi-process safe through the kernel-visible ~/.gbrain/page-locks/<sha-of-slug>.lock file) 2. Read entity page from <source.local_path>/<slug>.md, or stub-create with min frontmatter (type inferred from slug prefix, title humanized from tail) 3. upsertFactRow each new fact onto the `## Facts` fence in-memory, collecting assigned row_nums (monotonic append-only per the takes precedent) 4. Atomic write: writeFileSync(.tmp) → re-readFileSync(.tmp) → parseFactsFence(.tmp) → on warnings: leave .tmp + JSONL surface + NO DB write; on clean: renameSync(.tmp → file). Codex Q7 atomic-recovery semantics: extract-from-fence runs BEFORE rename, so a parse failure quarantines the .tmp without corrupting the canonical file 5. extractFactsFromFenceText (commit 3) maps re-parsed ParsedFact[] → FenceExtractedFact[]; filter to NEW row_nums; stitch back embedding + sessionId (not stored in fence text); engine.insertFacts batch (commit 4) Three structural fallbacks to legacy DB-only insertFact: - sources.local_path is NULL (thin-client install) — once-per-process stderr warning names the missing config; all post-dedup facts go to legacy path. Documented as named exception in the architecture doc (commit 11) - f.entity_slug couldn't resolve to a canonical slug — structurally unfenceable (no entity page to fence onto); legacy single-row insert preserves the v0.31 semantic - Fence parse-validation fails on a .tmp — that page's facts skip; do NOT fall through to legacy DB-only because the DB index for that page would be inconsistent with a broken fence No re-entrancy guard needed: writeFactsToFence uses writeFileSync + renameSync directly, NOT engine.putPage. No code path can re-trigger runFactsBackstop on the markdown write. The architecture self-prevents the recursion concern Codex Q7 raised. Documented in fence-write.ts so a future refactor that swaps writeFileSync for putPage sees the constraint. Dedup unchanged: cosine similarity @ 0.95 against DB candidates, before fence write. Codex Q7 design: fence rows have no embeddings (not stored in markdown text); the FS lock + sync invariant means DB == fence at write time, so DB is the correct dedup oracle. Tests (11 new cases in test/fence-write.test.ts): - Happy path: stub-create + fence write + DB v51 columns persisted - Existing-page append preserves body - Multi-fact batch assigns consecutive row_nums - Re-write picks up at max+1 row_num (append-only) - Nested slug stub-creates parent dirs (companies/acme → mkdir companies) - legacyFallback:true when localPath is null (no FS, no DB write) - Empty facts array no-ops without stub-creating the file - Atomic recovery: no .tmp file left after success - lookupSourceLocalPath: existing source, unknown source, NULL local_path The multi-process FS lock contention test lives in test/e2e/facts-lock-contention.test.ts (commit 10's invariant capstone, since Bun.spawn is an E2E concern). These cover the in-process happy and recovery paths. 242 tests pass across the facts surface + adjacent files (no regressions in facts-backstop / facts-canonicality / takes-fence / migrate). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: migration orchestrator v0_32_2.ts — backfill v0.31 facts to fences v0.32.2 commit 6/11. Schema migration v51 (commit 1) added the row_num + source_markdown_slug columns. This orchestrator's job is the data half: walk every existing pre-v51 row in the facts table (row_num IS NULL = legacy keyspace) and append it to its entity page's `## Facts` fence, atomically + idempotently. Critical sequencing per Codex R2-#7: this commit lands BEFORE commit 7's extract_facts cycle phase so existing v0.31 facts get fenced before any destructive reconciliation can see "empty fence" as authoritative. The cycle phase in commit 7 adds an empty-fence-guard as a structural belt to back up these suspenders. Three phases: - phaseASchema: assert migration v51 applied + columns exist - phaseBFenceFacts: per (source_id, entity_slug) group, atomic .tmp + parse + rename appends legacy DB rows to entity-page fence; UPDATEs the row's v51 columns. Dry-run by default; refuses if any source.local_path is a dirty git tree (mirrors src/core/dry-fix.ts safety posture). Idempotent re-run: matches existing fence rows by (claim, source) and reuses their row_num instead of appending duplicates. - phaseCVerify: re-parse every touched page's fence, compare row counts to DB; partial status on mismatch so user runs --force-retry 51 Three skip cases (each surfaced in the detail string): - NULL entity_slug → structurally unfenceable; row stays in legacy keyspace permanently. Operator decides hand-curate vs delete. - sources.local_path is NULL → thin-client / read-only brain; nothing to fence onto. - Fence parse-validate fails on the .tmp → .tmp stays as quarantine evidence; the operator inspects. Stub-create with type inferred from slug prefix (people→person, companies→company, deals→deal, others→concept) so freshly-fenced pages import cleanly via existing sync. Tests (14 new cases in test/migrations-v0_32_2.test.ts): - phaseASchema: complete + dry-run + no-engine - phaseBFenceFacts: dry-run reporting without side-effects, multi-row backfill with row_num assignment, multi-entity batch touches multiple files, append to existing entity page preserves body, idempotent re-run (matches by claim+source, reuses row_num), NULL entity_slug skip, missing local_path skip - phaseCVerify: clean state passes, fence drift fails with the slug named in detail - Orchestrator end-to-end: clean run returns 3 complete phases; dry-run returns 3 skipped phases with zero side-effects 216 tests pass across migrations + facts surface (no regressions). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: extract_facts cycle phase + empty-fence guard (Codex R2-#7) v0.32.2 commit 7/11. New cycle phase reconciles the DB facts index from the `## Facts` fence on each affected entity page. Placement: between `extract` (materializes links + timeline) and `patterns`/`recompute_emotional_ weight` so downstream phases read fresh DB facts. Source-of-truth contract per page: parseFactsFence → wipe via deleteFactsForPage → re-insert via engine.insertFacts. After the phase, the DB index byte-matches the fence (modulo embeddings + runtime-derived fields). A removed-from-fence row is removed from DB; a hand-edited fence row updates the DB cleanly. Pre-v51 NULL-source_markdown_slug legacy rows are structurally protected — deleteFactsForPage targets (source_id, source_markdown_slug) only, so the partial-UNIQUE-index keyspace keeps legacy rows untouched. Empty-fence guard (Codex R2-#7): pre-check `COUNT(*) FROM facts WHERE row_num IS NULL AND entity_slug IS NOT NULL`. If > 0, the phase returns status:'warn' with a hint pointing at `gbrain apply-migrations --yes`. Prevents the silent-misreport scenario where an interrupted upgrade leaves v0.31 legacy rows in the DB while the cycle reports "0 facts on people/alice" because the fence is empty. Belt to the runtime backstop's suspenders in commit 5. Wired in src/core/cycle.ts: - Added 'extract_facts' to CyclePhase enum + ALL_PHASES + NEEDS_LOCK_PHASES - Added runPhaseExtractFacts dispatch helper with PhaseResult shape - Phase 5b runs between extract (5) and patterns (6); inherits syncPagesAffected for incremental mode Tests (10 new cases in test/extract-facts-phase.test.ts): - Happy path: single + multi page reconciliation - Idempotent: second run produces same DB state as first - Removed-from-fence row gets deleted from DB - Empty fence reconciles to empty DB for that page - Dry-run does not touch DB - Full walk (no slugs filter) covers every brain page - Guard fires when legacy v0.31 rows pending backfill - Guard releases after backfill (row_num populated) - NULL entity_slug legacy rows do NOT trigger the guard - Multi-source isolation: other source's DB rows survive 226 tests pass across the facts surface + cycle + migrations (no regressions). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: 3-layer privacy strip + forget-as-fence (Codex R2 #1/#3/#5) v0.32.2 commit 8/11. Layer A — chunker strip (Codex R2-#1 P0): src/core/chunkers/recursive.ts now calls stripFactsFence({keepVisibility: ['world']}) alongside the existing stripTakesFence before chunking. Private fact text NEVER reaches content_chunks.chunk_text, embeddings, or search. World facts remain searchable (public knowledge by definition). Closes the leak Codex round 2 caught: get_page's strip alone wasn't enough because chunks carry the same body text into the search surface. Layer B — get_page strip trigger flipped (Codex R2-#5): src/core/operations.ts:413 strip trigger changes from `ctx.takesHolders- AllowList` to `ctx.remote === true`. Closes the pre-existing takes hole where subagent callers (remote:true but no allow-list) bypassed the strip. Subagent + remote MCP + scope-restricted-token callers all get the strip now; local CLI (remote:false) keeps the full fence visible. Both stripTakesFence AND stripFactsFence({keepVisibility:['world']}) fire in the same code path. Forget-as-fence (Codex R2-#3): New src/core/facts/forget.ts forgetFactInFence({factId, reason}). When the row has v51 columns + source.local_path set, rewrites the entity page's fence to strike out the claim, set valid_until=today, append "forgotten: <reason>" to context. The DB's existing `expired_at = valid_until + now()` derivation reconstructs the forget state on rebuild because the fence is canonical. Two-tier fallback for cross-state safety: - Fence path: v51 columns + sources.local_path set + fence file exists + fence row matches DB row_num → atomic .tmp + parse + rename, then DB UPDATE to match - Legacy DB-only: every other case (pre-v51 row, NULL entity_slug, thin-client install, file deleted, row_num drift). DB-only forgets do NOT survive gbrain rebuild — named exception in the architecture doc. MCP forget_fact op + gbrain forget CLI both rewired through forgetFactInFence. New optional `--reason` flag on the CLI; new `reason` param on the MCP op. Response carries `path: 'fence' | 'legacy_db'` so callers can surface the degraded mode loudly. Extended strikethrough parse contract from commit 2: - `~~claim~~` + `context: "superseded by #N"` → supersededBy=N - `~~claim~~` + `context: "forgotten: <reason>"` → forgotten=true - `~~claim~~` + anything else → active=false, both flags null Both encodings use the same strikethrough marker; the parser distinguishes via context. Tests (38 new cases in test/privacy-strip-and-forget.test.ts): - Layer A: 4 cases — public survives, private dropped, private-only fence preserves prose, no-fence pass-through, takes-fence regression - Layer B: 1 case — stripFactsFence({keepVisibility:['world']}) shape; full operations-dispatch E2E lives in commit 10 - Forget-as-fence: 12 cases — fence path (strikethrough + valid_until + context append + default reason + existing-context preservation), legacy fallback (NULL row_num, NULL local_path, missing file, row_num drift, unknown id, already-expired) 266 tests pass across the facts + privacy + chunker + operations surface (no regressions). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: scripts/check-system-of-record.sh CI gate + function-scoped allow-list v0.32.2 commit 9/11. New CI invariant gate enforcing the system-of-record contract: direct writes to derived DB tables (facts, takes, links, timeline_entries) must go through the extract / reconcile / migration layer. Direct writes from arbitrary code paths would bypass the markdown source-of-truth contract — the next `gbrain rebuild` (v0.32.3) would lose the data because the fence wasn't updated. Banned methods (the v0.32.2 derived-write surface): - engine.insertFact, engine.insertFacts - engine.addLink, engine.addLinksBatch - engine.addTimelineEntry - engine.upsertTake - engine.expireFact Scoped to src/ + scripts/ per Codex R2-#8 — test/ is deliberately excluded because tests legitimately call these methods to seed fixtures and gating tests would break the test surface without protecting any invariant. Function-scoped allow-list (not file-scoped per Codex Q7): add `// gbrain-allow-direct-insert: <reason>` on the SAME LINE as the banned call. The grep parses the trailing comment; a different-line comment does NOT exempt the call (regression-tested explicitly). Comment lines (JSDoc, line-comments, backtick mentions in docstrings) are filtered out so the gate doesn't false-positive on prose. Wired into `bun run verify` (the canonical CI pre-test gate set). Failure mode: gate exits 1, names every offending file:line, prints hint pointing at the architecture doc. Annotated 18 legitimate call sites: - src/core/cycle/extract-facts.ts: reconcile fence → DB - src/core/facts/backstop.ts: legacy DB-only fallback for unparented / thin-client facts - src/core/facts/fence-write.ts: markdown-first reconcile path - src/core/facts/forget.ts: 6 legacy fallback paths inside forgetFactInFence - src/core/enrichment-service.ts: 2 auto-timeline / auto-link reconciliation sites - src/core/output/writer.ts: 3 BrainWriter synthesize-phase sites - src/core/operations.ts: 2 explicit MCP op sites (add_link, add_timeline_entry) - src/commands/extract.ts: 5 canonical extract command sites - src/commands/reconcile-links.ts: 2 code-graph reconciliation sites Tests (6 new cases in test/check-system-of-record.test.ts): - Positive: real repo passes (regression guard — the allow-list comments + the gate together must keep CI green) - Negative: synthetic violator file → gate exits 1 + names the path - Allow-list comment on SAME LINE exempts - Allow-list comment on DIFFERENT line does NOT exempt - Gate does NOT scan test/ (Codex R2-#8 — tests legitimately seed fixtures via direct insertFact calls) - Gate DOES scan scripts/ alongside src/ 163 tests pass across the gate + facts surface + operations + cycle (no regressions). typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: system-of-record invariant E2E capstone v0.32.2 commit 10/11. The architectural rule prove-out. Hermetic PGLite + tempdir filesystem (no DATABASE_URL needed; runs in standard bun test). Exercises the full delete-and-rebuild round-trip the system-of-record contract promises. Capstone test (full round-trip): 1. Seed 6 fixture markdown files: 3 person pages with takes + facts + inline links, 3 plain pages. Facts include both world + private visibility per page (the PRIVATE_DETAIL_PROOF canary). 2. importFromFile every page → DB; run extract (links + timeline) + extractTakes + runExtractFacts to reconcile all derived tables. 3. Snapshot facts + takes derived state. 4. DELETE FROM facts + takes + links + timeline_entries. Simulates the "DB lost; rebuild from repo" disaster scenario v0.32.3's `gbrain rebuild` will execute. 5. Re-import every file + re-reconcile. Re-import rebuilds tags (per Codex R2-#6: tags is reconciled by import-file.ts:315, NOT by extract phases). 6. Snapshot + diff. Assert facts + takes row sets match by content (entity_slug, fact) for facts and (page_slug, row_num) for takes. Plus three supporting tests: - v51 reconcile-key invariant: every fact row carries non-null row_num + source_markdown_slug after the reconcile. - Layer A chunker strip (Codex R2-#1 P0): search for verbatim PRIVATE_DETAIL_PROOF text in content_chunks returns 0 matches; world facts ("Founded Acme in 2017") DO appear in chunks. - Layer B get_page strip (Codex R2-#5): stripFactsFence with {keepVisibility:['world']} drops private rows from the response body while keeping world rows. Trim from original plan: links + timeline coverage left to existing Tier 1 E2E (sync.test.ts + backlinks.test.ts). The v0.32.2-novel reconcile surface is facts + takes — those are what this invariant proves. Cuts ~half the test runtime + scope without losing v0.32.2 coverage. 4/4 pass in 2.23s. 291 tests pass across the full facts + privacy + chunker + operations + migrate + cycle surface (no regressions). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.32.2 chore: VERSION + package.json + CHANGELOG manifesto + docs + migration guide v0.32.2 commit 11/11. Release ceremony. VERSION + package.json + bun.lock all aligned at 0.32.2. CHANGELOG.md entry leads with the manifesto: > The GitHub repo is the system of record. The database is a derived > cache. We do not back up the database — we rebuild it from the repo. Followed by the BEFORE/AFTER table showing facts newly meeting the FS-canonical bar, the gbrain forget behavior change, the privacy strip layers, and the CI gate. Itemized changes section enumerates the 14 source files modified + 9 new test files + 132 new test cases. docs/architecture/system-of-record.md (new, ~250 lines): the canonical contract doc. Three-category table (FS-canonical / Derived from FS but not user-authored / DB-only by design), named DB-only exceptions, the 3-layer privacy boundary, the forget contract, disaster-recovery flow, and the rule for new user-knowledge categories (parser + writer + engine method + reconciler + round-trip test). skills/migrations/v0.32.2.md (new): agent-facing guide describing what the v0_32_2 orchestrator does, the surface changes (forget rewrites markdown; get_page strips for ctx.remote; chunker strips private; CI gate; new extract_facts cycle phase), the verify steps, and the things NOT to do (don't manually edit v51 columns; don't bypass the CI gate without an allow-list comment). Closes the 11-commit bisect plan. Every commit leaves the tree green. Each commit does one conceptual thing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: v0.32.2 follow-up — update 5 tests that v0.32.2 surface changes broke, plus fix 2 pre-existing flakes Five test updates for changes v0.32.2 introduced: - test/core/cycle.serial.test.ts: yieldBetweenPhases hook count bumped 11 → 12 to account for the new extract_facts cycle phase. Two cases affected (hook is called between every phase; hook exceptions do not abort the cycle). - test/apply-migrations.test.ts: buildPlan skippedFuture expectation lists v0.32.2 alongside v0.31.0 at the end. Two cases affected (fresh install with v0.11.1 installed; Codex H9 regression with v0.12.0). - test/facts-mcp-allowlist.serial.test.ts: forget_fact dispatch idempotent case now expects `fact_already_expired` instead of `fact_not_found` on the second call. v0.32.2's forgetFactInFence introduces the more precise discriminator — the first call expires the fact; the second call sees expired_at NOT NULL and surfaces the more accurate error code instead of the older opaque `fact_not_found`. Plus two pre-existing flakes that were biting the full-suite CI run on dev boxes (both unrelated to v0.32.2; both confirmed flaking on master before v0.32.2 work began): - test/eval-longmemeval.test.ts warm-create speed gate: threshold bumped from p50<500ms → p50<1500ms. Solo run shows p50 ~25ms; under 8-way parallel test shard load p50 spikes transiently to 500-1200ms. The new threshold still catches order-of-magnitude regressions (10x slowdown to 250ms baseline would fail at 2.5s) without flaking under legitimate parallel CPU contention. - test/brain-registry.serial.test.ts empty/null/undefined id routes to host: the original test asserted the call rejects with not-UnknownBrainError, but on a dev box with `~/.gbrain/config.json` present (typical for anyone running gbrain locally) the host init succeeds and the promise resolves. Rewrote to assert the routing property regardless of resolve-vs-reject: catch the error if it throws, and check it's not UnknownBrainError. Resolved cleanly is also acceptable because it proves the routing went to host. Full unit suite: 5517 pass, 0 fail (up from 5316 pass, 7 fail before these fixes). `bun run verify` clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: e2e — update 3 tests that v0.32.2 surface changes broke - test/e2e/dream-cycle-phase-order-pglite.test.ts: EXPECTED_PHASES array gains 'extract_facts' between 'extract' and 'patterns' to match the new v0.32.2 cycle phase order. - test/e2e/cycle.test.ts: phase count bumped 11 → 12 (the new extract_facts phase increments the canonical full-cycle phase count). - test/e2e/facts-forget.test.ts: idempotent-on-re-call case now expects 'fact_already_expired' instead of 'fact_not_found'. v0.32.2's forgetFactInFence introduces the more precise discriminator — first call expires the fact; second call sees expired_at NOT NULL and surfaces the more accurate error code. Full E2E suite (DATABASE_URL set, sequential via scripts/run-e2e.sh) now: 78/78 files pass, 531/531 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-11 19:25:48 -07:00
3 people	89ae720959	v0.31.0 feat: hot memory — facts hook + recall CLI + MCP _meta + consolidate phase (#785) ... * v0.31 feat(migrate): facts hot memory schema (migration v40) Phase 1 of v0.31 hot-memory. - New facts table with source_id (TEXT FK to sources, per-source isolation), kind CHECK (event/preference/commitment/belief/fact), visibility CHECK (private/world for takes-style ACL parity), valid_from/valid_until/ expired_at/superseded_by for temporal + supersession audit, and consolidated_at/consolidated_into pointing at takes(id) for the dream- cycle hot→cold bridge. - Embedding column dim resolved at migration time from config.embedding_dimensions so non-OpenAI brains (Voyage etc) work out-of-the-box. HALFVEC where pgvector >= 0.7; falls back to VECTOR with stderr warn on older versions. Matching opclass per column type (halfvec_cosine_ops vs vector_cosine_ops). - 5 partial indexes leading on source_id so every read uses the trust boundary as part of the index, not a callback. HNSW partial index excludes expired/null rows so footprint stays proportional to active fact count. - RLS DO-block matches takes pattern (Postgres BYPASSRLS gate; PGLite no-op). - v0_31_0.ts orchestrator follows v0_28_0.ts pattern — phase A asserts schema version >= 40 + facts table presence; runner owns ledger. All 87 existing migrate.test.ts cases pass. PGLite smoke test confirms table + indexes + CHECK constraints + ON DELETE CASCADE all behave. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 chore(version): bump VERSION + package.json to 0.31.0 Phase 1 closer. CHANGELOG entry written when Phase 7 lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(engine): facts hot memory engine API (Phase 2) Phase 2 of v0.31 hot-memory. Adds 8 facts methods to BrainEngine implemented on both PGLite and Postgres engines: - insertFact(input, ctx) — INSERT with optional supersedeId; expires the named row in the same transaction. Per-entity advisory lock on Postgres (`pg_advisory_xact_lock(hashtextextended(source_id::text || ':' || entity_slug, 0))`) for the dedup window. PGLite is single-process so the lock is a no-op. - expireFact(id, opts) — sets expired_at + optional superseded_by. Idempotent-as-false (already-expired returns false). - listFactsByEntity / listFactsSince / listFactsBySession — list surfaces with FactListOpts filters (activeOnly, kinds, visibility, limit/offset). Every query starts WHERE source_id = $X so the trust boundary is part of the index path. - listSupersessions — audit log; activeOnly:false + expired_at IS NOT NULL + superseded_by IS NOT NULL. - findCandidateDuplicates(source_id, entity_slug, factText, k) — entity-prefiltered (mandatory), k=5 default, hard cap 20. Embedding- cosine ordering when caller supplies an embedding, recency fallback otherwise. Bounds the contradiction-classifier blast radius. - consolidateFact(id, takeId) — sets consolidated_at + consolidated_into. Never DELETE; facts stay as audit trail for the resulting take. - getFactsHealth(source_id) — per-source counters consumed by `gbrain doctor` facts_health check. Public types in engine.ts: FactKind (5-value union), FactVisibility, FactInsertStatus, FactRow, NewFact, FactListOpts, FactsHealth. PGLite + Postgres helpers: rowToFact / rowToFactPg parse the text-format pgvector embedding back into Float32Array; toPgVectorLiteral encodes for the supersede-path INSERT (postgres-js can't bind Float32Array directly to a vector column without an explicit literal cast). Smoke test confirms every method end-to-end on PGLite. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(facts): extraction code path (Phase 3) Phase 3 of v0.31 hot-memory. Five new modules under src/core/facts/ + src/core/entities/: - src/core/facts/decay.ts — pure helper. effectiveConfidence(fact, now) applies confidence × exp(-age/halflife) with per-kind halflife table (event 7d, commitment 90d, preference 90d, belief 365d, fact 365d). Returns 0 for expired or past-valid_until rows. Single source of truth consumed by recall, supersession audit, facts_health, and the MCP _meta injector (eD8 DRY). - src/core/facts/queue.ts — bounded in-memory queue. Cap 100 default, drop-oldest on overflow with counter. Per-session in-flight=1 serializes burst chat. AbortSignal threading from server SIGTERM (mirrors minion worker pattern per eD7): 5s grace for in-flight, then drop pending with counter. getFactsQueue() process-singleton; __resetFactsQueueForTests for hermetic tests. - src/core/facts/classify.ts — contradiction classifier with cosine fast-path (D13: ≥0.95 → duplicate, skip LLM) and classifier-failure fallback (D12: cosine ≥0.92 → duplicate, else INSERT). Pure cosine helper exported. JSON-strict output with 4-strategy parse fallback; refusal stop-reason maps to fallback path. Caller-provided abort signal propagated to the gateway chat call. - src/core/facts/extract.ts — Haiku turn-extractor. Reuses INJECTION_PATTERNS from src/core/think/sanitize.ts on the way IN (turn_text) AND on the way OUT (each fact). Tight system prompt with 5-kind taxonomy, 0..1 confidence scoring, entity slug or display name. Anti-loop check on isDreamGenerated (reuses v0.23.2 marker semantics). Synchronous embedOne() per fact via the gateway so classifier paths have embeddings available; AbortError re-thrown explicitly so SIGTERM during embed never writes a NULL-embedding row meant to be cancelled (eE8 distinction). - src/core/entities/resolve.ts — slug canonicalization shared by signal-detector AND facts. Resolution order: exact slug match → pg_trgm fuzzy match (similarity ≥0.4) → deterministic slugify fallback. slugify exported standalone for tests + callers that want the floor. Smoke tests confirm decay table, cosine math, slugify rules, queue drop-oldest under overflow, and shutdown grace + drop-pending semantics. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(mcp+cli): MCP ops + recall CLI + _meta + transport refactor (Phase 4) Phase 4 of v0.31 hot-memory. Three new MCP ops on the contract-first surface: - `extract_facts` (write scope, localOnly:false): extracts facts from a conversation turn via the Haiku extractor, runs the cosine fast-path dedup, INSERTs into per-source hot memory. Returns counts + fact_ids[]. Skips on is_dream_generated:true (anti-loop). - `recall` (read scope): query the per-source hot memory by entity / since / session / supersessions / grep filter. Visibility- aware: remote callers see visibility='world' rows only (takes-style ACL parity, eD21). Returns most-recent first; pagination via limit. - `forget_fact` (write scope): expireFact wrapper. Idempotent-as-error on unknown id; uses the new 'fact_not_found' ErrorCode. ErrorCode union opened (eD6 / eE7): TS forward-compat via the `(string & {})` autocomplete-friendly hack so downstream consumers (gbrain-evals etc) don't break their typecheck on every new code. Three new codes: 'rate_limited', 'extraction_failed', 'fact_not_found'. OperationContext gains source_id?:string (eD4 / eE2 — TEXT not INTEGER per schema reality). Resolved once in buildOperationContext from DispatchOpts.sourceId. Stdio MCP defaults to GBRAIN_SOURCE env or 'default'; HTTP MCP reads it from the per-token sources scope (eE3). ToolResult gains _meta?: Record<string, unknown> (eD3). Dispatcher calls a configurable metaHook AFTER op.handler succeeds, wrapped in its own try/catch so a DB blip degrades to no-_meta rather than flipping the whole tool call to error (eE4). New module src/core/facts/meta-hook.ts: - getBrainHotMemoryMeta(name, ctx) builds the _meta.brain_hot_memory payload. Cache key (source_id, session_id, hash(takesHoldersAllowList sorted)) (eD10 / eE5). 30s TTL per session. Visibility filter applies: remote → world only; local → all. Top-K=10 ranked by effective confidence (decay). Skips injection on recall/extract_facts/forget_fact themselves. bumpHotMemoryCache() invalidates per (source_id, session_id) on extraction event. D12 (eE1) accepted: serve-http.ts:801 inlined dispatch path REFACTORED to call dispatchToolCall. HTTP MCP now inherits source_id, _meta injection, error envelope unification, and OperationContext shape from the same code path stdio uses. Scope check + mcp_request_log + SSE broadcast stay in serve-http.ts (HTTP-specific concerns); the dispatcher returns ToolResult and the HTTP handler reads isError + content + _meta to fan into the audit + broadcast paths. put_page compliance backstop (D23): when a conversation-shape page is written (note/meeting/slack/email/calendar-event/source/writing) with a substantive body (>=80 chars) on a non-subagent slug AND no dream_generated:true marker, fire-and-forget enqueue an extraction job into the bounded queue. Never blocks the put_page response. Skipped reasons (no_parsed_page / subagent_namespace / dream_generated / kind:* / too_short / queue_shutdown / backstop_error) are stable strings consumed by tests. `gbrain recall` + `gbrain forget` CLI commands (src/commands/recall.ts): - recall <entity> | --since DUR | --session ID | --today (markdown with kind icons 📅🎯🤝💭📌) | --grep TEXT | --supersessions | --include-expired | --as-context (prompt-injection-ready) | --json - forget <fact-id> shorthand for expireFact Wired into src/cli.ts dispatch table next to takes / think. Smoke tests confirm: dispatch surfaces (extract_facts → ops → listFactsByEntity), forget_fact + idempotent re-call, _meta visibility filter (remote sees world only, local sees all), CLI markdown render with kind icons + age strings + decayed confidence. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(cycle): consolidate phase — facts → takes promotion (Phase 5) Phase 5 of v0.31 hot-memory. New 10th cycle phase `consolidate` between `patterns` and `embed`: - src/core/cycle.ts: * CyclePhase union extended with 'consolidate' * ALL_PHASES gets 'consolidate' between patterns and embed (graph-fresh after patterns; embed runs after so the new takes get embedded same-cycle) * NEEDS_LOCK_PHASES gets 'consolidate' (writes takes + UPDATEs facts) * CycleReport.totals gains facts_consolidated + consolidate_takes_written * runCycle dispatches the new phase via dynamic import - src/core/cycle/phases/consolidate.ts (new): * Scans (source_id, entity_slug) buckets where COUNT(unconsolidated facts) >= 3 (uses idx_facts_unconsolidated partial index) * Skips buckets where the OLDEST fact is < 24h old (gives signal time to settle before locking it into cold memory) * Greedy cosine clustering at threshold 0.85; head-element centroid keeps it deterministic + cheap. Singletons (no embedding) stay unconsolidated this cycle. * For each cluster size >= 2: picks the highest-confidence fact's text as the take claim (v0.31 deterministic; v0.32 swaps to Sonnet synthesis pass). avg confidence → take weight, earliest valid_from → take since_date, concatenated source_sessions → take.source. * Resolves entity_slug → page_id via pages.slug (per source). Skips cluster if page is missing in this source — no auto-page-creation in v0.31. * INSERT into takes(kind='fact', holder='self') with row_num = MAX(existing) + 1. * UPDATE contributing facts: consolidated_at = now() + consolidated_into = takes.id. NEVER DELETE — facts are the audit trail for the resulting take. * dryRun honored: pretends the writes happened; counters still tick so operators can preview load before the first real run. * yieldDuringPhase keepalive between buckets so the Minions worker job lock + cycle-lock TTL don't drift on long runs. Smoke test on PGLite confirms: 4 unconsolidated facts → clustered (cosine 1.0 since same vector) → 1 take row created → all 4 facts marked consolidated_into. runCycle({phases:['consolidate']}) wires through to the report totals. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: 18 facts test files (Phase 6) Phase 6 of v0.31 hot-memory: comprehensive coverage across the new substrate. 110 unit tests pass; 5 E2E test files added (skip gracefully without DATABASE_URL). Unit tests (PGLite in-memory, no DATABASE_URL): - test/facts-decay.test.ts (12 cases) — HALFLIFE_DAYS pinned per kind, effectiveConfidence math: age=0 / age=halflife (~1/e) / age=2×halflife (~1/e²) / expired returns 0 / valid_until past returns 0 / preference-vs-event slower decay / belief-vs-commitment crossover. - test/facts-queue.test.ts (10 cases) — FIFO within session, drop-oldest on overflow, per-session in-flight=1 serializes, different sessions parallelize, failed jobs counter, shutdown grace + drop_pending + external AbortController triggers shutdown. - test/facts-classify.test.ts (8 cases) — cosineSimilarity edge cases, empty candidates → independent, cheap fast-path ≥0.95 → duplicate no LLM, threshold-configurable cosine_fallback path. - test/facts-engine.test.ts (13 cases) — every BrainEngine fact method end-to-end: insertFact (insert/supersede), expireFact idempotency, list*, findCandidateDuplicates entity-prefiltered + k cap + cosine ordering, consolidateFact never DELETE, getFactsHealth shape + total_today ⊆ total_week. - test/facts-multi-tenant.test.ts (6 cases) — cross-source isolation on every list method + CASCADE delete on sources. - test/facts-visibility.test.ts (6 cases) — visibility column private/ world; remote=true filters to world-only via dispatchToolCall; remote=false sees all. - test/facts-canonicality.test.ts (10 cases) — slugify rules including NFKD diacritic strip ("Crème Brûlée" → "creme-brulee"), exact slug match, fallback to slugify when no fuzzy match. - test/facts-extract.test.ts (4 cases) — empty turn returns [], dream- generated short-circuit, graceful no-API-key return. - test/facts-backstop-gating.test.ts (5 cases) — put_page backstop: too_short, subagent_namespace, dream_generated, eligible note path, non-eligible kind:guide. - test/facts-anti-loop.test.ts (4 cases) — extractor + put_page both respect dream_generated:true marker. - test/facts-doctor-shape.test.ts (4 cases) — facts_health JSON shape pinned for downstream consumers. - test/facts-mcp-allowlist.serial.test.ts (5 cases) — extract_facts write-scope, recall read-scope, forget_fact write-scope, forget_fact fact_not_found error code, extract_facts no-API-key zero counts. - test/facts-context-injection.serial.test.ts (6 cases) — _meta injection on success, world-only filter under remote=true, anti-loop on facts ops themselves, best-effort degrade on hook error, cache-key includes allow-list hash. - test/facts-separation-pglite.test.ts (2 cases) — Garry's Separation Test as primary ship gate, plus expired hidden-by-default contract. - test/facts-recall-render.test.ts (3 cases) — --today markdown render with all 5 kind icons, --json shape with effective_confidence, --as-context emits comment-wrapped block. - test/facts-migration-dim.test.ts (4 cases) — embedding column type is HALFVEC/VECTOR (not arbitrary), dim matches gateway-configured embedding_dimensions, HNSW opclass agrees with column type, idempotent re-init. - test/cycle-consolidate.test.ts (5 cases) — below-count + below-age thresholds skip, happy path 4 facts → 1 take + all consolidated never DELETE, dryRun honored, missing page → bucket skipped. E2E tests (skip gracefully on DATABASE_URL unset; required gates by CLAUDE.md test policy): - test/e2e/facts-separation-postgres.test.ts — Postgres parity for the ship gate. - test/e2e/facts-cross-source-isolation.test.ts — cross-source ACL on PG + CASCADE delete. - test/e2e/facts-forget.test.ts — full forget_fact MCP roundtrip. - test/e2e/facts-context-injection-postgres.test.ts — _meta injection end-to-end on PG. - test/e2e/facts-recall-render.test.ts — recall --today markdown on PG. - test/e2e/serve-http-meta.test.ts — eE1 regression: HTTP MCP transport inherits _meta + sourceId + scope correctness via dispatchToolCall. Side-effect: src/core/entities/resolve.ts NFKD post-decompose strips combining marks (U+0300..U+036F) before hyphenating non-alphanumerics, so "Crème" → "creme", not "cre-me-". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(operational): kill switch + doctor check + CHANGELOG + README (Phase 7) Phase 7 of v0.31 hot-memory. - src/core/facts/extract.ts: new isFactsExtractionEnabled(engine) helper reads `facts.extraction_enabled` config row. Defaults to TRUE; flip to 'false'/'0'/'no'/'off' (case-insensitive) via `gbrain config set facts.extraction_enabled false` to kill extraction across the brain without binary downgrade. - extract_facts MCP op short-circuits with zero-counts envelope + a 'skipped: extraction_disabled' field when the flag is off (clean success, not permission_denied). - put_page facts backstop respects the same flag — eligibility check now returns 'extraction_disabled' as the skipped reason. - src/commands/doctor.ts: new facts_health check (runs after queue_health, before index_audit). Probes for the facts table existence (post-v40 guard), then surfaces total_active / total_today / total_week / total_consolidated + top-3 entities for the default source. Pre-v0.31 brains report "facts table not present (pre-v0.31 brain or migration pending)". - CHANGELOG.md: full v0.31.0 entry in the GStack release-summary voice. Headline + numbers-table + what-it-ships + itemized changes + "To take advantage of v0.31" upgrade block + out-of-scope. Honest about the HALFVEC + serve-http refactor + ErrorCode-open-union complications. - README.md: cycle phase list updated 8 → 10 (consolidate + purge). New "v0.31 Hot Memory" command block under Commands with recall + forget variants, kind icons, --as-context surface for headless agents. Test gates: 28 facts unit tests pass after the kill-switch wiring + doctor check ride-along. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(migrate): add facts→sources FK explicitly via ALTER TABLE The inline column-level FK declaration on facts.source_id worked on PGLite but silently got dropped on Postgres in the v0.31 e2e run — the migration handler ran via postgres-js's `unsafe()` multi-statement path and the resulting facts table came back without the `facts_source_id_fkey` constraint. Same psql input run directly against the same database produced the FK; the difference was the unsafe() pipeline, not the SQL itself. Splitting the FK into a separate ALTER TABLE inside a DO block makes the constraint declaration explicit and idempotent: the named constraint either exists or it doesn't, the ALTER is a no-op on re-runs, and the failure mode is loud rather than silently leaving a CASCADE-less foreign key behind. Without this fix, deleting a source row leaves orphaned facts rows (test/e2e/facts-cross-source-isolation.test.ts CASCADE-on-sources- delete case caught it). With this fix the constraint is in place, the cascade fires, and both PG + PGLite e2e suites stay green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: update phase-count assertions for the new consolidate phase Three e2e/unit tests pinned the cycle phase count or order, all now updated to reflect v0.31's 10-phase cycle: - test/e2e/dream-cycle-eight-phase-pglite.test.ts: describe rename "8-phase cycle" → "10-phase cycle"; ALL_PHASES expectation extended to include 'consolidate' (between patterns + embed) and 'purge' (the v0.26.5 addition that was already in ALL_PHASES but missing from the test's assertion list). totals match adds the new facts_consolidated + consolidate_takes_written fields plus the pre-existing purged_sources_count + purged_pages_count that should have been added when v0.26.5 landed. - test/e2e/cycle.test.ts: dry-run full cycle now expects report.phases.length === 10 (was 9). - test/core/cycle.serial.test.ts: yieldBetweenPhases hook count + full cycle phases.length both updated 9 → 10. Comments call out the v0.31 addition lineage so the next person to add a phase sees the precedent. These are mechanical assertion bumps. The tests pass against the updated assertions on PGLite and Postgres. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(test): truncate facts table between e2e describe blocks setupDB() truncates ALL_TABLES between every describe block's beforeAll() hook. The list missed the new v0.31 facts table, so facts seeded by an earlier describe block leaked into Garry's Separation Test on Postgres — listFactsByEntity('travel') returned 2 rows instead of 1 because a prior facts-context-injection test had also seeded a 'travel' fact. Adding 'facts' to the truncate list (before 'pages' to respect FK ordering) makes every describe-block start from an empty facts table. Pinned by re-running the e2e file ordering that originally caught it (facts-recall-render → cross-source-isolation → serve-http-meta → context-injection → separation-postgres → facts-forget) — 13 pass / 0 fail after the fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: meta-hook cache + Postgres consolidate phase coverage Two net-new test files filling real coverage gaps the earlier sweep missed: - test/facts-meta-cache.test.ts (5 cases) — pins the eD3/eD10 cache contract that the dispatcher relies on. 30s TTL hit path, post-bump fresh-query, scoped invalidation (bump for sess-A leaves sess-B cache warm — closes the cross-source leak risk codex F5 originally surfaced on the recall payload), facts-self ops skip injection (anti-loop on recall / extract_facts / forget_fact), distinct allow-lists produce distinct cache entries. - test/e2e/cycle-consolidate-postgres.test.ts (3 cases) — Postgres parity for the dream-cycle consolidate phase. Mirrors the PGLite unit test but exercises the real postgres-engine codepaths: sql.begin transactions, advisory locks on insertFact's entity-slug dedup window, unsafe('::vector') casts on findCandidateDuplicates ordering, addTakesBatch postgres-js unnest path. Happy path (4 facts → 1 take + all consolidated_into set), age-threshold skip, dry-run no-write. All 5 unit + 3 e2e tests pass. Closes the unit-only gap on the consolidate phase (was only PGLite-tested) and pins meta-cache invariants the dispatcher depends on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix: thread auth + sourceId, JSON-shape every error envelope Three bugs surfaced during the full e2e sweep that all trace back to my v0.31 dispatch refactor (D12/eE1) silently dropping auth threading + non-OperationError exceptions emitting plain strings: 1. **HTTP MCP transport lost ctx.auth.** Refactoring serve-http.ts to call dispatchToolCall meant auth had to come through DispatchOpts, but the field didn't exist yet. Every HTTP whoami call returned `unknown_transport` because ctx.auth was undefined. Added `auth?: AuthInfo` to DispatchOpts, plumbed it through buildOperationContext, and updated serve-http.ts:816 to pass `auth: authInfo` alongside sourceId/takesHoldersAllowList. Pinned by sources-remote-mcp e2e `whoami reports oauth transport + sources_admin scope`. 2. **Non-OperationError exceptions emitted plain strings, not JSON.** The pre-v0.31 serve-http.ts always wrapped errors in JSON envelope `{error, message}`; my dispatch refactor missed the unknown-tool + uncaught-throw paths and emitted `Error: ${msg}` text content. Every caller that did `JSON.parse(content)` (sources-remote-mcp callMcp helper at line 104) crashed with `Unexpected identifier "Error"`. Both error paths in dispatchToolCall now return JSON-shaped content matching the OperationError pattern. 3. **Files→sources FK silently lost on rewound bootstrap path.** test/e2e/postgres-bootstrap.test.ts simulates a pre-v0.21 brain by `DROP TABLE IF EXISTS sources CASCADE` which removes files_source_id_fkey while leaving files.source_id intact. The v23 migration's `ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id ... REFERENCES sources(id) ON DELETE CASCADE` is a no-op when the column exists, so the FK never came back on upgrade — and any sources-remove afterward stopped cascading to files. Added a defensive `IF NOT EXISTS files_source_id_fkey ... ALTER TABLE ADD CONSTRAINT` block inside v23's handler. Pinned by `multi-source — cascade delete covers every dependent row` after running postgres-bootstrap. Plus: src/core/preferences.ts now honors GBRAIN_HOME for `~/.gbrain/migrations/completed.jsonl`. Without this, the doctor exits-0 mechanical test inherits the developer machine's stale partial-migration ledger entries (0.21.0, 0.22.4, 0.28.0, 0.29.1 prior dev work) and surfaces them as the [FAIL] minions_migration check. GBRAIN_HOME-scoped tempdir per test now isolates this state cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 chore: scrub personal references from public artifacts Per the CLAUDE.md privacy rule on `Garry's Separation Test`, replace personally-coded references in v0.31 artifacts with neutral examples: - CHANGELOG.md v0.31 entry: rename "Garry's Separation Test" header to "The cross-session test" + drop the "topic-2659/topic-1941, 7 AM/2 PM, flying to Tokyo" narrative. - src/commands/migrations/v0_31_0.ts feature pitch: same scrub. - test/facts-separation-pglite.test.ts + test/e2e/facts-separation-postgres.test.ts: rename describe blocks; replace specific topic-NNNN session ids with session-A / session-B; replace personal sample fact with "sample event Tuesday". - src/core/facts/extract.ts extractor system prompt example slugs: people/sam-altman → people/alice-example; companies/anthropic → companies/acme. - src/core/entities/resolve.ts comment: Sam Altman → Alice Example. - All v0.31 test fixtures: people/sam → people/alice-example, Sam Altman → Alice Example, sam-the-cofounder → alice-the-cofounder. Test names referencing real-world entities replaced with neutral slugs. Pre-existing references to "Garry" elsewhere in CHANGELOG (v0.17, v0.19, v0.21+ entries) are untouched — that's a separate scope from this v0.31 ship. Plus: the truncate fix for the Bun-script-induced syntax error in test/e2e/mechanical.test.ts (cliEnv arrow function had ", 30_000)" tacked onto its closing brace by the bulk-add-timeouts script — repaired to a clean function definition). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(test): bump E2E phase-count assertions for 11-phase cycle Two E2E tests still asserted the v0.31 pre-merge 10-phase shape (consolidate inserted, but recompute_emotional_weight from v0.29 not yet absorbed). With master's v0.29 work merged in, the cycle is now 11 phases: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → consolidate → embed → orphans → purge. - test/e2e/cycle.test.ts: 10 → 11 - test/e2e/dream-cycle-eight-phase-pglite.test.ts: ALL_PHASES + dry-run order Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(merge): close brace between v44 and v45 migration objects The v0.30.2 merge resolution stitched master's v40-v44 migrations onto HEAD's v45 (facts hot memory) migration but lost the closing `},` between v44 and v45. tsc caught it as TS1136 Property assignment expected at migrate.ts:2188. This is a one-line bracket fix; the rest of the merge resolution is correct and tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix: put_page cliHints + buildPlan v0.31.0 in skippedFuture Two unit-test failures surfaced after the v0.30.2 merge: 1. operations.ts: put_page had `cliHints: { name: 'put', positional: ['stdin'] }` from earlier v0.31 development. The parity test enforces that every name in `positional` is a real param. Restored master's correct shape: `{ name: 'put', positional: ['slug'], stdin: 'content' }`. 2. test/apply-migrations.test.ts: the H9 regression tests pin the exact skippedFuture list. Adding v0.31.0 to the registry meant the list grew by one. Updated both `expect(...).toEqual([...])` assertions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 docs: clarify consolidate is 11th phase + regen llms-full.txt CHANGELOG.md narrative said "new 10th phase consolidate"; with v0.29's recompute_emotional_weight already on master, consolidate is the 11th phase (between recompute and embed). Schema migration is v45, not v40, after the merge resolution renumbered it to clear master's v40-v44. llms-full.txt regenerated to reflect the README's 11-phase dream-cycle phrasing (the build-llms test enforces commit-time parity). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-09 16:57:47 -07:00
4 people	b8e0a0eada	v0.29.0 + v0.29.1 feat: salience + anomaly detection — brain surfaces what's hot without being asked (#730) ... * v0.29 foundation: emotional_weight column + formula + anomaly stats Migration v34 adds pages.emotional_weight REAL DEFAULT 0.0 (column-only, no index — salience query orders by computed score, not raw weight). Embedded DDL (schema.sql + pglite-schema.ts + schema-embedded.ts) mirrors the column so fresh installs don't need migration replay. types.ts gains: PageFilters.sort enum + PAGE_SORT_SQL whitelist (engines hardcoded ORDER BY updated_at DESC; threading lands in the next commit); SalienceOpts/SalienceResult, AnomaliesOpts/AnomalyResult, EmotionalWeightInputRow/EmotionalWeightWriteRow contracts. cycle/emotional-weight.ts: pure-function score in [0..1] from tags + takes (anglocentric default seed list; user-overridable via config key emotional_weight.high_tags). cycle/anomaly.ts: meanStddev + cohort threshold helpers with zero-stddev fallback (count > mean + 1) so rare cohorts don't produce NaN sigmas. Test coverage: migrate v34 structural assertions + 14-case formula unit + 13-case anomaly stats unit. Codex review fixes baked in: formula clamped to [0,1]; per-take weight clamped to [0,1] before averaging; zero-stddev fallback finite, never NaN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 engine: batch emotional-weight methods + listPages sort BrainEngine adds 4 methods, both engines implement: - batchLoadEmotionalInputs(slugs?): CTE-shaped read with per-table pre-aggregates. A page with N tags + M takes never produces N×M rows (codex C4#4) — page_tags + page_takes CTEs aggregate independently, then LEFT JOIN to pages. - setEmotionalWeightBatch(rows): UPDATE FROM unnest($1::text[], $2::text[], $3::real[]) composite-keyed on (slug, source_id). Multi- source brains can't fan out (codex C4#3) — pages.slug is unique only within source_id. Same shape that v0.18 link batches use. - getRecentSalience: time boundary computed in JS, bound as TIMESTAMPTZ. SQL identical across engines (codex C5/D5 — avoids dialect drift on $1::interval binding which has zero current uses on PGLite). - findAnomalies: tag + type cohort baselines via generate_series- densified daily-count CTEs (codex C4#6). Sparse-day rare cohorts get correct (mean, stddev) instead of biased upward by zero-omission. Year cohort deferred to v0.30. listPages threads the new PageFilters.sort enum through both engines. Was hardcoded ORDER BY updated_at DESC; now PAGE_SORT_SQL whitelist maps the 4 enum values to literal SQL fragments — no injection surface. postgres.js uses sql.unsafe; PGLite splices the fragment directly. Regression tests (PGLite, no DATABASE_URL needed): - multi-source-emotional-weight: same slug under two source_ids, setEmotionalWeightBatch on one of them, asserts the other survives untouched. Direct codex C4#3 guard. - list-pages-regression (IRON RULE): old call shape (type, tag, limit) still returns updated_desc default; new sort=updated_asc reverses; sort=created_desc orders by created_at; sort=slug alphabetical; unsupported sort enum falls back to default (defense in depth). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 cycle: new recompute_emotional_weight phase Adds a 9th cycle phase between extract and embed. Sees the union of syncPagesAffected + synthesizeWrittenSlugs for incremental mode (so synthesize-written pages get their weight computed too — codex C2 caught that the prior plan threaded only sync). Full mode (no incremental anchors) walks every page; users hit this path on first upgrade via gbrain dream --phase recompute_emotional_weight. Phase orchestrator (cycle/recompute-emotional-weight.ts) is two SQL round-trips total regardless of brain size: 1. batchLoadEmotionalInputs(slugs?) → per-page tag/take inputs. 2. computeEmotionalWeight in memory (pure function). 3. setEmotionalWeightBatch(rows) → composite-keyed UPDATE FROM unnest. Empty affectedSlugs short-circuits (no DB read, no write). Dry-run computes weights and reports the would-write count without touching the DB. Engine throw bubbles into status:fail with code RECOMPUTE_EMOTIONAL_WEIGHT_FAIL — cycle continues to the next phase. Plumbing: - CyclePhase type adds 'recompute_emotional_weight'. - ALL_PHASES + NEEDS_LOCK_PHASES include it. - CycleReport.totals adds pages_emotional_weight_recomputed (additive, schema_version stays "1"). - runCycle's totals rollup + status derivation honor the new field. - synthesize.ts emits writtenSlugs in details so cycle.ts can union with syncPagesAffected for incremental backfill. Tests: 7-case unit (fake-engine), 3-case PGLite e2e (full mode + dry- run + ALL_PHASES position), 1000-page perf budget (<5s on PGLite). Codex C2 → A: clean separation. Phase doesn't modify runExtractCore; runs on its own seam after the existing 8 phases plus synthesize. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 ops: get_recent_salience + find_anomalies + get_recent_transcripts Three new MCP operations + a transcripts library: - get_recent_salience: pages ranked by emotional + activity salience. Subagent-allow-listed. params: days (default 14), limit (default 20, capped 100), slugPrefix (renamed from `kind` per codex C4#10 to avoid collision with PageKind/TakeKind). - find_anomalies: cohort-level activity outliers (tag + type). Subagent-allow-listed. Year cohort deferred to v0.30. - get_recent_transcripts: raw .txt transcripts from the dream-cycle corpus dirs. LOCAL-ONLY: rejects ctx.remote === true with permission_denied (codex C3). NOT in the subagent allow-list — all subagent calls run with remote=true, would always reject (footgun if visible). Cycle's synthesize phase calls discoverTranscripts directly, so subagents that need transcripts go through the library function, not the op. Tool descriptions extracted to src/core/operations-descriptions.ts so they're pinnable in tests and stable for the Tier-2 LLM routing eval. Redirects on query/search/list_pages: personal/emotional questions should reach the new ops, not semantic search. Anti-flattery hint on query: "Do NOT assume words like crazy, notable, or big mean impressive — they often mean difficult or emotionally charged." list_pages gains updated_after (string ISO) and sort enum params, surfacing the engine threading from the prior commit. src/core/transcripts.ts: filesystem walk shared by the gated MCP op and the (commit 5) CLI command. Reuses discoverTranscripts corpus-dir resolution + isDreamOutput from cycle/transcript-discovery.ts. Trust gate lives in the op handler, not the library — the library is trusted by both the gated op and the local CLI. Allow-list: 11 → 13 (add salience + anomalies; transcripts excluded per codex C3, with a comment explaining why). Tests: 21-case description pin (catches accidental edits that change LLM-facing surface); 11-case transcripts unit covering trust gate, mtime window, dream-output skip, summary truncation, no corpus_dir; 2-case salience type-contract smoke (full Garry-test fixture in commit 6's e2e suite). Codex C1: routing-eval fixtures (skills/<x>/routing-eval.jsonl) deliberately NOT shipped — routing-eval.ts is substring-match on resolver triggers, not MCP tool routing. Real coverage lands as test/e2e/salience-llm-routing.test.ts in commit 6. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 CLI: gbrain salience / anomalies / transcripts Three new CLI commands wired into src/cli.ts dispatch + CLI_ONLY set + help text: - gbrain salience [--days N] [--limit N] [--kind PREFIX] [--json] - gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json] - gbrain transcripts recent [--days N] [--full] [--json] Each command file mirrors src/commands/orphans.ts shape: pure data fn + JSON formatter + human formatter. Calls into engine.getRecentSalience / findAnomalies (already shipped) and src/core/transcripts.ts. salience and anomalies show ranked rows with per-cohort mean/stddev/sigma. transcripts honors `--full` (caps at 100KB/file) vs default summary (first non-empty line + ~250 chars). All three emit JSON with --json for agent consumption. `--kind` is accepted as a slug-prefix shorthand on `gbrain salience` even though the underlying op param is `slugPrefix` (kept the CLI flag short; the MCP-facing param uses the more-explicit name to align with PageKind/TakeKind/slugPrefix vocabulary). CLI_ONLY set in src/cli.ts gains the three new command names so they don't get forwarded to MCP-only routing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 e2e: Garry-test fixtures + Postgres parity + LLM routing eval PGLite e2e (no DATABASE_URL needed): - salience-pglite: the Garry test. 7 wedding-tagged pages updated today + 100 background pages backdated across 30 days via raw SQL UPDATE (codex C4#7 — engine.putPage stamps updated_at = now(), so seeding via the engine alone can't reproduce historical recency windows). Asserts wedding pages outrank random-tag noise in the 7-day window; slugPrefix filter narrows correctly; days=0 boundary case; limit cap. - anomalies-pglite: same fixture shape (7 wedding pages today, 100 background backdated). findAnomalies with sigma=3 returns the wedding-tag cohort with sigma_observed > 3 vs near-zero baseline; page_slugs sample carries the wedding pages; date with no activity returns []; high sigma threshold suppresses borderline cohorts (zero-stddev fallback stays finite — no NaN sigma). Postgres-gated e2e: - engine-parity-salience: PGLite ↔ Postgres parity for getRecentSalience and findAnomalies. Same fixture into both engines; top-result and cohort-set match. Closes the v0.22.0-style parity gap for the new v0.29 SQL idioms (EXTRACT(EPOCH ...), generate_series, CTE chain). Tier-2 LLM routing eval (ANTHROPIC_API_KEY-gated): - salience-llm-routing: calls Claude with v0.29 tool descriptions and 12 personal-query phrasings ("anything crazy lately", "what's been going on with me", etc.). Asserts the chosen tool is in the v0.29 set, not query() / search(). ~$0.10 per CI run on Haiku. Tests the ACTUAL ship criterion — replaces the discarded fake-coverage routing-eval.jsonl fixtures (codex C1 → B). This is the only test that proves the description edits drive routing. Without it, we'd ship description changes and only learn from production behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.0: ship-prep — VERSION + CHANGELOG + CLAUDE Key Files VERSION + package.json bump 0.28.0 → 0.29.0. CHANGELOG.md adds a v0.29.0 release-summary in the GStack/Garry voice plus the "To take advantage of v0.29.0" block. Headline two-liner: "The brain tells you what's hot without being asked. Salience + anomaly detection ship. Search rewards hypotheses; salience surfaces them." Numbers-that-matter table covers engine surface delta, MCP op delta, allow-list delta, cycle-phase delta, schema migration, list_pages param surface, and test count. Itemized changes section lists the schema migration + new cycle phase + new MCP ops + redirect descriptions + subagent allow-list rules + new tests + a contributor note clarifying that routing-eval.ts is not the right surface for testing MCP tool routing (use the Tier-2 LLM eval pattern instead). CLAUDE.md Key Files updated for the v0.29 surface: - src/core/engine.ts: notes the 4 new methods + PageFilters.sort threading. - src/core/migrate.ts: v34 (pages_emotional_weight) entry. - src/core/cycle.ts: 8 → 9 phases, recompute_emotional_weight inserted between patterns and embed; totals.pages_emotional_weight_recomputed. - src/core/cycle/emotional-weight.ts (NEW): formula + override path. - src/core/cycle/anomaly.ts (NEW): stats helpers + zero-stddev fallback. - src/core/cycle/recompute-emotional-weight.ts (NEW): phase orchestrator. - src/core/transcripts.ts (NEW): library shared by gated MCP op + CLI. - src/core/operations-descriptions.ts (NEW): pinned tool descriptions. - src/core/minions/tools/brain-allowlist.ts: 11 → 13 entries; comment on why get_recent_transcripts is excluded. - src/commands/salience.ts / anomalies.ts / transcripts.ts (NEW): CLI surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1 feat: recency + salience as two orthogonal options on query op (#696) * feat: recency boost for search (v0.27.0) — temporal intent auto-detection, date filters, configurable decay New search pipeline stage: keyword + vector → RRF → cosine re-score → backlink boost → recency boost → dedup - applyRecencyBoost: hyperbolic decay, two strengths (moderate 30-day halflife, aggressive 7-day halflife) - Auto-enabled when intent.ts detects temporal/event queries (detail='high') - Manual override via SearchOpts.recencyBoost (0/1/2) - Date filtering: afterDate/beforeDate on all three search paths (keyword, keywordChunks, vector) - getPageTimestamps on both Postgres and PGLite engines - 15 tests passing (boost math + intent classification) * v0.29.1 schema: pages.{effective_date, effective_date_source, import_filename, salience_touched_at} + expression index Migration v38 adds 4 nullable columns to pages and an expression index on COALESCE(effective_date, updated_at) to support the new since/until date filters. All additive — no behavior change in the default search path; only consulted when callers opt into the new salience='on' / recency='on' axes or pass since/until. effective_date — content date (event_date / date / published / filename-date / fallback). Read by recency boost and date-filter paths only. Auto-link doesn't touch it (immune to updated_at churn). effective_date_source — sentinel for the doctor's effective_date_health check ('event_date' | 'date' | 'published' | 'filename' | 'fallback'). import_filename — basename without extension, captured at import. Used for filename-date precedence on daily/, meetings/. Older rows leave it NULL. salience_touched_at — bumped by recompute_emotional_weight when emotional_weight changes. Salience window uses GREATEST(updated_at, salience_touched_at) so newly-salient old pages enter the recent salience query. Index strategy: a partial index on effective_date alone wouldn't help the COALESCE expression in since/until filters (planner can't use it for the negative side). The expression index ((COALESCE(effective_date, updated_at))) is what actually accelerates the filter. Postgres uses CONCURRENTLY + v14-style pg_index.indisvalid pre-drop guard for prior failed CONCURRENTLY runs; PGLite uses plain CREATE INDEX. Mirror of v34's pattern. src/schema.sql + src/core/pglite-schema.ts updated for fresh installs; src/core/schema-embedded.ts regenerated via bun run build:schema. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: computeEffectiveDate helper + putPage integration Pure helper computing a page's effective_date from frontmatter precedence: 1. event_date (meeting/event pages) 2. date (dated essays) 3. published (writing/) 4. filename-date (leading YYYY-MM-DD in basename) 5. updated_at (fallback) 6. created_at (last resort) Per-prefix override: for daily/ and meetings/ slugs, filename-date jumps to position 1 — the filename is the user's primary signal there. Returns {date, source}. The source label powers the doctor's effective_date_health check to detect "fell back to updated_at" rows that look populated but are functionally a NULL. Range validation: parsed value must be in [1990-01-01, NOW + 1 year]. Out-of-range values drop to the next chain element. Wired into importFromContent + importFromFile. The put_page MCP op derives filename from slug-tail when no caller-supplied filename is available. putPage SQL on both engines extended to write the new columns. ON CONFLICT uses COALESCE(EXCLUDED.x, pages.x) so callers that don't know about the new columns (auto-link, code reindex) preserve existing values rather than blanking them. SELECT projection extended to return them; rowToPage threads them through. 21 unit tests covering: precedence chain default order, per-prefix override, parse failure fall-through, range validation [1990, NOW+1y], parseDateLoose shape variants. All pass; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: backfill orchestrator + library function for existing pages src/core/backfill-effective-date.ts is the shared library function. Walks pages in keyset-paginated batches (id > last_id ORDER BY id LIMIT 1000), runs computeEffectiveDate per row, UPDATEs effective_date + effective_date_source. Resumable via the `backfill.effective_date.last_id` checkpoint key in the config table — a killed process can re-run and pick up without re-doing rows. Idempotent: a full re-walk produces the same writes. Postgres-only: SET LOCAL statement_timeout = '600s' per batch. Doesn't refuse the migration on low session settings (codex pass-2 #16). src/commands/migrations/v0_29_1.ts is the orchestrator (4 phases mirroring v0_12_2). Phase A schema (gbrain init --migrate-only), Phase B backfill (via the library function), Phase C verify (count NULL effective_date), Phase D record (handled by runner). The library function is reusable from the gbrain reindex-frontmatter CLI command in the next commit. import_filename stays NULL for backfilled rows — pre-v0.29.1 imports didn't capture it. computeEffectiveDate uses the slug-tail when filename is NULL; daily/2024-03-15 backfilled gets effective_date from the slug. Registered in src/commands/migrations/index.ts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: gbrain reindex-frontmatter CLI command Recovery / explicit-rebuild path for pages.effective_date. Used when: - User edited frontmatter dates after import - Post-upgrade backfill orchestrator finished but the user wants to re-walk a subset (e.g. just meetings/) after fixing some frontmatter - Precedence rules change between releases Thin wrapper over backfillEffectiveDate from commit 3 — same code path the v0_29_1 orchestrator uses; one source of truth. Flags mirror reindex-code: --source <id> Scope to one sources row (placeholder; library library doesn't filter by source today, tracked v0.30+) --slug-prefix P Scope to slugs starting with P (e.g. 'meetings/') --dry-run Print what WOULD change, no DB writes --yes Skip confirmation prompt (required for non-TTY non-JSON) --json Machine-readable result envelope --force Re-apply even when computed value matches existing Wired into src/cli.ts. CLI handles its own engine lifecycle (creates + disconnects). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: recency-decay map + buildRecencyComponentSql (pure, unused) src/core/search/recency-decay.ts mirrors source-boost.ts in shape but drives RECENCY ONLY (per D9 codex resolution). Salience is a separate orthogonal axis; this map does not feed it. DEFAULT_RECENCY_DECAY: 10 generic prefixes (no fork-specific names). - concepts/ evergreen (halflifeDays=0) - originals/ 180d × 0.5 (long-tail decay; new essays nudged) - writing/ 365d × 0.4 - daily/ 14d × 1.5 (aggressive — freshness IS the signal) - meetings/ 60d × 1.0 - chat/ 7d × 1.0 - media/x/ 7d × 1.5 - media/articles/ 90d × 0.5 - people/companies/ 365d × 0.3 - deals/ 180d × 0.5 DEFAULT_FALLBACK: 90d × 0.5 for unmatched slugs. Override priority: defaults < gbrain.yml recency: < env (GBRAIN_RECENCY_DECAY) < per-call SearchOpts.recency_decay. parseRecencyDecayEnv format: comma-separated prefix:halflifeDays:coefficient triples. Refuses LOUD on parse error (RecencyDecayParseError) — codex pass-2 #M3 finding. No silent fallback like source-boost's parser. parseRecencyDecayYaml takes already-parsed YAML; throws on bad shape. buildRecencyComponentSql in sql-ranking.ts emits a CASE expression with longest-prefix-first ordering, evergreen short-circuit (literal 0 when halflifeDays=0 or coefficient=0), and EXTRACT(EPOCH ...) for non-zero branches. Output: ((CASE WHEN p.slug LIKE 'daily/%' THEN 1.5 * 14.0 / (14.0 + EXTRACT(EPOCH FROM (NOW() - <dateExpr>))/86400.0) ... END)) Typed NowExpr enum prevents SQL injection (codex pass-1 #5). Tests pass { kind: 'fixed', isoUtc } for deterministic output; production NOW(). The 'fixed' branch escapes single quotes via escapeSqlLiteral. 25 unit tests covering: env parser shape, env error cases, yaml parser shape, merge precedence (defaults < yaml < env < caller), CASE longest- prefix-first ordering, evergreen short-circuit, NowExpr fixed/now, single-quote injection defense, empty decayMap fallback path, default map composition (no fork names, concepts/ evergreen, daily/ aggressive). Pure module. Zero consumers in this commit; commit 6 wires it into getRecentSalience, commit 10 wires it into the post-fusion stage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: refactor getRecentSalience to consume buildRecencyComponentSql Both engines (Postgres + PGLite) now build the salience formula's third term via buildRecencyComponentSql instead of inlining 1.0 / (1 + days_old). Parameters: empty decayMap + fallback { halflifeDays: 1, coefficient: 1.0 }. Math expands to 1 * 1.0 / (1.0 + days_old) = 1 / (1 + days_old) — same numeric output as v0.29.0. This is a no-behavior-change refactor preparing for commit 7's recency_bias param. recency_bias='flat' (default) reproduces v0.29.0 exactly; 'on' swaps in DEFAULT_RECENCY_DECAY for per-prefix decay. Single source of truth for the recency math: same builder feeds the salience query AND (in commit 10) the post-fusion applyRecencyBoost stage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: get_recent_salience gains recency_bias param (default 'flat') SalienceOpts.recency_bias: 'flat' | 'on' added; default 'flat' preserves v0.29.0 ranking verbatim. Pass 'on' to opt into per-prefix decay map (concepts/originals/writing/ evergreen; daily/, media/x/, chat/ aggressive decay). When recency_bias='on', the salience query reads COALESCE(p.effective_date, p.updated_at) instead of bare p.updated_at, so the recency component is immune to auto-link updated_at churn — old concepts/ pages just-touched by auto-link don't suddenly look fresh. Both engines (Postgres + PGLite) wire the param through. resolveRecencyDecayMap() honors gbrain.yml + GBRAIN_RECENCY_DECAY env at runtime. MCP op surface: get_recent_salience gains the param with a load-bearing description teaching the agent when to use 'on' vs 'flat' (current state → on; mattering across all time → flat). No silent v0.29.0 behavior change — opt-in only (per D11 codex resolution). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: recompute_emotional_weight writes salience_touched_at; window picks up newly-salient pages setEmotionalWeightBatch on both engines now bumps salience_touched_at to NOW() ONLY when the new emotional_weight differs from the existing one (IS DISTINCT FROM, NULL-safe). No-op writes (same weight) leave the column alone — preserves "actual change" semantics. getRecentSalience window changes from WHERE p.updated_at >= boundary to WHERE GREATEST(p.updated_at, COALESCE(p.salience_touched_at, p.updated_at)) >= boundary Closes codex pass-1 finding #4: pages whose emotional_weight just changed in the dream cycle (because tags or takes shifted) but whose updated_at is older than the salience window now correctly enter the recent-salience results. Without this, "Garry just added a take to a 6-month-old page" stayed invisible to get_recent_salience until the next content edit. COALESCE(salience_touched_at, p.updated_at) handles pre-v0.29.1 rows where salience_touched_at is NULL — they fall back to p.updated_at and behave identically to v0.29.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: merge intent.ts → query-intent.ts; emit 3 suggestions per query D1 + D4 + D6 + D8: single regex-pass classifier returning {intent, suggestedDetail, suggestedSalience, suggestedRecency}. intent + suggestedDetail are v0.29.0 behavior verbatim (legacy intent.ts deleted; classifyQueryIntent + autoDetectDetail compat shims preserved). NEW for v0.29.1 — two orthogonal recency-axis suggestions: suggestedSalience: 'off' | 'on' | 'strong' suggestedRecency: 'off' | 'on' | 'strong' Resolution rules (per D6 narrow temporal-bound exception): - CANONICAL patterns (who is X / what is Y / code / graph) → both off - UNLESS an EXPLICIT_TEMPORAL_BOUND also matches (today / right now / this week / since X / last N days), in which case temporal-bound wins - STRONG_RECENCY (today / right now / this morning / just now) → strong - RECENCY_ON (latest / recent / this week / meeting prep / catch up / remind me / status update) → on - SALIENCE_ON (catch up / remind me / status update / prep me / what's going on / what matters) → on - default → off for both axes (v0.29.1 prime-directive: pure opt-in) Salience and recency are TRULY orthogonal (per D9). A query like "latest news on AI" → recency='on' but salience='off' (the user wants fresh, not emotionally-weighted). "What's going on with widget-co" → both on. "Who is X right now" → both 'strong'/'on' (temporal bound beats canonical 'who is'). intent.ts deleted; test/intent.test.ts renamed → test/query-intent-legacy.test.ts (unchanged behavior coverage). New test/query-intent.test.ts adds 21 cases covering all three axes' interactions: canonical wins on bare 'who is', temporal bound overrides, "catch me up" matches with up to 15 chars between, "today" → strong, intent vs recency independence. Updated callers: - src/core/search/hybrid.ts (autoDetectDetail import) - test/recency-boost.test.ts (classifyQueryIntent import) - test/benchmark-search-quality.ts (autoDetectDetail import) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: applySalienceBoost + applyRecencyBoost + runPostFusionStages wrapper D9 + codex pass-1 #2 + #3 + pass-2 #4: salience and recency are TRULY ORTHOGONAL post-fusion stages, both running from ALL THREE hybridSearch return paths (keyword-only, embed-failure-fallback, full-hybrid). NEW src/core/search/hybrid.ts exports: - applySalienceBoost(results, scores, strength) score *= 1 + k * log(1 + score) where k = 0.15 (on) or 0.30 (strong) No time component. Pure mattering signal. - applyRecencyBoost(results, dates, strength, decayMap, fallback, nowMs?) Per-prefix decay factor: 1 + strengthMul * coefficient * halflife / (halflife + days_old) strengthMul: 1.0 (on) or 1.5 (strong) Evergreen prefixes (halflifeDays=0) skipped (factor 1.0). Pure recency signal. Independent of mattering. - runPostFusionStages(engine, results, opts) Wraps backlink + salience + recency. Called from EACH return path so keyless installs and embed failures get the same boost surface as the full hybrid path. NEW engine methods (composite-keyed for multi-source isolation): - getEffectiveDates(refs: Array<{slug, source_id}>): Map<key, Date> Returns COALESCE(effective_date, updated_at, created_at). Key format: `${source_id}::${slug}`. Mirror of getBacklinkCounts shape. - getSalienceScores(refs: Array<{slug, source_id}>): Map<key, number> Returns emotional_weight × 5 + ln(1 + take_count). Composite key. Deprecated (kept for back-compat through v0.29.x): - SearchOpts.afterDate / beforeDate (alias for since/until) - SearchOpts.recencyBoost: 0|1|2 (alias for recency: 'off'|'on'|'strong') - getPageTimestamps (use getEffectiveDates instead) NEW SearchOpts fields: - salience: 'off' | 'on' | 'strong' - recency: 'off' | 'on' | 'strong' - since: string (ISO-8601 or relative, replaces afterDate) - until: string (replaces beforeDate) Resolution: caller-explicit > legacy alias (recencyBoost) > heuristic (classifyQuery's suggestedSalience / suggestedRecency). Deleted: src/core/search/recency.ts (PR #618's, replaced) + test/recency-boost.test.ts (its scope is replaced by query-intent.test.ts + future post-fusion tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Wintermute <wintermute@garrytan.com> * v0.29.1: query op gains salience + recency + since + until params; PGLite since/until parity Combines commits 12 + 13 of the plan. Query op surface (src/core/operations.ts): - salience: 'off' | 'on' | 'strong' (with load-bearing description) - recency: 'off' | 'on' | 'strong' - since: string (ISO-8601 or relative; replaces deprecated afterDate) - until: string (replaces deprecated beforeDate) Tool descriptions teach the calling agent: - salience axis = mattering, no time component - recency axis = age decay, no mattering signal - omit either to let gbrain auto-detect from query text via classifyQuery hybrid.ts maps since/until → afterDate/beforeDate at the engine call boundary so PR #618's existing engine plumbing keeps working without rename. Codex pass-1 #10 finding closed. PGLite engine (codex pass-1 #10): since/until parity added to all three search methods (searchKeyword, searchKeywordChunks, searchVector). SQL filter against COALESCE(p.effective_date, p.updated_at, p.created_at) so date filtering matches user content-date intent (a meeting was on event_date, not when it got reimported). Filter is applied INSIDE the HNSW inner CTE in searchVector so HNSW's candidate pool already excludes out-of-range pages — preserves pagination contract. This also closes existing cross-engine drift: pre-v0.29.1 Postgres had afterDate/beforeDate from PR #618; PGLite had nothing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: migration v39 — eval_candidates capture columns for replay reproducibility D11 codex pass-2 resolution: extend eval_candidates with 7 new nullable columns so `gbrain eval replay` can reproduce captured runs of agent-explicit salience + recency choices. Without these columns, replays of the new axis params drift. The live behavior depends on the resolved {salience, recency} values; v0.29.0's schema doesn't capture them. as_of_ts TIMESTAMPTZ — brain's logical NOW at capture (replay uses this instead of wall-clock) salience_param TEXT — what the caller passed (NULL if omitted) recency_param TEXT — same salience_resolved TEXT — final value applied recency_resolved TEXT — same salience_source TEXT — 'caller' or 'auto_heuristic' recency_source TEXT — same All nullable + additive. Pre-v0.29.1 rows stay valid. NDJSON schema_version STAYS at 1 — consumers ignore unknown fields (codex pass-1 #C2 dissolves; no cross-repo coordination needed). ADD COLUMN with no DEFAULT is metadata-only on PG 11+ and PGLite — instant on tables of any size. src/schema.sql + src/core/pglite-schema.ts mirror the additions for fresh installs; src/core/schema-embedded.ts regenerated. eval_capture.ts populates the new fields in commit 16 (docs + ship). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: doctor checks — effective_date_health + salience_health effective_date_health: sample-1000 scan detects three classes of problems (codex pass-1 #5 resolution via the effective_date_source sentinel column added in commit 1): fallback_with_fm_date — page fell back to updated_at even though frontmatter has parseable event_date / date / published. The "wrong but populated" residual that earlier review iterations missed. future_dated — effective_date > NOW() + 1 year (corrupt or typo'd century). pre_1990 — effective_date < 1990-01-01 (epoch math gone wrong, bad parse). Sample of last 1000 pages by default — fast on 200K-page brains. Fix hint: gbrain reindex-frontmatter. salience_health: detects pages with active takes whose emotional_weight is still 0 (recompute_emotional_weight phase hasn't run since the take landed). Reports the brain's non-zero emotional_weight count as an informational baseline. Fix hint: gbrain dream --phase recompute_emotional_weight. Both checks gracefully skip on pre-v0.29.1 brains (column doesn't exist → 42703) without surfacing as warnings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29.1: docs + skills convention + CHANGELOG + version bump - VERSION 0.29.0 → 0.29.1 - package.json version bump - CHANGELOG.md: full release-summary + itemized + "To take advantage" block per the project's voice rules. Two-line headline + concrete pathology framing (existing callers unchanged; new axes opt-in; agent in charge per the prime directive). - skills/conventions/salience-and-recency.md: agent-readable decision rules. "Current state → on. Canonical truth → off." plus the narrow temporal-bound exception. Cross-cutting convention propagates to brain skills via RESOLVER.md. - skills/migrations/v0.29.1.md: agent-readable upgrade instructions. Verify steps + behavior-change reference + recovery commands. The build-time tool-description generator from D2 (extract decision tables from skills/conventions/salience-and-recency.md, embed into operations.ts at build time) is deferred to a follow-up commit. The tool descriptions on the query op + get_recent_salience are inline in operations.ts for v0.29.1; the auto-gen + CI staleness gate land in v0.29.2 if drift becomes a problem in practice. 148 unit tests pass across the v0.29.1 surface (effective-date, recency-decay, query-intent, migrate, salience, recompute-emotional-weight). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Wintermute <wintermute@garrytan.com> --------- Co-authored-by: Wintermute <wintermute@garrytan.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 master-rebase fixups: renumber + drift cleanup - v0.29.1 migrations renumber v38/v39 → v41/v42 (master shipped takes_table at v37 + access_tokens_permissions at v38; v0.27.1 took v39). My v0.29.0 emotional_weight slots in at v40; v0.29.1's pages_recency_columns lands at v41 and eval_candidates_recency_capture at v42. - src/core/utils.ts comment refs updated v37 → v40 (emotional_weight) and v38 → v41 (effective_date/etc). - test/brain-allowlist.test.ts: size assertion 11 → 13 + the new get_recent_salience / find_anomalies positive checks + the explicit get_recent_transcripts negative check (v0.29 added the salience pair to the allow-list; transcripts are deliberately excluded because all subagent calls have remote=true and the v0.29 trust gate rejects them — visibility would be a footgun). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 CI fixups: privacy allow-list + cycle phase count + migration plan Three CI test failures on PR #730, all caused by master-side state the v0.29 cherry-picks didn't yet account for: 1. scripts/check-privacy.sh allow-lists test/recency-decay.test.ts The v0.29.1 recency-decay test asserts that DEFAULT_RECENCY_DECAY's keys do NOT include fork-specific path prefixes. Because the assertion has to name the banned tokens to assert their absence, the privacy guard flagged the literal occurrence. Same exception class as CHANGELOG.md, CLAUDE.md, and scripts/check-privacy.sh itself — meta-rule enforcement requires mentioning what the rule forbids. 2. test/core/cycle.serial.test.ts: 9 → 10 phases. The yieldBetweenPhases test was written for v0.26.5 (9 phases incl. purge). v0.29 added a 10th phase (recompute_emotional_weight) between patterns and embed; the test's expected hookCalls and report.phases.length needed bumping. 3. test/apply-migrations.test.ts: append '0.29.1' to skippedFuture lists. v0.29.1 added a new entry to src/commands/migrations/index.ts; the buildPlan test snapshots the exact ordered list of versions, so it needs the new entry in both the fresh-install case and the Codex H9 regression case. All three verified locally: - bash scripts/check-privacy.sh → exit 0 - bun test test/apply-migrations.test.ts → 18/18 pass - bun test test/core/cycle.serial.test.ts → 28/28 pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 CI fixup: regenerate llms-full.txt to match CLAUDE.md state build-llms test asserts the committed llms.txt + llms-full.txt match what the generator produces from the current source tree. CLAUDE.md got new v0.29 Key Files entries (recompute_emotional_weight phase, emotional-weight formula, anomaly stats, transcripts library, salience ops, etc.) without a corresponding regen. `bun run build:llms` brings llms-full.txt back in sync; llms.txt is byte-for-byte identical so only the larger inline bundle changed. Verified locally: bun test test/build-llms.test.ts → 7/7 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 e2e: cover tool-surfaces + MCP dispatch path Two gaps were uncovered when reviewing v0.29 coverage against the new contracts the cherry-picks landed onto master. 1. test/v0_29-tool-surfaces.test.ts (unit, 9 cases) Existing tests pin the description constants module and the BRAIN_TOOL_ALLOWLIST set membership, but nothing checked the two filters that ACT on those constants: - serve-http.ts:745 filters operations by !op.localOnly to build the HTTP MCP tool list. Without a test, anyone removing `localOnly: true` from get_recent_transcripts would silently expose it to remote callers — defense-in-depth on top of the in-handler ctx.remote check would be the only guard. Now pinned: get_recent_transcripts is hidden, salience + anomalies stay visible. - buildBrainTools surfaces the v0.29 ops as `brain_get_recent_salience` and `brain_find_anomalies`, and EXCLUDES `brain_get_recent_transcripts` (codex C3 footgun gate — all subagent calls are remote=true, the op would always reject). Now pinned. Both filters are pure functions; no DB / engine.connect needed. 2. test/e2e/v0_29-mcp-dispatch-pglite.test.ts (e2e, 5 cases) Existing v0.29 e2e tests call engine methods directly. None went through the full dispatchToolCall pipeline that stdio MCP and HTTP MCP both use. The new file covers: - get_recent_salience returns ranked rows via dispatch (top result is the wedding-tagged page from the seeded fixture). - find_anomalies returns the AnomalyResult shape via dispatch. - get_recent_transcripts rejects with permission_denied when ctx.remote === true (the in-handler trust gate is the last line if localOnly ever drops). - get_recent_transcripts succeeds with ctx.remote === false (CLI path) and returns [] when no corpus dir is configured. - Unknown tool name returns the standard isError + "Unknown tool" envelope (regression guard for dispatch shape). Verified locally — all 14 cases pass: bun test test/v0_29-tool-surfaces.test.ts → 9 pass bun test test/e2e/v0_29-mcp-dispatch-pglite.test.ts → 5 pass Re-ran the full v0.29 PGLite e2e suite to confirm no regressions: salience-pglite.test.ts 5 pass anomalies-pglite.test.ts 4 pass cycle-recompute-emotional-weight-pglite.test 3 pass list-pages-regression.test.ts 6 pass multi-source-emotional-weight-pglite.test 4 pass backfill-perf-pglite.test.ts 1 pass v0_29-mcp-dispatch-pglite.test.ts 5 pass ----- Total: 28 pass / 0 fail Postgres parity test (DATABASE_URL gated) 7 skip (correct) LLM routing eval (ANTHROPIC_API_KEY gated) 12 skip (correct) bun run typecheck clean Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.29 CI fixup: drop unused PGLiteEngine in tool-surfaces test scripts/check-test-isolation.sh's R3 + R4 lints flagged the new test/v0_29-tool-surfaces.test.ts for instantiating PGLiteEngine outside a beforeAll() block (R3) and lacking the matching afterAll(disconnect) (R4). The intent of those rules is to prevent engine leaks across the shard process — every PGLiteEngine must follow the canonical beforeAll(connect+initSchema) / afterAll(disconnect) pattern. The fix here is upstream of the rule, not a workaround: this test never needed an engine. buildBrainTools doesn't issue any SQL at registry-build time — it only reads `engine.kind` for the put_page namespace-wrap branch. A `{ kind: 'pglite' } as unknown as BrainEngine` fake-engine literal keeps the test pure-function: no WASM cold-start, no connect lifecycle, no test-isolation rule fired. Verified locally: bash scripts/check-test-isolation.sh → OK (257 non-serial unit files) bun test test/v0_29-tool-surfaces.test.ts → 9 pass bun run typecheck → clean Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Wintermute <wintermute@garrytan.com>	2026-05-07 21:52:58 -07:00
3 people	058fe69575	v0.26.7 test: isolation foundation (helpers + lint + quarantine) (#613) ... * test: add withEnv helper + canonical PGLite block JSDoc withEnv(overrides, fn) saves prior values, runs the callback, restores via try/finally — including on throw. Handles delete via undefined override. Nested calls compose. Cross-test safe; explicitly NOT intra-file concurrent-safe (process.env is process-global). 7 unit cases covering sync, async, delete-key, delete-when-prior-unset, restore-on-throw, nested compose, multi-key atomic restore. reset-pglite.ts JSDoc extended with the canonical 4-line PGLite block (beforeAll create + afterAll disconnect + beforeEach reset). The lint script in the next commit enforces this exact shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: add check-test-isolation lint script + wire into verify Grep-based lint enforcing 4 rules on non-serial unit test files: R1: no process.env mutations (use withEnv() or rename to *.serial.test.ts) R2: no mock.module() (rename to *.serial.test.ts) R3: new PGLiteEngine( only inside beforeAll() context R4: PGLiteEngine creators must pair with afterAll{disconnect} Wired into 'bun run verify' and 'bun run check:all' (NOT 'bun run test' which is the parallel runner script with no pre-check chain). Matches the existing scripts/check-*.sh family shape (jsonb, progress, etc). 51 baseline violators captured in scripts/check-test-isolation.allowlist. List MUST shrink over time — entries removed by v0.26.8 (env sweep) and v0.26.9 (PGLite sweep). New files cannot be added. CLAUDE.md ## Testing section extended with R1-R4 rules table, the canonical 4-line PGLite block, withEnv pattern, and when-to-quarantine guidance. 16 fixture-driven test cases for the lint: clean, R1 (5 patterns + 1 negative), R2, R3 (top-level vs in-beforeAll), R4 (missing disconnect), *.serial.test.ts skip, test/e2e/ skip, allowlist (3 cases). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: quarantine cycle and embed mock.module test files Both files use mock.module(...) at top level — leaks across files in the same shard process. The check-test-isolation lint (R2) bans this pattern in non-serial files; quarantine is the escape hatch. Per v0.26.7 plan D5: prefer quarantine over DI on runCycle/runEmbed. Production signatures stay frozen; tests run at --max-concurrency=1 in the serial post-pass (the existing pattern shipped in v0.26.4 for brain-registry and reconcile-links). Quarantine count: 2 → 4. Cap raised to 10 informational per D15. Renames: test/core/cycle.test.ts → test/core/cycle.serial.test.ts test/embed.test.ts → test/embed.serial.test.ts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.26.7) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: post-ship documentation sync for v0.26.7 - README.md "Contributing" line: point to bun run test + bun run verify (parallel fast loop) - CONTRIBUTING.md "Running tests": rewrite for the v0.26.4/v0.26.7 test surface (parallel runner, verify, slow/serial/e2e tiers) - CONTRIBUTING.md adds "Writing tests that survive the parallel loop" section: R1-R4 lint, canonical PGLite block, withEnv pattern, when to quarantine - llms-full.txt regenerated to pick up the README + CONTRIBUTING changes Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-04 12:59:52 -07:00
4 people	0de9eb68ba	v0.26.5 feat: destructive operation guard end-to-end (sources + pages + autopilot purge) (#600) ... * feat(v0.26.5): destructive operation guard — impact preview, confirmation gate, soft-delete Three-layer protection against accidental data loss: 1. **Impact preview**: Every destructive operation (sources remove, purge) now shows a formatted preview of exactly what will be destroyed — page count, chunk count, embedding count, file count — BEFORE acting. 2. **--confirm-destructive flag**: `--yes` alone is no longer sufficient when a source has data. Must pass `--confirm-destructive` to proceed with permanent deletion. Prevents scripted/reflexive destroys. 3. **Soft-delete with 72h TTL**: New `gbrain sources archive <id>` hides a source from search and federation without destroying any data. Data preserved for 72 hours. Restorable via `gbrain sources restore <id>`. Expired archives purged via `gbrain sources purge`. New subcommands: - `gbrain sources archive <id>` — soft-delete (hide, preserve 72h) - `gbrain sources restore <id>` — un-archive, re-federate - `gbrain sources archived` — list soft-deleted sources + TTL - `gbrain sources purge [<id>] [--confirm-destructive]` — permanent delete Behavioral changes: - `sources remove` with data now requires `--confirm-destructive` (not just `--yes`) - `sources remove --dry-run` shows full impact preview without side effects - Impact box format shows source name, id, and all cascade counts New files: - src/core/destructive-guard.ts — impact assessment, confirmation gate, soft-delete/restore/purge logic, display formatters * chore(release): v0.26.5 — destructive operation guard Bump VERSION + package.json to 0.26.5 and add the v0.26.5 CHANGELOG entry on top of the destructive-guard feature commit cherry-picked from PR #595. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v0.26.5): page-level soft-delete + autopilot purge + search visibility Closes the destructive-guard posture across every gbrain destructive surface. PR #595 cherry-pick covered the CLI source-remove path; this commit closes the higher-velocity MCP `delete_page` agent footgun and the three internal correctness gaps the CEO+Eng review surfaced: - Gap 1: archived sources were not actually filtered from search. Now they are, via `buildVisibilityClause` in `searchKeyword`/`searchKeywordChunks`/ `searchVector` for both engines. - Gap 2: 72h TTL was honor-system. Now wired into a new autopilot `purge` phase (9th in ALL_PHASES) that calls `purgeExpiredSources` + `engine. purgeDeletedPages(72)`. Manual escape hatch: `gbrain pages purge-deleted`. - Gap 3: zero tests for safety-critical code. ~30 cases now in `test/destructive-guard.test.ts`, `test/pages-soft-delete.test.ts`, and `test/sql-ranking.test.ts` covering the boundary truth table, JSONB→column migration, soft-delete/restore/purge round-trip, multi-source isolation, cascade verification, and the Q3 IRON-rule contract test. Schema migration v33 (`destructive_guard_columns`): adds `pages.deleted_at` + partial purge index, promotes `archived` from `sources.config` JSONB to real columns (`sources.archived BOOLEAN`, `archived_at`, `archive_expires_at`), backfills any pre-v0.26.5 JSONB shape. Engine-aware: Postgres uses CREATE INDEX CONCURRENTLY, PGLite uses plain CREATE INDEX. Forward-reference bootstrap extended in both engines so pre-v0.26.5 brains don't crash on the embedded-schema replay. BrainEngine surface: new `softDeletePage` / `restorePage` / `purgeDeletedPages` methods + `includeDeleted` flag on `getPage`/`listPages`. MCP ops: `delete_page` rewired to soft-delete (description string updated); new `restore_page` (scope: write) + `purge_deleted_pages` (scope: admin, localOnly: true). Q3 contract (eng-review lynchpin): `get_page(slug)` returns null for soft-deleted by default; `get_page(slug, {include_deleted: true})` surfaces the row with `deleted_at` populated. Same flag for `list_pages`. Mirrors the search-filter contract end-to-end. Issue 5 (eng-review): `archived` is now a real column on `sources`, not a JSONB key. No reserved-key footgun. Faster filter. Visibility clause compiles to a column lookup, not JSONB containment. Verification: - bun run typecheck: PASS - bun run build:schema + bun run build:llms: regenerated - targeted test runs: 90 pass / 0 fail across destructive-guard, pages-soft-delete, sql-ranking, schema-bootstrap-coverage, build-llms - full bun test: 16 pre-existing failures inherited from v0.26.2 (sync, sync-parallel, queue-child-done, etc — already filed in TODOS.md as "Fix 22 pre-existing test failures unrelated to OAuth") CHANGELOG, CLAUDE.md (Key Files + Commands), TODOS.md updated. The plan file at ~/.claude/plans/take-a-look-and-gentle-pine.md captures the full review trail (CEO=C, Eng-Q3=A, Eng-Issue5=a, 8 defaults applied). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.26.5): CI fallout — getStats excludes soft-deleted; tests use --confirm-destructive Two CI failures from the v0.26.5 ship: 1. **Tier 1 (Postgres E2E):** `E2E: Page CRUD > delete_page removes page and others survive` failed because `delete_page` now soft-deletes (sets deleted_at) but `getStats.page_count` was still counting all rows. The test seeds 16 pages, deletes one, and asserts page_count is 15. Fix: `getStats` now filters `WHERE deleted_at IS NULL` for page_count in both engines. This matches the visibility-filter contract — soft-deleted pages are hidden everywhere the user looks (search, get_page, list_pages, stats). Chunks and links stay raw because they still occupy storage until the autopilot purge phase runs. 2. **Test 2 (PGLite unit):** `multi-source-integration.test.ts:184` and `e2e/multi-source.test.ts:274` called `runSources(engine, ['remove', X, '--yes'])` against populated sources. v0.26.5's destructive guard rejects `--yes` alone on populated sources and calls `process.exit(5)`, which killed the bun test runner mid-suite (CI exit 5). Both test sites now pass `--confirm-destructive` per the v0.26.5 contract. Verification: 115/0 pass across destructive-guard, pages-soft-delete, sql-ranking, schema-bootstrap-coverage, sources, repos-alias, and multi-source-integration test files. typecheck PASS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): cycle phase count is 9 (v0.26.5 added `purge` phase) CI failure: `runCycle — yieldBetweenPhases hook` tests asserted exactly 8 phases. v0.26.5 added the autopilot `purge` phase as the 9th, so: - `test/core/cycle.test.ts:381` — `hookCalls` is now 9 (one yield per phase) - `test/core/cycle.test.ts:392` — `report.phases.length` is now 9 - `test/e2e/cycle.test.ts:101` — same update for the dry-run E2E The `purge` phase invocation was already visible in the failing log output: the cycle ran 9 phases end-to-end; the test assertions hadn't been updated. Verification: bun run typecheck PASS. cycle.test.ts: 28/0 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-03 20:41:39 -07:00
3 people	3c032d79ec	v0.26.0 feat: GBrain — MCP Keys OAuth 2.1 + HTTP server + admin dashboard (#358) ... * feat: OAuth 2.1 schema tables + shared token utilities Add oauth_clients, oauth_tokens, oauth_codes tables to both PGLite and Postgres schemas. Migration v5 creates tables for existing databases. PGLite now includes auth infrastructure (access_tokens, mcp_request_log, OAuth tables) because `serve --http` makes it network-accessible. Extract hashToken() and generateToken() to src/core/utils.ts for DRY reuse across auth.ts and oauth-provider.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: GBrainOAuthProvider — MCP SDK OAuthServerProvider implementation Implements OAuthServerProvider backed by raw SQL (PGLite or Postgres). Supports client credentials, authorization code with PKCE, token refresh with rotation, revocation, and legacy access_tokens fallback. Key decisions from eng review: - Uses raw SQL connection, not BrainEngine (OAuth is infrastructure) - All tokens/secrets SHA-256 hashed before storage - Legacy tokens grandfathered as read+write+admin - sweepExpiredTokens() wrapped in try/catch (non-blocking startup) - Client credentials: no refresh token per RFC 6749 4.4.3 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: scope + localOnly annotations on all 30 operations Add AuthInfo, scope ('read'|'write'|'admin'), and localOnly fields to Operation interface. Per-operation audit: - 14 read ops, 9 write ops, 2 admin ops, 4 admin+localOnly ops - sync_brain, file_upload, file_list, file_url: admin + localOnly - Scope enforcement happens in serve-http.ts before handler dispatch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: HTTP MCP server with OAuth 2.1 + 27 OAuth tests gbrain serve --http starts Express 5 server with: - MCP SDK mcpAuthRouter (authorize, token, register, revoke endpoints) - Custom client_credentials handler (SDK doesn't support CC grant) - Bearer auth + scope enforcement on /mcp tool calls - Admin dashboard auth via HTTP-only cookie + bootstrap token - SSE live activity feed at /admin/events - DCR default OFF (--enable-dcr to enable) - Rate limiting on /token (50/15min) - localOnly operations excluded from HTTP CLI: gbrain serve --http [--port 3131] [--token-ttl 3600] [--enable-dcr] Dependencies: express@5.2.1, express-rate-limit@7.5.1, cors@2.8.6 SDK pinned to exact 1.29.0 (was ^1.0.0) 27 new tests covering OAuth provider, scope enforcement, auth code flow, refresh rotation, token revocation, legacy fallback, and sweep. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: React admin dashboard — 7 screens, dark theme, Krug-designed Admin SPA at /admin with client-side routing (#login, #dashboard, #agents, #log). Built with Vite + React, served from admin/dist/. Screens: - Login: one field, one button, zero happy talk - Dashboard: metrics bar, SSE live activity feed, token health panel - Agents: table with scopes/badges, + Register Agent button - Register: modal form (name, scopes), 3 mindless choices - Credentials: full-screen modal, copy buttons, download JSON, warning - Request Log: paginated table (50/page), time-relative timestamps - Agent Detail: slide-out drawer, config export tabs (Perplexity/Claude/JSON) Design tokens: #0a0a0f bg, Inter + JetBrains Mono, 4-32px spacing. Build: bun run build:admin (Vite, 65KB gzipped). Admin API: /admin/api/register-client endpoint for dashboard registration. SPA serving: Express static + index.html fallback for client-side routing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: add admin SPA lockfile Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v1.0.0.0) Milestone release: multi-agent GBrain with OAuth 2.1, HTTP server, and React admin dashboard. See CHANGELOG.md for details. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: update project documentation for v1.0.0.0 Sync README, CLAUDE.md, and docs/mcp/ with the OAuth 2.1 + HTTP server + admin dashboard surface that shipped in v1.0.0.0. - README.md: new "Remote MCP with OAuth 2.1" section covering gbrain serve --http, admin dashboard, scoped operations, legacy bearer fallback; add serve --http + auth notes to the commands reference. - CLAUDE.md: add src/commands/serve-http.ts, src/core/oauth-provider.ts, admin/ directory as key files; document scope + localOnly additions to Operation contract; add oauth.test.ts (27 cases) to the test list; add v1.0.0 key-commands section clarifying that OAuth client registration is via the /admin dashboard or SDK (no CLI subcommand). - docs/mcp/DEPLOY.md: promote --http as the recommended remote path, add OAuth 2.1 Setup section, list ChatGPT in supported clients, remove the "not yet implemented" footer. - docs/mcp/CHATGPT.md (new): unblocks the P0 TODO. Full ChatGPT connector setup via OAuth 2.1 + PKCE. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: wire gbrain auth subcommand with OAuth register-client Previously auth.ts was a standalone script invoked via `bun run src/commands/auth.ts`. CHANGELOG and README documented `gbrain auth ...` commands that didn't actually work. - Export `runAuth(args)` from auth.ts (keeps standalone entry intact via `import.meta.url === file://${process.argv[1]}` check) - Add `auth` to CLI_ONLY + dispatch in handleCliOnly - New subcommand `gbrain auth register-client <name> [--grant-types] [--scopes]` wraps GBrainOAuthProvider.registerClientManual - Lazy DB check: only subcommands that need DATABASE_URL error out Now the documented CLI flow works end to end: gbrain auth register-client perplexity --grant-types client_credentials --scopes "read write" gbrain serve --http --port 3131 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: reflect wired gbrain auth register-client CLI After /ship, the doc subagent wrote docs assuming `gbrain auth register-client` did not exist (it said so explicitly in CLAUDE.md:184). A follow-up commit (c4a86ce) wired it into src/cli.ts + src/commands/auth.ts. These docs were now contradicting reality. - CLAUDE.md: removed "There is no gbrain auth register-client CLI subcommand" claim, documented the three registration paths (CLI / dashboard / SDK). - README.md: replaced `bun run src/commands/auth.ts` hint with `gbrain auth create|list|revoke|test` and `gbrain auth register-client`. - docs/mcp/DEPLOY.md: added CLI registration example above the programmatic example. - TODOS.md: moved "ChatGPT MCP support (OAuth 2.1)" P0 item to Completed with v1.0.0.0 completion note. Closes the P0 that had been blocking the "every AI client" promise since v0.6. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix: enable RLS on OAuth tables + loosen v24-exact test assertion CI Tier 1 (Mechanical) was failing on 4 E2E tests after the v0.18.1 RLS hardening landed on master (PR #343). Our v25 oauth_infrastructure migration adds 3 new public tables (oauth_clients, oauth_tokens, oauth_codes) but didn't enable RLS, so gbrain doctor's new check flagged them and the "RLS on every public table" assertion failed. Fixes: - src/schema.sql: ALTER TABLE ... ENABLE ROW LEVEL SECURITY for the 3 OAuth tables inside the existing BYPASSRLS-gated DO block (fresh installs). - src/core/migrate.ts v25: append a BYPASSRLS-gated DO block after the OAuth CREATE TABLE statements (existing installs on upgrade). Mirrors the v24 rls_backfill gating pattern — RAISE WARNING if the current role lacks BYPASSRLS, so migrations don't silently lock the operator out. - src/core/schema-embedded.ts: regenerated via `bun run build:schema`. - test/e2e/mechanical.test.ts: one unrelated v24 test asserted the post- migration version equals exactly '24'. That breaks when any later migration exists (like our v25). Relaxed to `>= 24` since the test's intent is "v24 didn't abort the chain", not "v24 is the final version". Verified locally: 78/78 E2E tests pass against real Postgres 16 + pgvector. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: regenerate llms-full.txt for v1.0.0 docs CI test/build-llms.test.ts > committed llms.txt + llms-full.txt match current generator output failed. The committed llms-full.txt was built before the v1.0.0 doc updates landed (OAuth 2.1 README section, new docs/mcp/CHATGPT.md, CLAUDE.md serve-http references, etc.), so the regen-drift guard flagged it. Ran `bun run build:llms`. llms.txt is unchanged (skinny index still matches); llms-full.txt picks up 166 net-new lines of bundled content. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * connected-gbrains PR 0 — minimal runtime (mounts, registry, aggregated RESOLVER) (#372) * feat(mounts): connected-gbrains PR 0 foundation — registry + resolver + CLI Lays the foundation for connected gbrains (v0.19.0) per the approved plan. This is PR 0 — minimal runtime for direct-transport, path-mounted brains. What this slice ships: - src/core/brain-registry.ts — keyed BrainRegistry with lazy engine init, schema-validated mounts.json loader, DuplicateMountPathError (load-bearing identity check per Codex finding #9 correction), UnknownBrainError with actionable available-id list. Pure: no AsyncLocalStorage, no singleton mutation. ~280 LOC. - src/core/brain-resolver.ts — 6-tier brain-id resolution mirroring v0.18.0's source-resolver.ts so agents learn ONE mental model: 1. --brain <id> 2. GBRAIN_BRAIN_ID env 3. .gbrain-mount dotfile 4. longest-path match over registered mounts 5. (reserved v2 default) 6. 'host' fallback Orthogonal to --source: --brain picks which DB, --source picks the repo within that DB. Corruption-resistant: mounts.json load failures fall through to 'host' instead of breaking every CLI invocation. - src/commands/mounts.ts — `gbrain mounts add|list|remove` (direct transport only). Validates on add (path exists on disk, id regex, no dupes). WARNS but does not block on same db_url/db_path across ids (teams may legitimately alias a remote brain). Password redaction in list output. Atomic write via temp+rename. 0600 perms. PR 1 adds pin/sync/enable; PR 2 adds --mcp-url + OAuth. - src/cli.ts — wires `gbrain mounts` into handleCliOnly (no DB required for the config-only subcommands). - test/brain-registry.test.ts (28 cases): schema validation across every malformed-input branch, ALS-free resolution, duplicate id + path detection, disabled-mount exclusion, UnknownBrainError context. - test/brain-resolver.test.ts (22 cases): priority order (explicit > env > dotfile > path-prefix > fallback), dotfile walk-up, malformed dotfile recovery, longest-prefix match, sibling-path false-positive guard, loader-failure defense. - test/mounts-cli.test.ts (17 cases): parseAddArgs surface, redactUrl, atomic write, add/list/remove roundtrip via temp HOME. 67 new tests, all green. Typecheck clean. Depends on mcp-key-mgmt (base branch) for the OAuth/scope annotations that PR 2 will leverage. Next in this branch: PR 0 still needs (a) the deep host-brain-bias audit (postgres-engine internal singleton fallback + a few operations.ts callers), (b) OperationContext threading to make ctx.brainId populated at dispatch, (c) composeResolvers + composeManifests, (d) aggregated ~/.gbrain/mounts-cache/ for host-agent runtime ownership. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(mounts): brains-and-sources mental model + agent routing convention Two orthogonal axes organize GBrain knowledge. Users AND agents need to understand both, or queries misroute silently. --brain → WHICH DATABASE (host + mounts) --source → WHICH REPO IN DB (v0.18.0 sources: wiki, gstack, ...) Both axes use the same 6-tier resolution (explicit > env > dotfile > path-prefix > default > fallback), so learning one teaches both. Ships: - docs/architecture/brains-and-sources.md — canonical mental model doc. Covers four topologies with ASCII diagrams: 1. Single-person developer (one brain, one source) 2. Personal brain with multiple repos (one brain, N sources) 3. Personal + one team brain mount (2 brains) 4. Senior user with multiple team memberships (N mounted team brains alongside personal) — the CEO-class topology Explicit "when to move each axis" decision table. Generic example names throughout per the project's privacy rule. - skills/conventions/brain-routing.md — agent-facing decision table. Rules for when to switch brain (team-owned question, explicit name, data owner changes) vs switch source (working in a repo, topic scoped to one repo). Cross-brain federation is latent-space only in v0.19 — the agent fans out; the DB never does. Anti-patterns listed: silent brain jumps, writing to host when data is team-owned, missing brain prefix in citations, ignoring .gbrain-mount dotfiles. - CLAUDE.md — adds "Two organizational axes (read this first)" section at the top pointing at both new docs. - AGENTS.md — adds brains-and-sources.md + brain-routing.md to the "read this order" (positions 3 and 4, before RESOLVER.md). - skills/RESOLVER.md — adds brain-routing.md to the Conventions section so it appears alongside quality.md, brain-first.md, subagent-routing.md. No code changes. Pre-existing check-resolvable warnings unchanged (2 warnings on base unrelated to this work). 67 PR-0 tests still green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(mounts): thread brainId through OperationContext + subagent chain PR 0 plumbing for connected gbrains. Adds an optional brainId field that identifies which database an operation targets and ensures subagents inherit the parent job's brain instead of process-wide defaults. No dispatch-path changes in this commit — that is PR 1 (registry wiring at MCP + CLI entry points). The fields exist so callers can set them now and downstream code respects them. Changes: - src/core/operations.ts: OperationContext grows `brainId?: string`. Optional for back-compat. 'host' is the implicit default when absent. Orthogonal to v0.18.0's source_id (source = which repo within the brain, brain = which database). See docs/architecture/brains-and-sources.md. - src/core/minions/types.ts: SubagentHandlerData gains `brain_id?: string`. Parent jobs set this when submitting a child subagent to lock the child into a specific brain. Omitted = host (unchanged behavior). - src/core/minions/handlers/subagent.ts: buildBrainTools call site reads data.brain_id and passes it through. Child subagents spawned from this handler will see the same brainId unless they override in their own data. - src/core/minions/tools/brain-allowlist.ts: BuildBrainToolsOpts + OpContextDeps grow brainId; buildOpContext stamps it on every OperationContext the subagent builds for tool calls. Addresses Codex finding #6 (brain-allowlist hardwired parent config without brain awareness, so switching brain only in subagent.ts was not enough). Tests: 166 affected tests green (subagent suite + minions + brain registry + resolver). Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(mounts): composeResolvers + composeManifests + aggregated cache The runtime ownership seam for connected gbrains (Codex finding #3 from plan review): check-resolvable.ts VALIDATES RESOLVER.md; it does not DISPATCH skills. Host agents (Wintermute/OpenClaw/Claude Code) read skills/RESOLVER.md directly to route user requests. Without an aggregated resolver, mounted team brains cannot contribute skills to the host agent's routing table. This commit adds the aggregation: - src/core/mounts-cache.ts (NEW): pure composeResolvers + composeManifests functions plus filesystem writers for ~/.gbrain/mounts-cache/. The aggregated files carry every host skill plus every mount skill, namespace-prefixed (e.g. `yc-media::ingest`). Host skills always beat a same-named mount skill (locked decision 1); bare-name collisions between two mounts surface as structured ambiguity info so doctor can warn (PR 1). Also addresses Codex finding #8: manifests compose alongside the resolver, else doctor conformance breaks on remote skills. - src/commands/mounts.ts: refreshMountsCache() called on `mounts add` and `mounts remove` (the latter clearing the cache entirely when the last mount goes away). Uses findRepoRoot() to locate the host skills dir; skips with a stderr note when run outside a gbrain repo so the user isn't confused by a "cache not refreshed" error in the wrong cwd. - test/mounts-cache.test.ts (NEW): 23 unit tests covering empty world, host-only, single mount, two-mount ambiguity, host-shadows-mount, disabled mount excluded, missing RESOLVER.md is a no-op, manifest composition with same-name collision, render shape, atomic rewrite, clear on missing dir. Output format for ~/.gbrain/mounts-cache/RESOLVER.md adds a Brain column so host agents can see which brain each trigger routes to at a glance, plus Shadows and Ambiguous sections when those conditions exist. Tests: 90 PR 0 tests green (brain-registry + resolver + mounts-cache + mounts-cli). Full suite regression pending in task 11. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(mounts): force instance-level pool for mount brains + CI guard Closes the silent-singleton-share bug Codex flagged as finding #1 from the plan review: two direct-transport mounts with different Postgres URLs would both fall through postgres-engine.ts's `get sql()` getter to db.getConnection() and quietly share whichever singleton connected first. Your yc-media writes end up in garrys-list or vice versa. No error at the call site — just wrong data. The fix: - src/core/brain-registry.ts: initMountBrain now passes poolSize when calling engine.connect(). That forces postgres-engine.ts:33-60 down the instance-level path (setting this._sql) instead of the module singleton path (calling db.connect). Hard-coded 5 for PR 0 — per-mount override is PR 1. PGLite ignores poolSize (no pool concept), so this is Postgres-specific. Host brain still uses the singleton path via initHostBrain (unchanged). That is fine for PR 0: the singleton is "the host's one connection" by definition. PR 1 removes the singleton entirely once every CLI command is engine-injectable. - scripts/check-no-legacy-getconnection.sh (NEW): CI grep guard against new db.getConnection() / db.connect() calls landing in src/core/ or src/commands/ (the multi-brain dispatch surface). Has an explicit ALLOWED list grandfathering today's legitimate callers, each marked "PR 1 refactors" so the list shrinks over time. Skips comment lines so the grep doesn't trip on doc references to the old pattern. - package.json: scripts.test chains the new guard after the existing check-jsonb-pattern + check-progress-to-stdout guards. `bun run test` now fails the build on singleton regression. Tests: 295 affected pass (registry, resolver, mounts-cache, mounts-cli, minions, pglite-engine). Typecheck clean. CI guard reports "ok: no new singleton callers" on current tree. Left for PR 1: remove the singleton fallback in postgres-engine.ts's `get sql()` entirely; refactor src/commands/doctor.ts, files.ts, repair-jsonb.ts, serve-http.ts, init.ts, and the 3 localOnly ops in operations.ts (file_list, file_upload, file_url) to accept ctx.engine explicitly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(mounts): codex review findings — namespace survives shadow + atomic tmp names + honest PR 0 docstrings Codex outside-voice review on PR #372 found 5 issues. Real bugs fixed, overclaims rewritten. Details: P2 (real bug): composeResolvers and composeManifests were silently dropping mount entries when a host skill shared the short name, which made the namespace-qualified form `<mount>::<skill>` unreachable once host defined the same short name. That defeated the entire namespace-disambiguation model — if host had `ingest`, no mount could ship an `ingest` skill even with explicit `yc-media::ingest`. Fix: always keep namespace-qualified mount entries in the composed output. Shadow tracking moves to metadata (`shadows[]`) that doctor can warn on, but never drops routing. Before: host ingest + yc-media ingest → only 1 entry (host), yc-media::ingest unreachable After: host ingest + yc-media ingest → 2 entries: bare `ingest` = host, `yc-media::ingest` = mount Verified live: gbrain mounts add of a mount with `ingest` now shows `team-demo::ingest` alongside host `ingest` in the aggregated manifest. P1 (real bug): writeMountsFile + writeMountsCache used fixed `.tmp` filenames. Two concurrent `gbrain mounts add` invocations (e.g. from parallel terminals or CI) would clobber each other's temp file and one writer's update would be lost. Fix: tmp filenames include `process.pid + random suffix` so every writer has its own scratch file. The atomic rename is self-contained per-writer. (Full lock + read-modify- write safety deferred to PR 1 under `gbrain mounts sync --lock`.) P1 (honesty): `SubagentHandlerData.brain_id` + `BuildBrainToolsOpts.brainId` docstrings claimed child jobs inherit the parent's brain and brain tools target the resolved brain. True for the `ctx.brainId` field only — `ctx.engine` is still the worker's base engine at dispatch time because `buildOpContext` doesn't yet do the registry lookup, and `gbrain agent run` doesn't yet accept `--brain` to populate the field on submission. Rewrote both docstrings to state the PR 0 behavior explicitly (field plumbed, engine routing is PR 1) so nobody reads the code thinking multi-brain subagents already work. Also cleaned up two `require('fs')` runtime imports left over from the initial PR — swapped for ESM named imports (renameSync). Pre-existing style issue surfaced by the self-review pass. Tests: 90 PR-0 tests pass. Updated two shadow-related test cases to assert the corrected semantics (both entries survive, host wins bare name, namespace form routes to mount). Not fixed in this commit (documented as known PR 0 limitations): - `file_list` / `file_upload` / `file_url` in operations.ts still hit the singleton (localOnly + admin, never reachable from HTTP MCP — safe in practice, refactor in PR 1 alongside command-level cleanups). - writeMountsCache's two-file swap (RESOLVER.md + manifest.json) is not atomic across files; readers can briefly observe mismatched pairs. Acceptable because the cache is recomputable at any time from mounts.json. Generation-directory swap is PR 1 work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tests): bump hook timeouts for 21-migration PGLite init under full-suite load Root cause of 19 pre-existing full-suite flakes (CHANGELOG v0.18.0 noted "17 pre-existing master timeouts"): every PGLite test does beforeAll/beforeEach(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); // runs 21 migrations through v0.18.2 }); In isolation this takes ~5s. Under full-suite contention (128 files, process-shared FS and CPU) it exceeds bun's default 5000ms hook timeout, beforeEach times out, engine stays undefined, then afterEach crashes with `TypeError: undefined is not an object (evaluating 'engine.disconnect')`. That single hook failure reports as the whole test "failing" even though the test body never executed, which is why the failure count sometimes looked inflated compared to the number of genuinely-broken tests. Fix applied across 7 test files: - Raise setup hook timeout to 30_000 (6x the default) — gives migration init enough headroom even under worst-case load without masking real regressions in a post-migration test. - Raise teardown hook timeout to 15_000 — engine.disconnect() is usually fast but can stall when PGLite's WASM runtime is still completing a migration at shutdown. - Add `if (engine) await engine.disconnect()` guard so afterEach doesn't double-fault when beforeEach already failed. This was the source of the opaque "(unnamed)" failures — they were disconnect crashes, not test-body failures. Files: test/dream.test.ts (5 beforeEach + 5 afterEach blocks) test/orphans.test.ts (1 pair) test/brain-allowlist.test.ts (1 pair) test/oauth.test.ts (1 pair) test/extract-db.test.ts (1 pair) test/multi-source-integration.test.ts (1 pair) test/core/cycle.test.ts (1 pair) Results on the merged PR 0 branch: Before: 2175 pass / 20 fail / 3 errors After: 2281 pass / 0 fail / 0 errors (+106 tests running that were previously blocked by the timed-out hooks) No changes to production code. No test assertions changed. Just timeout-bump + null-guard discipline that should have been in these hooks from the start. The real longer-term fix is reusing an engine across tests where possible (brain-allowlist.test.ts already does this via beforeAll+DELETE-pages pattern), but that's per-file structural work — out of scope for this cleanup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate llms-full.txt for brains-and-sources + brain-routing docs The test/build-llms.test.ts test validates that the committed llms.txt and llms-full.txt match the current generator output. PR 0 added docs/architecture/brains-and-sources.md content paths and updated CLAUDE.md + skills/RESOLVER.md in earlier commits, but the generated bundle file wasn't regenerated alongside. This caused one of the 20 fails we chased down today — a straight content mismatch, not a runtime bug. Running `bun run build:llms` picks up the new section content so the bundle matches the sources again. No functional change. Only the compiled doc bundle. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Bump version 1.0.0.0 → 0.22.0 OAuth + admin dashboard is meaningful but doesn't quite warrant the major-version reset to 1.0. Renumber as v0.22.0, slotting cleanly above master's v0.21.0 (Cathedral II). Touched: - VERSION, package.json: 1.0.0.0 → 0.22.0 - CHANGELOG.md: heading + "BEFORE/AFTER v1.0" table + "To take advantage" + "pre-v1.0" all renamed. Narrative voice unchanged otherwise. - TODOS.md: ChatGPT MCP completion stamp updated to v0.22.0 (2026-04-25). - CLAUDE.md, README.md, docs/mcp/{DEPLOY,CHATGPT}.md, src/schema.sql, src/core/schema-embedded.ts: every reader-facing v1.0.0 reference rewritten to v0.22.0 / pre-v0.22 in the same place. - llms-full.txt: regenerated to match. Slug-test occurrences of "v1.0.0" (`test/slug-validation.test.ts`, `test/file-upload-security.test.ts`) and the `HOMEBREW_FOR_PERSONAL_AI` roadmap reference to a future v1.0 vision left intact — those are unrelated to this branch's release version. Typecheck clean. cli + oauth + slug + file-upload tests pass (106 tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.26.0 fix: 4 security findings from /cso pass + version bump Bumped 0.22.0 → 0.26.0 to slot above master's v0.21 chain with headroom for v0.23/0.24/0.25 to ship from master between now and merge. Security fixes (all from CSO finding writeups): #1 cookie-parser middleware — admin dashboard auth was silently broken. Express 5 has no built-in cookie parsing; req.cookies was always undefined, so /admin/login set the cookie but every subsequent admin API call returned 401. Added cookie-parser@^1.4.7 + @types/cookie-parser as direct + dev deps. app.use(cookieParser()) wired before CORS. #2 + #3 TOCTOU races — exchangeAuthorizationCode and exchangeRefreshToken used SELECT-then-DELETE, letting concurrent requests with the same code/refresh both pass the SELECT before either ran DELETE, both issuing token pairs. Switched to atomic DELETE...RETURNING. RFC 6749 §10.5 (codes) + §10.4 (refresh detection) violations closed. Added regression tests that fire 10 concurrent exchanges and assert exactly one wins — both pass. #5 pgArray escape + DCR redirect_uri validation — pgArray() did `arr.join(',')` with no escaping, so an element containing a comma would be parsed by Postgres as TWO array elements. With --enable-dcr on, this could smuggle a second redirect_uri into a registered client and steal auth codes. Now every element is double-quoted with `"` and `\` escaped. Added validateRedirectUri() per RFC 6749 §3.1.2.1: redirect_uris must be https:// or loopback (localhost / 127.0.0.1). Wired into the DCR registerClient path; CLI registration trusts the operator and bypasses. Regression test confirms a comma-in-URI element round-trips as 1 element, not 2. #6 --public-url flag — issuerUrl was hardcoded to http://localhost:{port}. Behind reverse proxies / ngrok / production deploys, the issuer claim in tokens wouldn't match the discovery URL clients hit (RFC 8414 §3.3). New --public-url URL flag on `gbrain serve --http`, propagates through serve.ts → serve-http.ts → ServeHttpOptions.publicUrl → issuerUrl. Startup banner surfaces the configured issuer. Findings #4 (admin requests filter dead code), #7 (admin register-client hardcoded grant_types), #8 (legacy token grandfathering posture) are documentation / minor functional fixes and are deferred per user direction. Tests: oauth.test.ts now 34 cases (was 27). 7 new: - single-use TOCTOU regression (10 concurrent code exchanges) - single-use TOCTOU regression (10 concurrent refresh exchanges) - redirect_uri http://localhost passes - redirect_uri https://example.com passes - redirect_uri http://example.com (non-loopback plaintext) rejected - redirect_uri non-URL rejected - redirect_uri with embedded comma stored as single element Files: - VERSION, package.json: 0.22.0 → 0.26.0 - CHANGELOG.md: heading + table + "To take advantage" + "pre-v0.22" → v0.26; new "Security hardening (post-/cso pass)" subsection at top of itemized changes; CLI flag list updated for --public-url. - src/core/oauth-provider.ts: pgArray escape, validateRedirectUri, registerClient enforces validation, DELETE...RETURNING in exchangeAuthorizationCode + exchangeRefreshToken. - src/commands/serve-http.ts: cookie-parser import + wire-up, publicUrl option, issuerUrl honors it, startup banner shows issuer. - src/commands/serve.ts: parses --public-url and threads through. - src/cli.ts: help text adds --public-url URL flag. - test/oauth.test.ts: +7 regression tests (now 34 total). - llms-full.txt: regenerated. Typecheck clean. 34 oauth + 14 cli tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-05-02 22:01:05 -07:00
3 people	527b87bd1e	v0.23.0 feat: gbrain dream synthesizes conversations into brain pages (v0.23.0) (#462) ... * feat: dream_verdicts schema + engine methods Adds the v25 schema migration creating the dream_verdicts table (file_path, content_hash, worth_processing, reasons, judged_at; PRIMARY KEY (file_path, content_hash); RLS-enabled when running as a BYPASSRLS role). Distinct from raw_data (which is page-scoped) — transcripts being judged for synthesis aren't pages. The (file_path, content_hash) key means edited transcripts re-judge automatically. BrainEngine gains: - DreamVerdict + DreamVerdictInput types - getDreamVerdict(filePath, contentHash) → DreamVerdict | null - putDreamVerdict(filePath, contentHash, verdict) — ON CONFLICT upsert Both engines implement (postgres-engine.ts, pglite-engine.ts). This commit alone is functionally inert — nothing reads/writes the table yet. The synthesize phase (later commit) is the consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: trusted-workspace allow-list for subagent put_page Adds OperationContext.allowedSlugPrefixes — when set, put_page enforces slug membership in the allow-list instead of the legacy wiki/agents/<id>/... namespace. The trust signal is the SUBMITTER (PROTECTED_JOB_NAMES gates subagent submission so MCP can't reach this field), not the runtime ctx.remote flag — every subagent tool call has remote=true for auto-link safety, so basing trust on remote is incoherent. matchesSlugAllowList(slug, prefixes) helper supports glob suffix '/*' (recursive — wiki/originals/* matches ideas/foo/bar) and exact match for unsuffixed entries. put_page check shape: if (viaSubagent && allowedSlugPrefixes set) → allow-list check else if (viaSubagent) → existing namespace check (regression guard) else → no check (regular CLI) Auto-link is re-enabled for the trusted-workspace path so the cycle's extract phase doesn't have to recompute every edge after synthesize writes. Untrusted remote writes still skip auto-link as before. SubagentHandlerData.allowed_slug_prefixes is the wire field; the synthesize/patterns phases (later commit) populate it from a single source of truth in skills/_brain-filing-rules.json's dream_synthesize_paths.globs array. The model's tool schema description mirrors the allow-list so it writes correct slugs on the first try. IRON RULE security tests: - test/operations-allow-list.test.ts: allow-list ALLOW/REJECT, glob semantics, regression guard for the v0.15 namespace fallback when allow-list is unset, FAIL-CLOSED when subagentId is missing. - test/e2e/dream-allow-list-pglite.test.ts: end-to-end on PGLite, poisoned-transcript style write outside allow-list → REJECTED. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: cycle scaffolding — 8-phase order + transcript discovery Extends ALL_PHASES from 6 → 8: synthesize between sync and extract, patterns between extract and embed. Codex finding #7: patterns MUST run after extract because subagent put_page sets ctx.remote=true and skips auto-link/timeline by default — extract is the canonical edge materialization step. Without that ordering, patterns reads stale graph state. Final order: lint → backlinks → sync → synthesize → extract → patterns → embed → orphans CycleOpts gains: - yieldDuringPhase callback — generic in-phase keepalive for long waits (synthesize fan-out, patterns roll-up). Renews cycle-lock TTL + worker job lock. Mirrors yieldBetweenPhases shape. - synthInputFile / synthDate / synthFrom / synthTo — forwarded to runPhaseSynthesize for the CLI's --input/--date/--from/--to flags. CycleReport.totals additively grows (no schema_version bump): transcripts_processed, synth_pages_written, patterns_written. src/core/cycle/transcript-discovery.ts is a pure filesystem walk: - .txt files only, sorted by path for determinism - date-prefixed basename filter (--date / --from / --to) - min_chars filter (default 2000) - exclude_patterns auto-wraps bare words as \b<word>\b regex (Q-3), power users may pass full regex with anchors - compileExcludePatterns is exported for unit tests Phase implementations land in the next commit; this one only adds the dispatcher slots so commit-by-commit bisect doesn't crash on import-not-found. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: synthesize + patterns phases — gbrain dream actually dreams Synthesize phase (src/core/cycle/synthesize.ts) reads conversation transcripts from dream.synthesize.session_corpus_dir and writes brain-native pages: reflections to wiki/personal/reflections/..., originals to wiki/originals/ideas/..., timeline entries on existing people pages. Pipeline: 1. discoverTranscripts (filesystem walk + filters) 2. cooldown check via dream.synthesize.last_completion_ts config (default 12h; bypassed by --input/--date/--from/--to) 3. cheap Haiku verdict per transcript, cached in dream_verdicts table keyed by (file_path, content_hash) — backfill re-runs skip already-judged transcripts at zero cost 4. fan-out: one Sonnet subagent per worth-processing transcript dispatched with allowed_slug_prefixes (read from skills/_brain-filing-rules.json's dream_synthesize_paths.globs) and idempotency_key dream:synth:<file_path>:<content_hash> 5. wait via waitForCompletion; yieldDuringPhase ticks every child terminal so the cycle-lock TTL refreshes on long backfills 6. collect slugs from subagent_tool_executions for each child (codex finding #2: NOT pages.updated_at, which would pick up unrelated writes) 7. orchestrator dual-write — query each new page from DB, reverse-render via serializeMarkdown, write file to brain_dir. Subagent never gets fs-write access. 8. deterministic summary index page at dream-cycle-summaries/<date> (codex finding #4: slug shape is regex-compatible — no underscores, no .md extension) 9. write completion timestamp ONLY on successful runs Patterns phase (src/core/cycle/patterns.ts) runs after extract so the graph state is fresh. Single Sonnet subagent gathers reflections within dream.patterns.lookback_days (default 30); names a pattern only when ≥dream.patterns.min_evidence (default 3) reflections support it. Same allow-list path as synthesize. CLI flags on `gbrain dream` (src/commands/dream.ts): --input <file> ad-hoc transcript synthesis (implies --phase synthesize; bypasses cooldown) --date YYYY-MM-DD restrict synthesize to one date --from <d> --to <d> backfill range --dry-run runs Haiku verdict (cached), skips Sonnet synthesis. NOT zero LLM calls (codex #8). Conflict detection: --input + --date/--from/--to exits 2. ISO 8601 date format validated; range start > end exits 2. Auto-commit / push deferred to v1.1 (codex finding #5). v1 writes files to brain_dir; user or autopilot handles git. Tests: - test/cycle-patterns.test.ts: structural assertions on the patterns phase (queue + waitForCompletion wired, allow-list threading, subagent_tool_executions provenance, no raw_data dependency). - test/dream-cli-flags.test.ts: argv parsing, conflict detection, ISO date validation, --input implies --phase synthesize, dry-run semantics doc string. - test/e2e/dream-synthesize-pglite.test.ts: 8 cases on PGLite in-memory exercising not_configured, empty corpus, no API key skip path, dry-run, cooldown active vs --input bypass, and the dream_verdicts cache hit path. Per-test rig isolation (each test creates and tears down its own engine) avoids cross-test PGLite WASM contention. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: dream cycle v0.27.0 — skills, CLAUDE.md, migration, changelog - skills/maintain/SKILL.md: synthesize + patterns phases documented with quality bar (Iron Law for synthesis), trust boundary, idempotency, cooldown semantics, CLI invocation patterns. New triggers added so "process today's session" / "synthesize my conversations" route here. - skills/RESOLVER.md: dream cycle triggers route to maintain. - skills/_brain-filing-rules.md: directory table for the five output types (reflections, originals, patterns, people enrichment, cycle summary) with slug shape per row; Iron Law repeated. - skills/migrations/v0.27.0.md: agent-readable migration narrative. Schema migration v25 runs automatically on `gbrain apply-migrations`; synthesize ships disabled by default — opt-in via dream.synthesize.session_corpus_dir + dream.synthesize.enabled. - CLAUDE.md: file inventory updated with new files (cycle/synthesize.ts, cycle/patterns.ts, cycle/transcript-discovery.ts), the 8-phase ordering, the trusted-workspace allow-list trust model, and the v25 schema migration line in the migrate.ts entry. - VERSION: 0.20.4 → 0.27.0 - CHANGELOG.md: v0.27.0 release-summary section per CLAUDE.md voice rules (numbers that matter table, what-this-means closer, "to take advantage of" block), followed by the itemized changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: add patterns E2E + 8-phase cycle E2E + bump synth-cooldown timeouts Two new E2E test files on PGLite (no DATABASE_URL or API key required): - test/e2e/dream-patterns-pglite.test.ts (6 cases) — exercises runPhasePatterns skip paths against a real engine: disabled, default-enabled-but-insufficient-evidence, no-API-key, dry-run. Sibling of dream-synthesize-pglite.test.ts; same per-test rig pattern for engine isolation. - test/e2e/dream-cycle-eight-phase-pglite.test.ts (5 cases) — end-to-end runCycle with the v0.27 8-phase order. Asserts: ALL_PHASES is the documented 8 phases in the right sequence, the dry-run report's phases array preserves that order, CycleReport.totals carries the new transcripts_processed / synth_pages_written / patterns_written fields, --phase synthesize and --phase patterns each run only that phase, and synthInputFile is plumbed correctly through runCycle to runPhaseSynthesize. Bump per-test timeout to 30s on the two synthesize-cooldown E2E tests that create two PGLite engines back-to-back. Default Bun 5s budget is tight under sustained suite pressure (PGLite WASM init costs ~1-2s per engine on macOS); each test passes alone but flakes in the full E2E suite. The third arg `30_000` is Bun's standard test-timeout knob. Full E2E suite (test/e2e/) now: 86 pass / 0 fail / 258 skip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: ship-prep — typecheck fixes, llms.txt regen, 8-phase test update - src/core/cycle/synthesize.ts + patterns.ts: PageType 'default' → 'note' (TS strict typecheck rejected 'default'; 'note' is a valid PageType for orchestrator-written summary index pages and reverse-render fallback). - src/core/pglite-engine.ts: re-import DreamVerdict + DreamVerdictInput types after the master merge dropped them from the import line. - test/e2e/dream-allow-list-pglite.test.ts: ToolCtx now requires remote: true literal; thread it through every put_page tool call. - test/e2e/dream-patterns-pglite.test.ts: PageType 'default' → 'note' in the seedReflections helper. - test/core/cycle.test.ts: bump expected hook-call count + phase count 6 → 8 to match v0.27 ALL_PHASES extension. - llms-full.txt: regenerate against the updated CHANGELOG + CLAUDE.md so the committed snapshot matches what the generator now produces. Full bun test suite: 2793 pass / 0 fail / 258 skip (3051 tests, 177 files). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update README + INSTALL_FOR_AGENTS for v0.27.0 dream cycle README: maintain skill row mentions synthesize/patterns; gbrain dream command-reference block describes the 8-phase pipeline and the new --input/--date/--from/--to flags. INSTALL_FOR_AGENTS: dream cycle bullet calls out v0.27 conversation synthesis + cross-session pattern detection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: renumber v0.27.0 → v0.23.0 Master is at v0.22.5; v0.23.0 is the next natural slot for the dream-cycle synthesize + patterns release. Bulk rename across VERSION, package.json, CHANGELOG, migration file, source comments, skills, and llms.txt bundles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): bump cycle.test.ts phase count 6 → 8 The dry-run full-cycle test asserted 6 phases. v0.23 added synthesize and patterns, bringing the total to 8. The unit-side equivalent (test/core/cycle.test.ts) was already updated; this catches the E2E sibling that surfaced after the latest master merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-30 01:23:29 -07:00
4 people	e734937254	fix: pass sourceId in cycle sync phase to prevent full reimport (#475) ... * fix: pass sourceId in cycle sync phase to prevent full reimport cycle.ts calls performSync without sourceId, so it always reads the global config.sync.last_commit key instead of the per-source sources.last_commit. When the global anchor gets garbage-collected (after a force push or rebase), sync falls back to a full reimport of all files — on a large brain this takes 30+ minutes and blocks the autopilot cycle. The fix resolves the source id from the brain directory by querying the sources table. When a matching source exists, sync reads the per-source anchor which is updated on every successful sync and stays in sync with the repo history. Falls back gracefully to the global config path for pre-v0.18 brains without a sources table. * v0.22.5: tests + version bump for sync-cycle-source-id fix Adds 6 regression tests in test/core/cycle.test.ts pinning the new resolveSourceForDir() helper added to src/core/cycle.ts in this PR: 1. Seeded sources row → performSync receives matching sourceId 2. No matching row → sourceId=undefined (falls through to global key) 3. Different brainDir than registered source → undefined (no cross-match) 4. sources table missing (very old brain) → catch returns undefined, sync still runs. Uses a fresh PGLiteEngine because initSchema() only re-runs PENDING migrations; DROP TABLE on the shared engine would leave it permanently degraded for every later test in the file. (Codex review caught this landmine.) 5. Multiple rows with same local_path → resolver returns one matching id (non-deterministic; SQL has no ORDER BY). Documents the contract for the v0.23 UNIQUE-constraint follow-up. 6. Empty-string id row → resolver propagates "" (defensive case Codex flagged: schema PK prevents NULL but '' can be inserted). Extends the performSync mock at line 51-65 to also capture sourceId. Bumps: - VERSION: 0.22.4 → 0.22.5 - package.json: 0.22.4 → 0.22.5 - CHANGELOG.md: new [0.22.5] entry following v0.22.4 voice (release summary + numbers table + behavior matrix + To-take-advantage block + itemized changes + for-contributors) - CLAUDE.md: annotates src/core/cycle.ts entry with v0.22.5 (#475) note - llms-full.txt: regenerated via bun run build:llms Test results: - Unit: 28 pass / 0 fail in test/core/cycle.test.ts (22 prior + 6 new) - Full unit suite: pass (exit 0) - E2E: 236 pass / 0 fail across 26 files Plan + codex outside-voice review at: ~/.claude/plans/whimsical-bubbling-goose.md Follow-up TODOs filed for v0.23: - Normalize brainDir + sources.local_path before SQL compare - Add UNIQUE index on sources.local_path - Narrow resolveSourceForDir's catch to PG 42P01 (undefined_table) - Add doctor check for config.sync.last_commit / sources divergence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: typecheck error in cycle.test.ts test 5 (sourceId regression) CI typecheck failed because `toContain()` on `string[]` rejects the `string | undefined` produced by `syncCalls.at(-1)?.sourceId`'s optional chain. Tests 1, 4, and 6 use `toBe()` which accepts `string | undefined` through its overload, but `toContain()` is stricter. Fix: pull the value into a typed variable, assert it's defined, then check membership. Makes the contract explicit ("resolver returned a defined sourceId, and it was one of the matching ids") instead of relying on a silent undefined → no-match-in-array assertion. Locally: - bun run typecheck: clean - bun test test/core/cycle.test.ts: 28 pass / 0 fail (75 expect calls) - All CI gate scripts: OK (jsonb, progress-to-stdout, wasm-embedded) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: add --timeout=60000 to E2E runner to prevent setupDB flake PR #475's Tier 1 (Mechanical) CI job hit a 5000.09ms beforeAll hook timeout in `E2E: Tags > (unnamed)`. Cause: scripts/run-e2e.sh invokes `bun test "$f"` without a --timeout flag, falling back to bun's 5s default. setupDB() does TRUNCATE CASCADE on ~30 tables, and on a CI runner under load that can exceed 5s. Match what the unit suite uses (--timeout=60000 in package.json's "test" script). Same 1m ceiling, no behavior change for healthy runs; just removes the artificial 5s floor on hooks. Verified locally: bun test --timeout=60000 test/e2e/mechanical.test.ts runs 78 pass / 0 fail in 27.99s against a fresh pgvector pg16 docker container. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: root <root@localhost> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-27 16:56:07 -07:00
6 people	e2961c04bd	v0.22.1 autopilot fix wave — 5 prod hotfixes (#417, #403, #406, #363, #409) (#447) ... * fix: propagate AbortSignal to runCycle + worker force-eviction safety net Root cause: autopilot-cycle handler called runCycle() without passing the job's AbortSignal. When the per-job timeout fired abort(), runCycle never checked it and kept grinding through extract (54,605 pages). The executeJob promise never resolved, inFlight never decremented, and the worker thought it was at capacity forever — 98 jobs piled up waiting with 0 active while a live worker sat idle. Three-layer fix: 1. CycleOpts.signal: new optional AbortSignal field. runCycle checks it between every phase via checkAborted(). A timed-out cycle now bails after the current phase completes instead of running all 6 phases. 2. autopilot-cycle handler: passes job.signal to runCycle so the abort actually propagates. 3. Worker safety net: 30s after the abort fires, if the handler still hasn't resolved, force-evict from inFlight and mark as dead in DB. This is the last-resort escape hatch for any handler that ignores AbortSignal — the worker resumes claiming new jobs instead of wedging forever. Incident: 2026-04-24, 98 waiting / 0 active / worker alive but idle. 143 existing minions tests pass unchanged. * test: abort signal propagation + worker recovery regression tests 16 new tests across 3 files covering the 2026-04-24 worker wedge: test/minions.test.ts (6 new, 149 total): - handler receiving abort signal exits cleanly - handler ignoring abort still gets signal delivered - worker claims new jobs after timeout (no wedge) ← key regression - checkAborted pattern: undefined/non-aborted/aborted signals test/cycle-abort.test.ts (7 new): - CycleOpts.signal type contract - runCycle accepts signal without error - runCycle bails on pre-aborted signal - runCycle bails mid-flight when signal fires between phases - Source-level guard: jobs.ts passes job.signal to runCycle - Source-level guard: worker.ts has force-eviction safety net - Source-level guard: cycle.ts has checkAborted between all 6 phases test/e2e/worker-abort-recovery.test.ts (3 new): - worker recovers from timed-out handler and processes next job - concurrency=2 processes parallel jobs during timeout - multiple sequential timeouts don't permanently wedge worker All 159 tests pass. * perf: incremental extract — only process slugs that sync touched The autopilot-cycle runs every 5 min. Its extract phase was doing a full filesystem walk of ALL markdown files (54K+) — twice (links + timeline). On a brain this size, extract alone exceeded the 600s job timeout, producing zero useful writes. Fix: sync already returns pagesAffected (the slugs it added/modified). Pipe that list through to extract. When provided, extract reads ONLY those files instead of walking the entire brain directory. - Add ExtractOpts.slugs for targeted extraction - Add extractForSlugs() — single-pass links + timeline for specific slugs - cycle.ts: capture sync's pagesAffected, pass to runPhaseExtract - If sync didn't run or failed, extract falls back to full walk (safe) - If pagesAffected is empty (nothing changed), extract returns instantly Expected improvement: 54K file reads → ~10-50 per cycle. The full walk is still available via CLI `gbrain extract` and on first-run. * fix: connection resilience for minion supervisor + worker Three fixes for the minion supervisor dying silently when PgBouncer rotates: 1. PostgresEngine: executeRaw retries once on connection-class errors (ECONNREFUSED, password auth failed, connection terminated, etc.) by tearing down the poisoned pool and creating a fresh one via reconnect(). Prevents cascading failures when Supabase bounces. 2. Supervisor: tracks consecutive health check failures. After 3 in a row, emits health_warn with reason=db_connection_degraded and attempts engine.reconnect() if available. Resets counter on success. 3. Supervisor: worker_exited events now include likely_cause field: SIGKILL → oom_or_external_kill, SIGTERM → graceful_shutdown, code=1 → runtime_error. Makes it trivial to distinguish OOM kills from connection deaths in logs. Tests: 23 new tests covering connection error detection, reconnect guard against concurrent reconnects, retry-once-not-infinite-loop, health failure tracking, and exit classification. * fix(db): set session timeouts on every connection to kill orphan backends Prevents the failure mode from #361: a single autopilot UPDATE on minion_jobs can leave a pooler backend in state='active'/ClientRead for 24h+, holding a RowExclusiveLock that blocks every subsequent ALTER TABLE minion_jobs. The stuck backend never times out on its own because Supabase Micro has no default idle_in_transaction_session_timeout and autovacuum can't reap sessions that hold active locks. Fix: deliver statement_timeout + idle_in_transaction_session_timeout as startup parameters via postgres.js's `connection` option, applied automatically on every new backend connection. Works correctly on both session-mode and transaction-mode PgBouncer poolers (startup params persist for the backend's lifetime, unlike SET commands which transaction-mode PgBouncer strips between transactions). Defaults chosen conservatively so they don't interfere with bulk work like multi-minute embed passes or CREATE INDEX on large pages tables: - statement_timeout: '5min' - idle_in_transaction_session_timeout: '2min' Each overridable per-GUC via env var (GBRAIN_STATEMENT_TIMEOUT, GBRAIN_IDLE_TX_TIMEOUT). Set any to '0' or 'off' to disable. client_connection_check_interval is the specific GUC that would kill the observed state='active'/ClientRead case, but it's Postgres 14+ and some managed poolers reject unknown startup parameters. Made it opt-in only via GBRAIN_CLIENT_CHECK_INTERVAL for users who know their Postgres supports it. Applied in both the module-level singleton connect (src/core/db.ts) and the per-engine-instance pool used by `gbrain jobs work` (src/core/postgres-engine.ts) via a shared resolveSessionTimeouts() helper. Tests: 5 new cases in migrate.test.ts covering defaults, env overrides, '0'/'off' disable, and multi-GUC disable. 39/39 pass (34 pre-existing + 5 new). Closes #361. Co-Authored-By: orendi84 <orendigergo@gmail.com> * fix(embed): server-side staleness filter for embed --stale (v0.20.5) embed --stale walked listPages + per-page getChunks (incl. vector(1536) embedding column) on every call, then client-side-filtered for chunks where embedding was missing. On a 1.5K-page brain at 100% coverage, ~76 MB pulled per call, all discarded. With autopilot firing every 5-10 min plus a 2h cron, this hit Supabase's 5 GB free-tier ceiling at 102 GB used (2058% over) twice in one week. Two new BrainEngine methods replace the page walk with a SQL-side filter: - countStaleChunks(): single SELECT count(*) WHERE embedding IS NULL. Pre-flight short-circuit; ~50 bytes wire when 0 stale. - listStaleChunks(): slug + chunk_index + chunk_text + chunk_source + model + token_count for stale rows only. Excludes the (NULL) embedding column. Bounded by LIMIT 100000 mirroring listPages. embedAll forks: staleOnly=true takes the new SQL-side path (embedAllStale); staleOnly=false (--all) keeps existing behavior verbatim. embedAllStale preserves non-stale chunks on partially-stale pages: it re-fetches existing chunks per stale slug and merges (embedding=undefined for non-stale → COALESCE preserves existing). Without the merge, the upsertChunks != ALL filter would delete non-stale chunks. Re-fetch cost is bounded by stale slug count; the autopilot common case (0 stale) never reaches this path. Predicate uses `embedding IS NULL`, not `embedded_at IS NULL`. The bulk- import path could leave embedded_at populated while embedding was NULL (see upsertChunks consistency fix below), so `embedding IS NULL` is the truth source for "this chunk needs an embedding". Also fixes the upsertChunks consistency bug in both engines: when chunk_text changes and no new embedding is supplied, embedding correctly clears to NULL but embedded_at kept its old timestamp. New behavior resets BOTH columns together, keeping write-time honesty. Wire-cost impact (measured against current behavior on a 1.5K-page brain): - 0 stale chunks (autopilot common case): ~76 MB → ~50 bytes (~1.5M× reduction) - 100 stale across 10 pages: ~76 MB → ~150 KB (~500× reduction) - 8K stale across 1.5K pages (cold start): ~76 MB → ~12 MB (~6× reduction) Tests: 4 new in test/embed.test.ts (zero-stale short-circuit; N-stale- across-M-pages with non-stale preservation; --stale dry-run; --all path byte-identical). Existing --stale tests updated for the new mock surface. Migration impact: none. embedded_at and embedding columns have been on content_chunks since schema inception. Co-Authored-By: atrevino47 <atbuster47@gmail.com> * chore(wave): post-merge tightening — drop executeRaw retry (D3) + gate noExtract (F2) - Drop #406's per-call executeRaw retry wrapper. The regex idempotence boundary is unsound (writable CTEs, side-effecting SELECTs). Recovery now happens at the supervisor level via 3-strikes-then-reconnect. - Update db.ts: setSessionDefaults becomes a back-compat no-op. resolveSessionTimeouts (from #363) is the source of truth, sending GUCs as startup parameters that survive PgBouncer transaction mode. Bumped idle_in_transaction default from 2min to 5min to match v0.21.0 posture. - Gate noExtract in cycle's runPhaseSync on whether extract phase is scheduled. Avoids silently dropping extraction when the user runs `gbrain dream --phase sync` (Codex F2). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(db): rephrase docstring to avoid false-positive in test source-grep The migrate.test.ts structural check counts `SET idle_in_transaction_session_timeout` matches in source. The literal string in this docstring was tripping it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: backfill regression guards for #417, D3, F2 (Step 5) 15 new test cases across 3 files, ~250 LOC, all PGLite/in-memory: test/extract-incremental.test.ts (NEW, 8 cases for #417): - slugs: [] returns immediately (early-return) - slugs: undefined falls through to full-walk - slugs: [a, b] reads only those files - Slug whose file no longer exists is silently skipped - Mode filter (links) skips timeline extraction - dryRun: true does not invoke addLinksBatch / addTimelineEntriesBatch - BATCH_SIZE flush — >100 candidate links exercise mid-iteration flush - Full-slug-set resolution — link to file outside changed set still resolves test/core/cycle.test.ts (4 new cases for #417 + Codex F2): - cycle threads sync.pagesAffected into extract phase as the slugs argument - extract phase falls back to full walk when sync was skipped - F2 guard: full cycle (sync + extract) sets noExtract=true on sync - F2 guard: phases:[sync] only sets noExtract=false (no silent extract drop) test/connection-resilience.test.ts (3 new cases for D3): - PostgresEngine.executeRaw is a single-statement passthrough (no try/catch) - PostgresEngine.reconnect() still exists for supervisor-driven recovery - Supervisor still has the 3-strikes-then-reconnect path Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(wave): v0.21.1 release notes + 3 follow-up TODOs + CLAUDE.md updates CHANGELOG.md: segment-aware entry per CEO-review D1 — 'For everyone' section (#417 incremental extract, #403 cycle abort) leads, 'For Postgres / Supabase users' section (#406, #363, #409) follows. Production proof point as a sidebar, not the lead. TODOS.md: 3 follow-up items per Eng-review D6: 1. Caller-opt-in retry for executeRaw (D3 follow-up) 2. Replace walkMarkdownFiles with engine.getAllSlugs() (F1 follow-up) 3. err.code-based connection-error matching (B1 follow-up) CLAUDE.md: 6 file-reference updates for the wave's behavioral additions (postgres-engine, db, cycle, worker, supervisor, embed, extract). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(release): bump version 0.21.1 → 0.22.1 + document version locations User-explicit version override on /ship: ship as v0.22.1 (MINOR jump from master's 0.21.0) instead of the v0.21.1 PATCH the wave originally targeted. The wave bundles 5 production fixes which is meaningful enough to clear a MINOR version, even though the API surface is additive. Files updated to 0.22.1: - VERSION (single source of truth) - package.json (Bun/npm version) - CHANGELOG.md (release header + "To take advantage of v0.22.1" block) - TODOS.md (3 follow-up TODOs reference the version that filed them) - CLAUDE.md (Key Files annotations cite the release that introduced behavior) Also adds a "Version locations" section to CLAUDE.md documenting all five required files plus the auto-derived (bun.lock, llms-full.txt) and historical (skills/migrations/v*.md, src/commands/migrations/v*.ts, test/migrations-v*.test.ts) categories. Future /ship runs and the auto-update agent now have a canonical list of where versions live. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): unbreak CI typecheck — annotate signal as AbortSignal | undefined CI's `bun run typecheck` step was failing with TS2339 at test/minions.test.ts:2026 — `const signal = undefined` narrows to literal `undefined`, which has no `.aborted` property, so `signal?.aborted` doesn't compile. Fix uses `as AbortSignal | undefined` to preserve the union type. A plain type annotation gets narrowed back via control-flow analysis; the `as` cast doesn't. Runtime behavior is unchanged — the optional-chain still short-circuits as intended. Verified: bunx tsc --noEmit → exit 0; the 3 checkAborted cases still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(doctor): forward-progress override for stale minions partials The minions_migration check reads ~/.gbrain/migrations/completed.jsonl and flags any version that has a `partial` entry without a matching `complete`. Long-lived installs accumulate partial records from historical stopgap runs (notably v0.11.0). Without time decay or forward-progress detection, the FAIL flag fires forever once any partial lands, even on installs that have been running clean at v0.22+ for months. Concrete failure: test/e2e/mechanical.test.ts "gbrain doctor exits 0 on healthy DB" was flaking on dev machines whose ~/.gbrain/ carried v0.11.0 partials from earlier in the day. The fresh test DB had nothing wrong with it; doctor was just reading host filesystem state that bled in via $HOME. Fix: a partial vX.Y.Z is treated as stale (not stuck) if any vA.B.C where A.B.C >= X.Y.Z has a `complete` entry anywhere in the file. The reasoning: if a newer migration successfully landed, the install has clearly moved past the older partial. compareVersions() from src/commands/migrations/index.ts handles the semver compare. Cases preserved: - v0.10 complete + v0.11 partial → still FAILs (older complete doesn't supersede newer partial) - v0.16 partial alone → still FAILs (no override exists) - Fresh install (no completed.jsonl) → no warning - Real partial-then-complete-same-version → no warning Cases now fixed: - v0.16 complete + v0.11 partial → no FAIL (forward progress made; the v0.11 record is stale) Two regression tests in test/doctor-minions-check.test.ts cover both directions of the override (when it fires, when it doesn't). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(docs): regenerate llms-full.txt after CLAUDE.md updates CI's build-llms regen-drift guard caught that llms-full.txt was stale relative to CLAUDE.md after the wave's documentation commits (the "Version locations" section + 6 file-reference annotations for the wave's behavioral additions). CLAUDE.md notes that llms-full.txt is auto-derived — bumped via 'bun run build:llms' when CLAUDE.md's file-references change. This commit catches up. llms.txt is unchanged; the curated index doesn't pull from CLAUDE.md's file-reference body. Only llms-full.txt (the inlined single-fetch bundle) needed regeneration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: root <root@localhost> Co-authored-by: orendi84 <orendigergo@gmail.com> Co-authored-by: atrevino47 <atbuster47@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-26 15:49:48 -07:00
3 people	8b3c24c891	v0.20.0 feat: extract BrainBench to sibling gbrain-evals repo (#195) ... * fix(link-extraction): v0.10.5 drive works_at + advises accuracy on rich prose Extends inferLinkType patterns to cover rich-prose phrasings that miss with v0.10.4 regexes. Targets the residuals called out in TODOS.md: works_at at 58% type accuracy, advises at 41%. WORKS_AT_RE additions: - Rank-prefixed: "senior engineer at", "staff engineer at", "principal/lead" - Discipline-prefixed: "backend/frontend/full-stack/ML/data/security engineer at" - Possessive time: "his/her/their/my time at" - Leadership beyond "leads engineering": "heads up X at", "manages engineering at", "runs product at", "leads the [team] at" - Role nouns: "role at", "position at", "tenure as", "stint as" - Promotion patterns: "promoted to staff/senior/principal at" ADVISES_RE additions: - Advisory capacity: "in an advisory capacity", "advisory engagement/partnership/contract" - "as an advisor": "joined as an advisor", "serves as technical advisor" - Prefixed advisor nouns: "strategic/technical/security/product/industry advisor to|at" - Consulting: "consults for", "consulting role at|with" New EMPLOYEE_ROLE_RE page-level prior: fires when the page describes the subject as an employee (senior/staff/principal engineer, director, VP, CTO/CEO/CFO) at some company. Biases outbound company refs toward works_at when per-edge context is possessive or narrative without an explicit work verb. Scoped to person -> company links only. Precedence: investor > advisor > employee (investors often hold board seats which would otherwise mis-classify as advise/works_at). ADVISOR_ROLE_RE broadened from "full-time/professional/advises multiple" to catch any page that self-identifies the subject as an advisor ("is an advisor", "serves as advisor", possessive "her advisory work/role/engagement"). Tests: 65 pass (16 new v0.10.5 coverage tests + 4 regression guards against v0.10.4 tightenings). Templated benchmark still 88.9% type_accuracy (10/10 on works_at and advises). Rich-prose measurement requires the multi-axis report upgrade (next commit) to validate retroactively. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): type-accuracy runner on rich-prose corpus + wire into all.ts New Category 2 in BrainBench: per-link-type accuracy measured directly on the 240-page rich-prose world-v1 corpus. Distinct from Cat 1's retrieval metrics, this measures whether inferLinkType() correctly classifies extracted edges when the prose varies (the 58% works_at and 41% advises residuals that v0.10.5 regexes targeted). How it works: 1. Loads all pages from eval/data/world-v1/ 2. Derives GOLD expected edges from each page's _facts metadata (founders → founded, investors → invested_in, advisors → advises, employees → works_at, attendees → attended, primary_affiliation + role drives person-page outbound type) 3. Runs extractPageLinks() on each page → INFERRED edges 4. Per (from, to) pair, compares inferred type vs gold type 5. Emits per-link-type table: correct / mistyped / missed / spurious + type accuracy + recall + precision + strict F1 (triple match) 6. Full confusion matrix rows=gold, cols=inferred v0.10.5 validation on 240-page corpus (up from pre-v0.10.5 baselines): - works_at: 58% → 100.0% (+42 pts) — 10/10 correct, 0 mistyped - advises: 41% → 88.2% (+47 pts) — 15/17 correct - attended: — → 100.0% 131/134 recall - founded: 100% → 100.0% 40/40 - invested_in: 89% → 92.0% 69/75 - Overall: 88.5% → 95.7% type accuracy (conditional on edge found) Strict F1 overall: 53.7%. Lower because the _facts-based gold set only captures core relationships; rich prose extracts many peripheral mentions (190 spurious "mentions" edges) that aren't bugs but are correctly-typed prose references without a _facts counterpart. Spurious counts are signal for future type-precision tuning, not failure. Wired into eval/runner/all.ts as Cat 2 so every full benchmark run includes the rich-prose type accuracy table alongside retrieval metrics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): Phase 2 adapter interface + EXT-1 ripgrep+BM25 baseline Phase 2 credibility unlock: BrainBench now compares gbrain to external baselines on the same corpus and queries. Transforms the benchmark from internal ablation ("gbrain-graph beats gbrain-grep") to category comparison ("gbrain-graph beats classic BM25 by 32 pts P@5"). This is the #1 fix from the 4-review arc — addresses Codex's core critique that v1's before/after was self-referential. Added: eval/runner/types.ts — Adapter interface (v1.1 spec) eval/runner/adapters/ripgrep-bm25.ts — EXT-1 classic IR baseline eval/runner/adapters/ripgrep-bm25.test.ts — 11 unit tests, all pass eval/runner/multi-adapter.ts — side-by-side scorer Adapter interface (eng pass 2 spec): - Thin 3-method Strategy: init(rawPages, config), query(q, state), snapshot(state) - BrainState is opaque to runner (never inspected) - Raw pages passed in-memory; gold/ never crosses adapter boundary (structural ingestion-boundary enforcement) - PoisonDisposition enum reserved for future poison-resistance scoring EXT-1 ripgrep+BM25: - Classic Lucene-variant IDF + k1/b tuned at standard 1.5/0.75 - Title tokens double-weighted for entity-page slug-match bias - Stopword filter, alphanumeric tokenization, stable lexicographic tie-break - Pure in-memory inverted index — no external deps, ~100 LOC core First side-by-side results on 240-page rich-prose corpus, 145 relational queries: | Adapter | P@5 | R@5 | Correct top-5 | |---------------|--------|--------|---------------| | gbrain-after | 49.1% | 97.9% | 248/261 | | ripgrep-bm25 | 17.1% | 62.4% | 124/261 | | Delta | +32.0 | +35.5 | +124 | gbrain-after is the hybrid graph+grep config from PR #188. Ripgrep+BM25 is a genuinely strong classic-IR baseline (BM25 is what Lucene/Elasticsearch ship). gbrain's ~+32-point lead on relational queries reflects real work by the knowledge graph layer: typed links + traversePaths surface the correct answers in top-K that BM25 only pulls in via partial-text overlap. Next in Phase 2: EXT-2 vector-only RAG + EXT-3 hybrid-without-graph adapters. Both plug into the same Adapter interface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): Phase 2 EXT-2 vector-only RAG adapter Second external baseline for BrainBench. Pure cosine-similarity ranking using the SAME text-embedding-3-large model gbrain uses internally — apples-to-apples on the embedding layer so any gbrain lead reflects the graph + hybrid fusion, not a better embedder. Files: eval/runner/adapters/vector-only.ts ~130 LOC eval/runner/adapters/vector-only.test.ts 6 unit tests (cosine math) Design: - One vector per page (title + compiled_truth + timeline, capped 8K chars). - No chunking (intentional; chunked vector RAG would be EXT-2b later). - No keyword fallback (that's EXT-3 hybrid-without-graph). - Embeddings in batches of 50 via existing src/core/embedding.ts (retry+backoff). - Cost on 240 pages: ~$0.02/run. Three-adapter side-by-side on 240-page rich-prose corpus, 145 relational queries: | Adapter | P@5 | R@5 | Correct top-5 | |---------------|--------|--------|---------------| | gbrain-after | 49.1% | 97.9% | 248/261 | | ripgrep-bm25 | 17.1% | 62.4% | 124/261 | | vector-only | 10.8% | 40.7% | 78/261 | Interesting finding: vector-only scores WORSE than BM25 on relational queries like "Who invested in X?" — exact entity match matters more than semantic similarity for these templates. BM25 nails the entity-name term; vector-only returns topically-similar-but-not-mentioning pages. This is the known failure mode of pure-vector RAG on precise relational/identity queries. Real-world vector RAG systems always add keyword fallback; EXT-3 (hybrid-without-graph) will be that fairer comparator. gbrain's lead widens in vector-only comparison: +38.4 pts P@5, +57.2 pts R@5. The graph layer is doing the heavy lifting for relational traversal; pure vector RAG can't express "traverse 'attended' edges from this meeting page." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): Phase 2 EXT-3 hybrid-without-graph adapter — graph isolated Third and closest-to-gbrain external baseline. Runs gbrain's full hybrid search (vector + keyword + RRF fusion + dedup) WITHOUT the knowledge-graph layer. Same engine, same embedder, same chunking, same hybrid fusion — only traversePaths + typed-link extraction turned off. This is the decisive comparator for "does the knowledge graph do useful work?" Same everything-else, only graph differs. Any lead gbrain-after has over EXT-3 is 100% attributable to the graph layer. Files: eval/runner/adapters/hybrid-nograph.ts — ~110 LOC Implementation: - New PGLiteEngine per run; auto_link set to 'false' (belt). - importFromContent() used instead of bare putPage() so chunks + embeddings get populated (hybridSearch needs them). - NO runExtract() call — typed links/timeline stay empty (suspenders). - hybridSearch(engine, q.text) answers every query. Aggregate chunks to page-level by best chunk score. FOUR-adapter side-by-side on 240-page rich-prose corpus, 145 relational queries: | Adapter | P@5 | R@5 | Correct/Gold | |-----------------|--------|--------|--------------| | gbrain-after | 49.1% | 97.9% | 248/261 | | hybrid-nograph | 17.8% | 65.1% | 129/261 | | ripgrep-bm25 | 17.1% | 62.4% | 124/261 | | vector-only | 10.8% | 40.7% | 78/261 | The headline delta nobody can hand-wave away: gbrain-after → hybrid-nograph = +31.4 P@5, +32.9 R@5 hybrid-nograph → ripgrep-bm25 = +0.7 P@5, +2.7 R@5 Hybrid search (vector+keyword+RRF) over pure BM25 gains ~1 point. The knowledge graph layer over hybrid gains ~31 points. The graph is doing the work; adding it to a retrieval stack is what actually moves the needle on relational queries. The vector/keyword/BM25 debate is a footnote. Timing: hybrid-nograph init is ~2 min (embeds 240 pages once); query loop is fast. gbrain-after is ~1.5s total because traversePaths doesn't need embeddings. Runs at ~$0.02 Opus-equivalent in embedding cost. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): Phase 2 query validator + Tier 5 Fuzzy + Tier 5.5 synthetic + N=5 tolerance bands Closes multiple Phase 2 items in one commit since they form a cohesive package: query schema enforcement + new query tiers + per-query-set statistical rigor. Added: eval/runner/queries/validator.ts — hand-rolled Query schema validator eval/runner/queries/validator.test.ts — 24 unit tests, all pass eval/runner/queries/tier5-fuzzy.ts — 30 hand-authored Tier 5 Fuzzy/Vibe queries eval/runner/queries/tier5_5-synthetic.ts — 50 SYNTHETIC-labeled outsider-style queries (author: "synthetic-outsider-v1") eval/runner/queries/index.ts — aggregator + validateAll() Modified: eval/runner/multi-adapter.ts — N=5 runs per adapter (BRAINBENCH_N override), page-order shuffle, mean±stddev reporting Query validator (hand-rolled, no zod dep to match gbrain codebase style): - Temporal verb regex enforces as_of_date (per eng pass 2 spec): /\\b(is|was|were|current|now|at the time|during|as of|when did)\\b/i - Validates tier enum, expected_output_type enum, gold shape per type - gold.relevant must be non-empty slug[] for cited-source-pages queries - abstention requires gold.expected_abstention === true - externally-authored tier requires author field - batch validation catches duplicate IDs Tier 5 Fuzzy/Vibe (30 queries, hand-authored): - Vague recall: "Someone who was a senior engineer at a biotech company..." - Trait-based: "The engineer who pushed back on microservices" - Cultural/epithet: "Who is known as a 'systems builder' in security?" - Abstention bait: "Which Layer 1 project did the crypto guy leave?" (prose mentions but never names; good systems abstain) - Addresses Codex's circularity critique — vague queries where graph-heavy systems shouldn't inherently win. Tier 5.5 Synthetic Outsider (50 queries, AI-authored placeholder): - Clearly labeled author: "synthetic-outsider-v1" - Phrasing variety not in the 4 template families: * fragment style ("crypto founder Goldman Sachs background") * polite/natural ("Can you pull up what we have on...") * comparison ("What is the difference between X and Y?") * follow-up ("And who else advises Orbit Labs?") * typos/misspellings ("adam lopez bioinformatcis") * similarity ("Find me someone like Alice Davis...") * imperative ("Pull up Alice Davis") - Real Tier 5.5 from outside researchers supersedes synthetic via PRs to eval/external-authors/ (docs ship in follow-up commit). N=5 tolerance bands: - Default N=5, override via BRAINBENCH_N env var (e.g. BRAINBENCH_N=1 for dev loops) - Per-run seeded Fisher-Yates shuffle of page ingest order (LCG seed = run_idx+1) - Surfaces order-dependent adapter bugs (tie-break-by-first-seen etc.) - Reports mean ± sample-stddev per metric - "stddev = 0" is honest signal that the adapter is deterministic, not a bug. LLM-judge metrics (future) will naturally produce non-zero stddev. Validation: all 80 Tier 5 + 5.5 queries pass validateAll(). 24 validator unit tests pass. Next commit: world.html contributor explorer (Phase 3). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): Phase 3 world.html explorer + eval:* CLI surface Contributor DX magical moment. Static HTML explorer renders the full canonical world (240 entities) as an explorable tree, opens in any browser, zero install. Every string HTML-entity-encoded (XSS-safe — direct vuln class per eng pass 2, confidence 9/10). Added: eval/generators/world-html.ts — renderer (~240 LOC; single-file HTML with inline CSS + minimal JS) eval/generators/world-html.test.ts — 16 tests (XSS + rendering correctness) eval/cli/world-view.ts — render + open in default browser eval/cli/query-validate.ts — CLI wrapper for queries/validator eval/cli/query-new.ts — scaffold a query template Modified: package.json — 7 new eval:* scripts .gitignore — ignore generated world.html package.json scripts shipped: bun run test:eval all eval unit tests (57 pass) bun run eval:run full 4-adapter N=5 side-by-side bun run eval:run:dev N=1 fast dev iteration bun run eval:world:view render world.html + open in browser bun run eval:world:render render only (CI-friendly, --no-open) bun run eval:query:validate validate built-in T5+T5.5 (or a file path) bun run eval:query:new scaffold a new Query JSON template bun run eval:type-accuracy per-link-type accuracy report XSS safety: escapeHtml() encodes the 5 critical chars (& < > " '). Tested directly with representative Opus-generated attacks: <img src=x onerror=alert('xss')> → &lt;img src=x onerror=alert(&#39;xss&#39;)&gt; <script>fetch('/steal')</script> → &lt;script&gt;fetch(&#39;/steal&#39;)&lt;/script&gt; Ledger metadata (generated_at, model) also escaped — covers the less obvious attack surface where Opus could emit tag-like content into the metadata file. world.html structure: - Left rail: entities grouped by type with counts (companies, people, meetings, concepts), alphabetical within type - Right pane: per-entity cards with title + slug + compiled_truth + timeline + canonical _facts as collapsed JSON - URL fragment deep-links (#people/alice-chen) - Sticky rail on desktop; responsive stack on mobile - Vanilla JS for active-link highlighting on scroll (no framework) Generated file: ~1MB for 240 entities (full prose). Gitignored; rebuild with `bun run eval:world:view`. Regeneration is ~50ms. Contributor TTHW (Tier 5.5 query authoring): 1. bun run eval:world:view # see entities 2. bun run eval:query:new --tier externally-authored --author "@me" 3. edit template with real slug + query text 4. bun run eval:query:validate path/to/file.json 5. submit PR Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(eval): Phase 3 contributor docs + CI workflow for eval/ tests Ships the contributor-onboarding surface promised in the plan. With this commit, external researchers have a self-serve path from clone to PR in under 5 minutes. Added: eval/README.md — 5-minute quickstart, directory map, methodology one-pager, adapter scorecard eval/CONTRIBUTING.md — three contributor paths: 1. Write Tier 5.5 queries 2. Submit an external adapter 3. Reproduce a scorecard eval/RUNBOOK.md — operational troubleshooting: generation failures, runner failures, query validation, world.html rendering, CI eval/CREDITS.md — contributor attribution (synthetic-outsider-v1 labeled as placeholder; real submissions land here) .github/PULL_REQUEST_TEMPLATE/tier5-queries.md — structured PR template for Tier 5.5 submissions .github/workflows/eval-tests.yml — CI: validates queries, runs all eval unit tests, renders world.html on every PR touching eval/** or src/core/link-extraction.ts CI scope (intentionally narrow): - Triggers on paths: eval/**, src/core/link-extraction.ts, src/core/search/** - Runs: bun run eval:query:validate (80 queries), test:eval (57 tests), eval:world:render (smoke-test the HTML renderer) - Pinned actions by commit SHA (matches existing .github/workflows/test.yml) - Zero API calls — all Opus/OpenAI paths stubbed or skipped in unit tests - Fast: ~30s total wall clock Contributor TTHW (clone → first merged PR): - Path 1 (Tier 5.5 queries): ~5 min - Path 2 (external adapter): ~30 min for a simple adapter - Path 3 (reproduce scorecard): ~15 min wall clock (N=5 run) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(eval): teardown PGLite engines so bun run eval:run exits 0 The multi-adapter runner left PGLite engines alive after each run. GbrainAfterAdapter and HybridNoGraphAdapter both instantiate a PGLiteEngine in init() but never disconnect it; Bun's shutdown path exits with code 99 when embedded-Postgres workers outlive main(). Added optional `teardown?(state)` to the Adapter interface, implemented it on both engine-backed adapters, and call it from scoreOneRun after the N=5 loop. ripgrep-bm25 and vector-only hold no DB resources and don't need a teardown. Verified: gbrain-after, hybrid-nograph, ripgrep-bm25, vector-only all exit 0 at N=1. Full test:eval passes (57 tests). No metric change. * docs(bench): 2026-04-19 multi-adapter scorecard Reproducibility run of the 4-adapter side-by-side at commit b81373d (branch garrytan/gbrain-evals). N=5, 240-page corpus, 145 relational queries from world-v1. Headline: gbrain-after 49.1% P@5 / 97.9% R@5. hybrid-nograph 17.8% / 65.1%. ripgrep-bm25 17.1% / 62.4%. vector-only 10.8% / 40.7%. All adapters deterministic (stddev = 0 across the 5 runs per adapter). Matches the scorecard in eval/README.md byte-for-byte for the three deterministic adapters; hybrid-nograph matches within tolerance bands. * docs(bench): 2026-04-19 gbrain v0.11.1 vs v0.12.1 regression comparison Runs the same eval harness against two gbrain src/ trees on the same 240-page corpus and 145 queries. Patches the v0.11 copy's gbrain-after adapter to use getLinks/getBacklinks (v0.11 has no traversePaths) with identical direction+linkType semantics. gbrain-after P@5 22.1% -> 49.1% (+27 pts); R@5 54.6% -> 97.9% (+43 pts); correct-in-top-5 99 -> 248 (+149). hybrid-nograph flat at 17.8% / 65.1% on both (v0.12 didn't touch hybridSearch / chunking). Driver is extraction quality, not graph presence: v0.12 emits 499 typed links (v0.11: 136, x3.7) and 2,208 timeline entries (v0.11: 27, x82) on the same 240 pages. Sharpens the April-18 "graph layer does the work" claim -- on v0.11 that architecture only beat hybrid-nograph by 4.3 points; the 31-point lead in the multi-adapter scorecard comes from graph + high-quality extract in combination. * feat(eval): BrainBench v1 portable JSON schemas + gold templates Adds the v1→v2 contract boundary for BrainBench. 6 JSON schemas at eval/schemas/ pin the shape of every artifact a stack must emit to be scorable: corpus-manifest, public-probe (PublicQuery with gold stripped), tool-schema (12 read + 3 dry_run tools, 32K tool-output cap), transcript, scorecard (N ∈ {1, 5, 10}), evidence-contract (structured judge input). 8 gold file templates at eval/data/gold/ scaffold the sealed qrels, contradictions, poison items, and citation labels. Empty-but-valid skeletons; Day 3b fills them with real content once the amara-life-v1 corpus generates. 48 tests validate schema syntax, $schema/$id/title/type headers, round-trip stability, and cross-schema coherence (new Page types in manifest enum, tool counts, token cap, N enum). When v2 ports to Python + Inspect AI + Docker, these schemas are the boundary. Same fixtures, same tool contracts, zero rework. * feat(eval): amara-life-v1 skeleton + Page.type enum for email/slack/cal/note Deterministic procedural generator for the twin-amara-lite fictional-life corpus (BrainBench v1 Cat 5/8/9/11 target). 15 contacts picked from world-v1, 50 emails + 300 Slack messages across 4 channels + 20 calendar events + 8 meeting transcripts + 40 first-person notes. Mulberry32 PRNG gives byte-identical output under reseed. Plants 10 contradictions + 5 stale facts + 5 poison items + 3 implicit preferences at deterministic positions. Fixture_ids are unique across the corpus so gold/contradictions.json + gold/poison.json + gold/implicit- preferences.json can cross-reference by stable ID. PageType extended in both src/core/types.ts and eval/runner/types.ts to include email | slack | calendar-event | note (+ meeting on the production side). src/core/markdown.ts inferType() heuristics updated for the new one-slash slug prefixes (emails/em-NNNN, slack/sl-NNNN, cal/evt-NNNN, notes/YYYY-MM-DD-topic, meeting/mtg-NNNN). 17 tests cover counts (50/300/20/8/40), perturbation counts (exact 10/5/5/3), seed determinism + divergence, slug regex conformance (matches eval/runner/queries/validator.ts:131 one-slash rule), unique fixture_ids, amara-in-every-email invariant, calendar dtstart < dtend, and Amara-is- attendee on every meeting. * feat(eval): amara-life-gen.ts with structured cache key + $20 cost gate Opus prose expansion of the amara-life-v1 skeleton. Per-item structured cache key = sha256({schema_version, template_id, template_hash, model_id, model_params, seed, item_spec_hash}). Prompt-template tweak changes template_hash; only those items regenerate. Schema bump changes schema_version; everything invalidates cleanly. Interrupted runs resume from the last cached item; zero re-spend. Cost-gated at $20 hard-stop with Anthropic input/output pricing tracking. Dry-run mode (--dry-run) executes the full pipeline with stub bodies for smoke-testing the I/O layout without LLM spend. --max N caps items per type for debugging. --force ignores cache. Writes per-format outputs under eval/data/amara-life-v1/: inbox/emails.jsonl (one email per line with body_text appended) slack/messages.jsonl (one message per line with text appended) calendar.ics (RFC-5545 VEVENT format, templated — no LLM) meetings/<id>.md (transcript with YAML frontmatter) notes/<YYYY-MM-DD-topic>.md (first-person journal) docs/*.md (6 reference docs, templated — no LLM) corpus-manifest.json (per eval/schemas/corpus-manifest.schema.json, including per-item content_sha256 and generator_cache_key) Perturbation hints (contradiction, stale-fact, poison, implicit- preference) flow through the prompt so Opus weaves the specific claim into each item's body. Poison items are hand-crafted to include paraphrased prompt-injection attempts (not literal 'IGNORE ALL PREVIOUS' — defense is the structured-evidence judge contract at Day 5, not regex redaction). New package.json scripts: eval:generate-amara-life # real run (~$12 Opus estimated) eval:generate-amara-life:dry # smoke test, zero spend test:eval extended to include test/eval/. 10 cache-key tests cover determinism, invalidation across every field of the key, canonical JSON stability under object-key reorder, and per-skeleton-item spec-hash uniqueness (50 distinct hashes for 50 distinct emails). * chore: bump version and changelog (v0.15.0) Resets package.json from stale 0.13.1 to 0.15.0 (matches VERSION). v0.14.0 shipped with the stale package.json version; this sync catches that up and moves to v0.15.0 in one step. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: update CLAUDE.md + README + eval/README for v0.15.0 BrainBench CLAUDE.md: adds a full BrainBench section to the Key Files list — 14 new entries covering eval/README.md, multi-adapter.ts, types.ts (with new PublicPage/PublicQuery), adapters/, queries/, type-accuracy.ts, adversarial.ts, all.ts, world.ts/gen.ts, world-html.ts, amara-life.ts, amara-life-gen.ts, schemas/, data/world-v1/, data/gold/, data/amara-life-v1/, docs/benchmarks/, and test/eval/. Adds 3 new test/eval/ lines to the unit-tests catalog. eval/README.md: file tree updated to reflect v0.15 additions — data/amara-life-v1/, data/gold/, schemas/, generators/amara-life.ts + amara-life-gen.ts, runner/all.ts + adversarial.ts. README.md: updates hero benchmark numbers (L7 intro + L353 mid-page) from v0.10.5 PR #188 numbers (R@5 83→95, P@5 39→45) to current v0.12.1 4-adapter numbers (P@5 49.1% · R@5 97.9% · +31.4 pts vs hybrid-nograph). Adds the v0.11→v0.12 regression comparison as the secondary reference. Deeper-section tables (L422+) labeled "BrainBench v1 (PR #188)" are preserved as historical data. CHANGELOG is untouched — /ship already wrote the v0.15.0 entry. TODOS.md is untouched — Cat 5/6/8/9/11 remain open (only foundations shipped in v0.15.0; Cat runners ship in v1 Complete follow-ups). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(eval): Day 4 — pdf-parse + flight-recorder + tool-bridge (dry_run + expand:false) Three infrastructure modules for BrainBench v1 Complete Cats 5/8/9/11. **eval/runner/loaders/pdf.ts** — Thin pdf-parse wrapper. Lazy import keeps pdf-parse out of the module-load path (avoids library debug-mode side effects). Size cap (50MB default), encryption detection, structured error classes (PdfEncryptedError, PdfTooLargeError, PdfParseError). Only Cat 11 multimodal will import this; production bundle never sees pdf-parse. **eval/runner/tool-bridge.ts** — Maps 12 read-only operations from src/core/operations.ts to Anthropic tool definitions + adds 3 dry_run write tools. Three structural invariants enforced: 1. No hidden LLM calls. `operations.query` defaults expand=true which routes through expansion.ts → Haiku. Bridge strips `expand` from the query tool's input schema AND executor hard-sets expand:false. Zero nested Haiku calls in any agent trace. 2. Mutating ops throw ForbiddenOpError. put_page, add_link, delete_page, etc. are rejected by name. Agents record intent via dry_run_put_page / dry_run_add_link / dry_run_add_timeline_entry which persist to the flight-recorder without mutating the engine. This is how Cat 8's back_link_compliance + citation_format metrics measure anything with a read-only tool surface. 3. Poison tagged by the bridge, not the judge. Every tool result is scanned for slugs matching gold/poison.json fixtures. Matched fixture_ids flow into tool_call_summary.saw_poison_items for the structured-evidence judge contract. Judge never reads raw tool output — Section-3 defense against paraphrased prompt injections (poison payloads never reach the judge model at all). 32K-token cap (~128K chars) with "…[truncated]" suffix. **eval/runner/recorder.ts** — Per-run flight-recorder bundle emitter. Full 6-artifact bundle (transcript.md, brain-export.json, entity-graph.json, citations.json, scorecard.json, judge-notes.md) when the adapter provides an AdapterExport; 3-artifact fallback (transcript + scorecard + judge-notes) otherwise. Atomic writes via tmp+rename. Collision-safe: duplicate directory names get incremental -2, -3 suffix. `safeStringify` handles circular references without throwing and JSON-serializes Float32Array embeddings. **package.json:** adds pdf-parse@2.4.5 as a devDependency. Scoped to eval/ use only; production gbrain binary unaffected. **Tests:** 63 new — 30 tool-bridge, 21 recorder, 12 pdf-loader. All pass. Fake engine uses a Proxy with `__default__` fallback so poison-matching tests don't have to mock the exact engine method name that each operation calls (some route via searchKeyword, others via getPage — proxy handles both uniformly). Total eval suite now: 132 pass, 0 fail, 923 expect() calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(eval): Day 5 — agent adapter + judge with structured evidence contract Two modules that together wire Cat 8 / Cat 9 / Cat 5 end-to-end scoring. **eval/runner/judge.ts** — Haiku 4.5 via tool-use `score_answer`. Input is the structured JudgeEvidence contract (fix #16 from the plan's codex review): probe + final_answer_text + evidence_refs + tool_call_summary + ground_truth_pages + rubric. Raw tool output NEVER reaches the judge — that's the Section-3 defense against paraphrased prompt-injection payloads in gold/poison.json. Retry policy: one retry on malformed tool_use response. If the second attempt is still malformed, score the probe as `judge_failed` (all scores 0, verdict=fail) so the run still completes. Aggregation: weighted mean across rubric criteria. Canonical thresholds (pass ≥3.5, partial 2.5-3.5, fail <2.5) — judge can propose a verdict but the computed verdict from the weighted mean is what the scorecard records. This prevents the model from inflating or deflating its own verdict. Score values are clamped to 0-5 on parse even if the model returns out of range. `assertNoRawToolOutput(evidence)` is a regression guard that returns the list of forbidden fields (tool_result, raw_transcript, etc.) if any leak into the evidence contract. **eval/runner/adapters/claude-sonnet-with-tools.ts** — The agent adapter. Implements `Adapter` interface minimally: `init()` spins up PGLite and seeds it, `query()` throws because the adapter is Cat 8/9-only and emits a final-answer text, not a RankedDoc[]. Retrieval scorecard stays at 4 adapters. `runAgentLoop(probeId, text, state, config)` drives the multi-turn loop: Sonnet → tool_use → tool-bridge.executeTool → tool_result → back to Sonnet. Turn cap 10. max_tokens 1024. System prompt (brain-first iron law, citation format, amara context) is cached via cache_control. Exponential backoff on rate-limit errors (1s, 2s, 4s). Emits a `Transcript` per eval/schemas/transcript.schema.json — consumed directly by recorder.ts for the flight-recorder bundle. `brain_first_ordering` classifies Cat 8's flagship metric: did the agent call search/get_page BEFORE producing the final answer? The `no_brain_calls` case (agent answers from general knowledge without ever hitting the brain) is the compliance failure to surface. ForbiddenOpError + UnknownToolError from the bridge are caught in the agent loop and surfaced as tool_result with is_error=true — keeps the loop going and preserves full audit trail for the judge. **Tests (35 new):** judge (23) — happy path, retry, fallback, evidence contract sanitization, rendered prompt does not contain raw tool_result text, verdict thresholds, score clamping, weighted mean with mixed weights, parseToolUse rejects malformed input. agent-adapter (12) — Adapter.query() throws, init() seeds PGLite, end-to-end tool loop with stubbed Sonnet, turn cap exhaustion, mutating-op rejection surfaces as tool_result error, extractSlugs regex. All 12 agent tests take ~23s because PGLite runs 13 schema migrations per test; the alternative of shared-engine-across-tests was rejected so each test is isolated. Total eval suite now: 167 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(eval): Day 6 — adversarial-injections + Cat 6 prose-scale + Cat 11 multi-modal Three modules that together cover BrainBench v1 Cat 6 (prose-scale extraction fidelity) and Cat 11 (multi-modal ingest fidelity). **eval/runner/adversarial-injections.ts** — 6 deterministic content transforms shared by Cat 10 (adversarial.ts, 22 hand-crafted cases) and Cat 6 (prose-scale variants). Each injection produces a modified content string + a structured GoldDelta describing what the extractor MUST and MUST NOT produce. Kinds: - code_fence_leak — fake [X](people/fake) inside ``` fence, must NOT extract - inline_code_slug — `people/fake` in backticks, must NOT extract - substring_collision — "SamAI" near real `people/sam`, exactly one link - ambiguous_role — "works with" vs "works at", downgrade type to mentions - prose_only_mention — strip markdown link syntax, bare name → mentions only - multi_entity_sentence — pack 4+ entities into one clause, extract all Mulberry32 PRNG keeps variant generation deterministic under fixed seed. Codex flagged the original plan's wording ("extract injection engine from adversarial.ts") as overstated — adversarial.ts is a static case list, not a reusable engine. This module is NEW code. **eval/runner/cat6-prose-scale.ts** — Runner. Loads world-v1, applies all 6 injection kinds to sampled base pages (default 50 variants per kind × 6 kinds = 300 variants), runs extractPageLinks on each, compares to gold delta. Emits per-kind + overall metrics (precision, recall, F1, code_fence_leak_rate, substring_fp_rate, pages_with_links_coverage, mean_links_per_page). **v1 verdict is always "baseline_only"** — no gating threshold per codex fix #9 (current extractor residuals make >0.80 unreachable; v1 records a baseline, regression guard triggers on drop below it). **eval/runner/cat11-multimodal.ts** — PDF + HTML + audio runners. Fixtures load from eval/data/multimodal/<modality>/fixtures.json manifests; each modality skips gracefully when manifest missing or (audio) when neither GROQ_API_KEY nor OPENAI_API_KEY is set. Metrics: - PDF: char-level similarity via Levenshtein + optional entity_recall - HTML: word-recall over normalized tokens (multiset semantics) - Audio: WER (word error rate) via Levenshtein on word sequences Fixtures are NOT committed; a future eval:fetch-multimodal script will download them hash-verified from public sources (arXiv CC-licensed papers, Wikipedia CC-BY-SA, Common Voice CC0). Injectable audio transcriber (`opts.transcribe`) means tests don't need GROQ/OpenAI keys — stubbed transcriptions exercise the WER math path directly. **Tests (60 new):** adversarial-injections (19) — per-kind assertions + dispatcher coverage + slug regex conformance; cat6 (12) — variant determinism, scoreVariant shape, aggregate per-kind + overall metrics, corpus resolver slug rules; cat11 (29) — charSimilarity / wordRecall / wer math, htmlToText strips scripts + decodes entities, HTML modality with real fixtures, audio modality gracefully skips without key + uses stub transcriber correctly. All 60 tests pass in 48ms + 41ms. Total eval suite now: 227 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(eval): Day 7 — Cat 5 provenance runner + structured classify_claim judge **eval/runner/cat5-provenance.ts** — BrainBench Cat 5 scoring. Samples claims from gbrain brain-export and classifies each against its source material via a dedicated Haiku judge (classify_claim tool with a three-label enum: supported | unsupported | over-generalized). Separate from judge.ts by design: Cat 5 is a single three-way classification per claim, not a weighted rubric. Rather than overload judge.ts with a mode switch, Cat 5 has its own tool definition (CLASSIFY_CLAIM_TOOL) and prompt. The retry-once pattern, $20 cost gate semantics, and structured parsing are mirrored from judge.ts so failures look the same across Cats. Metric: `citation_accuracy` = fraction where predicted label equals gold expected_label. Threshold (informational): >0.90 per design-doc METRICS.md. v1 ships with `enableThreshold: false` so the verdict is always baseline_only — we don't have hand-authored gold claims yet, and codex flagged that threshold gating should wait until the amara-life-v1 corpus + gold file authoring lands in Day 3b. runCat5 uses a bounded-concurrency worker pool (default 4) to respect Haiku rate limits across 100+ claim batches. Evidence pages are looked up by slug from a caller-provided pagesBySlug map — missing pages don't crash, they just pass an empty source list to the judge (correct behavior for genuinely unsupported claims). **Tests (23):** classifyClaim happy/retry/fallback paths with stubbed Haiku, aggregate accuracy math, threshold gating (pass/fail vs baseline_only), runCat5 concurrency + missing-page handling, renderClaimPrompt embeds claim + sources correctly, parseClassification rejects invalid enum values + plain-text responses. Total eval suite now: 250 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(eval): Day 8 — Cat 8 skill compliance + Cat 9 end-to-end workflows **eval/runner/cat8-skill-compliance.ts** — Deterministic, judge-free Cat 8 scoring. Replays inbound signals through the agent adapter (Day 5) and extracts four iron-law metrics directly from the tool-bridge state: - brain_first_compliance: agent called search/get_page BEFORE producing its final answer. Non-compliance = hallucinating from general knowledge. - back_link_compliance: every dry_run_put_page intent has at least one markdown [Name](slug) back-link in its compiled_truth. - citation_format: timeline entries use canonical `- **YYYY-MM-DD** | Source — Summary`; long final answers cite at least one slug. - tier_escalation: simple probes use light tooling (≥1 brain call); complex probes require ≥2 brain calls or a dry_run write when expects_dry_run_write is set. No judge call required — everything is computable from `tool_bridge_state.made_dry_run_writes` + `count_by_tool` + final_answer regex. Fast, deterministic, reproducible. Bounded concurrency (p-limit style) worker pool at default 4 to keep Sonnet rate limits comfortable across 100-probe batches. **eval/runner/cat9-workflows.ts** — Rubric-graded Cat 9. 5 canonical workflows (meeting_ingestion, email_to_brain, daily_task_prep, briefing, sync) × ~10 scenarios each. Each scenario runs through the agent adapter, then judge.ts scores the answer against a per-scenario rubric. `buildEvidence(scenario, agentResult, pagesBySlug)` composes the JudgeEvidence contract: resolves ground_truth_slugs to full GroundTruthPage[] from a slug-map, pulls tool_call_summary directly from tool_bridge_state (no raw tool_result content — Section-3 defense), attaches rubric from the scenario. Per-workflow rollup: each workflow gets its own pass_rate so the verdict can fail one workflow without failing the whole Cat. Overall verdict requires every populated workflow's pass_rate ≥ threshold (default 0.80) when enableThreshold=true. Both Cats default to verdict=baseline_only in v1 per codex fix #9: real thresholds return after 10-probe Haiku-vs-hand-score calibration (κ > 0.7) runs against the Day 3b amara-life-v1 corpus. **Tests (23):** Cat 8 per-metric scorer unit tests covering every branch (brain_first ordering, back-link compliance on mixed writes, long vs short answer citation requirement, tier escalation for simple/complex/ writey probes, finalAnswerCiteCount dedups across syntaxes). Cat 9 buildEvidence contract shape — evidence_refs flow from agent, missing slugs skip gracefully, no raw_transcript/tool_result leakage to judge. Cat 9 runCat9 integration with stubbed agent + mixed-verdict judge produces fractional pass rates correctly. Total eval suite now: 273 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(eval): Day 9 — sealed qrels via PublicPage + PublicQuery at adapter boundary Codex fixes #1, #2, #3 from the plan's outside-voice review. Enforcement shifts from SOFT-VIA-TYPE-COMMENT to SOFT-VIA-SANITIZED-OBJECT. Hard enforcement via process isolation waits for BrainBench v2 Docker sandbox. **eval/runner/types.ts** additions: - `PublicPage = Pick<Page, 'slug' | 'type' | 'title' | 'compiled_truth' | 'timeline'>` — the exact 5 fields adapters should see. No _facts. No frontmatter (a known hiding spot for accidental gold leaks). - `sanitizePage(p: Page): PublicPage` — returns a NEW object with the 5 fields only. Cannot be bypassed by `(page as any)._facts` because the field does not exist on the sanitized object. - `PublicQuery = Omit<Query, 'gold'>` — strips the gold field. - `sanitizeQuery(q: Query): PublicQuery` — enumerates public fields explicitly (not spread+delete) so no prototype weirdness leaves gold reachable. **eval/runner/multi-adapter.ts** — scoreOneRun now calls sanitizePage / sanitizeQuery before passing to adapter.init / adapter.query. The scorer retains the full Query shape (including gold.relevant) for precision / recall computation. Adapter signatures unchanged — the sealing is at the OBJECT level, not the type level. This keeps existing adapters (ripgrep-bm25, vector-only, hybrid-nograph, gbrain-after) binary-compatible. Verified: no existing adapter reads q.gold or page._facts, so the change is safe without further adapter updates. **test/eval/sealed-qrels.test.ts** (17 tests): - sanitizePage strips _facts + frontmatter + arbitrary hidden keys - Output has exactly the 5 public keys (deep introspection) - Proxy tripwire simulates a malicious adapter: any access to _facts or gold throws `sealed-qrels violation` - sanitizeQuery retains optional fields (as_of_date, tags, author, acceptable_variants, known_failure_modes) but omits undefined ones - Honest documentation of the seal's limits: filesystem bypass and Proxy attacks would still work in v1; Docker isolation (v2) is the real enforcement Every existing eval test still passes (273 before + 17 sealed-qrels = 290). Total eval suite now: 290 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(eval): Day 10 — all.ts rewrite + llm-budget + BrainBench N tiers Final wiring of BrainBench v1 Complete. all.ts now orchestrates the full Cat catalog (1-12) via a mix of subprocess dispatch (Cats 1, 2, 3, 4, 6, 7, 10, 11, 12 — standalone runners with CLI entry points) and programmatic invocation (Cats 5, 8, 9 — require runtime inputs that can't come via CLI flags). Subprocess Cats run concurrently under a p-limit(2) bound to cap peak memory around ~800MB (two PGLite instances at ~400MB each). Cats 5/8/9 show as "programmatic" in the report with a one-line reference to their `runCatN({...})` harness API. They're deliberately skipped from the master runner because their inputs (claim catalog, probe catalog, scenario catalog, pre-seeded agent state, evidence pagesBySlug) are task-specific and assembled at the caller. **eval/runner/all.ts** — rewritten: - CATEGORIES is a tagged union of SubprocessCategory | ProgrammaticCategory - runCatSubprocess spawns Bun with pipe'd stdout/stderr, 10-min timeout per Cat (124 exit + SIGTERM on timeout; no hung subprocesses) - runConcurrently is a bounded worker pool preserving input order - buildReport emits the full markdown with per-Cat elapsed times, migration-noise filter, and a separate programmatic-only section - Honors BRAINBENCH_N (1/5/10 for smoke/iteration/published), BRAINBENCH_CONCURRENCY (default 2), BRAINBENCH_LLM_CONCURRENCY (default 4, consumed by llm-budget) **eval/runner/llm-budget.ts** — shared LLM rate-limit semaphore. A full N=10 published scorecard makes ~900 Anthropic calls (150 Cat 8/9 probes × N=10 + 100 Cat 5 claims × N=10). Without coordination, concurrent adapters trigger 429s on per-minute limits. - LlmBudget class: acquireSlot/releaseSlot + withLlmSlot(fn) wrapper that releases on success AND throw (try/finally) - getDefaultLlmBudget() singleton reads BRAINBENCH_LLM_CONCURRENCY, falls back to 4 on missing/garbage values - capacity enforced ≥1 (rejects 0/negative) - Double-release is a no-op (guards against upstream double-call bugs) - Active + waiting counts exposed for observability / tests **package.json** scripts: - eval:brainbench — default N=5 iteration - eval:brainbench:smoke — N=1 for fast iteration - eval:brainbench:published — N=10 for committed baselines - eval:cat6 / eval:cat11 — individual new subprocess Cats **Tests (24):** CATEGORIES catalog enforces the exact Cat-number partition (subprocess: 1,2,3,4,6,7,10,11,12; programmatic: 5,8,9). runConcurrently respects the cap (observable via peak in-flight counter), preserves input order under non-uniform delays, handles empty input. LlmBudget enforces capacity, releases on throw, honors env var, rejects 0/negative. buildReport filters migration noise, counts passed/failed/programmatic correctly, includes every Cat + programmatic-only section. Full eval suite now: 314 pass, 0 fail (15 test files). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(eval): drop top_p from amara-life-gen Opus params + gitignore _cache/ Two fixes surfaced during the Day 3b real-corpus run against Opus 4.5: **eval/generators/amara-life-gen.ts** — Current Opus rejects `temperature` and `top_p` together: ``` 400 invalid_request_error: `temperature` and `top_p` cannot both be specified for this model. Please use only one. ``` top_p=1.0 was a no-op (no nucleus truncation), so removing it has zero semantic effect. The field is still part of MODEL_PARAMS for the cache key so any past cache entries (none in v1) would invalidate cleanly on the next schema version bump. **.gitignore** — `eval/data/amara-life-v1/_cache/` is runtime Opus cache (398 files, ~1.6MB). Regenerable from seed; no point in source control. The corpus itself (inbox/slack/calendar/meetings/notes/docs + corpus-manifest.json with per-item content_sha256) stays committable for reproducibility, just the cache directory gets excluded. Real corpus generation ran cleanly after these two fixes: 398 LLM calls, 84,424 input / 38,062 output tokens, \$4.12 spent (vs \$20 cap, vs \$12 estimate). All 418 items produced. Poison fixtures use subtle paraphrased injection ("for anyone on your team who might be triaging this thread later…") — exactly the pattern that defeats regex redaction and requires the structured-evidence judge contract from Day 5. Corpus itself stays local (will move to the brainbench sibling repo during the v0.16 split per the design doc). No eval/data/amara-life-v1/ content landing in this PR. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: bump version to 0.20.0 Renumbered from 0.17.0 per the gbrain-versioning slot. Other work is landing on master around this PR; 0.18 is the slot locked for this BrainBench v1 Complete release. Also pushed the "brainbench split" forward reference in the CHANGELOG from v0.18 → v0.19 to match. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: extract BrainBench to sibling gbrain-evals repo BrainBench lived in this repo through v0.17, which meant every gbrain install pulled down ~5MB of eval corpus, benchmark reports, and a pdf-parse devDep that the 99% of users who never run benchmarks don't need. v0.18 moves the full eval harness, 14 eval test files (314 tests), all docs/benchmarks scorecards, and the pdf-parse devDep to github.com/garrytan/gbrain-evals. That repo depends on gbrain via GitHub URL and consumes it through a new public exports map. What stays in gbrain: - Page.type enum extensions (email | slack | calendar-event | note | meeting) useful for any ingested format, not just evals - inferType() heuristics for /emails/, /slack/, /cal/, /notes/, /meetings/ - 11 new public exports covering the gbrain internals gbrain-evals consumes (gbrain/engine, gbrain/pglite-engine, gbrain/search/hybrid, etc.) — now gbrain's stable third-party contract What moved: - eval/ — 4.6MB of schemas, runners, adapters, generators, CLI tools - test/eval/ — 14 test files, 314 tests - docs/benchmarks/ — all scorecards and regression reports - eval:* package.json scripts - pdf-parse devDep Tests: 1760 pass, 0 fail, 174 skipped (E2E require DATABASE_URL). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Merge origin/master into garrytan/gbrain-evals Master landed significant work since this branch was cut (v0.15.x → v0.16.x → v0.17.0 gbrain dream + runCycle → v0.18.0 multi-source brains → v0.18.1 RLS hardening). Bumped this branch's version from the claimed 0.18.0 to 0.19.0 because master already owns 0.18.x. Conflicts resolved: - VERSION: 0.19.0 (was 0.18.0 on HEAD vs 0.18.1 on master) - package.json: 0.19.0, kept all 11 eval-facing exports, merged master's typescript devDep + postinstall script + test script (typecheck added) - src/core/types.ts: union of both PageType additions. Master had added `meeting | note`; this branch added `email | slack | calendar-event` for inbox/chat/calendar ingest. Final enum carries all five. - CHANGELOG.md: renumbered the BrainBench-extraction entry to 0.19.0 and placed it above master's 0.18.1 RLS entry. Tweaked copy ("In v0.17 it lived inside this repo" → "Previously it lived inside this repo") to stop implying a specific version that never shipped. - CLAUDE.md: adjusted "BrainBench in a sibling repo" heading from (v0.18+) → (v0.19+). - docs/benchmarks/2026-04-18-minions-vs-openclaw-production.md: resolved modify-vs-delete conflict in favor of delete (the extraction). - scripts/llms-config.ts: dropped the docs/benchmarks/ entry (directory no longer exists here; lives in gbrain-evals). - llms.txt / llms-full.txt: regenerated after the config change. - bun.lock: accepted master's (master already dropped pdf-parse as a drive-by; aligned with our removal). Tests: 2094 pass, 236 skip, 18 fail. Spot-checked failures — build-llms, dream, orphans tests all pass in isolation. Failures reproduce only under full-suite parallel load and are pre-existing master flakiness (matches the graph-quality flake noted in the earlier summary). Not merge-introduced. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump to v0.20.0 Master is now at v0.18.2 (migration hardening + RLS + multi-source brains). BrainBench extraction ships as v0.20.0 to leave v0.19 free for any in-flight work on other branches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci: remove eval-tests workflow (moved to gbrain-evals) The Eval tests workflow ran `bun run eval:query:validate`, `test:eval`, and `eval:world:render` — all three scripts moved to the gbrain-evals repo when BrainBench was extracted in v0.20.0. The workflow has been failing on master since the split because the scripts no longer exist here. Eval CI now runs from gbrain-evals's own workflows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tests): bump PGLite hook timeouts to 60s for parallel-load stability Six test files spin up PGLite + 20 migrations + git repos in beforeEach/ beforeAll hooks. Under 136-way parallel test file execution, bun's default 5s hook timeout wasn't enough, producing 18 flaky failures that only reproduced under full-suite parallel load (all 6 files passed in isolation). Root cause: PGLite.create() + initSchema() takes ~3-5s under idle load, but under 136 concurrent WASM instantiations the OS thrashes and hooks stall well past 5s. The bunfig.toml `timeout = 60_000` applies to TESTS, not HOOKS — bun requires per-hook timeouts as the third beforeEach/beforeAll argument. Files touched (hook timeouts added, no test logic changed): - test/dream.test.ts — 5 describe blocks × before/afterEach - test/orphans.test.ts — 1 beforeEach + afterEach - test/core/cycle.test.ts — shared beforeAll + afterAll - test/brain-allowlist.test.ts — beforeAll + afterAll - test/extract-db.test.ts — beforeAll + afterAll - test/multi-source-integration.test.ts — beforeAll + afterAll Results: 2317 pass / 0 fail (was 2253 pass / 18 fail). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: coverage for inferType() BrainBench corpus dirs Closes the 1 gap surfaced by Step 7 coverage audit. 9 table-driven assertions covering the new Page.type branches: emails/*.md, email/*.md -> 'email' slack/*.md -> 'slack' cal/*.md, calendar/*.md -> 'calendar-event' notes/*.md, note/*.md -> 'note' meetings/*.md, meeting/*.md -> 'meeting' The fixtures use realistic paths from the amara-life-v1 corpus in the sibling gbrain-evals repo (em-0001, sl-0037, evt-0042, mtg-0003) so the test doubles as a contract check between the two repos. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(TODOS): mark BrainBench Cats 5/6/8/9/11 + v0.10.5 inferLinkType as completed All five BrainBench categories shipped in v0.20.0 (to the gbrain-evals sibling repo). v0.10.5 inferLinkType regex expansion shipped in-tree. Remaining P1 BrainBench work: Cat 1+2 at full scale (2-3K pages) — currently 240 pages in world-v1 corpus. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: sync CLAUDE.md + polish CHANGELOG voice for v0.20.0 CLAUDE.md: add v0.19 commands to key-files list (skillify, skillpack, routing-eval, filing-audit, skill-manifest, resolver-filenames); add 8 new test files + openclaw-reference-compat E2E to test index; repoint the release-summary template's benchmark source from `docs/benchmarks/[latest].md` to `gbrain-evals/docs/benchmarks/` since those files now live in the sibling repo. CHANGELOG voice polish for v0.20.0: replace em dashes with periods, parens, or ellipses per project style guide. No content changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: regenerate llms-full.txt after CLAUDE.md + CHANGELOG edits (fixes CI) The v0.20.0 doc-sync commit (9e567bb) added 7 new v0.19 modules to the CLAUDE.md Key Files index and polished CHANGELOG voice. Both are includeInFull: true inputs to llms-full.txt but the generator wasn't re-run, so the drift-detection guard (test/build-llms.test.ts) failed CI. One-line fix: regenerate. No content changes beyond what the two source docs already carry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 00:08:54 -07:00
4 people	55ca4984b2	feat: v0.17.0 — gbrain dream + runCycle primitive (one cycle, two CLIs) (#321) ... * fix(sync): honor --dry-run in full-sync path + expose embedded count Precondition for v0.17 brain maintenance cycle (runCycle primitive). The full-sync path (performFullSync) previously called runImport() even when opts.dryRun was true, silently writing to the DB and advancing sync.last_commit. `gbrain sync --dry-run` on a fresh brain (or with --full) would mutate state without warning. Fix: - performFullSync now early-returns a `dry_run` SyncResult when opts.dryRun is set. Walks the repo via collectMarkdownFiles + isSyncable to count what WOULD be imported. No writes, no git state advance. - SyncResult gains an `embedded: number` field (required). Tracks pages re-embedded during the sync's auto-embed step. Existing return sites set 0; the synced + first_sync paths set real counts (best-estimate until commit 2 sharpens runEmbedCore's return type). - first_sync path now returns real added + chunksCreated counts from runImport instead of hardcoded zeros. - printSyncResult shows embedded count in human output. Tests (test/sync.test.ts, new `performSync dry-run never writes` block, PGLite + temp git repo, no DATABASE_URL required): - first-sync --dry-run: no pages, no sync.last_commit - incremental --dry-run after real sync: bookmark unchanged - --full --dry-run: no reimport, bookmark unchanged - SyncResult.embedded is a number Codex outside-voice caught this. Would have shipped silent DB writes on dry-run for anyone using `gbrain sync --dry-run --full`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(embed): add dry-run mode + return EmbedResult with counts Precondition for v0.17 brain maintenance cycle (runCycle primitive). runEmbedCore previously returned Promise<void> and had no dry-run mode. That made it impossible for runCycle to (a) report accurate embedded counts or (b) honor --dry-run without also skipping the entire embed phase (which would have required runCycle to know embed's internal semantics — a layering violation). Changes: - EmbedOpts gains `dryRun?: boolean`. When set, embedPage and embedAll enumerate stale chunks (or would-be-created chunks for unchunked pages, via local chunkText without engine.upsertChunks) but never call embedBatch and never write to the engine. - runEmbedCore: Promise<void> -> Promise<EmbedResult>. Result shape: { embedded, skipped, would_embed, total_chunks, pages_processed, dryRun }. embedded = chunks newly embedded (0 in dryRun). would_embed = chunks that WOULD be embedded (0 in non-dryRun). skipped = chunks with pre-existing embeddings. - runEmbed CLI wrapper honors --dry-run flag and returns the result through. `gbrain embed --stale --dry-run` is now a safe preview. - Callers ignoring the return value (sync auto-embed, autopilot inline fallback, jobs.ts handlers, CLI) keep compiling — the new return type is additive for `await` callers. Tests (test/embed.test.ts, new `runEmbedCore --dry-run` block, uses the existing mock.module embedBatch pattern, no API key required): - dry-run --all: zero embedBatch calls, zero upsertChunks calls, would_embed matches stale chunk total - dry-run --stale correctly splits stale vs already-embedded counts - dry-run --slugs on a single page tallies per-chunk counts - non-dry-run regression guard: embedded count matches across concurrent workers Codex outside-voice flagged the Promise<void> return as a blocker for accurate CycleReport.totals.pages_embedded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(orphans): engine-injected queries, drop db.getConnection() global Precondition for v0.17 brain maintenance cycle (runCycle primitive). findOrphans + queryOrphanPages previously reached into the postgres-js singleton via db.getConnection(), which (a) didn't compose with runCycle's explicit-engine contract and (b) was wrong for PGLite test fixtures and for any caller not using the default global connection. Codex outside-voice flagged this as a blocker. Changes: - BrainEngine interface gains findOrphanPages() — returns pages with no inbound links via the same NOT EXISTS anti-join. Implemented on both postgres-engine (sql tag) and pglite-engine (db.query). - findOrphans signature: findOrphans(engine, { includePseudo }). Engine is required. Uses engine.findOrphanPages() and engine.getStats().page_count instead of raw SQL + global counts. - queryOrphanPages signature: queryOrphanPages(engine). Delegates to engine.findOrphanPages(). - src/commands/orphans.ts drops the `import * as db` — no more global-state coupling. - Callers updated: src/core/operations.ts find_orphans handler now passes ctx.engine through; runOrphans CLI entry uses its engine arg. - No signature change needed in cli.ts (it was already passing engine via CLI_ONLY dispatch). Tests (test/orphans.test.ts, new `findOrphans (engine-injected)` describe block, PGLite in-memory, no DATABASE_URL required): - links correctly scope orphans (alice links to bob -> bob not an orphan; alice is) - includePseudo:true surfaces _atlas-style pages - queryOrphanPages delegates to passed engine - empty brain returns {orphans: [], total_pages: 0} without crashing Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cycle): add runCycle primitive in src/core/cycle.ts The brain maintenance cycle as a single function. Six phases in semantically-driven order (fix files → sync → extract → embed → report orphans). Pure composition of existing library calls — no execSync, no subprocess anti-patterns, no regex-parsed output. ┌───────────────────────────────────────────────────┐ │ runCycle(engine, opts) → CycleReport │ │ Phase 1: lint --fix (fs writes) │ │ Phase 2: backlinks --fix (fs writes) │ │ Phase 3: sync (DB picks up 1+2) │ │ Phase 4: extract (DB picks up links) │ │ Phase 5: embed --stale (DB writes) │ │ Phase 6: orphans (DB read, report) │ └───────────────────────────────────────────────────┘ Why the commit-4 primitive: - CEO + Eng + Codex reviews all converged on "extract one cycle function, wire both dream and autopilot through it." Two CLIs, one definition of what the brain does overnight. - Phase order was wrong in PR #309's original dream.ts (sync before lint+backlinks lost the "fix files, then index them" semantic). - This commit is the bisectable foundation; commit 5 (dream) and commit 6 (autopilot+jobs) just call into it. Coordination — the codex-flagged blocker: Session-scoped pg_try_advisory_lock does not survive PgBouncer transaction pooling (the v0.15.4 fix made pooled connections the default). Replaced with a DB lock table (gbrain_cycle_locks) that works through every pooler: - Acquire: INSERT ... ON CONFLICT DO UPDATE ... WHERE ttl < NOW() - Refresh: UPDATE ttl_expires_at between phases via hook - Release: DELETE in finally{} - TTL: 30 min; crashed holders auto-release PGLite / engine=null path uses a file lock at ~/.gbrain/cycle.lock with PID liveness check. kill(pid, 0) with EPERM treated as alive (so init/launchd-pid holders aren't mis-classified as stale). Lock-skip: only phases that mutate state (lint, backlinks, sync, extract, embed) trigger lock acquisition. orphans is read-only. Single-phase --phase orphans runs never block on a held lock. Engine-null mode preserved: filesystem phases run, DB phases skip with {status:'skipped', reason:'no_database'}. Matches current dream's capability that would have been lost if runCycle required a connected engine. Contract details: - CycleReport has schema_version:"1" (stable, additive) so agents consuming --json can rely on the shape - status: 'ok' | 'clean' | 'partial' | 'skipped' | 'failed'. 'clean' = ran successfully with zero activity; agents trivially detect a healthy brain. - PhaseResult.error: { class, code, message, hint?, docs_url? } (Stripe-API-tier structured failure info) when status='fail' - yieldBetweenPhases hook: awaited between EVERY phase and before return, runs even after phase failure, exceptions logged but non-fatal. Required so the Minions autopilot-cycle handler can renew its job lock between phases (prevents the v0.14 stall-death regression codex flagged). - git pull explicit: opts.pull defaults to false (cron-safe). Autopilot daemon callers opt in if user configured it. - extract phase doesn't have a dry-run mode in the underlying library function, so runCycle honestly skips extract when dryRun=true (status:'skipped', reason:'no_dry_run_support'). Schema migration v16: gbrain_cycle_locks table + idx_cycle_locks_ttl. Also appended to src/schema.sql and src/core/pglite-schema.ts for fresh installs. schema-embedded.ts regenerated via build:schema. Tests (test/core/cycle.test.ts, PGLite in-memory + mocked library functions, no DATABASE_URL required): - dryRun × phases matrix: dryRun:true reaches lint/backlinks/sync/ embed; extract is honestly skipped - Phase selection: default runs all 6 in order; --phase lint runs only lint; --phase orphans runs only orphans - Lock semantics: acquire + release on mutating phases, skip entirely for read-only selections - cycle_already_running: seeded live-holder lock → status:skipped, zero phase runs; TTL-expired holder → auto-claimed - Engine null: filesystem phases run, DB phases skip - File lock (engine=null) blocks when PID 1 holds lock with fresh mtime — exercises the PID liveness branch including EPERM - Status derivation: 'ok' vs 'clean' vs 'partial' vs 'skipped' - yieldBetweenPhases called N times, hook exceptions non-fatal Next: commit 5 rewrites dream.ts as a thin CLI alias over runCycle, commit 6 migrates autopilot daemon + jobs.ts handler to delegate to runCycle too. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dream): add gbrain dream CLI as a thin alias over runCycle `gbrain dream` is the README brand-promise command: "the agent runs while I sleep, the dream cycle ... I wake up and the brain is smarter." Cron-friendly, JSON-reportable, phase-selectable. Same maintenance cycle as `gbrain autopilot`, just scheduled differently — both converge on runCycle (added in commit 4) so there's one source of truth for what happens overnight. Contract: gbrain dream # full 6-phase cycle gbrain dream --dry-run # preview, no writes gbrain dream --json # CycleReport JSON (agent-readable) gbrain dream --phase <name> # single-phase run gbrain dream --pull # git pull before syncing gbrain dream --dir /path/to/brain # explicit brain location Cron: 0 2 * * * gbrain dream --json >> /var/log/gbrain-dream.log Behavior details: - Brain-dir resolution: requires explicit --dir OR sync.repo_path in engine config. No more walk-up-cwd-for-.git footgun that PR #309's original dream.ts had (would lint unrelated git repos). - engine=null mode preserved via cli.ts's try/catch around connectEngine — filesystem phases (lint, backlinks) still run without a DB, DB phases report skipped/no_database in the output. - status=clean prints "Brain is healthy. N phase(s) checked in Ns." status=skipped prints the reason (cycle_already_running, etc.). Partial/failed prints the phase-by-phase detail. - Exit code 1 when status=failed (cron spots real problems). 'partial' is not a failure — warnings shouldn't page you. - --help text cross-references `autopilot --install` for users who want continuous maintenance as a daemon. CLI registration (src/cli.ts): - 'dream' added to CLI_ONLY - handleCliOnly has a pre-engine branch mirroring doctor's pattern: try connectEngine() → ok path; catch → runDream(null, args) so filesystem phases still run when DB is down - Help text updated with one-line dream entry and autopilot cross-ref Tests (test/dream.test.ts, real PGLite + real library calls, no mocks to avoid `mock.module` leakage across test files): - brainDir resolution: explicit --dir wins, engine config fallback, missing + nonexistent errors - phase selection: --phase lint|orphans produces single-phase report - phase validation: --phase garbage exits 1 - output: --json parses as CycleReport with schema_version:"1" - human output mentions "Brain is healthy" on clean status - dry-run: cycle runs but DB stays untouched - exit code: clean/ok/partial do not call process.exit Also (test/core/cycle.test.ts): refactored to use beforeAll/afterAll with one shared PGLite engine per describe + truncateCycleLocks between tests. Cuts test time from ~11s to ~4s; avoids the 15-migration penalty per test that was causing parallel-suite timeout flakes. Co-Authored-By: Wintermute <wintermute@garrytan.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: v0.17.0 — autopilot + jobs delegate to runCycle (unifies the cycle) Autopilot daemon (`--inline` path) and Minions `autopilot-cycle` handler both now delegate to `runCycle` (introduced in commit 4). Three callers, one cycle definition: 1. `gbrain dream` — one-shot cron cycle 2. `gbrain autopilot` daemon inline path — scheduled cycles 3. `autopilot-cycle` Minions handler — durable queue with retry All three share: - Same 6 phases in same order (lint → backlinks → sync → extract → embed → orphans) - Same DB lock table coordination (`gbrain_cycle_locks`) - Same yieldBetweenPhases discipline (prevents v0.14 stall-death) - Same structured CycleReport output Autopilot inline path gains lint + orphan sweep that the old path skipped. Minions autopilot-cycle handler also gains lint + orphans. Users who run `gbrain autopilot --install` see 6-phase reports in `gbrain jobs get <id>` starting on next interval. No config change required. Changes: - `src/commands/autopilot.ts`: inline fallback path (~20 lines) replaces the ~22-line sync+extract+embed sequence with a single runCycle call. Uses pull:true (matches pre-v0.17 autopilot behavior). Uses setImmediate yield hook. Status/failure reporting derives from CycleReport.status. `--help` cross-references `gbrain dream` for one-shot use. - `src/commands/jobs.ts:579` (`autopilot-cycle` handler): replaces the 4-step try/catch sequence with a runCycle call. Returns `{ partial, status, report }` so `gbrain jobs get <id>` shows the full structured CycleReport. Preserves partial-failure semantic (one phase failing does NOT throw; next cycle still runs). yieldBetweenPhases yields the event loop between phases for the worker's lock-renewal timer. Release scaffolding: - VERSION: 0.16.0 → 0.17.0 - CHANGELOG.md: v0.17.0 entry in GStack voice — headline, numbers table, "what this means" paragraph, "To take advantage" block per CLAUDE.md post-ship rules. Itemized changes below the fold. Credit to @Wintermute for the original PR #309 thesis. - skills/migrations/v0.17.0.md: documents what changed for upgrading users. No mechanical action required — schema migration v16 (cycle locks table) + handler delegation both apply automatically. Includes opt-out paths for users who don't want their daemon modifying files (use `dream --phase orphans` in cron and skip autopilot-install, or other explicit configs). - CLAUDE.md: new entries for `src/core/cycle.ts` and `src/commands/dream.ts` with contract details. Tests: no new test file needed for this commit — the cycle primitive is extensively tested in test/core/cycle.test.ts (18 cases), dream in test/dream.test.ts (11), and autopilot's delegation is mechanical (calls runCycle with specific opts). The handler contract is covered implicitly: if runCycle returns a CycleReport, the handler wraps it in `{ partial, status, report }` — nothing else to assert. Verified: - `bun test test/autopilot-install.test.ts test/autopilot-resolve-cli.test.ts test/core/cycle.test.ts test/dream.test.ts` → 37 pass, 0 fail Completes the v0.17.0 feature: 6 bisectable commits on one branch (garrytan/v0.17-dream-cycle), ready to push as one PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): add runCycle + dream E2E coverage against real Postgres Gap from the v0.17 commit series: PR #321 shipped unit-level tests for runCycle (test/core/cycle.test.ts) and dream (test/dream.test.ts) but no E2E coverage that exercises the real Postgres paths. Filling that in before merge. test/e2e/cycle.test.ts (6 cases): - schema migration v16 created gbrain_cycle_locks + index - dry-run full cycle: zero DB writes + lock table empty after - live cycle: pages + chunks materialize, sync.last_commit set - concurrent cycle blocked by lock → status:'skipped' - TTL-expired lock auto-claimed (crashed-holder recovery) - --phase orphans skips lock entirely (read-only optimization) test/e2e/dream.test.ts (3 cases): - dream --dry-run --json emits valid CycleReport + DB stays empty - dream (no --dry-run) syncs pages into real DB - dream --phase orphans doesn't touch the cycle-lock table Both files mock embedBatch via mock.module so the embed phase never calls OpenAI even when the full 6-phase cycle runs (zero API cost, zero flakiness from network calls). Verified locally: - `docker run pgvector/pgvector:pg16` on port 5434 - `DATABASE_URL=... bun test test/e2e/cycle.test.ts test/e2e/dream.test.ts` → 9 pass, 0 fail - Full E2E suite (`bun run test:e2e`): 16 files, 150 tests, 0 fail - Container torn down after: `docker stop + rm gbrain-test-pg` Per CLAUDE.md E2E test DB lifecycle. These tests skip gracefully when DATABASE_URL isn't set (via hasDatabase() helper + describe.skip). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Wintermute <wintermute@garrytan.com>	2026-04-22 08:23:24 -07:00