8 Commits

Include renames

Author	SHA1	Message	Date
3 people	d6db3f0ce3	v0.41.34.0 feat(search): retrieval cathedral — max-pool + title + alias + evidence (#1657) ... * feat(search): per-page max-pool in searchVector (both engines) T1 of the retrieval-cathedral wave (supersedes #1616). Vector search returned chunk-grain top-k with no DISTINCT ON, so a page could be represented by a weak chunk while a hub page's chunks crowded a distinct page's strong chunk out of the candidate set entirely. Keyword search always pooled per page; the vector path did not. - New shared buildBestPerPagePoolCte() in sql-ranking.ts — single source of truth consumed by searchKeyword + searchVector across postgres + pglite, so the two engines can't drift (the recurring parity bug class). - searchVector both engines: compute score as a select-list expr (HNSW ORDER BY stays pure-distance), pool DISTINCT ON (slug) over the full candidate set before the user LIMIT, deterministic tiebreak (slug, score DESC, page_id ASC, chunk_id ASC). - All keyword pooling blocks refactored onto the shared builder (DRY). - Regression test: a hub page's chunks no longer crowd out a distinct page's strong chunk; results are one-per-page by best chunk. Fails on old path. Verified: real-Postgres engine-parity 22/22, PGLite hermetic suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(search): title-phrase boost (page.title first-class signal) T2 of the retrieval-cathedral wave. A query that is a phrase from a page's title ("Greek amphitheater" -> "The Mingtang - Indoor Greek Amphitheater") matched a weak body chunk instead of being recognized as a title hit. Names of things deserve weight. - New pure title-match.ts: isTitlePhraseMatch (contiguous token-run inside page.title OR exact full-title match). Precision guards: >= 2 content tokens OR exact full-title; stopword filter; token-boundary match (no raw substring). Reused by the eval later so production + bench can't drift. - applyTitleBoost post-fusion stage in hybrid.ts: reads page.title (not the brittle "first chunk"), floor-ratio-gated, stamps title_match_boost for --explain, never touches base_score (the agent's dedup confidence). - ModeBundle.title_boost knob (1.25, on in all modes - cheap gated correctness fix), search.title_boost config key, dashboard description. - KNOBS_HASH_VERSION 6 -> 7 so a boost-on cache write can't serve a boost-off lookup; all version-pin + canonical-bundle assertions updated. - 18 new tests (matcher 13 + stage 5); typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(search): page_aliases data layer (T3 foundation) Free-text alias resolution for search. gbrain stored a page's chosen names in pages.frontmatter `aliases:` JSONB but search never consulted them, so a query like "Hall of Light" or "明堂" couldn't surface the "Mingtang" page. DELIBERATELY SEPARATE from slug_aliases (re-grounded against current code): - slug_aliases: old-slug -> canonical-slug (wikilink/get_page redirect, populated only from concept-redirect conversions) - page_aliases: normalized free-text name -> canonical slug (search hop) Overloading slug_aliases would muddy two distinct semantics, so this is a new table, not an extension (honors DRY by keeping concepts separate). - src/core/search/alias-normalize.ts: ONE normalizeAlias() (NFKC + lowercase + ws-collapse + quote-strip) + normalizeAliasList() shared by the write (ingest) and read (search) paths so they match on the same key (CQ2). - Migration v108 page_aliases (source_id, alias_norm, slug); btree (source_id, alias_norm) for indexed-equality hop, NOT ILIKE; unique TRIPLE (not source_id+alias_norm) so two pages may claim one alias — collisions reported + resolved at query time, not blocked at ingest (Codex#8). Mirror in pglite-schema.ts; Postgres fresh gets it from the migration. - engine.resolveAliases(aliasNorms, {sourceId|sourceIds}) read + setPageAliases(slug, source, aliasNorms) write, both engines, source-scoped. - 17 tests: normalize round-trip, collision, source-scope, replace, clear. Ingest projection + the hybridSearch alias hop land next (T3 wiring). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(search): alias hop + ingest projection (T3 wiring) Wires the page_aliases data layer into ingest (write) and hybridSearch (read) so a query that is a page's declared chosen name surfaces that page — the named-thing class neither max-pool nor title-boost can fix (true synonyms with zero surface overlap: "Hall of Light" / "明堂" -> the Mingtang page). - Ingest projection (import-file.ts): after the page write commits, normalizeAliasList(frontmatter.aliases) -> engine.setPageAliases. Always called (even []) so removing an alias clears its row; content_hash includes non-timestamp frontmatter so alias edits reach this path, not the skip branch. Fail-soft + pre-v108-safe (isUndefinedTableError swallowed). - applyAliasHop (hybrid.ts), AFTER rerank so a named query reliably surfaces its page: FULL normalized-query exact match only (no substring/n-grams), skip >6-token prose queries, present-boost 1.10x / inject absent canonical at top-of-organic + epsilon (never absolute 1.0, D3), collisions alpha-ordered + capped at 3, fail-open on pre-v108 / lookup error (D9). Stamps alias_hit for the T4 evidence contract. - SearchResult.alias_hit attribution field. - 8 tests: inject/boost/CJK/no-match/long-skip/collision + ingest projection round-trip + alias-removal-clears. 73 pass across the T1/T2/T3 + import suite. Backfill of existing pages' aliases lands as T8 (reindex --aliases). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(search): evidence/create_safety contract + search→cheap-hybrid + per-call mode (T4) The agent-facing fix for the incident's ROOT behavior: tonight the agent read a single blended 0.64 score, decided "no strong match, safe to write a new page", and wrote a duplicate on a developed concept page. A blended RRF/cosine score is not a calibrated probability, so the don't-duplicate decision must key off WHY a page matched, not a raw number. - evidence.ts: classifyEvidence (alias_hit > exact_title_match > high_vector_match > keyword_exact > weak_semantic) + createSafetyFor (exists | probable | unknown). stampEvidence runs at the end of every hybrid return path (main + both keyword fallbacks). SearchResult gains evidence + create_safety. The agent keys don't-duplicate off create_safety='exists', not a score threshold. - search op → cheap-hybrid everywhere (D4/D15): full vector+keyword+RRF+pool+ title+alias, expansion OFF (no per-call LLM cost); `query` stays full-control. search.mcp_keyword_only escape hatch (D17) keeps the old keyword-only behavior for operators who don't want query text sent to an embedding provider. - Alias hop + evidence now also run on the keyword-only fallback paths (the named-thing fix is most valuable exactly when vector is unavailable). - Per-call `mode` (D5): honored ONLY for local/trusted callers (ctx.remote=== false) so a remote OAuth client can't escalate to costly tokenmax; local + unknown mode rejects loudly; threaded into resolveSearchMode + the cache key. - 30 tests (evidence classifier incl. before/after-incident cases, per-call mode gate, alias hop). Updated mcp-eval-capture to the new cheap-hybrid contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): reconcile `gbrain search` dispatch (T5) After T4 made the `search` op cheap-hybrid, `gbrain search "x"` already does the right thing — but `gbrain search modes/stats/tune` would have run a hybrid search for the literal word "modes" instead of opening the config dashboard (the op intercepts before the unreachable handleCliOnly dashboard path). Add a pre-dispatch interception in main(): `search` + subArgs[0] in {modes,stats,tune} → runSearch dashboard (with the v0.41.6.0 read-only connect+ dispatch 10s timeout preserved); everything else (free-text) falls through to the cheap-hybrid `search` op. Subprocess test pins all three routes: modes/stats → dashboard, free-text → search op ("No results", not "Unknown subcommand"). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(eval): NamedThingBench retrieval-quality gate (T6) The eval that makes the retrieval-maxpool incident impossible to reintroduce silently. 7 query families, each a failure class the incident exposed: title-substring, generic-to-named, alias-synonym, multi-chunk-dilution, short-vs-rich, graph-relationship, hard-negative. - src/eval/retrieval-quality/harness.ts: pure scoring (Hit@1/Hit@3/MRR per family) + injected SearchFn (CLI uses hybridSearch; tests stub it) + evaluateGate. D12 gate: hard-gate the families that ARE the bug from day one (title-substring Hit@1>=0.95, alias-synonym Hit@1>=0.98, dilution Hit@3=1.0), warn-then-enforce the softer families. Env-overridable floors. - `gbrain eval retrieval-quality <fixture.jsonl> [--json] [--source]` + dispatch in eval.ts. Exit 0 PASS / 1 FAIL / 2 USAGE. - Synthetic fixture (placeholder names only, privacy-grep guarded) + hermetic gate test: seeds a synthetic brain, forces the keyword+title+alias path (embed transport stubbed to throw — free, deterministic), asserts the bug families pass. The vector max-pool guarantee is pinned separately by searchvector-maxpool.test.ts. - CI gate: the hermetic test is a normal unit test, so it runs in every PR shard — the gate is live on every change. - 23 tests (harness unit + hermetic gate + fixture privacy guard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(telemetry): rank-1 score drift signal (T7) Standing observability so a retrieval regression is caught before a human hits it in chat (like tonight). Aggregate columns on search_telemetry (NOT per-query rows, D10): sum_rank1_score + count_rank1 + 3 coarse buckets (<0.6 / 0.6-0.85 / >=0.85). The mean rank-1 base_score is the headline; a downward drift = retrieval quality regressing. - hybrid.ts: capture rank-1 base_score at all three return paths, thread through emitMeta → recordSearchTelemetry opts (like results_count). - telemetry.ts: Bucket + record + flush ON CONFLICT-add + readSearchStats expose avg_rank1_score (null when no samples — no NaN) + rank1_distribution. - Migration v109 ADD COLUMN IF NOT EXISTS (both engines; search_telemetry lives only in migration v57, so the v57+v109 chain covers fresh + upgrade). Columns exempted in schema-bootstrap-coverage (no forward-ref index → no bootstrap need). - `gbrain search stats` surfaces the avg + bucket line; JSON envelope auto-carries the fields. "true-positive" wording dropped per Codex#14 — production has no labels, so this is an unlabeled rank-1 score histogram; labeled calibration lives in NamedThingBench (T6). - 3 round-trip tests (mean+buckets, no-result excluded, empty=null). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(reindex): gbrain reindex --aliases backfill (T8) Import-time projection (T3) covers new + changed pages; this backfills EXISTING pages whose frontmatter `aliases:` predate v108 / the projection. Walks listAllPageRefs (cheap cross-source (source_id, slug) enumeration), reads each page's frontmatter aliases, writes page_aliases via setPageAliases. Idempotent (setPageAliases replaces) so re-running is convergent — no op-checkpoint needed (fast, no embedding). --dry-run reports would-write counts, --source narrows, --limit caps, --json envelope, progress reporter. Wired into the `reindex` dispatch alongside --markdown / --multimodal. 4 tests: backfill from array + comma-scalar frontmatter, --dry-run writes nothing, idempotent second run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(search): pre-migration fail-open regression (T9) Pins that pre-v108 brains (no page_aliases table) keep working: applyAliasHop returns input unchanged + doesn't throw, importFromContent with frontmatter aliases still imports (projection swallows table-missing via isUndefinedTableError), and resolveAliases surfaces the error for the caller to catch. Completes the T9 mandatory regression set (dilution → searchvector-maxpool, dispatch → cli-search-dispatch, MCP contract → mcp-eval-capture, engine parity → engine-parity 22/22, pre-migration → here). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(search): Phase-0 retrieval diagnostic — `gbrain search diagnose` (T0) The operator-facing trace the user runs against the production brain to pin which retrieval layer surfaces (or misses) a target page — the diagnostic the plan front-loaded so we don't ship a fix that doesn't move the incident. `gbrain search diagnose "<query>" --target <slug> [--json] [--source]` reports, for the target: keyword rank+score, vector rank+score (skipped/graceful if no embedding provider), whether the query is a registered alias, and the hybrid final rank + evidence + create_safety + which boosts fired (title/alias). The verdict names the layer that surfaces the target at rank 1 (or "none"), telling you whether the lever is max-pool/innerLimit (vector) vs title/alias. Wired into the `search` dispatch alongside modes/stats/tune (60s timeout since it runs real retrieval). 2 hermetic tests (alias-query trace + title-phrase trace). For the Mingtang incident, run: gbrain search diagnose "Greek amphitheater" --target projects/new-greek-theater/concept_v0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(retrieval): corrected incident record + named-thing layers + glossary (T10) - RETRIEVAL_MAXPOOL_INCIDENT.md: replaces closed PR #1616's RFC with the verified record — what happened, the disease, the corrections to the RFC's mechanics (search was keyword-only, --mode unthreaded, hybrid already pooled at dedup, aliases dead to search), the four-layer fix that shipped, and the triage commands (search diagnose / reindex --aliases / search stats / eval retrieval-quality). - RETRIEVAL.md: new "Named-thing retrieval" section documenting per-page pool + title boost + alias hop + the evidence contract, reconciling the doc with the shipped pipeline (closes the doc/reality gap). - metric-glossary.ts + regenerated METRIC_GLOSSARY.md: Hit@1, Hit@3, avg_rank1_score (drift signal, not labeled accuracy), and create_safety (the evidence contract) now carry plain-English glossary entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(eval): NamedThingBench fixture privacy guard via slug-shape (T6 fixup) The banned-name literal list itself tripped check-privacy/check-test-real-names. Replace it with the load-bearing assertion: every fixture slug must be an *-example placeholder (no real brain page can be referenced). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(search): source-isolation in per-page pool + alias hop (P0, codex adversarial) Codex outside-voice caught two source-isolation P0s in the retrieval wave — the exact class the v0.34.1 seal guards. Both fixed before merge. P0-1: buildBestPerPagePoolCte pooled on `slug` alone. In a federated brain, two pages with the same slug in different sources collapsed before ranking/pagination (the neighbor-source page dropped). Now DISTINCT ON (COALESCE(source_id,'default'), slug) — composite key matching dedup.ts's pageKey. Also fixes the PRE-EXISTING keyword-path bug (best_per_page was slug-only before this wave); real-PG parity 23/23. P0-2: the alias hop dropped source_id. resolveAliases returned bare slugs and applyAliasHop hydrated via getPage(slug, undefined), so a federated caller could get the default-source page injected or the right allowed-source page suppressed. resolveAliases now returns {slug, source_id} pairs; applyAliasHop matches by (source_id, slug) and fetches each canonical in its OWN source. Regression tests: alias hop boosts only the aliased source (not same-slug in another source); resolveAliases keeps cross-source same-slug distinct. Deferred as documented tradeoffs (TODO): evidence high_vector_match label uses blended base_score not pure cosine; deep-pagination candidate budget is chunk-bounded; telemetry writes swallow errors pre-v109 on rolling deploys. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.41.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: v0.41.30.0 retrieval cathedral — CLAUDE.md key files + llms regen Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: renumber release v0.41.30.0 → v0.41.34.0 (queue moved) Version trio + CHANGELOG header + CLAUDE.md key-file annotations + TODOS heading + regenerated llms bundles, all moved to 0.41.34.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): restore glossary roster + harden facts-anti-loop hook budget Two CI failures surfaced after the master merges that brought the branch to 111 migrations: 1. shard 1 — `ALL_METRICS roster > matches the renderer output (no orphans)`: the merge took master's `renderMetricGlossaryMarkdown` whose `groups` array lacked this branch's 4 retrieval-quality keys (hit@1, hit@3, avg_rank1_score, create_safety). `ALL_METRICS` (derived via Object.keys) kept them, so the roster test saw 4 orphans. The freshness check (check:eval-glossary) passed because renderer-output == committed doc — it can't catch a renderer that drops a metric; the roster test can. Restored the "Retrieval-Quality / Evidence Metrics (NamedThingBench)" group + regenerated docs/eval/METRIC_GLOSSARY.md. 2. shard 2 — facts-anti-loop's two engine-dependent put_page tests failed while the two engine-free extractFactsFromTurn tests passed (the signature of a partially-failed beforeAll). This file has a documented PGLite-cold-start-under-deep-shard-load timeout history; the 30s budget was tuned for 95 migrations and the chain is now 111. Bumped to 60s. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): isolate facts-anti-loop in its own process (serial) Follow-up to the prior hook-timeout bump, which was the wrong theory: the [58ms]/[71ms] body times in the re-run prove beforeAll did NOT time out — the engine connects and the two put_page tests run and fail for real, while the two engine-free extractFactsFromTurn tests in the same file pass. put_page (via dispatchToolCall) touches process-global singletons (the facts queue + the AI gateway used by importFromContent's embed step). Some sibling file in the 78-file shard-2 process leaves residual global state that makes put_page's pre-backstop path fail on the CI runner. The failure is NOT reproducible alone, in a Linux oven/bun:1 container, or in a full local shard-2 run (1172 pass) — only on the GitHub runner, deterministically. Per CLAUDE.md's test-isolation rules, a test coupled to shared process state belongs in its own process. Renamed to *.serial.test.ts so it runs in the dedicated serial-tests job (scripts/run-serial-tests.sh spawns a fresh `bun test` per serial file), where it passes deterministically; test-shard.sh excludes serial files from the matrix. Updated the comment to reflect the real cause and refreshed the test-weights.json key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): close cross-file gateway-config pollution in test shards The prior serial-move theory was incomplete. The real, single root cause behind all three shard failures (2, 5, 10) is cross-file AI-gateway config pollution within a shard's bun process: - A test calls configureGateway() and doesn't restore the gateway on exit. The legacy-embedding preload pins OpenAI/1536 ONCE at process start and re-pins per-test ONLY when the gateway slot is empty — so a leaker that reconfigured the gateway to the v0.37 default (zeroentropyai:zembed-1 / 1280-d) and never reset poisons every later file in the shard. - Victim A (shard 5, test/search/searchvector-maxpool.test.ts): runs initSchema in beforeAll under the leaked gateway → content_chunks.embedding becomes vector(1280) → inserting its hardcoded 1536-d basis vectors throws pgvector CheckExpectedDim. - Victims B/C (shard 10 facts-backstop-gating, shard 2 facts-anti-loop): put_page's importFromContent embeds by design (embed failure PROPAGATES, Codex C2). Under a leaked fake-key gateway the embed step 401s and put_page returns isError → the backstop assertions fail. My branch's shard re-partition (added test files + weight changes) merely co-located leakers with victims; the hazard was latent. Fixes (root cause + self-sufficient victims): - test/search/rerank.test.ts (the shard-5 leaker): add afterAll(resetGateway). Its stub omits embedding_model, so it fell back to the ZE/1280 default; now it restores the empty slot so the preload re-pins legacy for the next file. - test/search/searchvector-maxpool.test.ts: pin configureGateway(openai/1536) in beforeAll BEFORE initSchema (initSchema runs before any preload beforeEach, so it can't rely on the inherited slot). - test/facts-backstop-gating.test.ts + test/facts-anti-loop.test.ts: reset the gateway in beforeEach so put_page's embed is a graceful no-op; reverted anti-loop from the serial quarantine back into the matrix (the serial move was the wrong fix for a gateway-state problem). Validated deterministically: a non-resetting leaker that poisons the gateway to ZE, run first in one bun process, no longer breaks any of the three victims (14/14 pass). verify 29/29, typecheck clean, isolation lint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-05-30 12:07:38 -07:00
3 people	5d42f3295e	v0.41.22.0 feat: type-unification cathedral — 94 types → 15 canonical (closes #1479) (#1542) ... * Merge branch 'master' into garrytan/type-taxonomy-unification Resolve VERSION, package.json, CHANGELOG conflicts with v0.41.22.0 on top, preserving master's v0.41.19.0 entry below. * feat: v0.41.22.0 type-unification cathedral — collapse 94 types to 15 (closes #1479) Ships gbrain-base-v2 as the new install default (15 canonical types: 14 + note catch-all) and the unify-types PROTECTED Minion handler that runs the gbrain-base→v2 migration end-to-end on existing brains. What this delivers: - gbrain-base-v2.yaml standalone schema pack (no extends:) with 14 canonical page_types + 9 cluster mapping_rules + catch-all sentinel - 3 new schema-pack primitives: runRetypeCore (chunked UPDATE with legacy_type stamping), runPageToLinkCore (edge-shaped pages → link rows), runPageToAliasCore (concept-redirect → slug_aliases) - rewriteLinksBatch for N-pair atomic FK rewrite - Migration v104 slug_aliases table (forward-bootstrap probed on both engines for safe upgrade chain) - New engine method resolveSlugWithAlias(slug, sourceOrSources) on both Postgres + PGLite with multi-source ambiguity warning - inferTypeAndSubtypeFromPack overload + subtypes: + mapping_rules: + migration_from: schema-pack manifest extensions - findPackSuccessors version-range walker (1.x / 1.0.x / exact match) - expandTypeFilter for --type back-compat (D14): legacy aliases route through mapping_rules → canonical+subtype before the SQL filter fires - 3 new onboard checks: pack_upgrade_available, type_proliferation, dangling_aliases (source-scoped per F12) - unify-types Minion handler (PROTECTED, manual_only via render.ts allowlist per D17): retype-explicit → retype-catch-all → page-to-link → page-to-alias → final sync → active-pack flip - alias_resolved 1.05x post-fusion search boost stage; KNOBS_HASH_VERSION bumped 5→6 (one-time cache miss on upgrade, self-healing in TTL) - ELIGIBLE_TYPES for facts extraction extended with v2 canonicals (codex F-ELIGIBLE: blocker not v0.43 follow-up) Tests: 79 new unit/integration cases + 3 E2E cases covering all 9 production clusters end-to-end. 124-case verification on the cache-key + build-llms fixes. KNOBS_HASH_VERSION assertions updated in 3 tests. Plan: ~/.claude/plans/system-instruction-you-are-working-transient-elephant.md (16 locked decisions D1-D17, 12 baseline fixes F7-F21 absorbed from codex outside voice). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: CI verify failures — system-of-record allow-comment + schema-unify manifest registration Two CI failures on PR #1542: 1. check:system-of-record flagged page-to-link.ts:207 addLinksBatch as a direct write to a derived table. The call IS the reconcile surface for page_to_link mapping_rules — it converts edge-shaped pages into canonical link rows under the PROTECTED unify-types Minion handler, source-scoped, atomic per-rule. Added the canonical `// gbrain-allow-direct-insert: <reason>` comment on the same line. 2. check:resolver emitted 11 orphan_trigger warnings for `schema-unify` because the skill was added to skills/RESOLVER.md without a corresponding entry in skills/manifest.json. Added the registration under the existing skills[] array. bun run verify: 28/28 checks pass locally. * fix: CI test failures — schema-unify conformance + eligibility regression Six test failures across shards 2 + 10 on PR #1542: 1. resolver.test.ts: round-trip parser requires frontmatter triggers to be quoted (`- "..."` or `- '...'`). schema-unify shipped with bare YAML strings; quoted the 10 triggers to round-trip correctly. 2. skills-conformance.test.ts (×3): schema-unify SKILL.md was missing the required Contract, Anti-Patterns, and Output Format sections that every conformant skill must declare. Added all three: - Contract: inputs / outputs / side effects / failure modes - Anti-Patterns: 5 DON'Ts including the autopilot trust boundary - Output Format: per-phase stderr lines + celebration summary + JSON envelope shape 3. facts-eligibility.test.ts (×2): the v0.41.22 ELIGIBLE_TYPES expansion added `concept` to the eligible list, but the existing test suite pins concept as rejected (it's `extractable: true` in the schema pack but the v0.41.11 contract documented this as "cosmetic on the backstop path because backstop uses hardcoded ELIGIBLE_TYPES"). Removed `concept` from the expansion; other v2 canonicals (media, tweet, atom, analysis) stay. Comment updated to document the deliberate omission. All 6 failing tests now pass locally (370/370 across the 3 affected files). bun run verify: 28/28 checks green. * fix: harden findPackSuccessors test against shard pollution CI shard 8 reported 1 fail (1.00ms — too fast for any real loadActivePack file I/O) on `finds gbrain-base-v2 as successor of gbrain-base@1.0.0`. Local triple-run passes 9/9 in isolation. Root cause: the existing afterEach reset clears the module-level pack cache AFTER each test, but the FIRST test in the file inherits whatever state sibling files in the same bun shard process left behind. With 24+ schema-pack tests in shard 8 (mutate, mutate-audit, best-effort, registry-reload, manifest-v041_2, etc.) running before this file, the first test can read a poisoned cache. Fix: add `beforeEach(_resetPackCacheForTests)`. Two-sided reset guarantees clean state regardless of file ordering within the shard. bun run verify: 28/28 checks pass. * fix: quarantine two flaky tests to serial runner CI shard 1 + shard 8 each surfaced one intermittent failure: shard 1: buildBrainTools > execute() on put_page with valid namespace shard 8: findPackSuccessors > finds gbrain-base-v2 as successor Both pass cleanly in isolation. Both are concurrency races against shared in-shard state: - brain-allowlist.test.ts shares a singleton PGLiteEngine across 18 tests with a beforeEach DELETE FROM pages. With max-concurrency=4, two put_page tests can interleave their TRUNCATE + write phases, so the auto-link/extract sub-steps inside put_page race against the sibling test's DELETE. - schema-pack-find-pack-successors.test.ts reads bundled YAML packs via loadActivePack. The module-level pack cache is shared across parallel tests in the same shard; the previous beforeEach reset helped but didn't fully isolate against concurrent file reads under CI load. Fix per CLAUDE.md test-isolation lint rule R2 (concurrency-fragile files belong in the .serial.test.ts quarantine): rename both files to *.serial.test.ts. Serial runner picks them up at max-concurrency=1. 49/49 serial files pass locally. 28/28 verify checks pass. * fix: quarantine embed-stale test to serial runner CI shard 9 reported 6 failures, all from the embedStaleForSource describe block, all ~120-150ms each — classic shared-engine concurrency race shape. Passes 7/7 locally in isolation. Root cause: embed-stale.test.ts shares a singleton PGLiteEngine across 7 tests with beforeEach resetPgliteState. Under bun's max-concurrency=4 in the parallel shard, two tests can interleave their TRUNCATE + seedPage + upsertChunks + embedStaleForSource flow, so one test's stale-chunk count sees another test's mid-flight writes. Same fix as brain-allowlist.serial.test.ts and schema-pack-find-pack-successors.serial.test.ts: rename to *.serial.test.ts so the serial runner picks it up at max-concurrency=1. bun run verify: 28/28 checks pass. 7/7 embed-stale tests pass via serial. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-27 07:01:28 -07:00
3 people	d28be5d091	v0.40.4.0 feat(search): selective graph signals + per-stage attribution + audit-writer unification (#1300) ... * v0.40.4.0 T1: shared audit-writer primitive Extract createAuditWriter() helper. Five hand-rolled JSONL audit modules (rerank-audit, shell-audit, supervisor-audit, audit-slug- fallback, phantom-audit) duplicated the same ISO-week filename math, best-effort write loop, and read-current-plus-previous-week loop. T2 refactors all 5 onto this primitive. Behavior preservation: filename format, JSONL line shape, mkdir recursive, appendFileSync utf8, stderr-on-failure all byte-identical to the existing modules so their tests pass unchanged. resolveAuditDir() moves here from shell-audit.ts; shell-audit.ts will re-export for back-compat (T2). Honors GBRAIN_AUDIT_DIR with whitespace-trim, falls back to ~/.gbrain/audit/. Test coverage: 22 cases covering ISO-week math + year-boundary edges (2027-01-01 → 2026-W53), env override, mkdir-recursive, fail-open stderr-warn shape, cross-week readback, corrupt-row skip, non-finite- ts skip, round-trip with nested fields, computeFilename + resolveDir accessors. Plan ref: D5=B audit unification cathedral expansion. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T2: refactor 5 audit modules onto shared writer Replace the duplicated ISO-week filename math + best-effort write loop + read-current-plus-previous-week loop in: - src/core/rerank-audit.ts (rerank-failures-*.jsonl) - src/core/audit-slug-fallback.ts (slug-fallback-*.jsonl) - src/core/minions/handlers/shell-audit.ts (shell-jobs-*.jsonl) - src/core/minions/handlers/supervisor-audit.ts (supervisor-*.jsonl) - src/core/facts/phantom-audit.ts (phantoms-*.jsonl) All five now delegate file I/O to createAuditWriter from T1. Public API preserved bit-for-bit: - logRerankFailure, readRecentRerankFailures, computeRerankAuditFilename - logSlugFallback, readRecentSlugFallbacks, computeSlugFallbackAuditFilename - logShellSubmission, computeAuditFilename, resolveAuditDir - writeSupervisorEvent, readSupervisorEvents, computeSupervisorAuditFilename plus isCrashExit, summarizeCrashes, CrashSummary (domain-specific helpers stay in supervisor-audit.ts; only file I/O moves) - logPhantomEvent, readRecentPhantomEvents, computePhantomAuditFilename Domain-specific behavior preserved: - audit-slug-fallback emits per-call stderr (D7 dual logging) in the caller; the shared writer is failure-only stderr - rerank-audit truncates error_summary to 200 chars before write - phantom-audit spreads optional fields conditionally (skip undefined) - supervisor-audit keeps single-file readback (no cross-week walk) to preserve pre-v0.40.4 doctor assertions resolveAuditDir lives in src/core/audit/audit-writer.ts; shell-audit.ts re-exports it so existing imports keep working (every other audit module + gbrain-home-isolation.test.ts + minions.test.ts + minions-shell.test.ts pull resolveAuditDir from shell-audit.ts). Operator-visible drift: rerank-audit stderr line drops the 'rerank-failure audit' qualifier — was '[gbrain] rerank-failure audit write failed (...)' now '[gbrain] write failed (...); search continues'. Stderr is human-debugging, not machine-parsed; the file written gives the qualifier away in `tail -f audit/*`. Test coverage: 128/128 audit-touching tests pass unchanged. Plan ref: D5=B audit unification cathedral expansion. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T3: getAdjacencyBoosts engine method (PG+PGLite parity) Add BrainEngine.getAdjacencyBoosts(pageIds) returning Map<page_id, AdjacencyRow{hits, cross_source_hits}>. Returns ALL pages with hits >= 1 (callers apply their own threshold). Cross-source semantic (D15=A): cross_source_hits EXCLUDES the target page's own source. A page in source A linked from 2 pages in source A reports cross_source_hits = 0. Linked from 1 in source B + 1 in source C reports 2. Source-scope contract: pageIds MUST already be source-scoped by the caller. Method does NOT filter by source_id. The in-set restriction makes cross-source leakage impossible by construction. JSDoc spells this out; same trust posture as cosineReScore's chunk_id handling. COALESCE(p.source_id, 'default') on both target and from-page sides for defense-in-depth even though pages.source_id is NOT NULL today. JSDoc/SQL contract alignment (codex #2): HAVING >= 1 matches the "returns ALL pages with hits >= 1" contract; threshold of 2 is the caller's call in applyGraphSignals. Known limitation (codex #15): cross_source_hits cannot distinguish "genuinely linked from another team" from "mirrored imports from another source." T-todo-4 captures the v0.41+ refinement. SearchResult type extension (D4=A flat fields, D12=A attribution): - graph_adjacency_hits, graph_cross_source_hits, graph_session_demoted, graph_session_prefix - base_score, backlink_boost, salience_boost, recency_boost, exact_match_boost, graph_adjacency_boost, graph_cross_source_boost, session_demote_factor, reranker_delta All optional; T4-T6 populate them. Test coverage: 7/7 hermetic PGLite cases. Empty input, singleton, same-source hub, cross-source attribution including the "linked-only-from-other-source" case (widget in source b, linked from alice+bob in source a → cross_source_hits=1), JSDoc HAVING>=1 contract. Postgres parity asserted by SQL-shape identity (will get a mirror Postgres E2E in T10's eval gate work via DATABASE_URL when set; PGLite hermetic case shipped now). NULL source_id COALESCE branch noted as untestable in current PGLite schema (pages.source_id is NOT NULL); kept as defense-in-depth. Plan ref: T3 in v0.40.4.0 wave plan; D1=A, D3=A, D15=A. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T4+T11: applyGraphSignals 4th stage in runPostFusionStages New file src/core/search/graph-signals.ts. Three signals: 1. Adjacency-within-top-K (×1.05): hits >= 2 inbound from in-set. 2. Cross-source adjacency (×1.10, stacks): cross_source_hits >= 2. Dormant on single-source brains. 3. Session diversification (×0.95): if multiple top-K share a slug prefix, keep highest scoring, DEMOTE the rest. NOT amplify — codex caught the original framing was backwards (amplification of redundancy makes the cited "weak chunks compete for budget" problem worse, not better). Conservative magnitudes (D14=B): 1.05/1.10/0.95. Score-distribution probe (onScoreDistribution) collects min/p25/p50/p75/p95/max + reorder_band_width to feed T-todo-2 magnitude calibration wave. Slot: 4th stage inside runPostFusionStages (hybrid.ts:248), AFTER backlink/salience/recency, pre-dedup. Inherits the v0.35.6.0 floor-ratio gate from computeFloorThreshold — this is the structural protection that prevents a low-cosine hub from outranking a strong non-hub (codex T2 / D1=A). PostFusionOpts extends with graphSignalsEnabled, onGraphMeta, onScoreDistribution. Caller (hybridSearch in subsequent T5 work) resolves graph_signals from the mode bundle. Source-scope contract preserved: getAdjacencyBoosts takes raw page_ids, no source filter. Adjacency is in-set restricted so cross-source leakage is impossible by construction (D3=A). Fail-open: engine throw → JSONL audit row via shared createAuditWriter (T1/T2 primitive, featureName='graph-signals-failures') + meta.errored + caller's results unchanged. Session diversification ALSO skips on failure (predictable all-or-nothing posture). Mutation note (codex #9): score mutated in place. base_score must be stamped at runPostFusionStages entry BEFORE this stage so eval-capture sees pre-boost score (T6 attribution wave). Test coverage (24 cases, including T11 IRON RULE regression): - sessionPrefix multi/single/empty cases - computeScoreDistribution percentile math - Disabled + empty short-circuits - Adjacency hit, no-hit, cross-source stacking, cross-source alone - Session diversification 3-share + single-segment + singleton - Test seam injection (no engine call) - Fail-open: throw → audit row + meta.errored + unchanged - Empty Map → session still runs - Score-distribution always emits when enabled - Meta carries fire counts + duration_ms - Missing page_id silently skipped from dedup set - **T11 IRON RULE regression (3 cases):** * weak hub BELOW floor_threshold does NOT get boosted past above-floor non-hub (the bug class the floor gate exists for) * hub AT floor still gets boosted (gate is < not <=) * NaN score → NaN >= threshold is false → no boost Plan ref: T4 + T11 in v0.40.4.0 wave plan; D1=A, D2=A, D11=B, D14=B, D9=A, D5=B. Codex outside-voice #1 + #2 + #6 + #8 + #9 addressed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T5: graph_signals mode-bundle knob + KNOBS_HASH bump 3→4 ModeBundle gains graph_signals: boolean. Per-mode defaults: - conservative: false (cost-sensitive tier) - balanced: true (the wave's primary surface for default-on) - tokenmax: true (power-user tier, capstone fit) SearchKeyOverrides + SearchPerCallOpts gain optional graph_signals field. resolveSearchMode picks via the standard per-call → config override → mode bundle chain. loadOverridesFromConfig parses 'search.graph_signals' from the config table ('1' or 'true' → true). SEARCH_MODE_CONFIG_KEYS adds the key so `gbrain search modes --reset` clears it alongside other knobs. KNOBS_HASH_VERSION bump 3→4 (append-only per CDX2-F13). New `gs=` parts entry appended AFTER cross-modal + column + prov entries. A graph-on cache write cannot be served to a graph-off lookup — mid-deploy hit-rate dip clears within cache.ttl_seconds (3600s). src/commands/search.ts KNOB_DESCRIPTIONS gains graph_signals entry so `gbrain search modes` dashboard renders the new knob. Test coverage: - test/search-mode.test.ts (+ 8 new cases): per-mode defaults canonical, config override both directions, per-call override wins, knobsHash distinct for on/off, config key registered, attributeKnob reports per-call + mode sources correctly. - test/search/knobs-hash-reranker.test.ts: version assertion bumped 3→4 with v0.40.4 rationale comment. - test/cross-modal-phase1.test.ts: version assertion bumped 3→4 with v0.40.4 rationale comment. - Canonical-bundle assertions updated to include graph_signals in expected shape (3 cases). 50/50 search-mode tests pass. 45/45 cross-modal pass. 17/17 knobs-hash-reranker pass. 10/10 balanced-reranker pass. Plan ref: T5 in v0.40.4.0 wave plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T6: per-stage attribution stamping in every boost Every boost stage that mutates SearchResult.score now stamps a field recording WHAT it multiplied: - applyBacklinkBoost → backlink_boost (skipped when count == 0) - applySalienceBoost → salience_boost (skipped when score == 0) - applyRecencyBoost → recency_boost (skipped on evergreen prefix) - applyExactMatchBoost → exact_match_boost (skipped on no-match OR when intent's exactMatchBoost == 1.0 no-op) - runPostFusionStages → base_score stamped ONCE at entry, BEFORE any boost mutates r.score. Idempotent: caller-pre-stamped value preserved. Empty-results short-circuit unchanged. - applyReranker → reranker_delta = original_index - new_index (positive = rank improved; raw rerank score stays in rerank_score) - applyGraphSignals → graph_adjacency_boost, graph_cross_source_boost, session_demote_factor (T4 already stamped these) Why: feeds the T7 `gbrain search --explain` formatter so it can attribute the final score to its components. Without these stamps, "why did this rank where it did?" is grep-and-guess. SearchResult.reranker_delta doc updated to clarify it's a RANK delta (positive = improved), not a score delta. The raw relevance score stays in `rerank_score` (untyped, for back-compat with telemetry that already reads it). Test coverage: 16 new cases in test/search/attribution-stamping.test.ts. Pins: every boost stamps when it fires AND skips stamping when it doesn't (no false attribution on no-op stages). base_score idempotency preserved. reranker_delta computed correctly across rank-improved + rank-degraded cases. All 178/178 search tests pass (no regressions). Plan ref: T6 cathedral expansion in v0.40.4.0 wave plan; D12=A. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T7: gbrain search --explain per-stage attribution New file src/core/search/explain-formatter.ts renders SearchResult[] as a multi-line breakdown of how the final score was formed: 1. people/alice (score=12.4) base=10.2 (rrf+cosine) + backlink ×1.08 + salience ×1.05 + adjacency ×1.05 (hits=3) + cross_source ×1.10 (other_sources=2) ↑ reranker rank +2 = final 12.4 Reads the boost_* / base_score / *_hits fields populated by T4 + T6. Empty path: "no boosts applied" when no stage stamped anything. Session demote rendered with `-` prefix (not `+`) so the demotion direction is visually distinct from boosts. CliOptions gains `explain: boolean`; parseGlobalFlags recognizes `--explain` anywhere in argv. cli.ts formatResult for `search` + `query` cases reads CliOptions.explain via the module-level singleton and routes to formatResultsExplain when set. Lazy import keeps the hot path narrow for the common non-explain case. Number formatting: 4-decimal precision, trailing zeros stripped ('1.0000' → '1', '0.1234' → '0.1234'). NaN preserved as 'NaN'. Test coverage: - test/search/explain-formatter.test.ts: 19 cases pin output format. Each boost type renders correctly, every-stage stacking composes, reranker_delta=0 doesn't render, empty list short- circuits, rank numbering 1-based, number formatting edge cases. - test/cli-options.test.ts: 3 new cases for --explain parsing (basic, absent default, any-argv-position). Existing CliOptions literals in test/cli-options.test.ts + test/thin-client-upgrade-prompt.test.ts updated for new required explain field. JSON envelope unchanged — the same attribution fields surface in existing --json output via JSON.stringify; no separate JSON formatter needed. Plan ref: T7 cathedral expansion in v0.40.4.0 wave plan; D12=A + D6=A. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T8: doctor check graph_signals_coverage New checkGraphSignalsCoverage in src/commands/doctor.ts. Wired into both runDoctor (local engine) and doctorReportRemote (HTTP MCP / JSON path) so local AND remote-server brains both surface the metric. Logic: 1. Resolve active graph_signals setting: config override 'search.graph_signals' wins, else mode bundle default ('search.mode' → conservative=false, balanced/tokenmax=true). 2. When disabled → silent ok ("disabled — coverage not checked"). Avoids polluting doctor output on installs that don't use the feature. 3. When enabled, compute global inbound-link density: COUNT(DISTINCT to_page_id) / COUNT(*) across non-deleted pages. 4. <10% → warn ("signal will rarely fire") with paste-ready `gbrain extract all` fix hint. 5. >=30% → ok ("fire on most queries") with metric. 6. 10-29% → ok ("fire occasionally") with metric. Known limitation (codex outside-voice #14): global density is an imperfect proxy for "top-K subgraphs have enough edges to fire." T-todo-5 captures the v0.41+ refinement that measures actual fire rate from search-stats after 30 days of data. Best-effort: SQL errors → warn with the underlying message. Never breaks doctor. Test coverage (7 new cases in test/doctor.test.ts): - conservative mode → silent ok regardless of coverage - balanced default + 0 links → warn at 0% with fix hint - balanced default + 40% inbound → ok "fire on most queries" - balanced default + 20% inbound → ok "fire occasionally" - explicit search.graph_signals=false overrides mode default - empty brain → ok with explanation - check is wired into runDoctor (source-grep regression guard) All 55/55 doctor.test.ts cases pass. Plan ref: T8 in v0.40.4.0 wave plan; D6=A. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T9: gbrain search stats graph_signals section runStatsSubcommand in src/commands/search.ts gains a graph_signals section in both --json and human output: Graph signals: enabled: true (mode default) failures: 3 fail-open event(s) ECONNREFUSED 2 timeout 1 Data sources: - config: 'search.graph_signals' override → enabled + source=config, otherwise mode-bundle default → enabled + source=mode_default. - JSONL audit: readRecentGraphSignalsFailures(days) returns events; failures_count is len, failures_by_reason buckets by first word of error_summary (e.g. 'ECONNREFUSED', 'timeout'). JSON envelope (schema_version 2 unchanged; graph_signals is a new sibling property of stats, so consumers reading the existing fields keep working): { "schema_version": 2, ...stats..., "graph_signals": { "enabled": bool, "source": "config" | "mode_default", "failures_count": int, "failures_by_reason": { reason: count } }, "_meta": { metric_glossary: { ..., graph_signals_enabled: ..., graph_signals_failures_count: ... } } } Fire-rate metrics (adjacency_fires, cross_source_fires, session_demotions) and score-distribution stats are NOT in this section yet — they require telemetry-table writes from the applyGraphSignals onMeta callback. Wired in v0.41+ via T-todo-2 calibration wave (the wave that needs them). For v0.40.4: status + error count is the actionable surface for "is graph_signals on, and is it failing?" Human output: prints the section after the existing stats block. Edge case: when total_calls is 0 BUT graph_signals is enabled OR has historical failures, still prints the section so operators don't lose the signal on a brain with no telemetry yet. Test coverage (6 cases in test/search/search-stats-graph-signals.test.ts): - search.graph_signals=true → enabled true, source=config - mode=conservative → enabled false, source=mode_default - no config → enabled true (balanced default), source=mode_default - JSONL failures bucketed by first word of error_summary - empty audit → failures_count 0, empty failures_by_reason - human output includes "Graph signals:" header Plan ref: T9 in v0.40.4.0 wave plan; D6=A. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T10: eval gates (longmemeval-mini A/B + paired bootstrap) New test/e2e/graph-signals-eval.test.ts runs each longmemeval-mini question twice (graph_signals off, graph_signals on) and asserts: Gate 1 (QUALITY) — paired bootstrap, 10,000 resamples: - If signals-on is significantly WORSE than off (delta < 0 AND p < 0.05) → fail. - Otherwise pass. p>=0.05 either direction OR delta >= 0 → ok. Gate 2a (CHANGE-MAGNITUDE): mean Jaccard@5 over result-set overlap must be >= 0.5. If results overlap less than half, the change is too large and needs human review before default-on. Gate 2b (CHANGE-MAGNITUDE): top-1 stability rate >= 0.7. If 30%+ of top picks change, hard look required. Gate 3 (HARD ABSOLUTE FLOOR): recall@5 drop <= 5pt. Catastrophic regression catch (codex outside-voice #18 — addresses the "top-5 must not drop at all" brittleness on tiny fixtures). Bootstrap implementation: - Per-question observation is binary (recall@5 hit/miss). - Paired pairing on question_id between on/off branches. - Centered distribution under null (subtract observed mean) per standard paired-bootstrap-shift approach for binary outcomes. - Two-tailed p-value: |resampled delta| >= |observed delta|. - Deterministic seeded RNG so test runs are stable across CI. pairedBootstrapPValue exported as a pure function with separate tests for edge cases (empty input, all-equal, strong positive, strong negative, determinism). Reusable from future calibration waves. Hermetic: in-memory PGLite via createBenchmarkBrain + resetTables between questions. No API keys needed (--no-embed import path exercises keyword-only retrieval). Skips gracefully via describe.skip when the fixture is missing. Plan ref: T10 in v0.40.4.0 wave plan; D7=C absolute floor + D13=A paired bootstrap; codex #4 + #18 stability-vs-quality distinction. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 T12: VERSION + package.json + CHANGELOG + TODOS VERSION: 0.37.11.0 → 0.40.4.0 package.json: 0.37.11.0 → 0.40.4.0 CHANGELOG.md: top entry for v0.40.4.0 in ELI10-lead voice per CLAUDE.md release rules. Lead is plain-English ("Your search now notices when a page is a hub for your query"); precise file paths / SQL semantics / numbers live in the "Itemized changes" section below. Includes the cathedral-expansion notes (D5=B audit unification, D12=A per-stage attribution, D13=A eval gates) and the "To take advantage of v0.40.4.0" verify-and-fix block. TODOS.md: 5 new items captured under "v0.40.4 graph signals — deferred follow-ups (v0.41+)": - T-todo-1: profile graph-signal SQL latency, merge if hot (D8=C) - T-todo-2: magnitude calibration wave from probe data (D14=B / D17) - T-todo-3: DB-backed audit table for cross-deploy observability (codex #15) - T-todo-4: sync-topology-aware cross-source signal (codex #11) - T-todo-5: replace doctor's global density with fire-rate (codex #14) Verified the 3-line audit: VERSION + package.json + CHANGELOG topmost all match 0.40.4.0. `bun install` ran (lockfile unchanged — root package version isn't stored in bun.lock). `bun run build:llms` refreshed llms.txt + llms-full.txt for the next commit. Plan ref: T12 in v0.40.4.0 wave plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 TODO: document pre-existing shard-2 flake noticed during ship 3 isCacheSafe test failures in shard 2 reproduce on stashed clean master. Confirmed pre-existing — not introduced by v0.40.4. Filed under "Pre-existing flake on master (noticed during v0.40.4 ship)" with reproduction commands + remediation options. Shipping v0.40.4 through it; future wave can fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 privacy scrub: replace wintermute → media in example slugs CLAUDE.md line 550 bans the private OpenClaw fork name in public artifacts. Example session prefix in sessionPrefix() docs + 3 test fixtures swept to 'media/chat/...' instead. Pre-existing scripts/check-privacy.sh in `bun run verify` caught it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 fix: wire graph_signals from mode bundle to runPostFusionStages CRITICAL: pre-landing review (codex outside-voice via /ship Step 9) caught that hybrid.ts's `postFusionOpts` literal at line 566 was building PostFusionOpts WITHOUT threading `resolvedMode.graph_signals` to `graphSignalsEnabled`. The gate at hybrid.ts:358 read the field from a literal that never set it. Result before this fix: the entire v0.40.4 graph-signals wave was dead code in production. Mode bundles set `balanced.graph_signals = true` and `tokenmax.graph_signals = true`, but no production call site ever reached applyGraphSignals. The KNOBS_HASH bump 3→4 correctly varied the cache key by the flag, so contamination was prevented — but the feature itself never fired. All shipped infrastructure (engine SQL, fail-open audit, attribution stamps, --explain formatter, doctor coverage check, search-stats section) was reachable only through the unit-test seam (`opts.adjacencyFn`). The CHANGELOG-advertised behavior never landed in user-visible search. Fix: thread `graphSignalsEnabled: resolvedMode.graph_signals` into the postFusionOpts literal (1 line). Inline comment names codex's catch so future refactors see the regression class. Tests: new test/search/graph-signals-wire-integration.test.ts pins the wire end-to-end. Three cases: 1. balanced mode → hybridSearch on a seeded brain with adjacency hub produces a result with base_score stamped (proves runPostFusionStages actually ran). 2. search.graph_signals=false config override → no graph_* fields stamped (proves the gate honors the override path). 3. Source-grep regression guard pinning the `graphSignalsEnabled: resolvedMode.graph_signals` literal in hybrid.ts so a future refactor can't silently disconnect. All 57 existing v0.40.4 wave tests still pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 fix: pre-landing review AUTO-FIX findings (audit msg drift + deleted_at) Two informational findings from /ship pre-landing review (Step 9): 1. Stderr message qualifier drift (rerank/slug-fallback/phantom audits) Pre-v0.40.4 messages included a per-feature qualifier: [gbrain] rerank-failure audit write failed (...) [gbrain] slug-fallback audit write failed (...) [gbrain] phantom audit write failed (...) The T2 refactor dropped the qualifier (plan promised "byte-identical" operator-visible behavior, but stderr lines did drift). Restored via new `errorMessagePrefix` option on `createAuditWriter` (optional, '' default). Three modules pass the per-feature qualifier; shell-audit and supervisor-audit unaffected (their pre-v0.40.4 messages didn't have a separate qualifier — label already carried the feature name). 2. Defense-in-depth `deleted_at IS NULL` on getAdjacencyBoosts SQL was previously protected by-construction (hybridSearch's visibility filter ensures input pageIds are live), but matches the v0.35.5.0 findOrphanPages pattern and closes the bug class if a future caller bypasses hybridSearch. Added to both Postgres and PGLite engines for parity. Three JOIN sites guarded (targets CTE, FROM-pages join). One inline comment per engine cites the codex review and the v0.35.5.0 precedent. Plan ref: /ship pre-landing review v0.40.4.0 (codex finding C and F). All 84 audit+graph-signals tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 fix: adversarial review HIGH findings (codex H1+H2 + Claude F1) Three HIGH-severity issues from /ship adversarial pass: H1 (Codex): Eval gate was a no-op. Test passed `graph_signals: graphSignalsOn` via `as any` cast, but SearchOpts had no field and hybridSearch's perCall didn't thread it. Both off/on branches resolved to the mode-bundle default — gate measured identical behavior, could pass while detecting nothing. Fix: add `graph_signals?: boolean` to SearchOpts (types.ts:794). Thread `opts.graph_signals` into perCall in both hybridSearch (hybrid.ts:425) AND hybridSearchCached (hybrid.ts:1027) so the cache-key resolver also sees the override. Drop the `as any` from the eval test — types are real now. H2 (Codex): Session diversification fired on entity directories. sessionPrefix() used "any shared parent directory" as the session signal. Result: a search for "people in SF" returned `people/alice` + `people/bob` + `people/charlie` and the latter two got demoted to 0.95×. Every common entity-search query silently penalized legitimate same-type results. Default-on for balanced/tokenmax means production behavior was wrong. Fix: narrow sessionPrefix() to fire ONLY when the slug contains a session-like marker (`chat`/`session`/`sessions` segment OR a `YYYY-MM-DD` date segment). Entity directories (`people/`, `companies/`, `docs/`) return null → diversification skips. Returns NULL (not the slug itself) so the loop skips clean. Examples in JSDoc: your-agent/chat/2026-05-20-foo → 'your-agent/chat/2026-05-20-foo' daily/2026-05-20/journal-entry-1 → 'daily/2026-05-20' transcripts/chat/funding-discussion → 'transcripts/chat/funding-discussion' people/alice → null ← codex H2 regression docs/quickstart → null F1 (Claude adversarial subagent): case-sensitivity drift across 3 sites. loadOverridesFromConfig in mode.ts is case-insensitive + whitespace-trimmed for 'search.graph_signals' values. But doctor's checkGraphSignalsCoverage (doctor.ts:899) AND search-stats's readGraphSignalsStats (search.ts:288) used case-sensitive compare. User sets `search.graph_signals TRUE`: production enables the feature, but doctor + search-stats both silently report disabled. Operators lose the only observability surface for the new feature on values like 'True'/'TRUE'. Fix: trim + lowercase parity at both sites. Mirror the parser's semantic. Also case-normalized `search.mode` reads at both sites for the same divergence class. Tests: - sessionPrefix block rewritten with 7 cases covering chat marker + date anchor + entity dirs (now-NULL) + degenerate (no /). - Added regression test pinning codex H2: people/alice + people/bob + people/charlie do NOT get diversified. - graph-signals-eval.test.ts drops `as any` — typed field works. - Existing tests using `chat/a`/`chat/b` updated to session-shaped `media/2026-05-20/chunk-a` so the date anchor actually fires. 111/111 graph-signals + doctor + search-stats tests pass. Typecheck clean. Plan ref: /ship adversarial review v0.40.4.0 (codex H1, H2; Claude F1). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.4.0 TODOs: capture 11 LOW adversarial findings for v0.41+ Codex L1 (audit window underreport) + Claude F2/F3/F5-F8/F11/F12/F14/F16 from /ship adversarial review. None are load-bearing; all captured under 'v0.40.4 adversarial review LOW findings — captured for v0.41+'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update project documentation for v0.40.4.0 - README: surface v0.40.4.0 graph signals + --explain in Hybrid search capability - CLAUDE.md: annotate engine.ts getAdjacencyBoosts, new graph-signals.ts / explain-formatter.ts / audit/audit-writer.ts, plus hybrid.ts post-fusion 4th stage, mode.ts graph_signals knob + KNOBS_HASH 3→4, cli-options.ts --explain flag, search stats + doctor coverage check - llms-full.txt: regenerated from CLAUDE.md per the build:llms chaser rule Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(ci): pin bun-version to 1.3.13 across all workflows setup-bun action with `bun-version: latest` calls the GitHub API (https://api.github.com/repos/oven-sh/bun/git/refs/tags) to resolve the tag. CI started failing today with HTTP 401 "Bad credentials" even though the action receives a token (visible as `token: ***` in the run log). Pinning the version eliminates the API call entirely. Affected workflows: test.yml, e2e.yml, release.yml, heavy-tests.yml (5 invocations total). Pinned to 1.3.13 — matches package.json engines (`bun >= 1.3.10`) and the version v0.40.4.0 was developed against. Bump cadence: when a new bun version is required, update this pin in one PR. Trading "always-latest" for "always-deterministic" is the right trade for a 5-shard CI matrix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-23 10:01:08 -07:00
3 people	43608c1856	v0.40.3.0 feat: contextual retrieval + cache invalidation gate + 4 deferred-item closures (#1323) ... * v0.40.3.0 T1: migration v81 + CRMode type substrate Five additive columns + Page/SourceRow type extensions + CRMode discriminated union land the schema foundation for v0.40.3.0 contextual retrieval. All columns are NULL-tolerant; existing rows continue working unchanged until the post-upgrade reembed sweep catches up. Schema (migration v81 + schema.sql + pglite-schema.ts mirror): - pages.contextual_retrieval_mode TEXT NULL — tier the page was last embedded under. NULL on pre-v81 rows; drift detection treats NULL as 'none' for reindex predicates. - pages.corpus_generation TEXT NULL — composite hash of (synopsis_prompt_version, haiku_model, title_wrapper_version, embedding_model) per D27 P1-5. Document-side provenance for the v0.40.3.0 query_cache.page_generations invalidation contract. - sources.contextual_retrieval_mode TEXT NULL — per-source override. CLI-write-only per D15 security gate. - sources.trust_frontmatter_overrides BOOLEAN DEFAULT FALSE — per-source mount-frontmatter trust gate per D15. Host source (id='default') is always trusted in the resolver regardless of column value. - query_cache.page_generations JSONB DEFAULT '{}' — D27 P1-5 invalidation contract foundation. Per-row tag of {page_id: corpus_generation} so lookup can LEFT JOIN against current pages and exclude stale rows. Types (src/core/types.ts + src/core/sources-ops.ts): - New CR_MODES = ['none', 'title', 'per_chunk_synopsis'] as const + CRMode type union + isCRMode() type guard for parsing untrusted frontmatter / config values. - Page interface extended with contextual_retrieval_mode + corpus_generation (optional, NULL-tolerant for pre-v81 rows). - SourceRow interface extended with contextual_retrieval_mode + trust_frontmatter_overrides (optional for pre-v81 brains). Bootstrap coverage: - All four pages/sources columns are in PGLITE_SCHEMA_SQL CREATE TABLE bodies (fresh installs get them at initSchema time). - query_cache.page_generations is exempt because query_cache itself is migration-created (added in v55, not in PGLITE_SCHEMA_SQL). Same rationale as the existing query_cache.knobs_hash exemption. Pinned by the migrate.test.ts v81 round-trip + the schema-bootstrap-coverage parser (which also gained the query_cache.page_generations exemption). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T2: MARKDOWN_CHUNKER_VERSION 2→3 (contextual wrapper signal) Bumps the markdown chunker version so the post-upgrade reembed sweep finds every page on the old chunker version and re-embeds it through the new contextual-retrieval wrapper path. Chunk boundaries themselves are unchanged from v2 — the bump forces re-embed (not re-chunk) so existing pages pick up the wrapper without recomputing chunk splits. JSDoc on MARKDOWN_CHUNKER_VERSION updated to document the v3 semantic ("chunks embed with optional contextual retrieval wrapper per Anthropic's published methodology"). Pins the dependency between the chunker version bump and the upcoming src/core/contextual-retrieval-service.ts (T5). Test fixture in test/chunkers/recursive.test.ts updated to assert v3 with a brief comment on the bump rationale so future contributors see the v0.40.3.0 reason inline. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T3: pure modules — resolver, wrapper, synopsis, audit Four new pure modules under src/core/ that the upcoming service layer (T5) and Minion handler (T6) compose. All four are testable in isolation; no engine I/O, no filesystem reads outside the synopsis source-text fallback chain (which is invoked by the service, not the modules themselves). src/core/contextual-retrieval-resolver.ts (D5+D6+D15+D26 P0-4): - resolveContextualRetrievalMode() walks the three-source override chain: page frontmatter > source row > global mode bundle. Returns a tagged result with source attribution + invalid_frontmatter_value (D13) + frontmatter_rejected_untrusted_mount (D15) for doctor surfacing. - crModeDistinct() helper for D26 P0-4 IS DISTINCT FROM semantics on app-side CRMode comparisons (NULL-aware, defeats the != misses NULL drift bug Codex pass 2 caught). - HOST_SOURCE_ID = 'default' always trusted regardless of trust_frontmatter_overrides; mount sources require the explicit flag per D15 security gate. src/core/embedding-context.ts (D20-T1 + D20-T4 + Codex T5 title-weakness): - buildContextualPrefix(title, synopsis) → null | wrapped block. Handles title-only, summary-only, both, or neither. - wrapChunkForEmbedding(text, prefix, chunkSource) short-circuits on chunk_source='fenced_code' per D20-T4 (code chunks inside markdown pages skip the wrapper — prepending page title to a code block doesn't help cross-modal retrieval). - sanitizeTitle/sanitizeSynopsis strip </context> (injection vector) and collapse whitespace + cap at 300 chars. - extractFirstTwoSentences() pure regex with CJK_SENTENCE_DELIMITERS from src/core/cjk.ts for the title-tier free fallback path. src/core/page-summary.ts (D27 P1-2 + D27 P1-4 + D21 reversal): - generatePerChunkSynopsis() routes through gateway.chat(tier='utility'). - Richer failure envelope per D27 P1-2: refusal/empty/malformed (→ D14 page-level fall-back) vs auth_failure/rate_limit/timeout/network/ provider_5xx (→ retry per gateway, or throw to Minion retry). - buildSynopsisCacheKey() composes the LRU key per D27 P1-4: (content_hash, chunk_index, corpus_generation, source_text_hash). - DELIBERATELY no calibration injection — D21 reversed D7's calibration- aware acceptance. Mutable answer-time bias tags don't belong in static document vectors. Query-side personalization is the v0.41+ home. src/core/audit-synopsis.ts (D17, mirrors v0.35.0.0 rerank-audit precedent): - Failure-only JSONL writer at ~/.gbrain/audit/synopsis-failures-YYYY-Www.jsonl with ISO-week rotation. Deliberately no success logging (10K+ pages per backfill would generate 10K+ JSONL rows of noise; failure signal is the actionable one). - summarizeSynopsisFailures() aggregator returns SynopsisFailureSummary for doctor's synopsis_refusal_rate check. Clean typecheck across the four modules. Tests land in T14 alongside the service + Minion handler so the test layer can integrate the full path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T4: ModeBundle.contextual_retrieval + KNOBS_HASH_VERSION 3→4 Three-tier wrapper ladder gated by search.mode lands in the bundle. The per-mode defaults match the cost-tier philosophy (D2): conservative → 'none' (minimum surface) balanced → 'title' (free at runtime; pure string concat) tokenmax → 'per_chunk_synopsis' (Anthropic's published method) Plus the D18 soft kill switch (contextual_retrieval_disabled) so a single config-key flip neutralizes wrapping for queries AND new embeds without touching the migration path. src/core/search/mode.ts: - ModeBundle: contextual_retrieval: CRMode + contextual_retrieval_disabled. - All three frozen MODE_BUNDLES updated with the per-tier defaults. - SearchKeyOverrides + SearchPerCallOpts: both fields optional in the per-key config + per-call surfaces. - resolveSearchMode's pick chain threads both new fields through the standard per-call > per-key > mode bundle precedence ladder. - KNOBS_HASH_VERSION 3→4. Two new entries appended to knobsHash() parts list (append-only per CDX2-F13 convention): cr=${cr_mode} + crd=${0|1}. A query against a tokenmax-mode brain can no longer be served from a cache row written when the brain was on balanced — they sit in different embedding spaces. - SEARCH_MODE_CONFIG_KEYS: 'search.contextual_retrieval' + 'search.contextual_retrieval_disabled' added. - loadOverridesFromConfig reads both keys; CR_MODES guard rejects typos (drift typos still fall through to mode default per D13 sync-failure semantics; this is the no-typo path). - Imports CR_MODES + CRMode from src/core/types.ts. src/commands/search.ts: - KNOB_DESCRIPTIONS picks up the two new entries so `gbrain search modes` dashboard renders them with description copy. test/search-mode.test.ts: - Three canonical bundle tests updated with the per-tier CR defaults. - KNOBS_HASH_VERSION expectation bumped 3→4 with inline rationale. Clean typecheck + 42 search-mode tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T8: NULL→non-NULL upsert race fix (D24, closes v0.35.x TODO) Two writers racing on the same chunk (autopilot sync + manual `embed --stale` + contextual reindex) previously raced last-writer-wins via the text- unchanged branch's `COALESCE(EXCLUDED.embedding, content_chunks.embedding)`. Pre-v0.40.3 the cost of an overwrite was one wasted ~$0.000001 text- embedding-3-large call. With v0.40.3's per-chunk Haiku synopsis on tokenmax, the cost rises ~300x to ~$0.0003 per overwritten chunk plus the discarded synopsis work. On a 10K-page tokenmax brain, a few percent overwrite rate during concurrent backfill+sync wastes $1-5 of Haiku spend silently. Fix (mirrored exactly in postgres-engine.ts + pglite-engine.ts so both engines stay parity-pinned): embedding = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding WHEN content_chunks.embedding IS NULL THEN EXCLUDED.embedding WHEN EXCLUDED.embedded_at IS NOT NULL AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at) THEN EXCLUDED.embedding ELSE content_chunks.embedding END, embedded_at = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL WHEN content_chunks.embedding IS NULL AND EXCLUDED.embedding IS NOT NULL THEN EXCLUDED.embedded_at WHEN EXCLUDED.embedded_at IS NOT NULL AND (content_chunks.embedded_at IS NULL OR EXCLUDED.embedded_at > content_chunks.embedded_at) THEN EXCLUDED.embedded_at ELSE content_chunks.embedded_at END, The two columns move together via aligned CASE WHEN logic — embedding + embedded_at stay consistent so `embed --stale` (predicate `embedding IS NULL`) keeps working correctly. Behavior summary for the text-unchanged branch: - existing embedding NULL → take new (cold path, no race) - new is fresher (embedded_at > existing) → take new - otherwise → keep existing (slower writer with stale embedding loses) Closes the v0.35.x TODOS.md item that flagged this race pre-existing. v0.40.3 fold-in lands the fix when the wave amplifies the cost vector, per D24 in the eng-review pass. 100 pglite-engine tests pass + clean typecheck. E2E concurrent-writer test lands in T14 alongside the broader test suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T5: contextual-retrieval-service + two-phase build (D27 P1-1) Centerpiece service module. Single source of truth for "re-embed one page with the active CR mode" — composed by import-file.ts (sync time), reindex.ts (batch sweep), and the contextual-reindex-per-chunk Minion handler (T6). Closes the drift class Codex pass 2 P1-1 flagged: each consumer no longer hand-rolls the embed-then-stamp flow, so there's literally no way for them to diverge. src/core/contextual-retrieval-service.ts: - reembedPageWithContextualRetrieval() implements the D26 P0-2 two-phase build pattern. PHASE 1 (in-memory, no DB writes): - Load page + source + chunks - Resolve effective CR mode (resolver) with optional kill-switch short-circuit per D18 - 'none' tier: skip wrap, stamp column, return early (records page is up-to-date relative to current state so reindex sweep doesn't re-walk it) - 'title' tier: pure string concat with sanitized title prefix - 'per_chunk_synopsis' tier: read source text via fallback chain (D11), generate synopsis per chunk SEQUENTIALLY within page (D10), batch embedBatch ONCE per page (D27 P2-2). Rate-leasing hooks (acquireSynopsisLease/releaseSynopsisLease) supplied by the Minion handler; inline callers rely on gateway-level retry. - On refusal/empty/malformed (per D27 P1-2): RESTART PHASE 1 at 'title' tier — D14 page-level consistency (whole page demoted, no mid-state on disk). PHASE 2 (single DB transaction): - tx.upsertChunks() — chunk_text stays canonical per D20-T1; only the wrapped string went to the embedder, not into the column. - tx.updatePageContextualRetrievalState() — stamps both columns atomically with PHASE 1 chunk writes. - computeCorpusGeneration() composes the document-side provenance hash per D27 P1-5: sha256(cr_mode + synopsis_prompt_version + haiku_model + title_wrapper_version + embedding_model_tag).slice(0,16). Future prompt edits or model bumps invalidate prior cache rows via the query_cache.page_generations LEFT JOIN (lands in T11). - computeSourceTextHash() for D27 P1-4 synopsis cache key composition. - expectedModeForPageSourceOnly() helper for the T9 reindex sweep predicate. - ReembedPageResult discriminated union: success | skipped (4 reasons) | page_fallback (refusal triggered D14) | transient_error | permanent_error. Each consumer dispatches on `kind` to decide retry / surface / commit. New engine method (added to BrainEngine interface + both engines): - updatePageContextualRetrievalState(slug, sourceId, mode, corpusGeneration): narrow UPDATE of just the two CR-state columns + updated_at. Skips soft-deleted rows. Mirrors refreshPageBody's narrow-update pattern so we don't fire createVersion on every tier upgrade (which would bloat page_versions). Clean typecheck + 272 existing tests pass (no regressions). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T6: contextual_reindex_per_chunk Minion handler + protection Thin handler (D23) that wires the global Haiku rate-leaser (D26 P0-3) + delegates re-embed work to contextual-retrieval-service.ts (T5). One job per page (D10). Submitted by the mode-switch hook (T10), the reindex sweep (T9), and doctor --remediate (T13). src/core/minions/handlers/contextual-reindex-per-chunk.ts: - makeContextualReindexHandler(opts) factory closure. - Per-chunk Haiku call wrapped in acquireLease/releaseLease against the shared key 'anthropic:utility:contextual-synopsis'. Default RPM cap is 50 (Anthropic Haiku 4.5 published limit); operators on a tier with higher quota override via GBRAIN_CONTEXTUAL_HAIKU_RPM env var. - D27 P2-1 source-id derivation: payload carries only page_slug; handler loads the page row and uses its source_id as authoritative. Optional expected_source_id field on the payload triggers UnrecoverableError on mismatch (stale/malicious payload defense). - Result classification: success / page_fallback (D14) → ok transient_error → throw (Minion retries) permanent_error → UnrecoverableError → dead-letter - 60s poll-wait per Haiku call when the rate-lease is saturated; gives up with explicit error rather than blocking forever. src/core/minions/protected-names.ts: - contextual_reindex_per_chunk added to PROTECTED_JOB_NAMES with comment documenting the cost vector (1-50 Haiku calls per page, bulk MCP submission could drain user's Anthropic budget). src/commands/jobs.ts: - registerBuiltinHandlers wires the new handler via dynamic import. - Registered ABOVE autopilot-cycle so the handler is available when doctor --remediate proposes contextual_retrieval_coverage steps. Clean typecheck. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T7: import-file.ts wraps at embed time, stamps CR state columns import-file.ts now resolves the effective CR mode for each page at embed time and applies the wrapper inline. Per D20-T1 critical invariant, the stored chunk_text stays canonical (powers FTS, snippets, reranker, debug); only the wrapped string goes to the embedder. Inline path scope (cost-discipline choice): - title-tier: inline wrap is free (pure string concat). Applied directly. - per_chunk_synopsis tier: TOO EXPENSIVE for the inline import path (one Haiku call per chunk on every sync would compound into hours of blocking per `gbrain sync`). The inline path lands the page at the title tier; the Minion-driven contextual reindex (T6 handler) upgrades it to per_chunk_synopsis later when the user accepts the cost prompt in the mode-switch hook (T10). Per D3 explicit-consent contract. - 'none' tier (conservative mode, kill-switch disabled): no wrapping, raw chunk_text → embedder unchanged from pre-v40.3 behavior. Code chunks (chunk_source='fenced_code') always bypass wrapping per D20-T4 — wrapChunkForEmbedding short-circuits. Stamping (alongside putPage in the same transaction): - pages.contextual_retrieval_mode → tier the page was just embedded at - pages.corpus_generation → composite hash via computeCorpusGeneration from the service module. NULL when 'none' tier or noEmbed=true. Override chain: page frontmatter > source row > global mode bundle (D5+D6). Mount-frontmatter trust gate (D15) — currently lookup uses defaults for source row; future T9 reindex sweep + T10 mode-switch hook can pass a richer source row when the per-source override lands. Kill switch (D18): when search.contextual_retrieval_disabled=true, the resolver short-circuits to 'none' and the wrapper is skipped. Clean typecheck + 251 unit tests pass (migrate + pglite-engine + import-file all green). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T9: reindex --markdown extends to catch CR state drift `gbrain reindex --markdown` predicate widens from chunker_version drift alone to also catch contextual_retrieval_mode IS NULL — the v0.40.3.0 upgrade-path signal that a page has never been evaluated against the CR ladder (pre-v81 brains where the column is freshly NULL after the migration ran). Pages enter the sweep when EITHER: (a) chunker_version < MARKDOWN_CHUNKER_VERSION (existing behavior) (b) contextual_retrieval_mode IS NULL (new — D26 P0-1 + D26 P0-4 prep) Since chunker_version 2→3 (T2) already forces every pre-v40 page into (a), the IS NULL clause is effectively a belt-and-suspenders for the case where a brain upgrades migrate but somehow the chunker_version bump didn't propagate (concurrent upgrade race, manual SQL edit, etc.). The re-import path uses importFromContent with forceRechunk:true (existing v0.32.7 behavior) which bypasses the content_hash short- circuit so the v0.40.3.0 import-file.ts wrapper application path (T7) actually applies. Each re-imported page picks up the active CR tier and stamps contextual_retrieval_mode + corpus_generation atomically. Page-frontmatter overrides are honored at re-import time (importFromFile re-parses YAML and the resolver picks the per-page tier). The frontmatter- mismatch drift case Codex P0-1 called for (user removes override after initial import) is partially handled here via the IS NULL+forceRechunk path; a v0.41+ wave can add the explicit "frontmatter may contain override" candidate path if real users hit drift the current predicate misses. Clean typecheck + 230 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T10: post-upgrade cost prompt explains contextual retrieval The existing post-upgrade-reembed.ts prompt fires automatically on `gbrain upgrade` because T2 bumped MARKDOWN_CHUNKER_VERSION 2→3. Prompt copy extended to explain WHY the re-embed is happening — without this, users see a "chunker-bump" prompt and wonder if it's a routine internal refresh vs the actual headline feature ship. formatReembedPrompt now appends a [contextual retrieval] line below the chunker-bump cost summary, mentioning that v0.40.3.0 wraps each chunk with its page title before embedding (Anthropic's published method). What the user sees on upgrade: [chunker-bump] Will re-embed ~N markdown pages via {model}, est. ~$X.XX, ~Ymin. Press Ctrl-C within Zs to abort. [contextual retrieval] v0.40.3.0 wraps each chunk with its page title before embedding (Anthropic's published method). Title-tier wrap is free at runtime (pure string concat, no Haiku) so the cost number stays unchanged from the chunker-bump-only case. The per-chunk Haiku synopsis tier is OPT-IN via `gbrain config set search.mode tokenmax` post-upgrade, which fires the contextual_reindex_per_chunk Minion handler (T6) for the backfill. T10 mode-switch hook in src/commands/config.ts (the explicit per-mode cost prompt UX on `gbrain config set search.mode tokenmax`) is deferred to v0.40.3.1 — the explicit-consent contract (D3) is satisfied by the existing post-upgrade prompt for the title-tier path that the wave ships by default. The Minion handler from T6 + the protected-name guard ensure that any direct Minion submission for the per-chunk path is gated on the CLI/doctor-remediate trust boundary. Kill switch (D18): the contextual_retrieval_disabled config key is honored at import time (T7) and in the service (T5) — when true, the resolver short-circuits to 'none' regardless of mode bundle. No hybridSearch changes needed: queries embed raw text already; the kill switch only affects NEW embeds. Existing wrapped vectors keep serving queries via cosine similarity (asymmetric retrieval is preserved). 11 upgrade-reembed-prompt tests pass + clean typecheck. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T11-T13: query cache notes + remediation note + doctor check T11 (query_cache.page_generations contract): the DB column shipped in T1 migration v81 + KNOBS_HASH_VERSION 4 bump in T4 invalidates the common-case cache contamination (full-brain mode upgrade). The LEFT JOIN read-side gate per Codex P1-5 — for the edge case where a brain is mid- reindex and some pages are stamped at corpus_generation N+1 while others are still at N — is deferred to v0.40.3.1. In practice, the post-upgrade reembed prompt fires automatically + completes before search resumes on healthy brains, so the edge case is narrow. CHANGELOG documents the limitation. T12 (generic RemediationStep contract): the existing recommendation registry shape (sync/embed/backlinks/extract hardcoded) is extended via the doctor check below rather than refactored to a generic registry. Codex P1-6 called for the refactor; v0.40.3.1+ can absorb it once a real second consumer requires the same registration shape. T13 (contextual_retrieval_coverage doctor check): - New checkContextualRetrievalCoverage() in src/commands/doctor.ts. - Two SQL signals: pages.chunker_version < current + pages.contextual_ retrieval_mode IS NULL. Single COUNT...FILTER query is cheap on every brain size. - Audit summary line: reads ~/.gbrain/audit/synopsis-failures-*.jsonl via the v0.40.3.0 audit-synopsis module (T3). >5% page-level fallback rate surfaces explicitly so operators see the Haiku refusal signal. - Paste-ready fix: `gbrain reindex --markdown` — the v0.32.7 + v0.40.3.0 sweep covers both chunker_version drift AND CR mode drift per T9. - Status: ok when fully aligned + no recent failures; warn when drift exists (with the paste-ready fix in the message). - Wired into the standard doctor run alongside the other v0.36+ checks (abandoned_threads, calibration_freshness, etc.). Sources/mounts CLI surfaces (set-cr-mode + trust-frontmatter) deferred — the post-upgrade-reembed prompt + the per-page frontmatter override path cover the v0.40.3.0 operational workflow. Per-source override CLI is a power-user feature that can ship in v0.40.4+ once real federated- brain users surface specific friction. 48 doctor tests pass + clean typecheck. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T14: 5 test files, 77 new tests, IRON-RULE regression coverage Test suite for the v0.40.3.0 contextual retrieval wave. 77 new test cases across 5 files, all green. Pins every IRON-RULE invariant end-to-end so future contributors can't silently regress the wave. test/contextual-retrieval-resolver.test.ts (29 tests): - 9-combo override matrix (page-fm > source-row > global, all permutations). - D15 mount-trust gate: host always trusted, mounts honor only when trust_frontmatter_overrides=true, rejected frontmatter surfaces via result.frontmatter_rejected_untrusted_mount for doctor. - D13 invalid frontmatter (typo + non-string + empty): falls through to source/global with raw value in invalid_frontmatter_value. - D18 kill switch: short-circuits to 'none' regardless of overrides. - D26 P0-4 crModeDistinct: NULL-aware comparison, matches SQL IS DISTINCT FROM semantics on every combination of NULL/defined args. test/embedding-context.test.ts (21 tests): - buildContextualPrefix: title-only, synopsis-only, both, neither. - wrapChunkForEmbedding: non-code wraps; D20-T4 fenced_code ALWAYS bypasses; null prefix passes through; image_asset wraps as text. - sanitizeTitle: </context> injection stripped (case-insensitive), whitespace collapsed, 300-char cap, trim semantics. - extractFirstTwoSentences: English boundaries, question marks, CJK delimiters, run-on cap, empty input, no-delimiter passthrough. - modeRequiresHaiku / modeRequiresWrapper guards. - D20-T1 IRON-RULE regression test: wrapping does not mutate input string reference (so caller's chunk_text safely flows to upsert). test/contextual-retrieval-service-pure.test.ts (16 tests): - computeCorpusGeneration: 16-char hex, deterministic, mode-sensitive, model-sensitive, TITLE_WRAPPER_VERSION stable. - computeSourceTextHash: D27 P1-4 cache invalidation key composition. - expectedModeForPageSourceOnly (T9 reindex predicate helper): kill switch returns none, source override beats global, invalid override falls through, all CR modes round-trip. test/audit-synopsis.test.ts (11 tests): - ISO-week filename rotation (stable for same week, different days). - logSynopsisFailure round-trip: kind, page_level_fallback flag, multi-event accumulation, detail 200-char cap. - summarizeSynopsisFailures aggregation: null on empty, by_kind counts, page_level_fallback_rate math. - Missing audit file returns empty (silent no-op). test/e2e/contextual-retrieval-pglite.test.ts (5 tests, hermetic PGLite + gateway stub): - IRON RULE #1 (D20-T1): wrapper text in embedder input but NEVER in content_chunks.chunk_text after import — pins the canonical chunk_text separation invariant end-to-end. - IRON RULE #2 (D14 stamping): pages.contextual_retrieval_mode AND pages.corpus_generation are set after every import. - IRON RULE: chunker_version stamps to current MARKDOWN_CHUNKER_VERSION (3 for v0.40.3.0). - D5 per-page frontmatter override: `contextual_retrieval: none` makes the embedder receive UNWRAPPED text; mode column stamped 'none'. - T9 reindex predicate: pages with contextual_retrieval_mode IS NULL enter the sweep regardless of chunker_version. 462 tests pass across all v0.40.3.0 + adjacent suites (migrate, pglite-engine, search-mode, doctor, import-file, upgrade-reembed-prompt, schema-bootstrap-coverage, recursive chunker, all five new files). Zero regressions, clean typecheck. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 T15: VERSION + CHANGELOG + migration self-repair + llms regen VERSION 0.37.11.0 → 0.40.3.0 with package.json sync. CHANGELOG entry follows the CLAUDE.md ELI10-lead voice rule: opens with "Your search now understands what each chunk is about, not just what words are in it," lays out the tier ladder with a real cost table, calls out the chunk_text storage separation (D20-T1) with a concrete example, and includes the "Things to watch" + "What we caught and fixed before merging" sections per the format spec. CHANGELOG also includes the canonical "To take advantage of v0.40.3.0" self-repair block with the manual `gbrain apply-migrations --yes` + `gbrain reindex --markdown` recovery path for users whose `gbrain upgrade` post-upgrade-reembed didn't fully fire. skills/migrations/v0.40.3.0.md walks the agent through the mechanical upgrade flow, the opt-up to tokenmax path with the realistic backfill cost table, the opt-out soft kill switch flip, and the per-page frontmatter override with the D15 mount-trust note. Matches the v0.13.0 + v0.32.7 migration doc structure so agent muscle memory works. llms-full.txt + llms.txt regenerated via `bun run build:llms` to pick up the CHANGELOG + migration doc additions. test/build-llms.test.ts passes. Also moved test/audit-synopsis.test.ts → test/audit-synopsis.serial.test.ts to satisfy the check-test-isolation lint (the test mutates GBRAIN_AUDIT_DIR via beforeAll/afterAll for a fixture dir, which the parallel runner forbids in *.test.ts files; serial quarantine is the canonical fix per CLAUDE.md test-isolation rules). `bun run verify` passes (typecheck + 4 CI gate checks). 469 tests across all v0.40.3.0 + adjacent suites pass with 0 failures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.3.0 test gaps: doctor check coverage + concurrent race regression Post-T15 test gap-fill: covers the two highest-leverage spots that the T14 suite didn't exercise. test/contextual-retrieval-doctor.serial.test.ts (8 tests, .serial because the doctor check reads the audit JSONL via GBRAIN_AUDIT_DIR env mutation): - empty-brain → ok - fully-aligned brain (chunker_version current + mode stamped) → ok - chunker_version drift → warn with paste-ready `gbrain reindex --markdown` - NULL mode column → warn surfaces "never evaluated against CR ladder" - both drift conditions together → warn with both messages - soft-deleted pages NOT counted (deleted_at filter works) - non-markdown (code) pages NOT counted (page_kind filter works) - audit JSONL refusal event surfaces in the failure-summary line test/e2e/concurrent-embed-race.test.ts (3 tests, D24 regression guard): - cold path: existing embedding NULL → take new (no-race case) - IRON RULE: fresher write wins over stale write when text unchanged. Pre-fix this would have last-writer-wins via COALESCE; post-fix the fresher embedded_at survives. Pinned by raw SQL upsert with an explicit -5min embedded_at to simulate the slower writer. - text change with no new embedding → both embedding + embedded_at reset to NULL (consistent state so embed --stale picks up). Cross-shard contamination fix: race test calls configureGateway with embedding_dimensions=1536 BEFORE initSchema so the PGLite vector column sizes consistently regardless of what other tests in the same shard process configured first. Without this, running the race test alongside the pglite-e2e test triggered "expected 1280 dimensions, not 1536" when the gateway was left in its default ZE-1280 state by a prior file. `bun run verify` passes (typecheck + 5 CI gate checks). 88 tests pass across all v0.40.3.0 + new gap-fill files in one combined run; zero shared-state contamination. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.40.5.0 T2: schema — v90 contextual_retrieval_columns + v91 trigger + index Migration v90 (renamed from v0.40.3.0 v81 on master merge per D2/D7): - 5 additive columns (pages.contextual_retrieval_mode, pages.corpus_generation, sources.contextual_retrieval_mode, sources.trust_frontmatter_overrides, query_cache.page_generations) for the contextual retrieval wave. Migration v91 (NEW per D6 + codex #4 + codex #8): - pages.generation BIGINT NOT NULL DEFAULT 1 (per-page generation counter) - query_cache.max_generation_at_store BIGINT NOT NULL DEFAULT 0 (Layer 1 bookmark) - bump_page_generation_fn() trigger function: - BEFORE INSERT: NEW.generation := COALESCE(MAX(generation), 0) + 1 — codex #4 INSERT coverage so cache rows stored before a new page existed invalidate correctly. - BEFORE UPDATE: bumps generation only when allow-list columns IS DISTINCT FROM (compiled_truth, timeline, frontmatter, deleted_at, contextual_retrieval_mode, title, type, page_kind, corpus_generation, content_hash) per D6 widened to catch user-visible mutations. - CREATE INDEX CONCURRENTLY pages_generation_idx ON pages (generation) so MAX(generation) for the bookmark check is O(log N) — codex #8 confirmed plain btree, no DESC necessary. Mirrored in src/schema.sql, src/core/pglite-schema.ts CREATE TABLE body (trigger included so fresh PGLite installs get it from the schema blob, not just migration replay). Extended REQUIRED_BOOTSTRAP_COVERAGE with pages.contextual_retrieval_mode, pages.corpus_generation, sources.contextual_retrieval_mode, sources.trust_frontmatter_overrides, pages.generation. Probes added to applyForwardReferenceBootstrap on both engines + matching ALTER blocks for pre-v90/pre-v91 brains. COLUMN_EXEMPTIONS extended: query_cache.max_generation_at_store (same rationale as page_generations — query_cache is migration-only, not in PGLITE_SCHEMA_SQL). Test results: - bun test test/migrate.test.ts: 140 pass / 0 fail - bun test test/schema-bootstrap-coverage.test.ts: 9 pass / 0 fail - bun run typecheck: clean * v0.40.5.0 T3: cache gate — query-cache-gate.ts + lookup/store rewrites New pure module src/core/search/query-cache-gate.ts: - buildPageGenerationsSnapshot(engine, pageIds) builds the {pageId: gen} snapshot + MAX(generation) bookmark in one round trip via UNION ALL. Pre-v91 brains (no generation column) fall back to empty snapshot + zero bookmark — backward compat with legacy rows preserved. - validateCacheRowAgainstPages() — pure validator for unit testing. - CACHE_GATE_WHERE_CLAUSE exported as a SQL fragment that lookup() embeds in its WHERE clause. Two-layer gate per D11: Layer 1 (cheap): (SELECT MAX(generation) FROM pages) <= qc.max_generation_at_store Layer 2 (per-page): jsonb_each + LEFT JOIN pages to detect deletes + bumped pages on the cached result set. Legacy compat: rows with empty {} snapshot are vacuously valid (Layer 2 short-circuits) — IRON-RULE pinned. query-cache.ts wiring: - lookup() table-aliased to `qc` so the gate fragment can reference qc.max_generation_at_store + qc.page_generations. WHERE clause adds `AND ${CACHE_GATE_WHERE_CLAUSE}` after the existing similarity + TTL + knobs_hash filters. - store() captures the snapshot via the pure helper, then INSERTs both page_generations JSONB and max_generation_at_store BIGINT alongside the existing columns. ON CONFLICT (id) DO UPDATE refreshes both. Test coverage (15 unit + 6 e2e): - test/query-cache-gate.test.ts: 15 cases covering pure validator branches (vacuous valid, bookmark short-circuit, single/multi/partial bumps, deleted page, codex D11 critical case), PGLite-backed snapshot builder (empty pageIds, populated pageIds, integer JSONB shape, non-existent IDs skipped, bump-after-update), SQL shape regression on CACHE_GATE_WHERE_CLAUSE. - test/e2e/cache-gate-pglite.test.ts: 6 cases covering store → HIT, content UPDATE → MISS, INSERT new page → HIT (codex #4 case where bookmark fires but snapshot intact serves correctly), legacy row → HIT (IRON-RULE backward compat), soft-delete → MISS (trigger path), multi-page partial bump → MISS. Test results: - bun test test/query-cache-gate.test.ts test/query-cache.test.ts test/query-cache-isolation.test.ts test/e2e/cache-gate-pglite.test.ts: 33 pass / 0 fail - bun run typecheck: clean Note: hard-delete (raw DELETE FROM pages) is not covered by the trigger (BEFORE INSERT OR UPDATE doesn't fire on DELETE). Production uses soft-delete via deleted_at (trigger allow-list catches NULL → timestamp distinction). Hard-delete via admin-only `gbrain pages purge-deleted` is best-effort cache-wise — acceptable for the rare admin path. * v0.40.5.0 T5: mode-switch UX at gbrain config set search.mode New module src/core/search/mode-switch-ux.ts: - summarizeTransition(old, new): pure 5-cell matrix (no_change / narrowing / broadening / tokenmax_opt_in / invalid_new_mode) + reindex command + cost estimate + paste-ready callout lines. - probeWorkerAvailable(engine): worker liveness proxy. gbrain has no minion_workers heartbeat table yet (B7 follow-up from v0.19.1), so we use a proxy: minion_jobs activity within 10-min query window. Within 2 min = active; >2min but <10min = stale; nothing = never_seen. - buildReindexIdempotencyKey(): content-stable per codex D12 Bug 1. Pattern: cr-backfill:<source_id>:<chunker_version>:<mode>. NOT timestamp-based — two retries against same brain state dedupe. - runModeSwitchUx(): orchestrator. Honors GBRAIN_NO_MODE_SWITCH_UX=1 (full skip), non-TTY (print paste-ready hints to stderr), yesFlag (auto-submit reindex). For tokenmax_opt_in + TTY + worker probe active: submits via MinionQueue.add with allowProtectedSubmit=true. For probe = stale or never_seen: loud-fail per D3 with a "start a worker OR run inline" recovery hint — closes the silent-stall footgun. src/commands/config.ts hook (~30 LOC): - Captures the OLD search.mode BEFORE setConfig so summarizeTransition classifies correctly. - Fires runModeSwitchUx() AFTER setConfig persisted, wrapped in try/catch so UX failures never break the config-set that already landed. - Best-effort: failures emit `[mode-switch] UX hook failed (non-fatal)` to stderr. Test coverage (18 cases): - summarizeTransition: 8 cases covering all 5 transition kinds + null inputs + tokenmax-as-first-set + invalid mode. - probeWorkerAvailable: 4 cases via real PGLite — never_seen / active / stale (seeded via minion_jobs) + threshold constant assertion. - buildReindexIdempotencyKey: 6 cases pinning content-stable contract (codex D12 Bug 1) — identical inputs match, different inputs differ, consecutive calls match despite time delta (NOT timestamp-based). Test results: - bun test test/mode-switch-ux.test.ts: 18 pass / 0 fail - bun run typecheck: clean * v0.40.5.0 T6: gbrain mounts {enable,disable,trust-frontmatter,untrust-frontmatter} Four new mounts CLI verbs per D4: - gbrain mounts enable <id> — re-enable a disabled mount - gbrain mounts disable <id> — toggle off without removing - gbrain mounts trust-frontmatter <id> — let this mount's per-page contextual_retrieval_mode frontmatter override the source default. Off by default for mounted brains; host is always trusted. - gbrain mounts untrust-frontmatter <id> — clear the trust flag. Implementation: - src/core/brain-registry.ts MountEntry interface extended with trust_frontmatter_overrides?: boolean. loadMounts() projection threads the field through with default false (mounts opt in explicitly per D4 + D15 security posture). - src/commands/mounts.ts: new runSetMountFlag() helper handles all 4 verbs via a shared file-write path. Missing-mount loud rejection (GBrainError with list-hint). Host brain rejection. Idempotent: no-op when current value already matches. Cache refresh after each write so host agents see the new flag immediately. Test infrastructure: - GBRAIN_MOUNTS_PATH env override on getMountsPath() in BOTH brain-registry.ts AND mounts.ts (the latter has its own copy — two source-of-truth paths). Reason: libuv caches homedir() on some platforms, so withFakeHome's HOME mutation isn't picked up by tests calling runMounts(). Production callers don't set the env. Test coverage (5 new cases): - enable → disable → enable cycle persists - trust-frontmatter → untrust → trust cycle preserves other fields - missing mount id → loud rejection with list-hint (closes the critical gap from idempotent-pebble Failure Modes table) - host brain rejection: cannot trust-frontmatter "host" - enable on already-enabled mount: no-op (idempotent) Test results: - bun test test/mounts-cli.test.ts test/brain-registry.serial.test.ts: 54 pass / 0 fail - bun run typecheck: clean * v0.40.5.0 T7: gbrain sources set-cr-mode + missing-source loud rejection New verb `gbrain sources set-cr-mode <id> <mode>` per D5: - Mode argument validated against CR_MODES via isCRMode (closed enum: none | title | per_chunk_synopsis). - "unset" / "default" / "" clears the column to NULL (falls through to the global search.mode bundle). - Loud rejection on: - Missing id/mode → exit 2, prints usage - Invalid mode → exit 2, lists valid options - Missing source id → exit 4, paste-ready `gbrain sources list` hint (closes the idempotent-pebble Failure Modes critical gap) src/commands/sources.ts wired into the switch dispatch + help text updated. isCRMode + CR_MODES lazy-imported per existing import pattern in this file. Test coverage (10 cases): - happy path for all 3 valid CRMode values - unset path via "unset" + "default" both clear to NULL - invalid mode → exit 2 + no mutation - missing source id → exit 4 - missing arguments → exit 2 with usage - missing mode (only id) → exit 2 + no mutation - round-trip preserves other fields (name) Test results: - bun test test/sources-set-cr-mode.test.ts: 10 pass / 0 fail - bun run typecheck: clean * v0.40.5.0 T8: RemediationStep refactor + makeRemediationStep factory New canonical module src/core/remediation-step.ts: - RemediationStep interface (lifted from brain-score-recommendations.ts). Same shape; rename to "Step" suffix per D6 for clarity ("a step in a remediation plan"). - RemediationSeverity + RemediationStatus type re-exports. - canonicalJson(value): zero-dep canonical serialization — sorts object keys recursively before stringify. Per codex D12 Bug 2: identical logical params hash identically regardless of insertion order. - idempotencyKey(source, job, params): shape <source>:<job>:sha8(canonicalJson(params)). Lifted from the legacy inline idemKey helper so future check authors don't drift. - makeRemediationStep(opts): canonical factory. Defaults id to the idempotency key (override for human-readable like 'sync.repo'). Status defaults to 'remediable'. All check authors should use this; hand-rolling is the drift hazard the refactor closes. src/core/brain-score-recommendations.ts: - Removed the local Remediation + RemediationSeverity + RemediationStatus definitions. - Re-exports them from remediation-step.ts so existing callers (e.g. doctor.ts) still resolve. Also re-exports Remediation as an alias for RemediationStep so import paths can migrate gradually. - Imports type Remediation alias internally so the (substantial) existing computeRecommendations body keeps compiling without sed pass. Test coverage (17 cases): - canonicalJson: key-ordering determinism (3 cases), nested objects, array order preservation, primitive types, codex D12 Bug 2 regression - idempotencyKey: shape regex, content invariance, key-ordering invariance, source/job/params differentiation - makeRemediationStep: default id, explicit id override, default status, canonical-JSON invariance, all-opts threadthrough - back-compat: `import { Remediation } from brain-score-recommendations` still resolves to RemediationStep (compile + runtime check) Test results: - bun test test/remediation-step.test.ts: 17 pass / 0 fail - bun test test/brain-score-recommendations.test.ts test/doctor.test.ts: 70 pass / 0 fail (back-compat preserved) - bun run typecheck: clean Per D6 + D8: T8b in next commit wires lint, integrity, sync_failures doctor checks to emit RemediationStep via the new factory. * v0.40.5.0 T8b: RemediationStep consumers — integrity + sync_failures + 3 Minion handlers Doctor checks now emit RemediationStep via makeRemediationStep(): - `integrity` check (when bareHits > 0) emits integrity-auto step. Severity escalates to 'high' when bareHits > 50. Deterministic; $0 cost. - `sync_failures` check (when unacked > 0) emits sync-retry-failed step. Severity escalates to 'high' when count >= 10. Content-stable params (failure_count + oldest_failure timestamp) per codex D12 Bug 2. - sync-skip-failed DELIBERATELY NOT emitted per D12 Bug 3 (auto-skipping failed syncs hides data loss). Operators retain `gbrain sync --skip-failed` as a direct CLI option. Lint doctor check NOT wired — there is no `lint` check in doctor.ts today; the lint workflow is the standalone `gbrain lint` command. Adding a doctor lint check is a v0.41+ TODO when it justifies its own complete section. Three new Minion handlers in registerBuiltinHandlers (NOT in PROTECTED_JOB_NAMES — they're thin wrappers around already-shipping CLI commands, idempotent, no shell exec, MCP-safe): - lint-fix → runLintCore({ fix: true }) - integrity-auto → runIntegrity(['auto']) - sync-retry-failed → runSync(['--retry-failed']) Check.remediation field shape upgrade: - Was: inline Array<{...}> shape. - Now: RemediationStep[] from the canonical src/core/remediation-step.ts. Check authors `import { makeRemediationStep }` and emit through the factory. Test results: - bun test test/doctor.test.ts: 48 pass / 0 fail (zero regression on the doctor surface; new remediation fields are additive) - bun run typecheck: clean * v0.40.5.0 T11: capture-generation regression test (D3 + codex #5) The v0.38 ingestion cathedral added a new write path to pages via the `ingest_capture` Minion handler. The v0.40.5.0 cache-invalidation gate relies on pages.generation being bumped by EVERY write path via the BEFORE INSERT OR UPDATE trigger. This file pins that the new v0.38 capture write path correctly bumps generation through three scenarios: 1. INSERT path (codex #4 INSERT coverage): ingest_capture with a fresh slug creates a page with generation = MAX(generation) + 1 so any cache row stored before the new page existed has its bookmark fire. 2. UPDATE path: ingest_capture with an existing slug + new content → trigger fires on content-column IS DISTINCT FROM and bumps generation. 3. Idempotent UPDATE: capture with the SAME content → trigger short-circuits, no bump. Cache freshness preserved on re-runs. Per codex #5 strengthening: noEmbed: true is set explicitly so the test doesn't require API keys (test runs against pure PGLite). Test results: - bun test test/e2e/capture-generation-regression.test.ts: 3 pass / 0 fail - bun run typecheck: clean * v0.40.5.0 T9: docs — CHANGELOG fold-in + CLAUDE.md + migration skill + llms regen Single combined v0.40.5.0 CHANGELOG entry folds in v0.40.3.0 contextual retrieval content + v0.40.5.0 wave additions (cache gate + mode-switch UX + mounts/sources CLI + RemediationStep refactor). Voice per CLAUDE.md: ELI10 lead, plain language, paste-ready commands, tier table, "Things to watch", "What we caught and fixed before merging" (summarizes the 8 codex findings + 3 design decisions in user-facing terms), "Itemized changes", "## To take advantage of v0.40.5.0" mandatory self-repair block. CLAUDE.md: new section "Key commands added in v0.40.5.0 (contextual retrieval + cache gate + 4 CLI verbs)" listing the 4 new mount verbs, sources set-cr-mode, mode-switch UX, KNOBS_HASH_VERSION bump, 3 new Minion handlers, and the 3 new modules (remediation-step, query-cache-gate, mode-switch-ux). skills/migrations/v0.40.5.0.md: new migration skill with feature_pitch frontmatter for the auto-update agent. Documents the 6 master commits merged in, migration v90 (renumber from v81) + v91 (trigger), the optional opt-up to tokenmax, per-source CR mode overrides, mount frontmatter trust, the soft kill switch, and the backward-compat guarantees. bun run build:llms refreshed llms.txt + llms-full.txt: - llms.txt: 4314 bytes - llms-full.txt: 578257 bytes Test results: - bun test test/build-llms.test.ts: 7 pass / 0 fail (committed bundles byte-match generator output) * v0.40.5.0 T10: fix 5 unit-suite drift failures from the wave KNOBS_HASH_VERSION bumped 4→5 per D8 (sequenced behind salem's pending v=4 graph-signals work). Three test files held stale ==3 / ==4 assertions: - test/search-mode.test.ts: assertion + comment updated to v=5. - test/search/knobs-hash-reranker.test.ts: assertion + describe name updated to v=5 ladder. - test/cross-modal-phase1.test.ts: assertion + name updated to v=5. reindex.test.ts "skips pages already at current chunker_version" — the v0.40.3.0 reindex predicate (`chunker_version < CURRENT OR contextual_retrieval_mode IS NULL`) caught the should-skip page because its CR mode was NULL. Fixed by seeding `contextual_retrieval_mode = 'title'` on the should-skip row. reindex.test.ts "idempotent: re-run on a fully-updated brain reports nothing to do" — by design, `--no-embed` reindex bumps chunker_version but skips CR-state stamping (import-file.ts:457-466 documents this). Fixed by manually stamping `contextual_retrieval_mode = 'title'` between the first and second reindex calls so the brain matches the "fully updated" state the idempotency test name implies. Production embed flow stamps both in one pass; the test uses --no-embed only to avoid requiring API keys. Test results: - bun run verify (typecheck + 4 pre-checks): clean - bun run test: 9482 pass / 0 fail / 0 skip across 410s * v0.40.3.0: rename version from 0.40.5.0 → 0.40.3.0 (clean slot above master) Master is at v0.40.2.0; v0.40.3.0 is genuinely the next free slot. The wave was originally planned as v0.40.5.0 sequenced behind salem (PR #1300 = v0.40.4.0) but the user is shipping THIS branch as v0.40.3.0 because: 1. v0.40.3.0 IS the canonical version slot for the contextual retrieval cathedral (matches branch name garrytan/v0.40.3.0-contextual-retrieval). 2. Master is at v0.40.2.0 — v0.40.3.0 is the immediate next slot, not a collision. 3. salem's v0.40.4.0 + any v0.40.5.0 work sit ON TOP of this in the landing train, not under it. Mechanical rename only — no content changes from the v0.40.5.0 commit sequence (T1-T11 wave is preserved verbatim, just relabeled): - VERSION + package.json: 0.40.5.0 → 0.40.3.0 - bun.lock: refreshed (no dep changes) - CHANGELOG.md: ## [0.40.5.0] header → ## [0.40.3.0] + body references - skills/migrations/v0.40.5.0.md → skills/migrations/v0.40.3.0.md (previous v0.40.3.0.md file overwritten with the richer T9 content) - CLAUDE.md: "Key commands added in v0.40.5.0" → "v0.40.3.0" - 30 source + test files: comment references swept via sed s/0.40.5.0/0.40.3.0/g - llms.txt + llms-full.txt: regenerated Migration numbering UNCHANGED: v90 (renamed from original v81 because master took v82-v88) and v91 (new trigger migration) stay at v90/v91 — the version slot is orthogonal to the migration ledger collision. KNOBS_HASH_VERSION = 5 stays — sequenced behind master's v=4 schema-pack work; salem's v=4 graph-signals will rebump to v=5 if it lands first. Test results after rename: - bun run verify: clean (typecheck + 7 pre-checks) - bun run test: 9482 pass / 0 fail / 0 skip Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(migrate): v91 CREATE INDEX CONCURRENTLY can't run inside a transaction (CI Tier 1) CI Tier 1 (Mechanical) failed on real Postgres with: ERROR: CREATE INDEX CONCURRENTLY cannot run inside a transaction block STATEMENT: <v91 multi-statement SQL block including CREATE INDEX CONCURRENTLY ...> Root cause: postgres.js's multi-statement `.unsafe()` wraps the entire block in an implicit transaction. `transaction: false` on the migration entry doesn't help — the implicit wrap happens at the driver layer, below the migration runner. CONCURRENTLY refuses to run inside any transaction. Fix: rewrite v91 using the v14 pages_updated_at_index handler pattern — `sql: ''` + `handler:` function that splits the work into separate `engine.runMigration()` calls: 1. Columns + trigger function + trigger (single multi-statement runMigration — ALTER/CREATE FUNCTION/CREATE TRIGGER are transaction-safe). 2. On Postgres only: pre-drop invalid index remnant via `pg_index.indisvalid` (matches v14 pattern for retry safety after a failed CONCURRENTLY left a half-built index with the target name). 3. CREATE INDEX CONCURRENTLY as a standalone runMigration call (separate statement = no implicit transaction wrap). 4. PGLite: plain CREATE INDEX (no CONCURRENTLY needed — single writer). Verified against real Postgres (pgvector:pg16): - schema_version=91 after init - pages_generation_idx exists with btree shape - bump_page_generation_trg installed - test/e2e/postgres-bootstrap.test.ts + test/e2e/schema-drift.test.ts: 8 pass / 0 fail - bun test test/migrate.test.ts test/schema-bootstrap-coverage.test.ts: 161 pass / 0 fail - bun run typecheck: clean --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-23 08:49:28 -07:00
3 people	e60b60244f	v0.36.6.0 feat: cross-modal search wave (text↔image + unified column + LLM intent) (#1165) ... * feat(cross-modal/0): batched multimodal + query helpers + SSRF helper Commit 0 of the cross-modal search wave. Foundation for Phase 1-3: - embedMultimodal accepts MultimodalInput text variant + EmbedMultimodalOpts with inputType: 'document' | 'query' (D22-2). Default unchanged so importImageFile keeps document-side embedding. - embedQueryMultimodal(text) + embedQueryMultimodalImage(input) wrappers for hybridSearch + searchByImage query paths. - embedMultimodalSafe binary-search retry on transient batch failure + failed_indices surfacing. Phase 3 reindex uses this so a single bad chunk doesn't discard the 31 in-flight embeddings around it. - Voyage path: text + image inputs in one batch via content arrays. - openai-compat path: text + image inputs in one request per input. - src/core/ssrf-validate.ts (D19): DNS-resolve-and-fetch-by-IP defense for redirect chains. Closes the DNS-rebinding gap that url-safety.ts' static check leaves open. Uses node:dns/promises with {all: true, family: 0} to inspect every A and AAAA record before connecting. fetchWithSSRFGuard helper validates per-redirect-hop and limits chain depth (default 3). - Re-exports from src/core/embedding.ts public seam. Tests: - test/embed-multimodal-batching.test.ts (13 cases): text variant, query inputType discipline, mixed text+image batches, embedQueryMultimodal, embedQueryMultimodalImage, embedMultimodalSafe happy/empty/all-fail/ mid-batch-recovery/permanent-misconfig. - test/ssrf-validate.test.ts (20 cases): static rejections via isInternalUrl, scheme + credentials rejection, DNS rebinding defense (single-record + multi-record), public happy path, IPv6 literals, malformed URLs. No regression in existing voyage-multimodal.test.ts or openai-compat-multimodal.test.ts (33 cases all pass). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cross-modal/1): Phase 1 text→image routing + knobsHash + RRF + backfill Phase 1 of the cross-modal search wave. Wires the existing 1024d Voyage multimodal embedding space (already populated for image chunks via importImageFile) into the user-facing query path. Text queries that match cross-modal intent regex route through Voyage multimodal-3 instead of the text embedding model, then search content_chunks.embedding_image. - query-intent.ts: new `suggestedModality: 'text' | 'image' | 'both'` axis on `QuerySuggestions`. Module-scope CROSS_MODAL_PATTERNS regex array (D15 — compiled once at module load). Conservative on purpose; LLM intent escalation (Commit 4) catches genuinely ambiguous phrasings. - query-intent.ts: new `isAmbiguousModalityQuery(query)` pure heuristic for Commit 4's escalation gate. Returns true ONLY when regex misses AND a visual noun + reference marker both fire. - types.ts: `SearchOpts.crossModal: 'text' | 'image' | 'both' | 'auto'` + `SearchResult.modality: 'text' | 'image'` for downstream renderers. - mode.ts: 7 new knobs in ModeBundle (D2): cross_modal_both_text_weight, cross_modal_both_image_weight, image_query_text_refinement_weight, image_query_image_refinement_weight, unified_multimodal, unified_multimodal_only, cross_modal_llm_intent. All three mode bundles default to the same values (cross-modal is opt-in). - mode.ts: D2 cache-key fix — KNOBS_HASH_VERSION bumped 2→3, all 7 new knobs participate in knobsHash so a text-mode cache hit can't be served to an image-mode caller. - mode.ts: D3 registry — all 7 keys land in SEARCH_MODE_CONFIG_KEYS so `gbrain search modes` / `stats` / `tune` see them. - hybrid.ts: routing branch at the embed step. Resolves effective modality from (per-call opts → suggestions → 'text'). Image route: embedQueryMultimodal + searchVector(embedding_image), skip expansion + keyword (D9 mode-bundle override). Both route: parallel text + image vector searches merged via weighted RRF (D6) with cross_modal_both_* weights. Fail-open: multimodal misconfigured → structured warn + text fallback. 'auto' literal normalized to undefined (D22-1). - operations.ts: thread `cross_modal` param through `query` op. - backfill-registry.ts: new `modality` backfill kind. SQL filter requires `chunk_source='image_asset'` (D22-7 defensive guard). Idempotent. - doctor.ts: `cross_modal_modality_backfill` check surfaces unflagged image-asset chunks with paste-ready `gbrain backfill modality` hint. Tests: - cross-modal-phase1.test.ts (45 cases): regex classification (positive + negative + plural-safe), isAmbiguousModalityQuery, D3 registry, D2 knobsHash diffs across all 7 new knobs, MODE_BUNDLES defaults, resolveSearchMode precedence chain. - cross-modal-hybrid-integration.test.ts (7 cases): PGLite + stubbed gateway. Verifies image-modality calls Voyage and not OpenAI, text calls OpenAI and not Voyage, 'auto' literal normalizes, 'both' mode hits both endpoints, fail-open routes to text on multimodal misconfig. - search-mode.test.ts: updated MODE_BUNDLES + KNOBS_HASH_VERSION assertions (148 cross-suite tests still pass; no regression). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cross-modal/2): Phase 2 image-as-query + D18 path ban + D23-#6 spend cap Phase 2 of the cross-modal search wave. Adds the `search_by_image` MCP op, the SSRF-defended image loader, and the daily per-OAuth-client spend cap on paid Voyage multimodal calls. D17 honest framing applied: Phase 2 ships image→similar-images + image-OCR-text retrieval. True image→full-text- knowledge requires Phase 3's unified column. - src/core/search/image-loader.ts: loadImageInput accepts local path, data: URI, or http(s):// URL. Magic-byte sniff for PNG/JPEG/WebP (no other formats). Hard size cap (10MB local default, 2MB remote default). http(s) path uses fetchWithSSRFGuard from Commit 0: every redirect hop re-resolved via DNS lookup + every record checked against the internal IP deny list. Max 3 redirect hops. 5s total fetch timeout. Pre-flight Content-Length check + post-fetch size guard for lying servers. - src/core/search/by-image.ts: searchByImage runs the image branch always; D13 hybrid intersect runs a parallel text branch when `query` is provided, merged via weighted RRF. Phase 3 will widen the column routing to embedding_multimodal once that lands. - src/core/operations.ts: new search_by_image op (scope: read, NOT localOnly). D18 P0 — when ctx.remote === true AND image_path is set, rejects with permission_denied at handler entry (validateParams would catch it again at dispatch). D5 source-id thread via sourceScopeOpts. D12 per-param length cap enforced via remote-vs-local maxBytes config read at handler entry. D23-#6 pre-flight checkBudget + post-call recordSpend (best-effort; failures don't block response). - src/core/spend-log.ts: BudgetExceededError + checkBudget + recordSpend + getTodaySpendCents. UTC day-aligned aggregation so the cap rolls over deterministically. Local CLI callers (no clientId) bypass the gate entirely. Pre-v0.36 brains without the mcp_spend_log table fail open to spend=0; the migration brings the table in on first start. - src/core/migrate.ts: new migration v67 mcp_spend_log table + indexes for the (client_id, day) and (token_name, day) hot reads. PGLite parity via sqlFor.pglite. - src/core/search/hybrid.ts: RRF_K constant exported so by-image.ts can share the same effective-K math as the main hybrid path. Tests: - cross-modal-phase2.test.ts (15 cases): magic-byte sniffing (PNG + JPEG + WebP positive, GIF rejection), oversized rejection (default + custom cap), data: URI happy path + malformed + decoded-non-image + oversized, invalid input shapes (empty + ftp), SSRF defense via DNS rebinding stub. - search-by-image-op.test.ts (7 cases): D18 remote image_path rejection + local CLI accepts; input validation (missing all three / multiple together); D23-#6 budget block-at-cap + allow-under-cap + local-CLI-bypass; migration v67 mcp_spend_log table applied cleanly. All 166 tests across the cross-modal suite pass; no regression in existing voyage-multimodal / openai-compat-multimodal / search-mode suites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cross-modal/3): Phase 3 unified column + reindex + D8 fail-open + D23-#2 Phase 3 of the cross-modal search wave. Adds the unified multimodal column on content_chunks + the `gbrain reindex --multimodal` sweep + the `search.unified_multimodal` routing flag with D8 source-aware coverage guard + fail-open behavior. D17 honest framing: this is the phase that unlocks true image→full-text-knowledge — Phase 2's searchByImage transparently upgrades to the richer retrieval once the unified column has coverage. D10 reindex-core extraction filed as a follow-up TODO. The existing markdown reindex walks pages and re-imports via importFromFile; this walks content_chunks and re-embeds via the gateway. Patterns rhyme but cores diverge enough that extraction balloons the diff. Both commands stand alone with their own checkpoint + cost-prompt logic. - migrate.ts v68 (embedding_multimodal_column): column-only ALTER on content_chunks. HNSW partial index deferred to post-reindex build (D20: pgvector docs recommend post-load build for HNSW). Both engines. - types.ts SearchOpts.embeddingColumn type widened to include 'embedding_multimodal'. - postgres-engine.ts + pglite-engine.ts searchVector: route to embedding_multimodal column when opts.embeddingColumn set. NO modality filter (unified column carries both text + image content). - hybrid.ts unified routing branch: when search.unified_multimodal=true, bypasses dual-column branching and runs embedQueryMultimodal + searchVector(embedding_multimodal). D8 fail-open: zero rows + not strict-mode → falls through to dual-column text path with structured warning. search.unified_multimodal_only=true bypasses the fallback. - src/commands/reindex-multimodal.ts: `gbrain reindex --multimodal`. D7 lock via tryAcquireDbLock('gbrain-reindex-multimodal'); 6h TTL. Cost prompt + 10s Ctrl-C grace window in TTY; auto-proceeds non-TTY. GBRAIN_NO_REEMBED=1 bypass. Checkpoint at ~/.gbrain/reindex-multimodal-checkpoint.json for resume. D23-#2 auto-flip prompt at coverage=100% completion. - cli.ts: `gbrain reindex --multimodal` dispatch with --limit, --dry-run, --cost-estimate, --no-embed, --yes, --json flags. - doctor.ts: unified_multimodal_coverage check (D21 source-aware) + reports per-source % when search.unified_multimodal is on. Warns at <95% lowest source; fails when unified_multimodal_only=true AND lowest source <99%. Falls open to OK when column not yet present. Tests: - unified-multimodal.test.ts (8 cases): schema migration v68 applies, reindex --dry-run + --cost-estimate + GBRAIN_NO_REEMBED bypass + zero-pending fast-path, hybridSearch unified routing forces voyage endpoint, D8 fail-open routes to text on empty unified, D8 strict blocks text fallback. All 211 tests across the cross-modal + related suite pass; no regression in voyage-multimodal / openai-compat-multimodal / search-mode / intent / search base suites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cross-modal/4): LLM intent escalation for ambiguous modality Commit 4 of the cross-modal search wave (opt-in default off). When `search.cross_modal.llm_intent` is true AND the regex classifier returned 'text' AND `isAmbiguousModalityQuery(query)` fires, hybridSearch awaits a Haiku tie-break via gateway.chat() before routing. The ambiguous-modality gate (introduced in Commit 1) ensures the LLM call only fires on the narrow band where regex misses but a visual noun + reference marker both fire — roughly <1% of queries with the flag on. - src/core/search/llm-intent.ts: new module. `classifyModalityWithLLM` routes through gateway.chat() with a fixed system prompt ("Output exactly one word: text, image, or both"). 1s timeout via AbortController. `parseModality` is a pure exported helper that tolerates trailing punctuation + casing. Fail-open on every error path (gateway unavailable, timeout, parse failure, unrecognized output). - src/core/search/hybrid.ts: escalation branch slots BEFORE the unified routing branch. Gated by: no explicit per-call crossModal opt, regex result == 'text', config flag on, ambiguity heuristic fires. Fail-open to regex result on any error from the LLM tie-break. Tests: - llm-intent-escalation.test.ts (14 cases): parseModality tolerance matrix (text / image / both / trailing punct / whitespace / unrecognized / empty), classifyModalityWithLLM happy paths for all 3 outputs, fail-open on throw / unrecognized output / gateway-not- configured, explicit-fallback-honored. - llm-intent-hybrid-integration.test.ts (6 cases): hybridSearch escalation gate fires ONLY when flag-on + ambiguous; off when flag-off, unambiguous, regex-confident, or explicit per-call opt set; fail-open on LLM throw. All 231 tests across the cross-modal + related suite pass; no regression in voyage-multimodal / openai-compat-multimodal / search-mode / intent / search base suites. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cross-modal/3): verify-gate fixes for full test suite Three small fixes to pass the full unit + E2E sweep after the cross-modal wave commits land. - migrate.ts v67: drop date_trunc('day', created_at) from mcp_spend_log indexes. TIMESTAMPTZ truncation depends on session timezone and isn't IMMUTABLE, so Postgres rejects the function in the index expression with SQLSTATE 42P17. BTREE on (client_id, created_at) covers the per-day rollup query via range scan on created_at — same performance, no IMMUTABLE constraint. - pglite-schema.ts + src/schema.sql: shorten the embedding_multimodal column comment. The longer version contained a comma inside a SQL line comment ("...search.unified_multimodal=true, all queries..."), which broke parseBaseTableColumns in test/schema-bootstrap-coverage (the parser splits on commas at depth-0 before stripping comments, so the comma inside the comment shortened the column-definition part and an "all" token from "all queries" got picked up as the next column name — silently hiding embedding_multimodal from coverage). - schema-embedded.ts: regenerated via `bun run build:schema`. - test/e2e/v030_1-integration-pglite.test.ts: listBackfills assertion extended to include the new `modality` entry registered in src/core/backfill-registry.ts as part of Commit 1. - test/search/knobs-hash-reranker.test.ts: KNOBS_HASH_VERSION assertion updated from 2→3 to match the cross-modal-wave hash-key extension (D2 cache contamination fix). Same shape as the prior v0.32→v0.35 bump. - test/unified-multimodal.test.ts: migrated process.env mutation to withEnv() helper to satisfy the scripts/check-test-isolation R1 rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(cross-modal): VERSION + CHANGELOG + CLAUDE.md + spec doc + llms regen Final docs commit for the cross-modal wave (v0.36.0.0). - VERSION + package.json: bump 0.35.5.1 → 0.36.0.0 - CHANGELOG.md: full Garry-voice release entry with five-commit breakdown, the-numbers-that-matter table, what-this-means-for-you, and the required to-take-advantage-of-v0.36.0.0 block - docs/issues/cross-modal-search.md: cherry-picked from PR #1127 head (164 lines, the original spec doc preserved as historical reference for Phase 2 + 3 background) - CLAUDE.md: Key Files entries for src/core/ssrf-validate.ts, src/core/search/image-loader.ts, src/core/search/by-image.ts, src/core/search/llm-intent.ts, src/core/spend-log.ts, src/commands/reindex-multimodal.ts, plus extension annotations on src/core/search/query-intent.ts, src/core/search/mode.ts, src/core/search/hybrid.ts, src/core/backfill-registry.ts, src/core/migrate.ts (v67 + v68) - llms-full.txt + llms.txt: regenerated via `bun run build:llms` `bun run verify` clean (privacy + proposal-pii + test-names + jsonb + source-id-projection + progress + test-isolation + wasm + admin-build + admin-scope-drift + cli-exec + system-of-record + eval-glossary + typecheck). `bun test test/build-llms.test.ts` clean (7/7). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cross-modal): renumber migrations 67→69 + 68→70 post-master-merge Master shipped its own v67 (`facts_typed_claim_columns`) during the cross-modal wave's review cycle. The merge picked up both side's v67 entries, breaking the migration-distinct-versions test. Renumbering moves cross-modal's table + column ALTER off the collision: - v67 mcp_spend_log → v69 mcp_spend_log - v68 embedding_multimodal_column → v70 embedding_multimodal_column References updated in CHANGELOG, CLAUDE.md, pglite-schema.ts, schema.sql. schema-embedded.ts regenerated. llms-full.txt regenerated. 7006 unit tests pass, 0 fail. No test code touched — just version renumbering plus comment refs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version 0.36.0.0 → 0.36.4.0 Bumping to v0.36.4.0 to land in the queue slot the user requested. No behavior change; pure version bump across VERSION, package.json, CHANGELOG.md header, llms-full.txt regen. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-19 17:14:53 -07:00
3 people	1d5f69fe7a	v0.36.3.0 feat: dynamic embedding column selection for search (#1164) ... * feat: migration v68 — eval_candidates.embedding_column Schema migration ALTERs eval_candidates to add a nullable embedding_column TEXT column. Per-row capture metadata so `gbrain eval replay` reproduces the same column the capture ran against (D16 / CDX-10). NULL-tolerant: pre-v0.36 rows fall back to current default. Renumbered v67→v68 because master claimed v67 for facts_typed_claim_columns during this branch's lifetime. PGLite parity via sqlFor.pglite — same ALTER IF NOT EXISTS. * feat: dynamic embedding column — core (resolver, types, gateway, engines) The read-path foundation for routing search through any populated embedding column, not just OpenAI 1536. src/core/search/embedding-column.ts (new) is the canonical seam. Single source of truth for column → provider/dim/type lookup. Validates registry keys via regex (/^[a-z_][a-z0-9_]*$/), uses Object.create(null) + Object.hasOwn so 'constructor' and other inherited names can't masquerade as registered columns. Identifier-quoting on SQL interpolation as defense in depth. src/core/types.ts widens SearchOpts.embeddingColumn to accept ResolvedColumn descriptors at the engine boundary; adds EmbeddingColumnConfig + ResolvedColumn exports. src/core/config.ts merges embedding_columns + search_embedding_column from the DB plane via loadConfigWithEngine, mirroring the existing embedding_multimodal_model pattern. Handles the no-file case so env-only Postgres installs see DB-plane overrides (codex /ship #3). src/core/ai/gateway.ts: embedQuery(text, opts) + embed(texts, opts) accept embeddingModel + dimensions overrides. isAvailable(touchpoint, modelOverride?) so hybrid asks 'is the active column's provider reachable?' not 'is the global default reachable?' (CDX-4 / D10). Engines: searchVector accepts ResolvedColumn descriptors via normalizeEngineColumn; engine code is config-free and unit-testable. getEmbeddingsByChunkIds(ids, column?) so cosineReScore hydrates from the active column instead of always 'embedding' (CDX-3 / D9). Identifier-quoting belt at the SQL boundary. src/core/eval-capture.ts threads embedding_column from hybridSearch meta into the persisted capture row. * feat: dynamic embedding column — integration (hybrid, ops, doctor) Wires the resolver into hybridSearch, the query op, doctor, and the config command. src/core/search/hybrid.ts: resolves the column once at the boundary, threads the descriptor into engine calls, routes embedQuery through the resolved column's provider/dims, and calls isCacheSafe (not isDefaultColumn) for cache skip so user overrides of the 'embedding' builtin can't leak across vector spaces (CDX-4). cosineReScore now hydrates from the active column. src/core/search/mode.ts: KNOBS_HASH_VERSION 2→3, append-only new fields col= and prov= alongside floor_ratio. Cache rows from different columns or providers now sit in different keyspaces — cross-column contamination impossible. src/core/operations.ts: query op accepts embedding_column param for per-call A/B benchmarking. search op (keyword-only) deliberately does NOT (CDX-9 / D15) — would be silent UX. src/commands/doctor.ts: new embedding_column_registry check. Batch format_type probe (D13) catches dim drift that information_schema.columns.udt_name can't. Batch pg_indexes probe (D5) warns on missing HNSW. Coverage % on active column, gates at <90% (D14), short-circuits on empty brains (codex /ship #5). src/commands/config.ts: validates embedding_columns JSON shape at set time, runs the coverage gate when setting search_embedding_column, uses Object.hasOwn for the registry lookup. src/commands/eval-replay.ts: replay re-runs queries against the captured embedding_column so post-flip-config replays don't surface as false-positive regressions. * test: dynamic embedding column — unit + e2e coverage 50 unit cases for the resolver (resolution chain, registry merge, validation, prototype pollution, descriptor passthrough, isCacheSafe, normalizeEngineColumn). 8 gateway override cases — embeddingModel + dimensions flow into providerOptions, isAvailable(touchpoint, override) routes to the right recipe, unknown models throw clean. 4 cosineReScore + 6 ops + 5 knobs-hash + 7 mode + 9 PGLite E2E + 7 Postgres E2E + 5 eval-replay column metadata. Postgres E2E (gated on DATABASE_URL) covers halfvec(2560) end-to-end on real pgvector, EXPLAIN-visible HNSW index on the alternate column, format_type-based dim drift catch, and the <90% coverage gate. Pins every codex /ship fix: prototype-pollution rejection ('constructor' as column name), descriptor passthrough validation (rejects SQL-shaped strings in dimensions), isCacheSafe semantics (space-based, not name-based). Total: 141 new + extended cases, all green. * chore: bump version and changelog (v0.36.3.0) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: sync to v0.36.3.0 Add CLAUDE.md key-files entry for src/core/search/embedding-column.ts. Annotate hybrid.ts, gateway.ts, doctor.ts, and migrate.ts entries with v0.36.3.0 wave changes (ResolvedColumn threading, embedQuery model override, embedding_column_registry check, migration v68). Document knobs_hash v=2 → v=3 bump under the Search Mode section. Regenerate llms-full.txt from the updated CLAUDE.md so the auto-checked bundle matches source (build-llms.test.ts CI guard). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): two CI failures from v0.36.3.0 1. test/loadConfig-merge.test.ts: update the 'returns null when base config is null' contract test. Pre-v0.36 the function returned null for null base; the codex /ship #3 fix changed that to synthesize a minimal `{ engine: 'postgres' }` so env-only installs see DB-plane overrides. Test now pins the new contract + adds a round-trip case asserting the merge actually surfaces `embedding_columns` / `search_embedding_column` set via gbrain config set on a null base. 2. test/schema-bootstrap-coverage.test.ts was failing because eval_candidates.embedding_column (added by migration v68) wasn't covered by applyForwardReferenceBootstrap. Fix: add the column to PGLITE_SCHEMA_SQL's eval_candidates CREATE TABLE definition (and src/schema.sql for parity) so fresh installs get it natively. The coverage test's third tier (schemaCreateTableCols) now finds it. Regenerated schema-embedded.ts via bun run build:schema. Schema-blob path is cleaner than COLUMN_EXEMPTIONS — fresh installs skip the migration entirely; upgrade installs still run v68. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-18 21:26:12 -07:00
4 people	af7e5379c2	v0.35.6.0 feat(search): floor-ratio gate for metadata boost stages (closes #1091) (#1129) ... * v0.35.6.0 feat(search): floor-ratio gate for metadata boost stages Opt-in score-based gate on the three metadata-axis boost stages (backlink, salience, recency) inside `runPostFusionStages`. When `SearchOpts.floorRatio` or `search.floor_ratio` config is set, each stage skips results whose post-cosine-rescore score is below `floorRatio * topScore`. Default undefined preserves prior behavior bit-for-bit. Prevents weak-overlap candidates from accumulating metadata boosts and leapfrogging the legitimate primary hit on dense-embedder corpora. Built on the contributor PR from @jayzalowitz (PR #1091, SkyTwin twin-memory layer). Refactored on top: threshold is computed ONCE at runPostFusionStages entry instead of per-stage (single-baseline semantic, order-independent); knobsHash bumped 2->3 so a no-floor cache write can't be served to a floor-enabled lookup; NaN scores skip the boost instead of bypassing the gate; SearchOpts/config/MODE_BUNDLES integration replaces the PR's PostFusionOpts-only surface; no env var (resolveSearchMode is pure by design). Three correctness issues codex outside-voice review caught and this landed with fixed: - Cache contamination via knobsHash() (same bug class as v0.32.3 CDX-4 hotfix for the other search-lite knobs) - NaN scores would have bypassed the gate (NaN < threshold is false in JS); realistic on Voyage flexible-dim / zembed-1 Matryoshka dim drift - Negative top scores would have broken the "single result trivially eligible" claim; gate now disables on no-positive-signal inputs Scope: gates metadata stages only. Exact-match boost (applyExactMatchBoost) runs independently as a lexical-relevance signal by design. Cross-source floor stays global (per-source deferred to v0.36 if federated-read users hit the suppression). Default-on for any mode bundle deferred until gbrain-side ablation against longmemeval / whoknows / suspected-contradictions / BrainBench-Real (TODOS.md). Plan + 9-decision review trail (D1-D9): ~/.claude/plans/swift-sniffing-nygaard.md. Empirical motivation, failure-mode framing, dense-embedder targeting, and the 0.85 starting value all from @jayzalowitz's labeled-retrieval ablation. Integration shape is gbrain-side. Test surface: 30+ new cases (computeFloorThreshold edge cases including T1a NaN / T1b negative top, three boost-function gate parity tests including T6 IRON-RULE applyRecencyBoost regression, runPostFusionStages single-baseline composition pin, KNOBS_HASH_VERSION bump from 2 to 3, floor-ratio-changes-hash cache-contamination prevention, loadOverridesFromConfig coverage for search.floor_ratio config key). bun run verify clean; full unit suite 6753 pass / 0 fail. Co-Authored-By: Jay Zalowitz <jayzalowitz@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: rewrite v0.35.6.0 CHANGELOG ELI10-lead-first; codify the rule in CLAUDE.md CHANGELOG entry for v0.35.6.0 was readable only by someone who already understood gbrain's internals (RRF, knobsHash, MODE_BUNDLES, runPostFusionStages, Matryoshka, CDX-4). Rewrote it so the first ~150 words explain what shipped in everyday English, with a concrete worked example, before any file paths or function names appear. Itemized changes section keeps the technical precision for engineers who need it. Then codified the rule in CLAUDE.md so future release entries land the same way. The "Release-summary template" section now has an iron rule: "lead ELI10, get precise after." No file paths or internal constants in the first 150 words; user-visible behavior change first; everyday-language column headers in any tables. Technical precision is required (the entry is still the technical record) but lives BELOW the plain-English lead, never before it. Smell test: if a reader who has never opened gbrain can walk away from the first 150 words knowing what shipped and whether they care, the entry passes. bun run build:llms regenerated to pick up the CLAUDE.md change (CI guard test/build-llms.test.ts pins committed bundles against fresh generator output). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jay Zalowitz <jayzalowitz@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-17 15:41:43 -07:00
3 people	baf1a47798	v0.35.0.0 feat: ZeroEntropy zembed-1 + zerank-2 reranker (#1008) ... * feat(ai): add ZeroEntropy recipe + reranker touchpoint type Widens `TouchpointKind` with `'reranker'`, adds `RerankerTouchpoint` interface, extends `Recipe.touchpoints` and `AIGatewayConfig` to carry reranker model state. Registers `zeroentropyai` recipe (zembed-1 embeddings + zerank-{2,1,1-small} rerankers) in the recipe registry. Recipe declares the 7 Matryoshka dims (2560/1280/640/320/160/80/40), Voyage-style dense-payload hedge (chars_per_token=1, safety_factor=0.5), and 5MB rerank payload cap. Pinned by test/ai/zeroentropy-recipe.test.ts including F1 regression (implementation literal is 'openai-compatible') and F2 regression (base_url_default ends with /v1, no doubling). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai/dims): thread input_type 4th-arg + ZE flexible-dim allowlist `dimsProviderOptions` gains an optional `inputType?: 'query' | 'document'` 4th param so asymmetric providers (ZE zembed-1, Voyage v3+) can route query-side vs document-side encoding. Per-model filtering inside the openai-compatible branch keeps `input_type` from leaking to symmetric providers (OpenAI text-3, DashScope, Zhipu) that would 400 on it. Adds `ZEROENTROPY_VALID_DIMS` allowlist (2560/1280/640/320/160/80/40), `supportsZeroEntropyDimension(modelId)`, and `isValidZeroEntropyDim(dims)`. Throws `AIConfigError` with paste-ready fix hint when zembed-1 is configured with an invalid dim (most common: defaulting to 1536 from DEFAULT_EMBEDDING_DIMENSIONS). The 4th-arg is optional; existing call sites (1 production + N tests across Voyage/OpenAI/DashScope/Zhipu/MiniMax) compile unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai/gateway): zeroEntropyCompatFetch + embedQuery + gateway.rerank() Two seams land together because they share the same recipe + auth path. zeroEntropyCompatFetch handles ZE's non-OpenAI-compatible wire shape: - URL rewrite: SDK's `${base_url}/embeddings` -> `${base_url}/models/embed` - Body inject: `input_type` (default 'document'; 'query' when threaded via providerOptions) + explicit `encoding_format: 'float'` - Response rewrite: `{results: [{embedding}]}` -> `{data: [{embedding, index}]}` so the AI SDK's openai-compat schema validates - `usage.prompt_tokens` injected from `total_tokens` (Voyage hit the same SDK schema requirement at :655) - Layer 1 (Content-Length) + Layer 2 (per-embedding size) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because the Voyage cap tests do structural source-text greps pinning the Voyage name) - Wired in `instantiateEmbedding()` via the existing `recipe.id === 'voyage' ? voyageCompatFetch : ...` ternary pattern embedQuery(text) routes `inputType: 'query'` through dimsProviderOptions for the search hot path. Companion to embed(texts) which now takes an optional 2nd-arg inputType (defaults to undefined -> 'document' for asymmetric providers). gateway.rerank() is the new native HTTP path (no AI-SDK reranking abstraction). Resolves the configured reranker model via `getRerankerModel()` (new accessor), parses + asserts the model is in the recipe's touchpoint.reranker.models allowlist (CDX2-F11: assertTouchpoint does not enforce allowlists for openai-compatible recipes — rerank() does it directly). Posts to `${recipe.base_url}/models/rerank` with bearer auth. Returns `RerankResult[]` sorted by `relevanceScore`. Errors classify into `RerankError.reason: 'auth' | 'rate_limit' | 'network' | 'timeout' | 'payload_too_large' | 'unknown'`. 5s default timeout. Pre-flight payload guard rejects bodies over `recipe.max_payload_bytes` BEFORE any HTTP call so applyReranker can fail-open without burning a round-trip. `_rerankTransport` + `__setRerankTransportForTests` mirror the embed test seam. `AIGatewayConfig.reranker_model` + isAvailable('reranker') branch + configureGateway / reconfigureGatewayWithEngine extensions thread the reranker model through the same state path as embedding/expansion/chat. `applyResolveAuth` + `defaultResolveAuth` widen the touchpoint param to include `'reranker'`. `KnownTouchpointKey` + `getTouchpoint()` in model-resolver widen to cover `'reranker'`. Pinned by: - test/ai/embedQuery.test.ts (8): returns single Float32Array, threads input_type='query' for ZE, drops field for OpenAI text-3, back-compat: legacy embed() callers without 4th arg keep their previous Voyage no-input_type shape - test/ai/rerank.test.ts (21): URL (F2 regression — no /v1/v1/), body shape, bearer header, response parsing, error classification across 6 HTTP shapes, payload pre-flight (no transport call), allowlist enforcement - test/ai/zeroentropy-compat-fetch.test.ts (14): structural source assertions for the shim that mirror test/voyage-response-cap.test.ts — URL rewrite path, body injection, response rewrite, usage.prompt_tokens injection, OOM caps Layer 1 + Layer 2 + instanceof rethrow, instantiateEmbedding wiring branch Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(search): applyReranker + rerank-failure audit + hybrid wire-in src/core/search/rerank.ts — the call-site abstraction. Slices the top `opts.topNIn` deduped candidates, sends to gateway.rerank(), reorders by relevanceScore desc, appends the un-reranked tail in its original RRF order (recall protection). Fail-open on every RerankError.reason: logs via `logRerankFailure` and returns the input array unchanged. Stamps `rerank_score` onto reordered items. `topNOut: null` is the explicit "don't truncate" signal — distinct from `undefined` (fall through to mode bundle); pin in test (CDX2-F16). src/core/rerank-audit.ts — failure-only JSONL audit at `~/.gbrain/audit/rerank-failures-YYYY-Www.jsonl` (ISO-week rotation; mirrors `src/core/audit-slug-fallback.ts`). Exports `logRerankFailure` + `readRecentRerankFailures(days)`. **No `logRerankSuccess`** — CDX2-F22 deliberately drops success-event logging: writing once per tokenmax search is hot-path I/O churn AND success events leak query volume + timing into a local audit. The doctor check reads `search.reranker.enabled` first so "no events in window" gets interpreted correctly (disabled -> healthy by definition; enabled -> healthy because nothing failed). Query text is SHA-256-prefix-hashed (8 hex chars) for privacy. Honors `GBRAIN_AUDIT_DIR`. src/core/search/hybrid.ts — slots `applyReranker` between `dedupResults()` and `enforceTokenBudget()` in the main RRF path. Resolution: per-call `opts.reranker` overrides; otherwise pulled from the resolved mode bundle (tokenmax -> enabled, others -> disabled in commit 5). Cache rows store final reranked results; the bumped knobsHash (commit 5) ensures rows can't leak across reranker configs. src/core/types.ts — adds `SearchOpts.reranker` as a structural type so callers can pass per-call overrides; runtime type lives in src/core/search/rerank.ts (avoids circular import). Tests: - test/search/rerank.test.ts (14): reorder, tail preserve, fail-open on every error class, topNOut null vs number, score stamping, empty + enabled=false pass-through - test/rerank-audit.test.ts (10): JSONL round-trip, error_summary truncated to 200, corrupt rows skipped, missing dir -> [], ISO-week rotation walks current + previous week, no logRerankSuccess export (CDX2-F22 contract) - test/search/hybrid-reranker-integration.test.ts (6): reranker fires when enabled, doesn't when disabled, reorders correctly, preserves tail, stamps rerank_score, fail-opens on rerankerFn throw — uses PGLite + stubbed embed transport, no API keys Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(search/mode): reranker mode-bundle fields + KNOBS_HASH_VERSION v=2 Extends `ModeBundle` with five reranker fields: `reranker_enabled`, `reranker_model`, `reranker_top_n_in`, `reranker_top_n_out`, `reranker_timeout_ms`. Per-mode defaults: - conservative -> enabled=false (cost-sensitive) - balanced -> enabled=false (opt-in via search.reranker.enabled) - tokenmax -> enabled=true (the high-cost-tolerant tier; ~$0.0003/query) Defaults model to `zeroentropyai:zerank-2`, topNIn=30, topNOut=null (no truncate by default; preserves tokenmax's searchLimit=50 end-to-end per CDX2-F16), timeout_ms=5000. `SearchKeyOverrides` + `SearchPerCallOpts` + `resolveSearchMode.pick` all extend to thread the new fields through the resolution chain (per-call -> per-key config -> mode bundle -> default). `loadOverridesFromConfig` adds parsers for the five new `search.reranker.*` config keys. `top_n_out` parsing distinguishes three input shapes (CDX2-F15): key absent -> undefined (fall through to mode bundle) 'null'|'none'|empty -> explicit null (no truncate) positive integer -> that number `SEARCH_MODE_CONFIG_KEYS` extends so `gbrain search modes --reset` clears the reranker overrides too. **KNOBS_HASH_VERSION bumps 1 -> 2** (CDX1-F14). Five new entries appended to `parts[]` (append-only convention CDX2-F13; reordering existing fields would silently rebuild every existing cache row). Includes `reranker_timeout_ms` so a 5s -> 100ms change invalidates stale rows (CDX2-F14: more fail-opens = different search behavior). Mid-rolling-deploy note (CDX2-F12): v=1 and v=2 processes produce distinct cacheRowIds for the same (source_id, query_text). Expect a temporary hit-rate dip + cache-row doubling for hot queries. Clears naturally within `cache.ttl_seconds` (default 3600s). src/commands/search.ts extends `KNOB_DESCRIPTIONS` with five new entries so `gbrain search modes` renders them. test/search-mode.test.ts extends the three bundle fixtures and bumps the KNOBS_HASH_VERSION expectation to 2. Pinned by test/search/knobs-hash-reranker.test.ts (13): each of the 5 reranker fields independently flips the hash, top_n_out=null renders stable, append-only convention enforced via source-position assertion. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(doctor): probeRerankerConfig + reranker_health check `gbrain models doctor` gains two new probes: - `probeRerankerConfig` (zero-network) validates that the configured reranker model resolves through the recipe registry, that the recipe declares a `reranker` touchpoint, and that the model is in `touchpoint.models[]`. Direct allowlist check here — assertTouchpoint does not enforce allowlists for openai-compatible recipes (CDX2-F11). Surfaces paste-ready `gbrain config set search.reranker.model <zerank-2|zerank-1|zerank-1-small>` fix hint. - `probeRerankerReachability` (1-token-equivalent) sends a minimal `{query: "probe", documents: ["probe"]}` rerank to verify auth + URL. Failures classify via `classifyError` into auth/rate_limit/network/ unknown. Skipped silently when reranker is unconfigured. Also extends `probeEmbeddingConfig` with a `providerId === 'zeroentropyai'` branch that catches the silent-1536-default bug class for zembed-1 configurations (same posture as the existing Voyage branch). `ProbeResult.touchpoint` widens to include `'reranker_config'`. `gbrain doctor` adds `checkRerankerHealth` to both the abbreviated (doctorReportRemote) and full (runDoctor) check sets. Logic: 1) Read `search.reranker.enabled` first. Disabled + no failures => 'reranker disabled'. Enabled + no failures => healthy. 2) Walk last 7 days of ~/.gbrain/audit/rerank-failures-*.jsonl. 3) ANY auth failure warns (config-time problem the probe should have caught — surface it). 4) ANY payload_too_large failure warns (workload mismatch). 5) Transient (network/timeout/rate_limit) warns at >=5 in window. Below that they're noise; reranker fails open anyway. CDX2-F21 blind-spot fix: reading enabled state first means "no events" gets interpreted correctly — never confuses "never-used" with "success logging broken" (the latter is impossible because there is no success logging by design, CDX2-F22). Engine-agnostic; file-based + one config-key read. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): ZeroEntropy live API round-trip + wire into Tier 2 CI test/e2e/zeroentropy-live.test.ts exercises the full stack against the real api.zeroentropy.dev: embed (default 2560-dim + flexible 1280), embedQuery (asymmetric query side), batch embed (3 distinct vectors), rerank (3 docs sorted by relevance score, photosynthesis-relevant docs beat the irrelevant cat doc), rerank with topN truncation. Gated on `ZEROENTROPY_API_KEY`: every test prints `[skip]` and returns early without assertions when the env var is unset, so fork PRs and contributor machines without a ZE account stay green. CI wire-up: `.github/workflows/e2e.yml` Tier 2 step adds `test/e2e/zeroentropy-live.test.ts` to its `bun test` invocation and exposes `ZEROENTROPY_API_KEY: ${{ secrets.ZEROENTROPY_API_KEY }}` to the runner. The secret is set on garrytan/gbrain at the repo scope (separately from this commit — set via `gh secret set` so the value never lands in source). Tier 1 stays mechanical (no API keys); Tier 2 is the natural home for provider-live tests because it's already the API-keyed lane. Cost: each full run fires ~6 small HTTP calls totaling well under a cent at the published $0.025/1M-token rate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.33.3.0 feat: ZeroEntropy zembed-1 + zerank-2 reranker Release notes for the ZeroEntropy support wave: zembed-1 embeddings (flexible-dim 2560/1280/640/320/160/80/40, asymmetric input_type) and zerank-2 cross-encoder reranking land as a new openai-compatible recipe alongside OpenAI/Voyage. Reranker defaults ON for tokenmax mode, OFF for conservative/balanced (~$0.0003/query at tokenmax topNIn=30; rounding error vs the tier's $700/mo Opus pairing per the CLAUDE.md cost matrix). Search now ends with `RRF -> dedup -> reranker -> token-budget` when reranker is enabled; fails open to RRF order on any error class (audit-logged at ~/.gbrain/audit/rerank-failures-*.jsonl). `KNOBS_HASH_VERSION` bumps 1 -> 2 to fold reranker config into the query_cache row key. Rolling-deploy operators should expect a temporary cache hit-rate dip + cache-row doubling for hot queries (clears naturally within `cache.ttl_seconds`, default 3600s). Files in this commit are pure docs / version bump: - VERSION + package.json bump to 0.33.3.0 - CHANGELOG.md release-summary entry with "How to take advantage" block - CLAUDE.md Key Files annotations for the new recipe + rerank.ts + rerank-audit.ts + gateway extensions - docs/ai-providers/zeroentropy.md one-pager (setup, knob reference, failure observability, troubleshooting table) - skills/migrations/v0.33.3.md (purely informational: no required user action; reranker is opt-in everywhere, ZE embedding is opt-in) - llms-full.txt regenerated to match CLAUDE.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-15 06:48:36 -07:00