From d6db3f0ce320ab9e3dd3d4ff60655cff8cb01389 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 30 May 2026 12:07:38 -0700 Subject: [PATCH] =?UTF-8?q?v0.41.34.0=20feat(search):=20retrieval=20cathed?= =?UTF-8?q?ral=20=E2=80=94=20max-pool=20+=20title=20+=20alias=20+=20eviden?= =?UTF-8?q?ce=20(#1657)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(search): per-page max-pool in searchVector (both engines) T1 of the retrieval-cathedral wave (supersedes #1616). Vector search returned chunk-grain top-k with no DISTINCT ON, so a page could be represented by a weak chunk while a hub page's chunks crowded a distinct page's strong chunk out of the candidate set entirely. Keyword search always pooled per page; the vector path did not. - New shared buildBestPerPagePoolCte() in sql-ranking.ts — single source of truth consumed by searchKeyword + searchVector across postgres + pglite, so the two engines can't drift (the recurring parity bug class). - searchVector both engines: compute score as a select-list expr (HNSW ORDER BY stays pure-distance), pool DISTINCT ON (slug) over the full candidate set before the user LIMIT, deterministic tiebreak (slug, score DESC, page_id ASC, chunk_id ASC). - All keyword pooling blocks refactored onto the shared builder (DRY). - Regression test: a hub page's chunks no longer crowd out a distinct page's strong chunk; results are one-per-page by best chunk. Fails on old path. Verified: real-Postgres engine-parity 22/22, PGLite hermetic suite green. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(search): title-phrase boost (page.title first-class signal) T2 of the retrieval-cathedral wave. A query that is a phrase from a page's title ("Greek amphitheater" -> "The Mingtang - Indoor Greek Amphitheater") matched a weak body chunk instead of being recognized as a title hit. Names of things deserve weight. - New pure title-match.ts: isTitlePhraseMatch (contiguous token-run inside page.title OR exact full-title match). Precision guards: >= 2 content tokens OR exact full-title; stopword filter; token-boundary match (no raw substring). Reused by the eval later so production + bench can't drift. - applyTitleBoost post-fusion stage in hybrid.ts: reads page.title (not the brittle "first chunk"), floor-ratio-gated, stamps title_match_boost for --explain, never touches base_score (the agent's dedup confidence). - ModeBundle.title_boost knob (1.25, on in all modes - cheap gated correctness fix), search.title_boost config key, dashboard description. - KNOBS_HASH_VERSION 6 -> 7 so a boost-on cache write can't serve a boost-off lookup; all version-pin + canonical-bundle assertions updated. - 18 new tests (matcher 13 + stage 5); typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(search): page_aliases data layer (T3 foundation) Free-text alias resolution for search. gbrain stored a page's chosen names in pages.frontmatter `aliases:` JSONB but search never consulted them, so a query like "Hall of Light" or "明堂" couldn't surface the "Mingtang" page. DELIBERATELY SEPARATE from slug_aliases (re-grounded against current code): - slug_aliases: old-slug -> canonical-slug (wikilink/get_page redirect, populated only from concept-redirect conversions) - page_aliases: normalized free-text name -> canonical slug (search hop) Overloading slug_aliases would muddy two distinct semantics, so this is a new table, not an extension (honors DRY by keeping concepts separate). - src/core/search/alias-normalize.ts: ONE normalizeAlias() (NFKC + lowercase + ws-collapse + quote-strip) + normalizeAliasList() shared by the write (ingest) and read (search) paths so they match on the same key (CQ2). - Migration v108 page_aliases (source_id, alias_norm, slug); btree (source_id, alias_norm) for indexed-equality hop, NOT ILIKE; unique TRIPLE (not source_id+alias_norm) so two pages may claim one alias — collisions reported + resolved at query time, not blocked at ingest (Codex#8). Mirror in pglite-schema.ts; Postgres fresh gets it from the migration. - engine.resolveAliases(aliasNorms, {sourceId|sourceIds}) read + setPageAliases(slug, source, aliasNorms) write, both engines, source-scoped. - 17 tests: normalize round-trip, collision, source-scope, replace, clear. Ingest projection + the hybridSearch alias hop land next (T3 wiring). Co-Authored-By: Claude Opus 4.8 (1M context) * feat(search): alias hop + ingest projection (T3 wiring) Wires the page_aliases data layer into ingest (write) and hybridSearch (read) so a query that is a page's declared chosen name surfaces that page — the named-thing class neither max-pool nor title-boost can fix (true synonyms with zero surface overlap: "Hall of Light" / "明堂" -> the Mingtang page). - Ingest projection (import-file.ts): after the page write commits, normalizeAliasList(frontmatter.aliases) -> engine.setPageAliases. Always called (even []) so removing an alias clears its row; content_hash includes non-timestamp frontmatter so alias edits reach this path, not the skip branch. Fail-soft + pre-v108-safe (isUndefinedTableError swallowed). - applyAliasHop (hybrid.ts), AFTER rerank so a named query reliably surfaces its page: FULL normalized-query exact match only (no substring/n-grams), skip >6-token prose queries, present-boost 1.10x / inject absent canonical at top-of-organic + epsilon (never absolute 1.0, D3), collisions alpha-ordered + capped at 3, fail-open on pre-v108 / lookup error (D9). Stamps alias_hit for the T4 evidence contract. - SearchResult.alias_hit attribution field. - 8 tests: inject/boost/CJK/no-match/long-skip/collision + ingest projection round-trip + alias-removal-clears. 73 pass across the T1/T2/T3 + import suite. Backfill of existing pages' aliases lands as T8 (reindex --aliases). Co-Authored-By: Claude Opus 4.8 (1M context) * feat(search): evidence/create_safety contract + search→cheap-hybrid + per-call mode (T4) The agent-facing fix for the incident's ROOT behavior: tonight the agent read a single blended 0.64 score, decided "no strong match, safe to write a new page", and wrote a duplicate on a developed concept page. A blended RRF/cosine score is not a calibrated probability, so the don't-duplicate decision must key off WHY a page matched, not a raw number. - evidence.ts: classifyEvidence (alias_hit > exact_title_match > high_vector_match > keyword_exact > weak_semantic) + createSafetyFor (exists | probable | unknown). stampEvidence runs at the end of every hybrid return path (main + both keyword fallbacks). SearchResult gains evidence + create_safety. The agent keys don't-duplicate off create_safety='exists', not a score threshold. - search op → cheap-hybrid everywhere (D4/D15): full vector+keyword+RRF+pool+ title+alias, expansion OFF (no per-call LLM cost); `query` stays full-control. search.mcp_keyword_only escape hatch (D17) keeps the old keyword-only behavior for operators who don't want query text sent to an embedding provider. - Alias hop + evidence now also run on the keyword-only fallback paths (the named-thing fix is most valuable exactly when vector is unavailable). - Per-call `mode` (D5): honored ONLY for local/trusted callers (ctx.remote=== false) so a remote OAuth client can't escalate to costly tokenmax; local + unknown mode rejects loudly; threaded into resolveSearchMode + the cache key. - 30 tests (evidence classifier incl. before/after-incident cases, per-call mode gate, alias hop). Updated mcp-eval-capture to the new cheap-hybrid contract. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(cli): reconcile `gbrain search` dispatch (T5) After T4 made the `search` op cheap-hybrid, `gbrain search "x"` already does the right thing — but `gbrain search modes/stats/tune` would have run a hybrid search for the literal word "modes" instead of opening the config dashboard (the op intercepts before the unreachable handleCliOnly dashboard path). Add a pre-dispatch interception in main(): `search` + subArgs[0] in {modes,stats,tune} → runSearch dashboard (with the v0.41.6.0 read-only connect+ dispatch 10s timeout preserved); everything else (free-text) falls through to the cheap-hybrid `search` op. Subprocess test pins all three routes: modes/stats → dashboard, free-text → search op ("No results", not "Unknown subcommand"). Co-Authored-By: Claude Opus 4.8 (1M context) * feat(eval): NamedThingBench retrieval-quality gate (T6) The eval that makes the retrieval-maxpool incident impossible to reintroduce silently. 7 query families, each a failure class the incident exposed: title-substring, generic-to-named, alias-synonym, multi-chunk-dilution, short-vs-rich, graph-relationship, hard-negative. - src/eval/retrieval-quality/harness.ts: pure scoring (Hit@1/Hit@3/MRR per family) + injected SearchFn (CLI uses hybridSearch; tests stub it) + evaluateGate. D12 gate: hard-gate the families that ARE the bug from day one (title-substring Hit@1>=0.95, alias-synonym Hit@1>=0.98, dilution Hit@3=1.0), warn-then-enforce the softer families. Env-overridable floors. - `gbrain eval retrieval-quality [--json] [--source]` + dispatch in eval.ts. Exit 0 PASS / 1 FAIL / 2 USAGE. - Synthetic fixture (placeholder names only, privacy-grep guarded) + hermetic gate test: seeds a synthetic brain, forces the keyword+title+alias path (embed transport stubbed to throw — free, deterministic), asserts the bug families pass. The vector max-pool guarantee is pinned separately by searchvector-maxpool.test.ts. - CI gate: the hermetic test is a normal unit test, so it runs in every PR shard — the gate is live on every change. - 23 tests (harness unit + hermetic gate + fixture privacy guard). Co-Authored-By: Claude Opus 4.8 (1M context) * feat(telemetry): rank-1 score drift signal (T7) Standing observability so a retrieval regression is caught before a human hits it in chat (like tonight). Aggregate columns on search_telemetry (NOT per-query rows, D10): sum_rank1_score + count_rank1 + 3 coarse buckets (<0.6 / 0.6-0.85 / >=0.85). The mean rank-1 base_score is the headline; a downward drift = retrieval quality regressing. - hybrid.ts: capture rank-1 base_score at all three return paths, thread through emitMeta → recordSearchTelemetry opts (like results_count). - telemetry.ts: Bucket + record + flush ON CONFLICT-add + readSearchStats expose avg_rank1_score (null when no samples — no NaN) + rank1_distribution. - Migration v109 ADD COLUMN IF NOT EXISTS (both engines; search_telemetry lives only in migration v57, so the v57+v109 chain covers fresh + upgrade). Columns exempted in schema-bootstrap-coverage (no forward-ref index → no bootstrap need). - `gbrain search stats` surfaces the avg + bucket line; JSON envelope auto-carries the fields. "true-positive" wording dropped per Codex#14 — production has no labels, so this is an unlabeled rank-1 score histogram; labeled calibration lives in NamedThingBench (T6). - 3 round-trip tests (mean+buckets, no-result excluded, empty=null). Co-Authored-By: Claude Opus 4.8 (1M context) * feat(reindex): gbrain reindex --aliases backfill (T8) Import-time projection (T3) covers new + changed pages; this backfills EXISTING pages whose frontmatter `aliases:` predate v108 / the projection. Walks listAllPageRefs (cheap cross-source (source_id, slug) enumeration), reads each page's frontmatter aliases, writes page_aliases via setPageAliases. Idempotent (setPageAliases replaces) so re-running is convergent — no op-checkpoint needed (fast, no embedding). --dry-run reports would-write counts, --source narrows, --limit caps, --json envelope, progress reporter. Wired into the `reindex` dispatch alongside --markdown / --multimodal. 4 tests: backfill from array + comma-scalar frontmatter, --dry-run writes nothing, idempotent second run. Co-Authored-By: Claude Opus 4.8 (1M context) * test(search): pre-migration fail-open regression (T9) Pins that pre-v108 brains (no page_aliases table) keep working: applyAliasHop returns input unchanged + doesn't throw, importFromContent with frontmatter aliases still imports (projection swallows table-missing via isUndefinedTableError), and resolveAliases surfaces the error for the caller to catch. Completes the T9 mandatory regression set (dilution → searchvector-maxpool, dispatch → cli-search-dispatch, MCP contract → mcp-eval-capture, engine parity → engine-parity 22/22, pre-migration → here). Co-Authored-By: Claude Opus 4.8 (1M context) * feat(search): Phase-0 retrieval diagnostic — `gbrain search diagnose` (T0) The operator-facing trace the user runs against the production brain to pin which retrieval layer surfaces (or misses) a target page — the diagnostic the plan front-loaded so we don't ship a fix that doesn't move the incident. `gbrain search diagnose "" --target [--json] [--source]` reports, for the target: keyword rank+score, vector rank+score (skipped/graceful if no embedding provider), whether the query is a registered alias, and the hybrid final rank + evidence + create_safety + which boosts fired (title/alias). The verdict names the layer that surfaces the target at rank 1 (or "none"), telling you whether the lever is max-pool/innerLimit (vector) vs title/alias. Wired into the `search` dispatch alongside modes/stats/tune (60s timeout since it runs real retrieval). 2 hermetic tests (alias-query trace + title-phrase trace). For the Mingtang incident, run: gbrain search diagnose "Greek amphitheater" --target projects/new-greek-theater/concept_v0 Co-Authored-By: Claude Opus 4.8 (1M context) * docs(retrieval): corrected incident record + named-thing layers + glossary (T10) - RETRIEVAL_MAXPOOL_INCIDENT.md: replaces closed PR #1616's RFC with the verified record — what happened, the disease, the corrections to the RFC's mechanics (search was keyword-only, --mode unthreaded, hybrid already pooled at dedup, aliases dead to search), the four-layer fix that shipped, and the triage commands (search diagnose / reindex --aliases / search stats / eval retrieval-quality). - RETRIEVAL.md: new "Named-thing retrieval" section documenting per-page pool + title boost + alias hop + the evidence contract, reconciling the doc with the shipped pipeline (closes the doc/reality gap). - metric-glossary.ts + regenerated METRIC_GLOSSARY.md: Hit@1, Hit@3, avg_rank1_score (drift signal, not labeled accuracy), and create_safety (the evidence contract) now carry plain-English glossary entries. Co-Authored-By: Claude Opus 4.8 (1M context) * test(eval): NamedThingBench fixture privacy guard via slug-shape (T6 fixup) The banned-name literal list itself tripped check-privacy/check-test-real-names. Replace it with the load-bearing assertion: every fixture slug must be an *-example placeholder (no real brain page can be referenced). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(search): source-isolation in per-page pool + alias hop (P0, codex adversarial) Codex outside-voice caught two source-isolation P0s in the retrieval wave — the exact class the v0.34.1 seal guards. Both fixed before merge. P0-1: buildBestPerPagePoolCte pooled on `slug` alone. In a federated brain, two pages with the same slug in different sources collapsed before ranking/pagination (the neighbor-source page dropped). Now DISTINCT ON (COALESCE(source_id,'default'), slug) — composite key matching dedup.ts's pageKey. Also fixes the PRE-EXISTING keyword-path bug (best_per_page was slug-only before this wave); real-PG parity 23/23. P0-2: the alias hop dropped source_id. resolveAliases returned bare slugs and applyAliasHop hydrated via getPage(slug, undefined), so a federated caller could get the default-source page injected or the right allowed-source page suppressed. resolveAliases now returns {slug, source_id} pairs; applyAliasHop matches by (source_id, slug) and fetches each canonical in its OWN source. Regression tests: alias hop boosts only the aliased source (not same-slug in another source); resolveAliases keeps cross-source same-slug distinct. Deferred as documented tradeoffs (TODO): evidence high_vector_match label uses blended base_score not pure cosine; deep-pagination candidate budget is chunk-bounded; telemetry writes swallow errors pre-v109 on rolling deploys. Co-Authored-By: Claude Opus 4.8 (1M context) * chore: bump version and changelog (v0.41.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) * docs: v0.41.30.0 retrieval cathedral — CLAUDE.md key files + llms regen Co-Authored-By: Claude Opus 4.8 (1M context) * chore: renumber release v0.41.30.0 → v0.41.34.0 (queue moved) Version trio + CHANGELOG header + CLAUDE.md key-file annotations + TODOS heading + regenerated llms bundles, all moved to 0.41.34.0. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(ci): restore glossary roster + harden facts-anti-loop hook budget Two CI failures surfaced after the master merges that brought the branch to 111 migrations: 1. shard 1 — `ALL_METRICS roster > matches the renderer output (no orphans)`: the merge took master's `renderMetricGlossaryMarkdown` whose `groups` array lacked this branch's 4 retrieval-quality keys (hit@1, hit@3, avg_rank1_score, create_safety). `ALL_METRICS` (derived via Object.keys) kept them, so the roster test saw 4 orphans. The freshness check (check:eval-glossary) passed because renderer-output == committed doc — it can't catch a renderer that drops a metric; the roster test can. Restored the "Retrieval-Quality / Evidence Metrics (NamedThingBench)" group + regenerated docs/eval/METRIC_GLOSSARY.md. 2. shard 2 — facts-anti-loop's two engine-dependent put_page tests failed while the two engine-free extractFactsFromTurn tests passed (the signature of a partially-failed beforeAll). This file has a documented PGLite-cold-start-under-deep-shard-load timeout history; the 30s budget was tuned for 95 migrations and the chain is now 111. Bumped to 60s. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(ci): isolate facts-anti-loop in its own process (serial) Follow-up to the prior hook-timeout bump, which was the wrong theory: the [58ms]/[71ms] body times in the re-run prove beforeAll did NOT time out — the engine connects and the two put_page tests run and fail for real, while the two engine-free extractFactsFromTurn tests in the same file pass. put_page (via dispatchToolCall) touches process-global singletons (the facts queue + the AI gateway used by importFromContent's embed step). Some sibling file in the 78-file shard-2 process leaves residual global state that makes put_page's pre-backstop path fail on the CI runner. The failure is NOT reproducible alone, in a Linux oven/bun:1 container, or in a full local shard-2 run (1172 pass) — only on the GitHub runner, deterministically. Per CLAUDE.md's test-isolation rules, a test coupled to shared process state belongs in its own process. Renamed to *.serial.test.ts so it runs in the dedicated serial-tests job (scripts/run-serial-tests.sh spawns a fresh `bun test` per serial file), where it passes deterministically; test-shard.sh excludes serial files from the matrix. Updated the comment to reflect the real cause and refreshed the test-weights.json key. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(ci): close cross-file gateway-config pollution in test shards The prior serial-move theory was incomplete. The real, single root cause behind all three shard failures (2, 5, 10) is cross-file AI-gateway config pollution within a shard's bun process: - A test calls configureGateway() and doesn't restore the gateway on exit. The legacy-embedding preload pins OpenAI/1536 ONCE at process start and re-pins per-test ONLY when the gateway slot is empty — so a leaker that reconfigured the gateway to the v0.37 default (zeroentropyai:zembed-1 / 1280-d) and never reset poisons every later file in the shard. - Victim A (shard 5, test/search/searchvector-maxpool.test.ts): runs initSchema in beforeAll under the leaked gateway → content_chunks.embedding becomes vector(1280) → inserting its hardcoded 1536-d basis vectors throws pgvector CheckExpectedDim. - Victims B/C (shard 10 facts-backstop-gating, shard 2 facts-anti-loop): put_page's importFromContent embeds by design (embed failure PROPAGATES, Codex C2). Under a leaked fake-key gateway the embed step 401s and put_page returns isError → the backstop assertions fail. My branch's shard re-partition (added test files + weight changes) merely co-located leakers with victims; the hazard was latent. Fixes (root cause + self-sufficient victims): - test/search/rerank.test.ts (the shard-5 leaker): add afterAll(resetGateway). Its stub omits embedding_model, so it fell back to the ZE/1280 default; now it restores the empty slot so the preload re-pins legacy for the next file. - test/search/searchvector-maxpool.test.ts: pin configureGateway(openai/1536) in beforeAll BEFORE initSchema (initSchema runs before any preload beforeEach, so it can't rely on the inherited slot). - test/facts-backstop-gating.test.ts + test/facts-anti-loop.test.ts: reset the gateway in beforeEach so put_page's embed is a graceful no-op; reverted anti-loop from the serial quarantine back into the matrix (the serial move was the wrong fix for a gateway-state problem). Validated deterministically: a non-resetting leaker that poisons the gateway to ZE, run first in one bun process, no longer breaks any of the three victims (14/14 pass). verify 29/29, typecheck clean, isolation lint clean. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 100 ++++++++ CLAUDE.md | 17 +- README.md | 4 +- TODOS.md | 29 +++ VERSION | 2 +- docs/architecture/RETRIEVAL.md | 32 +++ .../RETRIEVAL_MAXPOOL_INCIDENT.md | 97 ++++++++ docs/eval/METRIC_GLOSSARY.md | 34 +++ llms-full.txt | 21 +- package.json | 2 +- src/cli.ts | 39 ++++ src/commands/eval-retrieval-quality.ts | 75 ++++++ src/commands/eval.ts | 6 + src/commands/reindex-aliases.ts | 86 +++++++ src/commands/search-diagnose.ts | 131 +++++++++++ src/commands/search.ts | 7 + src/core/engine.ts | 32 +++ src/core/eval/metric-glossary.ts | 25 ++ src/core/import-file.ts | 21 ++ src/core/migrate.ts | 61 +++++ src/core/operations.ts | 127 +++++++--- src/core/pglite-engine.ts | 93 ++++++-- src/core/pglite-schema.ts | 20 ++ src/core/postgres-engine.ts | 73 +++++- src/core/search/alias-normalize.ts | 56 +++++ src/core/search/evidence.ts | 78 +++++++ src/core/search/hybrid.ts | 217 +++++++++++++++++- src/core/search/mode.ts | 39 +++- src/core/search/sql-ranking.ts | 51 ++++ src/core/search/telemetry.ts | 74 +++++- src/core/search/title-match.ts | 93 ++++++++ src/core/types.ts | 32 +++ src/eval/retrieval-quality/harness.ts | 184 +++++++++++++++ test/cli-search-dispatch.test.ts | 60 +++++ test/cross-modal-phase1.test.ts | 2 +- test/eval-retrieval-quality.test.ts | 98 ++++++++ test/facts-anti-loop.test.ts | 27 ++- test/facts-backstop-gating.test.ts | 15 +- .../retrieval-quality/namedthing.jsonl | 10 + test/mcp-eval-capture.test.ts | 11 +- test/retrieval-quality-harness.test.ts | 114 +++++++++ test/schema-bootstrap-coverage.test.ts | 9 + test/search-alias-resolved-boost.test.ts | 2 +- test/search-mode.test.ts | 5 +- test/search/alias-hop.test.ts | 119 ++++++++++ test/search/alias-normalize.test.ts | 50 ++++ test/search/evidence.test.ts | 75 ++++++ test/search/knobs-hash-reranker.test.ts | 2 +- test/search/page-aliases-engine.test.ts | 83 +++++++ test/search/per-call-mode.test.ts | 33 +++ test/search/pre-migration-failopen.test.ts | 56 +++++ test/search/reindex-aliases.test.ts | 65 ++++++ test/search/rerank.test.ts | 14 +- test/search/search-diagnose.test.ts | 65 ++++++ test/search/searchvector-maxpool.test.ts | 130 +++++++++++ test/search/telemetry-rank1.test.ts | 58 +++++ test/search/title-boost-stage.test.ts | 56 +++++ test/search/title-match.test.ts | 68 ++++++ 58 files changed, 3075 insertions(+), 110 deletions(-) create mode 100644 docs/architecture/RETRIEVAL_MAXPOOL_INCIDENT.md create mode 100644 src/commands/eval-retrieval-quality.ts create mode 100644 src/commands/reindex-aliases.ts create mode 100644 src/commands/search-diagnose.ts create mode 100644 src/core/search/alias-normalize.ts create mode 100644 src/core/search/evidence.ts create mode 100644 src/core/search/title-match.ts create mode 100644 src/eval/retrieval-quality/harness.ts create mode 100644 test/cli-search-dispatch.test.ts create mode 100644 test/eval-retrieval-quality.test.ts create mode 100644 test/fixtures/retrieval-quality/namedthing.jsonl create mode 100644 test/retrieval-quality-harness.test.ts create mode 100644 test/search/alias-hop.test.ts create mode 100644 test/search/alias-normalize.test.ts create mode 100644 test/search/evidence.test.ts create mode 100644 test/search/page-aliases-engine.test.ts create mode 100644 test/search/per-call-mode.test.ts create mode 100644 test/search/pre-migration-failopen.test.ts create mode 100644 test/search/reindex-aliases.test.ts create mode 100644 test/search/search-diagnose.test.ts create mode 100644 test/search/searchvector-maxpool.test.ts create mode 100644 test/search/telemetry-rank1.test.ts create mode 100644 test/search/title-boost-stage.test.ts create mode 100644 test/search/title-match.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index ce3e356dc..0d9925aec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,106 @@ All notable changes to GBrain will be documented in this file. +## [0.41.34.0] - 2026-05-30 + +**Search now finds a page by its name, even when you only remember what you +called it.** Ask for "Greek amphitheater" and the page you titled "The Mingtang — +Indoor Greek Amphitheater" comes back first, not buried at a weak 0.64 behind two +unrelated pages. Give a page `aliases:` in its frontmatter ("Hall of Light", "明堂") +and any of those names finds it too. And every result now tells your agent *why* +it matched, so it stops writing duplicate pages on top of ones you already have. + +A real miss prompted this: an agent searched "Greek amphitheater," didn't +confidently find the existing concept page, and wrote a duplicate stub on top of +2,000 words of work. The page was right there — search just represented it by a +throwaway body chunk instead of its title. Four things changed so that can't +happen again: + +- **A page is now scored by its best chunk, not a random one.** Vector search + pools to one row per page (its strongest match) before ranking, so a strong + page can't get crowded out of the results by another page's chunks. Same + behavior on both the embedded (PGLite) and Postgres engines. +- **A query that's a phrase in the title gets a boost.** "Greek amphitheater" is + in the title, so that page wins by design instead of by luck. Bounded and gated + so it can't float an irrelevant page to the top. +- **Chosen names actually resolve.** Add `aliases:` to a page's frontmatter and + those names route to it at query time — the only thing that bridges "Hall of + Light" to the Mingtang when no words overlap. Run `gbrain reindex --aliases` + once to backfill names on pages you already have. +- **Results carry evidence, not just a number.** Each result says how it matched + (`exact_title_match`, `alias_hit`, `weak_semantic`, …) and a `create_safety` + hint (`exists` / `probable` / `unknown`). Your agent keys "is this already in + the brain?" off that, not a fuzzy score — which is what stops the duplicate. + +`gbrain search ""` is now the good hybrid path (it used to be keyword-only); +`gbrain search modes/stats/tune` still work. New: `gbrain search diagnose +"" --target ` traces exactly which layer surfaces (or misses) a +page. `gbrain search stats` now shows an average rank-1 match score so a +retrieval regression shows up before you hit it in chat. And `gbrain eval +retrieval-quality` is a CI gate (NamedThingBench) that fails any change which +regresses name-based search. + +### To take advantage of v0.41.34.0 + +`gbrain upgrade` applies migrations v110 (page_aliases) and v111 (telemetry +columns) automatically. If `gbrain doctor` warns about a partial migration: + +1. **Run the orchestrator manually:** + ```bash + gbrain apply-migrations --yes + ``` +2. **Backfill aliases for pages you already have** (one time): + ```bash + gbrain reindex --aliases + ``` + To make a page findable by a chosen name, add `aliases:` to its frontmatter + and re-sync, or pass the names there before the backfill. +3. **Verify:** + ```bash + gbrain search "" + gbrain search diagnose "" --target + gbrain doctor + ``` +4. **If anything looks wrong,** file an issue with `gbrain doctor` output: + https://github.com/garrytan/gbrain/issues + +Note: the `search` operation (CLI + MCP) now runs hybrid retrieval by default +instead of keyword-only. If you specifically need the old cheap keyword-only +behavior for an automated caller, set `gbrain config set search.mcp_keyword_only true`. + +### Itemized changes + +#### Added +- Per-page max-pool in `searchVector` (both engines) via a shared + `buildBestPerPagePoolCte` SQL builder; collapse key is the composite + `(source_id, slug)` so federated brains never collapse same-slug pages across sources. +- Title-phrase boost (`search.title_boost`, on in all modes): floor-ratio-gated, + capped, reads `page.title`, stamped for `--explain`. +- Free-text alias layer: `page_aliases` table (migration v110), shared + `normalizeAlias`, ingest projection from frontmatter `aliases:`, query-time + alias hop (full-match, source-scoped, bounded inject, fail-open pre-migration), + and `gbrain reindex --aliases` backfill. +- Evidence contract: every `SearchResult` carries `evidence` + + `create_safety` (+ `base_score`/`final_score`), so the agent's + don't-duplicate decision keys off why a page matched. +- `gbrain search diagnose "" --target ` — Phase-0 retrieval trace + (keyword / vector / alias / hybrid ranks + boosts + verdict). +- NamedThingBench: `gbrain eval retrieval-quality ` + a hermetic + CI gate over 7 query families. +- Rank-1 match-score drift telemetry (migration v111) surfaced in `gbrain search stats`. +- Per-call `--mode` for local/trusted callers (remote uses configured mode). + +#### Changed +- The `search` operation (CLI + MCP) is now cheap-hybrid by default + (vector + keyword + RRF + pool + title + alias, expansion off); `query` + remains the full-control variant. Opt back into keyword-only with + `search.mcp_keyword_only`. +- `gbrain search ""` routes to hybrid; `modes/stats/tune` stay subcommands. + +#### Fixed +- A page represented by its weakest chunk in vector search (the incident). +- Same-slug pages in different sources collapsing in per-page pooling and the + alias hop (source-isolation). ## [0.41.33.0] - 2026-05-29 **`gbrain search` and `query` can now return a tight, intent-sized set of diff --git a/CLAUDE.md b/CLAUDE.md index 8df4a658d..859c50c86 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,7 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. **v0.41.30.0:** `put_page`'s v0.38 inline disk write-through (the bare `writeFileSync` into `sync.repo_path`) extracted into the shared `writePageThrough` helper (`src/core/write-through.ts`) that `put_page` now calls — behavior preserved (same render-from-row + frontmatter-override stamping) and upgraded to ATOMIC (temp-sibling + rename), so a crash or concurrent `gbrain sync` can no longer read a half-written `.md`. The same helper now also backs `gbrain brainstorm/lsd --save`. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. **v0.41.31.0 (stale-embedding-semantics wave):** four engine-interface changes for real model/dims-swap stale detection. (1) `countStaleChunks(opts?)` gains an optional `signature?: string` — when set, the stale predicate widens from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (a model/dims swap). NULL signature is GRANDFATHERED (never counted) so the post-migration-v108 corpus isn't flagged en masse; omit `signature` for the legacy NULL-only count. (2) New `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` — `SUM(LENGTH(chunk_text))` over stale chunks (same stale predicate + embed_skip filter + optional sourceId scope as `countStaleChunks`); used by the `gbrain sync --all` cost preview to price the embedding backlog via `estimateCostFromChars`. (3) New `setPageEmbeddingSignature(slug, {sourceId?, signature})` — stamps `pages.embedding_signature` for one page after its chunks are (re)embedded; idempotent, no-op when the page doesn't exist. (4) New `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` — NULLs `embedding` + `embedded_at` on every chunk whose page `embedding_signature` is set AND differs from `signature`, returns the count invalidated; the embed-stale loop calls it BEFORE `listStaleChunks` so signature-drift pages flow through the existing NULL-embedding keyset cursor unchanged (GRANDFATHER: NULL signature never invalidated). Implementation parity across postgres-engine.ts + pglite-engine.ts. Also widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` per the v0.41.29.0 orphan-source-scoping wave (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. **v0.41.31.0 (stale-embedding-semantics wave):** four engine-interface changes for real model/dims-swap stale detection. (1) `countStaleChunks(opts?)` gains an optional `signature?: string` — when set, the stale predicate widens from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (a model/dims swap). NULL signature is GRANDFATHERED (never counted) so the post-migration-v108 corpus isn't flagged en masse; omit `signature` for the legacy NULL-only count. (2) New `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` — `SUM(LENGTH(chunk_text))` over stale chunks (same stale predicate + embed_skip filter + optional sourceId scope as `countStaleChunks`); used by the `gbrain sync --all` cost preview to price the embedding backlog via `estimateCostFromChars`. (3) New `setPageEmbeddingSignature(slug, {sourceId?, signature})` — stamps `pages.embedding_signature` for one page after its chunks are (re)embedded; idempotent, no-op when the page doesn't exist. (4) New `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` — NULLs `embedding` + `embedded_at` on every chunk whose page `embedding_signature` is set AND differs from `signature`, returns the count invalidated; the embed-stale loop calls it BEFORE `listStaleChunks` so signature-drift pages flow through the existing NULL-embedding keyset cursor unchanged (GRANDFATHER: NULL signature never invalidated). Implementation parity across postgres-engine.ts + pglite-engine.ts. Also widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` per the v0.41.29.0 orphan-source-scoping wave (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. **v0.41.34.0 (retrieval-maxpool incident, T3):** two new methods for the free-text alias layer — `resolveAliases(aliasNorms, opts?): Promise>>` (READ side; maps each normalized alias to the `(slug, source_id)` pairs that declare it, source-scoped via `sourceId`/`sourceIds`) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE side; replaces the full alias set for one `(slug, source)` — delete then insert, empty clears, idempotent on the unique triple). Called by the ingest projection in `importFromContent` and the `reindex --aliases` backfill so a page's declared aliases stay in lockstep with its frontmatter. Implementation parity across both engines; pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines now injects the shared `buildBestPerPagePoolCte` per-page max-pool (T1) so a page surfaces on its strongest chunk, not whichever single chunk won the inner LIMIT. - `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. @@ -84,7 +84,16 @@ strict behavior when unset. - `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level) - `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator - `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. -- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. +- `src/core/search/sql-ranking.ts` (v0.22.0, extended v0.41.34.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. **v0.41.34.0 (retrieval-maxpool incident, T1):** new `buildBestPerPagePoolCte(...)` is the shared per-page max-pool CTE both engines' `searchVector` inject — instead of returning the single best chunk per page from an inner `ORDER BY embedding <=> vec LIMIT N` (which let a page lose to a neighbor on ONE weak chunk while its strong chunk sat just below the inner cut), the CTE pools the BEST chunk score per `(source_id, slug)` composite key, so a page surfaces on its strongest evidence. Composite key (not bare slug) keeps multi-source brains correct. Single source of truth so the two engines can't drift on the pooling SQL. +- `src/core/search/title-match.ts` (v0.41.34.0, retrieval-maxpool incident, T2) — pure, zero-I/O title-phrase matcher shared by the production title boost AND NamedThingBench (no drift). `isTitlePhraseMatch(query, title)` returns true when the normalized query is a contiguous token run inside the title with `>= MIN_CONTENT_TOKENS=2` non-stopword tokens, OR an exact full-title match (covers deliberate 1-word chosen names like "Mingtang"). Token-boundary matching (never raw substring, so "art" doesn't match "Bartholomew"); small conservative English stopword set excluded from the content-token floor (Codex#10 guard against promoting generic pages on stopword-y queries); NFKC normalize so CJK / width variants converge. Exports `tokenizeTitle` + `__test__` internals. Closes the direct regression: "Greek amphitheater" matched a weak body chunk instead of the page whose title contained the phrase. +- `src/core/search/alias-normalize.ts` (v0.41.34.0, retrieval-maxpool incident, T3) — ONE normalizer shared by the WRITE path (ingest projects frontmatter `aliases:` into `page_aliases`) and the READ path (search matches query against `page_aliases`), so stored aliases can't silently fail to match queries via divergent normalization (same single-source posture as `cjk.ts`). `normalizeAlias(raw)` does NFKC + lowercase + whitespace-collapse + trim + strip one layer of wrapping quotes/brackets; returns `''` for empty (callers MUST skip empty aliases). `normalizeAliasList(value)` coerces a frontmatter scalar / array / comma-list / garbage into a deduped list of normalized non-empty aliases — used by both the ingest projection and the `reindex --aliases` backfill. +- `src/core/search/evidence.ts` (v0.41.34.0, retrieval-maxpool incident, T4) — the agent-facing why-it-matched contract that closes the incident's root behavior (agent read one blended score 0.64, decided "no strong match, safe to create", wrote a duplicate over a fully-developed page). `classifyEvidence(r)` names the strongest signal (precedence: `alias_hit` > `exact_title_match` > `high_vector_match` (base ≥ `HIGH_MATCH_FLOOR=0.85`) > `keyword_exact` (base ≥ `SOLID_MATCH_FLOOR=0.6`) > `weak_semantic`). `createSafetyFor(evidence)` derives the don't-duplicate hint (`exists` / `probable` / `unknown`) the agent keys off INSTEAD of a raw threshold (a blended RRF/cosine score is not a calibrated probability, Codex#5). `stampEvidence(results)` stamps `evidence` + `create_safety` in place once at pipeline end (after the alias hop, before slice); idempotent. +- `src/core/search/mode.ts` extension (v0.41.34.0, retrieval-maxpool incident, T2) — new `title_boost: number | undefined` knob in `ModeBundle` (default `1.25` for all three modes; multiplier for the post-fusion title-phrase boost). Override chain: per-call `SearchOpts` → `search.title_boost` config (clamped `[1.0, 5.0]`) → bundle. `KNOBS_HASH_VERSION` bumps 6→7 with a `tib=` parts entry so a title-boost-on cache write can't be served to a title-boost-off lookup. `SEARCH_MODE_CONFIG_KEYS` gains `search.title_boost`. +- `src/commands/search-diagnose.ts` (v0.41.34.0, retrieval-maxpool incident, T0) — `gbrain search diagnose "" --target [--json] [--source ]`: Phase-0 retrieval diagnostic. Traces WHERE a target page surfaces (or fails to) across keyword / vector (per-page max-pool) / alias / hybrid layers and names the layer responsible for an incident, so an operator can pin whether the fix is max-pool/innerLimit (vector) vs title/alias. The verdict names the layer that DOES surface the target (or "none"). Pinned by `test/search/search-diagnose.test.ts`. +- `src/commands/reindex-aliases.ts` (v0.41.34.0, retrieval-maxpool incident, T8) — `gbrain reindex --aliases [--limit N] [--dry-run] [--json] [--source ]`: backfills the free-text alias layer for EXISTING pages whose frontmatter `aliases:` predate v110 (the import-time projection covers new + changed pages). Reads each page's frontmatter `aliases:`, writes via `engine.setPageAliases`. Idempotent + convergent (setPageAliases replaces a page's alias set) so no op-checkpoint needed; walks `listAllPageRefs` (cheap cross-source enumeration), `--source` narrows. Pinned by `test/search/reindex-aliases.test.ts`. +- `src/eval/retrieval-quality/harness.ts` + `src/commands/eval-retrieval-quality.ts` + `test/fixtures/retrieval-quality/namedthing.jsonl` (v0.41.34.0, retrieval-maxpool incident, T6) — NamedThingBench, the retrieval-quality eval that makes the incident impossible to reintroduce silently. Seven query families, each a distinct failure class: `title-substring` (the direct regression), `generic-to-named` (tourist label → named thing), `alias-synonym` (declared alias / romanization → canonical), `multi-chunk-dilution` (one strong chunk among many weak — stresses max-pool), `short-vs-rich`, `graph-relationship` (guardrail), `hard-negative` (precision guard, must NOT return a page). `gbrain eval retrieval-quality ` runs it; hard gates (e.g. title-substring Hit@1 ≥ 0.95, alias Hit@1 ≥ 0.98, multi-chunk-dilution Hit@3 = 1.0). Pure: caller injects a `SearchFn` (CLI uses `hybridSearch`, tests stub) so it's engine-agnostic. Metric glossary entries (`hit@1` / `hit@3`) added to `src/core/eval/metric-glossary.ts`. Pinned by `test/eval-retrieval-quality.test.ts` + `test/retrieval-quality-harness.test.ts`. +- `docs/architecture/RETRIEVAL.md` + `docs/architecture/RETRIEVAL_MAXPOOL_INCIDENT.md` (v0.41.34.0) — retrieval-pipeline architecture reference + the named-thing-miss incident write-up (root cause, the five-layer fix, the eval that pins it). +- `src/core/types.ts` extension + `src/core/operations.ts:search` + `src/core/import-file.ts` + `src/cli.ts` + `src/core/search/telemetry.ts` (v0.41.34.0, retrieval-maxpool incident) — the wiring layer for the cathedral. `SearchResult` gains `evidence`, `create_safety`, `title_match_boost`, and `alias_hit` (all optional; evidence/create_safety reference the union types in `evidence.ts`). The `search` MCP op (T4/D4/D15) switches from keyword-only to a cheap-hybrid path by default and accepts a per-call `mode` (conservative|balanced|tokenmax) honored ONLY for trusted/local callers (`resolvePerCallMode(ctx, ...)` — remote callers use the configured mode so a remote provider can't force tokenmax spend); every search path stamps evidence fail-soft. `importFromContent` projects frontmatter `aliases:` into `page_aliases` via `normalizeAliasList` + `engine.setPageAliases` (T3 WRITE side) so new + changed pages register aliases at ingest. `src/cli.ts` adds the `gbrain search diagnose` dispatch (lazy import) and reconciles the `search` CLI path with the cheap-hybrid op. `src/core/search/telemetry.ts` extends the rollup with the T7 rank-1 base_score drift signal (sum/count + 3 coarse buckets, aggregate not per-query per D10) surfaced via `gbrain search stats`, backed by migration v111's `search_telemetry` columns. Tests: `test/cli-search-dispatch.test.ts`, `test/search/per-call-mode.test.ts`, `test/search/telemetry-rank1.test.ts`, `test/search/title-boost-stage.test.ts`, `test/search/alias-hop.test.ts`, `test/search/evidence.test.ts`, `test/search/searchvector-maxpool.test.ts`, `test/search/pre-migration-failopen.test.ts`. - `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. v0.27.x adds `gbrain eval cross-modal` to the dispatch (the user-facing path is the cli.ts no-DB branch — `src/commands/eval.ts:cross-modal` only fires when callers re-enter with an existing engine). - `src/commands/eval-cross-modal.ts` (v0.27.x, extended v0.40.1.0 Track D) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes per Q3=A in plans/radiant-napping-lerdorf.md). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (T11=B partial cost guardrail). Receipts land at `gbrainPath('eval-receipts')/-.json`. The full `--budget-usd` cap is a v0.27.x follow-up TODO. **v0.40.1.0 Track D (T3+T4, per D1/D5/D6/D10):** new `--batch [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` flags fan out cross-modal scoring across a LongMemEval-shape JSONL. Mutually exclusive with `--task` (fail-fast usage error if both set). Filters `kind: "by_type_summary"` rows from the input (Codex #6). Pre-flight cost estimate refuses if `> --max-usd` without `--yes`; default cap 5.00 USD. Semaphore-bounded fan-out via inline `runWithLimit(items, limit, fn)` (~15 LOC, exported for unit tests): max N questions in-flight at once × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run (per D10 — keeps `~/.gbrain/eval-receipts/` clean); the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (NEW batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})` at eval-longmemeval.ts:299; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by 15 cases in `test/eval-cross-modal-batch.test.ts`. - `src/core/cross-modal-eval/json-repair.ts` (v0.27.x) — `parseModelJSON(raw)` named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes. @@ -110,7 +119,7 @@ strict behavior when unset. - `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". - `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. - `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. -- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. +- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. **v0.41.34.0 (retrieval-maxpool incident):** two new post-fusion stages plus the evidence stamp. `applyTitleBoost(results, query, titleBoost, floorThreshold)` (T2) multiplies a result's score by the resolved `title_boost` when `isTitlePhraseMatch` fires, stamps `title_match_boost`, and inherits the floor-ratio gate so a title match can't shove a much-stronger page below it. `applyAliasHop(engine, results, query, opts)` (T3) normalizes the query, calls `engine.resolveAliases`, and when the normalized query exactly matches a page's declared alias, surfaces that page at top-of-organic + epsilon with `alias_hit=true`. `stampEvidence(...)` (T4) runs LAST (after the alias hop, before slice) on every path — keyword-only, no-embed, and full hybrid — so MCP callers and `--explain` read the same `evidence` + `create_safety` contract. `title_boost` is resolved from the mode bundle and threaded in. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. **v0.41.31.0 (cost-model + stale-semantics wave):** `estimateEmbeddingCostUsd(tokens)` now prices against the CURRENTLY-CONFIGURED model's rate instead of the hardcoded OpenAI `EMBEDDING_COST_PER_1K_TOKENS`. Pre-fix, a brain on a cheaper model (e.g. ZeroEntropy) saw a `sync --all` preview that named OpenAI and over-stated spend ~2.6x. New `currentEmbeddingPricePerMTok()` resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate (0.13) only when the gateway is unconfigured (unit tests, pre-connect preview) or the model is unknown to the pricing table. `EMBEDDING_COST_PER_1K_TOKENS` is retained for back-compat with direct importers/tests. New `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (that's tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from the current one and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. New `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so the cost gate and the actual embed decision can never drift. New `shouldBlockSync(costUsd, floorUsd, mode): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. @@ -265,7 +274,7 @@ strict behavior when unset. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). **v0.41.32.0 (supersedes #1623):** `checkSyncFreshness` short-circuit now passes `requireCleanWorkingTree: 'ignore-untracked'` (was `true`) — the headline fix: a quiet repo whose only dirt is untracked dirs is `unchanged`, not SEVERE. The SELECT widens to carry `newest_content_at`; the REMOTE (non-`localOnly`) path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column — NO git subprocess on a DB-supplied `local_path` (trust boundary intact). LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock (A1). `checkCycleFreshness` deliberately NOT content-relativized (CM2 — different axis, `last_full_cycle_at`; filed in TODOS as a probe-phase follow-up). Pinned by the "v0.41.32.0 — commit-relative staleness" describe in `test/doctor.test.ts` (T1 untracked-folders headline bug + T2/T2b/T2c remote-never-shells-out trust boundary). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. **v109 (v0.41.32.0, supersedes #1623):** `sources_newest_content_at` adds `sources.newest_content_at TIMESTAMPTZ` — durable newest-COMMIT timestamp (HEAD committer time) written at sync time by `writeSyncAnchor`. The REMOTE staleness path (federation_health, get_status_snapshot MCP op) reads it instead of shelling out to git on a DB-supplied local_path. Renumbered 108→109 on the master merge that landed v0.41.31 pages_embedding_signature at v108. Metadata-only ADD COLUMN; mirror in pglite-schema.ts + schema.sql + bootstrap probe coverage. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. **v109 (v0.41.32.0, supersedes #1623):** `sources_newest_content_at` adds `sources.newest_content_at TIMESTAMPTZ` — durable newest-COMMIT timestamp (HEAD committer time) written at sync time by `writeSyncAnchor`. The REMOTE staleness path (federation_health, get_status_snapshot MCP op) reads it instead of shelling out to git on a DB-supplied local_path. Renumbered 108→109 on the master merge that landed v0.41.31 pages_embedding_signature at v108. Metadata-only ADD COLUMN; mirror in pglite-schema.ts + schema.sql + bootstrap probe coverage. **v0.41.34.0 (retrieval-maxpool incident, v110 + v111):** v110 (`page_aliases`) creates the free-text alias table — `(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` and lookup indexes on `(source_id, alias_norm)` + `(source_id, slug)`. `alias_norm` is the output of `normalizeAlias()` so the WRITE side (ingest projection) and READ side (search) key on the same normalized form. Also lands in `src/core/pglite-schema.ts` for fresh PGLite installs. v111 (`search_telemetry_rank1_columns`) is `ADD COLUMN IF NOT EXISTS` on both engines for the rank-1 base_score drift signal: `sum_rank1_score`, `count_rank1`, and three coarse distribution buckets (`rank1_lt_solid` / `rank1_solid` / `rank1_high`) on the existing `search_telemetry` rollup — aggregate columns (NOT per-query rows, D10) so a downward drift in the median rank-1 match score is computable with bounded growth. `search_telemetry` lives only in migration v57, so these ALTER right after v57 runs. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. diff --git a/README.md b/README.md index a443ed0ee..9c64ee8ab 100644 --- a/README.md +++ b/README.md @@ -230,7 +230,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec ## Capabilities -**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. +**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. **Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. @@ -238,7 +238,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. -**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). +**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). **Brain consistency.** `gbrain eval suspected-contradictions` samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle. diff --git a/TODOS.md b/TODOS.md index dab631c19..cac097918 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,32 @@ # TODOS +## v0.41.34.0 retrieval-cathedral follow-ups (v0.42+) + +Deferred from the v0.41.34.0 wave (codex adversarial P1/P2 — documented tradeoffs, +not blockers; the P0 source-isolation issues were fixed in-wave). + +- [ ] **P1 — Calibrate the `evidence` classifier.** `high_vector_match` is assigned + from `base_score >= 0.85`, but `base_score` is the pre-boost RRF/keyword/title/alias + pipeline score, not a pure cosine. A generic high-scoring page can read as + `create_safety='exists'`. Add a true vector-cosine signal (or a `keyword_exact` + exact-token check) so the evidence labels are grounded, not inferred from the blend. + File: `src/core/search/evidence.ts`. **Why:** the evidence contract is what stops + the duplicate-page class; mislabeled evidence weakens it. + +- [ ] **P1 — Page-bounded vector pagination.** `searchVector` innerLimit is + `offset + max(limit*5, 100)` counted BEFORE `DISTINCT ON`, so on a dense page one + page can consume the candidate budget and a deep `OFFSET` can underfill even when + more pages exist. Restructure to a two-stage pull (top-N chunks → pool → re-expand) + or raise innerLimit adaptively for deep offsets. Files: both engines' `searchVector`. + **Why:** deep search pagination on big brains can return short pages. + +- [ ] **P2 — Telemetry rolling-deploy gap.** Pre-v111 (mid rolling deploy), rank-1 + telemetry INSERTs reference missing columns and the write is swallowed, so a window + of telemetry is silently lost and `search stats` reads empty on old tables. Either + feature-detect the columns before writing the extended INSERT, or accept the gap + (documented). File: `src/core/search/telemetry.ts`. **Why:** brief observability + blind spot during upgrades. + ## v0.41.33.0 adaptive return-sizing follow-ups (v0.42+) Filed from the v0.41.33.0 wave (intent-aware adaptive return-sizing, born from @@ -11,6 +38,7 @@ default-off; these are the gates and extensions before any default flip. - [ ] **v0.42+: gentle adaptive gate on `think`'s gather stage (A3).** The plan's A3 decision was a gentler return-gate on `runThink`'s gather candidates (cleaner context, fewer tokens per reasoning call). Deferred because the benefit is unvalidated without a longmemeval answer-quality run, and trimming the answer path (even default-off) carries regression risk. gather fuses 4 streams (page / takes-keyword / takes-vector / graph); the gate must operate on the fused output with a higher min-keep than search, validated on `gbrain eval longmemeval` answer quality (not retrieval precision). Also: `RunThinkOpts` has no `sourceId` today, so think's gather runs unscoped (codex finding) — scope-isolated think needs that plumbing first. Priority: P2. - [ ] **v0.42+: `--explain` human header for adaptive_return.** The decision is in `HybridSearchMeta.adaptive_return` and surfaces in `--json` today. The per-result `explain-formatter.ts` is result-scoped and can't render a per-query meta line; the human `gbrain search --explain` header needs the meta threaded through `cli.ts:formatResult` (it currently only receives `results`). Add a one-line gate-decision header (intent / cap / kept of total). Priority: P3. - [ ] **v0.42+: structured-alias / facts-mode fidelity for the PrecisionMemBench eval.** The gbrain-evals benchmark seeds beliefs as pages with aliases in the body (real FTS). A second fidelity that exercises gbrain's structured alias/entity-resolution layer (facts with `valid_until` + entity resolution) would measure gbrain's structured-belief path on the 23 alias cases. Lives in gbrain-evals (`eval/precisionmembench/seed.ts` throws on `fidelity:'structured'` today). Priority: P3. + ## v0.41.32.0 content-relative staleness follow-ups (v0.42+) Filed from the v0.41.32.0 wave (supersedes #1623 — commit-relative sync @@ -44,6 +72,7 @@ were deliberately scoped out (CM2 + the remote post-sync-divergence residual). different axis from sync staleness). Content-relativizing it (a source whose newest commit predates its last full cycle doesn't need re-cycling) is a natural companion to this probe phase. Priority: P3. + ## brainstorm/lsd --save source-awareness (v0.42+) Filed from the `--save` dual-sink hardening wave (route through the canonical diff --git a/VERSION b/VERSION index 60c06192c..bbd1e5c01 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.33.0 +0.41.34.0 \ No newline at end of file diff --git a/docs/architecture/RETRIEVAL.md b/docs/architecture/RETRIEVAL.md index ecd227c75..64ec0aed5 100644 --- a/docs/architecture/RETRIEVAL.md +++ b/docs/architecture/RETRIEVAL.md @@ -58,6 +58,38 @@ Hybrid search applies a source-factor CASE expression at the SQL layer (lives in The boost map is configurable via `GBRAIN_SOURCE_BOOST` env var or per-call `SearchOpts.exclude_slug_prefixes`. Temporal queries (`detail: 'high'`) bypass the boost so chat pages re-surface for time-sensitive lookups. +## Named-thing retrieval (per-page pool + title + alias + evidence) + +A brain organized around *chosen names* (Mingtang, Hall of Light) needs more than +embedding proximity. Four layers, added after the incident in +[`RETRIEVAL_MAXPOOL_INCIDENT.md`](./RETRIEVAL_MAXPOOL_INCIDENT.md): + +- **Per-page max-pool** — `searchVector` (both engines) collapses chunk-grain + candidates to the best chunk per page (`DISTINCT ON (slug)`) over the full + candidate set before the user `LIMIT`, via the shared `buildBestPerPagePoolCte` + in `sql-ranking.ts`. The vector side returns N distinct pages by best chunk, + not N chunks that collapse to fewer pages downstream. +- **Title-phrase boost** — when the normalized query is a contiguous token-run + inside `page.title` (or an exact full-title match), a floor-ratio-gated, + bounded multiplier fires (`applyTitleBoost`, `search.title_boost` knob). A + query that is a phrase from the title can't lose to a body chunk by luck. +- **Alias hop** — free-text `aliases:` frontmatter is projected into a + `page_aliases` table (separate from the `slug_aliases` wikilink redirect) and + consulted at query time: a full normalized-query match injects/boosts the + canonical page (`applyAliasHop`). The only layer that bridges true synonyms + with zero surface overlap ("Hall of Light" → the Mingtang page). Backfill + existing pages with `gbrain reindex --aliases`. +- **Evidence contract** — every result carries `evidence` + (`alias_hit | exact_title_match | high_vector_match | keyword_exact | + weak_semantic`) and `create_safety` (`exists | probable | unknown`). An agent + deciding "is this page already here, safe to NOT write a duplicate?" keys off + `create_safety`, not a raw blended score. + +The `search` MCP/CLI op is **cheap-hybrid** (vector + keyword + RRF + pool + +title + alias, expansion off); `query` is the full-control variant. NamedThingBench +(`gbrain eval retrieval-quality`) gates these families on every PR. Diagnose a +specific miss with `gbrain search diagnose "" --target `. + ## Intent-aware query rewriting `src/core/search/intent.ts` classifies queries into `entity`, `temporal`, `event`, or `general`. Each routes through different ranking knobs: diff --git a/docs/architecture/RETRIEVAL_MAXPOOL_INCIDENT.md b/docs/architecture/RETRIEVAL_MAXPOOL_INCIDENT.md new file mode 100644 index 000000000..e5c514d79 --- /dev/null +++ b/docs/architecture/RETRIEVAL_MAXPOOL_INCIDENT.md @@ -0,0 +1,97 @@ +# Retrieval Incident: a chosen-name page was missed, and the fix + +**Status:** Resolved (retrieval-cathedral wave). Supersedes the docs-only RFC in +closed PR #1616 — the diagnosis there was directionally right about the disease +but wrong on several mechanics; this is the corrected record + what shipped. +**Original author:** Garry Tan's OpenClaw. **Severity at the time:** High. +**Related:** [`RETRIEVAL.md`](./RETRIEVAL.md), [`../eval/METRIC_GLOSSARY.md`](../eval/METRIC_GLOSSARY.md). + +--- + +## 1. What happened + +The agent was asked to log that Garry "wants to build a Greek amphitheater." It +ran a retrieval for the concept, the canonical concept page (titled "...Indoor +Greek Amphitheater...") did **not** surface with enough confidence to be +recognized as the existing page, and the agent wrote a **duplicate stub** on top +of a fully-developed concept doc. Garry caught it: "It's in the brain. It's the +Hall of Light. Why did you forget?" + +The page is *about* a Greek amphitheater — the phrase is in its title and first +sentence. A healthy index returns it at the top. It didn't. + +## 2. The disease (the RFC got this right) + +The brain is stored by **meaning and chosen name** (Mingtang, Hall of Light) but +was retrieved by **literal embedding proximity to a body chunk**, and the agent's +"is this already here?" decision keyed off a single fuzzy blended score. Three +retrieval gaps plus one contract gap produced the miss. + +## 3. Verified ground truth (corrections to the RFC) + +These were checked in code during the fix; several change the remedy: + +1. **`gbrain search` was keyword-only**, not hybrid — so the RFC's cosine scores + (0.64/0.98) came from the hybrid `query`/MCP path the agent actually hit, not + `gbrain search`. The repro command in the RFC was mislabeled. +2. **`--mode` was never a CLI param** — mode resolves server-side from the + `search.mode` config key, which is why all three "modes" returned identical + results (the flag was silently dropped; `thorough` isn't a real mode). +3. **`hybridSearch` already max-pooled per page at the dedup layer.** So the + per-page max-pool fix's real win is *candidate-set page recall* (the vector + side returned N chunks that could collapse to fewer pages), and it is + necessary-but-not-sufficient: if a page's title chunk scores below a body + chunk on a 2-word query, or falls outside the candidate pool, pooling alone + doesn't rescue it. +4. **Frontmatter `aliases:` was dead to search** — stored in `pages.frontmatter` + JSONB, never consulted. `slug_aliases` is a *slug→slug* wikilink redirect, a + different concept. + +## 4. The fix that shipped (four layers + a contract) + +| Layer | Fixes | Where | +|---|---|---| +| **Per-page max-pool** (T1) | a page scored by its weakest chunk; vector page-recall | `searchVector` both engines, shared `buildBestPerPagePoolCte` | +| **Title-phrase boost** (T2) | query is a phrase in the title but matched a body chunk | `applyTitleBoost` (reads `page.title`), `title_boost` mode knob | +| **Alias hop** (T3) | true synonyms with zero surface overlap ("Hall of Light" → Mingtang) | `page_aliases` table, `applyAliasHop`, ingest projection + `reindex --aliases` backfill | +| **Evidence contract** (T4) | the agent keyed "don't duplicate" off a fuzzy score | `evidence` + `create_safety` on every result; the agent keys off `create_safety='exists'`, not a threshold | + +Plus: `gbrain search ""` is now cheap-hybrid (the obvious verb gives the +good path); `modes/stats/tune` stay subcommands; `--mode` works per-call for +local callers; rank-1 score drift telemetry; and **NamedThingBench**, a CI gate +that hard-gates the families that ARE this incident. + +## 5. How to confirm / triage a recurrence + +``` +# Which layer surfaces (or misses) the target page? +gbrain search diagnose "Greek amphitheater" --target projects/new-greek-theater/concept_v0 + +# Backfill aliases for existing pages whose frontmatter predates the alias layer: +gbrain reindex --aliases + +# Watch retrieval quality over time (a downward avg rank-1 score = regressing): +gbrain search stats --days 30 + +# The gate that prevents silent reintroduction: +gbrain eval retrieval-quality test/fixtures/retrieval-quality/namedthing.jsonl +``` + +For a page to be reliably found by its chosen name, give it `aliases:` frontmatter: + +```yaml +--- +title: The Mingtang — Indoor Greek Amphitheater +aliases: + - Hall of Light + - 明堂 +--- +``` + +## 6. The discipline this teaches + +A benchmark that scores 97.9 R@5 while production returns a flagship page at 0.64 +means the benchmark and the shipped path diverged. NamedThingBench runs the same +families through the real pipeline on every PR, and the evidence contract means +the agent's duplicate-or-not decision is grounded in *why* a page matched, not a +number that was never a calibrated probability. diff --git a/docs/eval/METRIC_GLOSSARY.md b/docs/eval/METRIC_GLOSSARY.md index b596b0aa8..ac7e2bc66 100644 --- a/docs/eval/METRIC_GLOSSARY.md +++ b/docs/eval/METRIC_GLOSSARY.md @@ -38,6 +38,40 @@ Every metric `gbrain eval *` and `gbrain search stats` reports has a plain-Engli **Range:** 0..1, higher is better. nDCG@10 above 0.65 is the common "ship it" threshold for hybrid retrieval on technical corpora. +## Retrieval-Quality / Evidence Metrics (NamedThingBench) + +### Hit rate at 1 (Hit@1) + +**Key:** `hit@1` + +**Plain English:** Fraction of queries where the right page is the very first result. NamedThingBench hard-gates title-substring Hit@1 >= 0.95 and alias Hit@1 >= 0.98 — a query that is a page's name or title phrase should land it at rank 1, not "somewhere in the top 10". + +**Range:** 0..1, higher is better. + +### Hit rate at 3 (Hit@3) + +**Key:** `hit@3` + +**Plain English:** Fraction of queries where the right page is in the top 3 results. NamedThingBench requires the multi-chunk-dilution family to hit 1.0 — a page with one strong chunk among many weak ones must never be buried. + +**Range:** 0..1, higher is better. + +### Average rank-1 match score + +**Key:** `avg_rank1_score` + +**Plain English:** The mean base (pre-boost) retrieval score of the TOP result across recent searches, from `gbrain search stats`. It is NOT a labeled accuracy number — it is a drift signal: if this trends DOWN over time, retrieval quality is regressing (the early warning that would have caught the duplicate-page incident before a human did). + +**Range:** 0..1. Watch the trend, not the absolute value; pair with the <0.6 / 0.6-0.85 / >=0.85 bucket counts for shape. + +### Create-safety hint (evidence contract) + +**Key:** `create_safety` + +**Plain English:** A result's answer to "is this page already in the brain — safe to NOT write a new one?" Derived from the strongest evidence, NOT a raw score: exists (alias_hit / exact_title_match / high_vector_match — do not duplicate), probable (solid keyword match — prefer updating), unknown (weak match — look closer). An agent keys its don't-duplicate decision off this, which is what prevents the incident's duplicate-stub class. + +**Range:** enum: exists | probable | unknown + ## Set-Similarity / Stability Metrics ### Jaccard similarity at k (set Jaccard @k) diff --git a/llms-full.txt b/llms-full.txt index a397b47d4..9eb0f74b9 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -183,7 +183,7 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. **v0.41.30.0:** `put_page`'s v0.38 inline disk write-through (the bare `writeFileSync` into `sync.repo_path`) extracted into the shared `writePageThrough` helper (`src/core/write-through.ts`) that `put_page` now calls — behavior preserved (same render-from-row + frontmatter-override stamping) and upgraded to ATOMIC (temp-sibling + rename), so a crash or concurrent `gbrain sync` can no longer read a half-written `.md`. The same helper now also backs `gbrain brainstorm/lsd --save`. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. **v0.41.31.0 (stale-embedding-semantics wave):** four engine-interface changes for real model/dims-swap stale detection. (1) `countStaleChunks(opts?)` gains an optional `signature?: string` — when set, the stale predicate widens from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (a model/dims swap). NULL signature is GRANDFATHERED (never counted) so the post-migration-v108 corpus isn't flagged en masse; omit `signature` for the legacy NULL-only count. (2) New `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` — `SUM(LENGTH(chunk_text))` over stale chunks (same stale predicate + embed_skip filter + optional sourceId scope as `countStaleChunks`); used by the `gbrain sync --all` cost preview to price the embedding backlog via `estimateCostFromChars`. (3) New `setPageEmbeddingSignature(slug, {sourceId?, signature})` — stamps `pages.embedding_signature` for one page after its chunks are (re)embedded; idempotent, no-op when the page doesn't exist. (4) New `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` — NULLs `embedding` + `embedded_at` on every chunk whose page `embedding_signature` is set AND differs from `signature`, returns the count invalidated; the embed-stale loop calls it BEFORE `listStaleChunks` so signature-drift pages flow through the existing NULL-embedding keyset cursor unchanged (GRANDFATHER: NULL signature never invalidated). Implementation parity across postgres-engine.ts + pglite-engine.ts. Also widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` per the v0.41.29.0 orphan-source-scoping wave (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. **v0.41.31.0 (stale-embedding-semantics wave):** four engine-interface changes for real model/dims-swap stale detection. (1) `countStaleChunks(opts?)` gains an optional `signature?: string` — when set, the stale predicate widens from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (a model/dims swap). NULL signature is GRANDFATHERED (never counted) so the post-migration-v108 corpus isn't flagged en masse; omit `signature` for the legacy NULL-only count. (2) New `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` — `SUM(LENGTH(chunk_text))` over stale chunks (same stale predicate + embed_skip filter + optional sourceId scope as `countStaleChunks`); used by the `gbrain sync --all` cost preview to price the embedding backlog via `estimateCostFromChars`. (3) New `setPageEmbeddingSignature(slug, {sourceId?, signature})` — stamps `pages.embedding_signature` for one page after its chunks are (re)embedded; idempotent, no-op when the page doesn't exist. (4) New `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` — NULLs `embedding` + `embedded_at` on every chunk whose page `embedding_signature` is set AND differs from `signature`, returns the count invalidated; the embed-stale loop calls it BEFORE `listStaleChunks` so signature-drift pages flow through the existing NULL-embedding keyset cursor unchanged (GRANDFATHER: NULL signature never invalidated). Implementation parity across postgres-engine.ts + pglite-engine.ts. Also widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` per the v0.41.29.0 orphan-source-scoping wave (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. **v0.41.34.0 (retrieval-maxpool incident, T3):** two new methods for the free-text alias layer — `resolveAliases(aliasNorms, opts?): Promise>>` (READ side; maps each normalized alias to the `(slug, source_id)` pairs that declare it, source-scoped via `sourceId`/`sourceIds`) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE side; replaces the full alias set for one `(slug, source)` — delete then insert, empty clears, idempotent on the unique triple). Called by the ingest projection in `importFromContent` and the `reindex --aliases` backfill so a page's declared aliases stay in lockstep with its frontmatter. Implementation parity across both engines; pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines now injects the shared `buildBestPerPagePoolCte` per-page max-pool (T1) so a page surfaces on its strongest chunk, not whichever single chunk won the inner LIMIT. - `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. @@ -226,7 +226,16 @@ strict behavior when unset. - `src/core/search/intent.ts` — Query intent classifier (entity/temporal/event/general → auto-selects detail level) - `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator - `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. -- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. +- `src/core/search/sql-ranking.ts` (v0.22.0, extended v0.41.34.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. **v0.41.34.0 (retrieval-maxpool incident, T1):** new `buildBestPerPagePoolCte(...)` is the shared per-page max-pool CTE both engines' `searchVector` inject — instead of returning the single best chunk per page from an inner `ORDER BY embedding <=> vec LIMIT N` (which let a page lose to a neighbor on ONE weak chunk while its strong chunk sat just below the inner cut), the CTE pools the BEST chunk score per `(source_id, slug)` composite key, so a page surfaces on its strongest evidence. Composite key (not bare slug) keeps multi-source brains correct. Single source of truth so the two engines can't drift on the pooling SQL. +- `src/core/search/title-match.ts` (v0.41.34.0, retrieval-maxpool incident, T2) — pure, zero-I/O title-phrase matcher shared by the production title boost AND NamedThingBench (no drift). `isTitlePhraseMatch(query, title)` returns true when the normalized query is a contiguous token run inside the title with `>= MIN_CONTENT_TOKENS=2` non-stopword tokens, OR an exact full-title match (covers deliberate 1-word chosen names like "Mingtang"). Token-boundary matching (never raw substring, so "art" doesn't match "Bartholomew"); small conservative English stopword set excluded from the content-token floor (Codex#10 guard against promoting generic pages on stopword-y queries); NFKC normalize so CJK / width variants converge. Exports `tokenizeTitle` + `__test__` internals. Closes the direct regression: "Greek amphitheater" matched a weak body chunk instead of the page whose title contained the phrase. +- `src/core/search/alias-normalize.ts` (v0.41.34.0, retrieval-maxpool incident, T3) — ONE normalizer shared by the WRITE path (ingest projects frontmatter `aliases:` into `page_aliases`) and the READ path (search matches query against `page_aliases`), so stored aliases can't silently fail to match queries via divergent normalization (same single-source posture as `cjk.ts`). `normalizeAlias(raw)` does NFKC + lowercase + whitespace-collapse + trim + strip one layer of wrapping quotes/brackets; returns `''` for empty (callers MUST skip empty aliases). `normalizeAliasList(value)` coerces a frontmatter scalar / array / comma-list / garbage into a deduped list of normalized non-empty aliases — used by both the ingest projection and the `reindex --aliases` backfill. +- `src/core/search/evidence.ts` (v0.41.34.0, retrieval-maxpool incident, T4) — the agent-facing why-it-matched contract that closes the incident's root behavior (agent read one blended score 0.64, decided "no strong match, safe to create", wrote a duplicate over a fully-developed page). `classifyEvidence(r)` names the strongest signal (precedence: `alias_hit` > `exact_title_match` > `high_vector_match` (base ≥ `HIGH_MATCH_FLOOR=0.85`) > `keyword_exact` (base ≥ `SOLID_MATCH_FLOOR=0.6`) > `weak_semantic`). `createSafetyFor(evidence)` derives the don't-duplicate hint (`exists` / `probable` / `unknown`) the agent keys off INSTEAD of a raw threshold (a blended RRF/cosine score is not a calibrated probability, Codex#5). `stampEvidence(results)` stamps `evidence` + `create_safety` in place once at pipeline end (after the alias hop, before slice); idempotent. +- `src/core/search/mode.ts` extension (v0.41.34.0, retrieval-maxpool incident, T2) — new `title_boost: number | undefined` knob in `ModeBundle` (default `1.25` for all three modes; multiplier for the post-fusion title-phrase boost). Override chain: per-call `SearchOpts` → `search.title_boost` config (clamped `[1.0, 5.0]`) → bundle. `KNOBS_HASH_VERSION` bumps 6→7 with a `tib=` parts entry so a title-boost-on cache write can't be served to a title-boost-off lookup. `SEARCH_MODE_CONFIG_KEYS` gains `search.title_boost`. +- `src/commands/search-diagnose.ts` (v0.41.34.0, retrieval-maxpool incident, T0) — `gbrain search diagnose "" --target [--json] [--source ]`: Phase-0 retrieval diagnostic. Traces WHERE a target page surfaces (or fails to) across keyword / vector (per-page max-pool) / alias / hybrid layers and names the layer responsible for an incident, so an operator can pin whether the fix is max-pool/innerLimit (vector) vs title/alias. The verdict names the layer that DOES surface the target (or "none"). Pinned by `test/search/search-diagnose.test.ts`. +- `src/commands/reindex-aliases.ts` (v0.41.34.0, retrieval-maxpool incident, T8) — `gbrain reindex --aliases [--limit N] [--dry-run] [--json] [--source ]`: backfills the free-text alias layer for EXISTING pages whose frontmatter `aliases:` predate v110 (the import-time projection covers new + changed pages). Reads each page's frontmatter `aliases:`, writes via `engine.setPageAliases`. Idempotent + convergent (setPageAliases replaces a page's alias set) so no op-checkpoint needed; walks `listAllPageRefs` (cheap cross-source enumeration), `--source` narrows. Pinned by `test/search/reindex-aliases.test.ts`. +- `src/eval/retrieval-quality/harness.ts` + `src/commands/eval-retrieval-quality.ts` + `test/fixtures/retrieval-quality/namedthing.jsonl` (v0.41.34.0, retrieval-maxpool incident, T6) — NamedThingBench, the retrieval-quality eval that makes the incident impossible to reintroduce silently. Seven query families, each a distinct failure class: `title-substring` (the direct regression), `generic-to-named` (tourist label → named thing), `alias-synonym` (declared alias / romanization → canonical), `multi-chunk-dilution` (one strong chunk among many weak — stresses max-pool), `short-vs-rich`, `graph-relationship` (guardrail), `hard-negative` (precision guard, must NOT return a page). `gbrain eval retrieval-quality ` runs it; hard gates (e.g. title-substring Hit@1 ≥ 0.95, alias Hit@1 ≥ 0.98, multi-chunk-dilution Hit@3 = 1.0). Pure: caller injects a `SearchFn` (CLI uses `hybridSearch`, tests stub) so it's engine-agnostic. Metric glossary entries (`hit@1` / `hit@3`) added to `src/core/eval/metric-glossary.ts`. Pinned by `test/eval-retrieval-quality.test.ts` + `test/retrieval-quality-harness.test.ts`. +- `docs/architecture/RETRIEVAL.md` + `docs/architecture/RETRIEVAL_MAXPOOL_INCIDENT.md` (v0.41.34.0) — retrieval-pipeline architecture reference + the named-thing-miss incident write-up (root cause, the five-layer fix, the eval that pins it). +- `src/core/types.ts` extension + `src/core/operations.ts:search` + `src/core/import-file.ts` + `src/cli.ts` + `src/core/search/telemetry.ts` (v0.41.34.0, retrieval-maxpool incident) — the wiring layer for the cathedral. `SearchResult` gains `evidence`, `create_safety`, `title_match_boost`, and `alias_hit` (all optional; evidence/create_safety reference the union types in `evidence.ts`). The `search` MCP op (T4/D4/D15) switches from keyword-only to a cheap-hybrid path by default and accepts a per-call `mode` (conservative|balanced|tokenmax) honored ONLY for trusted/local callers (`resolvePerCallMode(ctx, ...)` — remote callers use the configured mode so a remote provider can't force tokenmax spend); every search path stamps evidence fail-soft. `importFromContent` projects frontmatter `aliases:` into `page_aliases` via `normalizeAliasList` + `engine.setPageAliases` (T3 WRITE side) so new + changed pages register aliases at ingest. `src/cli.ts` adds the `gbrain search diagnose` dispatch (lazy import) and reconciles the `search` CLI path with the cheap-hybrid op. `src/core/search/telemetry.ts` extends the rollup with the T7 rank-1 base_score drift signal (sum/count + 3 coarse buckets, aggregate not per-query per D10) surfaced via `gbrain search stats`, backed by migration v111's `search_telemetry` columns. Tests: `test/cli-search-dispatch.test.ts`, `test/search/per-call-mode.test.ts`, `test/search/telemetry-rank1.test.ts`, `test/search/title-boost-stage.test.ts`, `test/search/alias-hop.test.ts`, `test/search/evidence.test.ts`, `test/search/searchvector-maxpool.test.ts`, `test/search/pre-migration-failopen.test.ts`. - `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. v0.27.x adds `gbrain eval cross-modal` to the dispatch (the user-facing path is the cli.ts no-DB branch — `src/commands/eval.ts:cross-modal` only fires when callers re-enter with an existing engine). - `src/commands/eval-cross-modal.ts` (v0.27.x, extended v0.40.1.0 Track D) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes per Q3=A in plans/radiant-napping-lerdorf.md). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (T11=B partial cost guardrail). Receipts land at `gbrainPath('eval-receipts')/-.json`. The full `--budget-usd` cap is a v0.27.x follow-up TODO. **v0.40.1.0 Track D (T3+T4, per D1/D5/D6/D10):** new `--batch [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` flags fan out cross-modal scoring across a LongMemEval-shape JSONL. Mutually exclusive with `--task` (fail-fast usage error if both set). Filters `kind: "by_type_summary"` rows from the input (Codex #6). Pre-flight cost estimate refuses if `> --max-usd` without `--yes`; default cap 5.00 USD. Semaphore-bounded fan-out via inline `runWithLimit(items, limit, fn)` (~15 LOC, exported for unit tests): max N questions in-flight at once × 3 model slots = ceiling of 3N parallel API calls (default `--concurrent 3` → 9). Per-question receipts land in a per-batch tempdir and are deleted at end of run (per D10 — keeps `~/.gbrain/eval-receipts/` clean); the summary receipt inlines per-question verdicts as JSON, not file paths. Exit precedence (NEW batch-level policy, NOT inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS. DI seam: `runEvalCrossModal(args, opts?: {runEval?: typeof runEval})` mirrors `runEvalLongMemEval(args, {client?})` at eval-longmemeval.ts:299; tests pass `opts.runEval` to bypass real LLM calls AND the gateway availability check. Pinned by 15 cases in `test/eval-cross-modal-batch.test.ts`. - `src/core/cross-modal-eval/json-repair.ts` (v0.27.x) — `parseModelJSON(raw)` named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes. @@ -252,7 +261,7 @@ strict behavior when unset. - `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". - `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. - `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. -- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. +- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. **v0.33:** `HybridSearchOpts.types?: PageType[]` (defined on `SearchOpts`) threads a multi-type filter through to per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks`, where it lands as `AND p.type = ANY($N::text[])`. Primary consumer is `gbrain whoknows` (filters to `['person','company']`). AND-applies alongside the existing single-value `type` filter; either or both can be used. **v0.36.3.0:** `hybridSearch` now resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor into per-engine `searchVector` (not a raw string), and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision (replaces the prior name-based `isDefaultColumn` check that leaked across vector spaces when a user repointed the `embedding` builtin). `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B benchmarking; `search` (keyword-only) deliberately rejects the param. **v0.41.34.0 (retrieval-maxpool incident):** two new post-fusion stages plus the evidence stamp. `applyTitleBoost(results, query, titleBoost, floorThreshold)` (T2) multiplies a result's score by the resolved `title_boost` when `isTitlePhraseMatch` fires, stamps `title_match_boost`, and inherits the floor-ratio gate so a title match can't shove a much-stronger page below it. `applyAliasHop(engine, results, query, opts)` (T3) normalizes the query, calls `engine.resolveAliases`, and when the normalized query exactly matches a page's declared alias, surfaces that page at top-of-organic + epsilon with `alias_hit=true`. `stampEvidence(...)` (T4) runs LAST (after the alias hop, before slice) on every path — keyword-only, no-embed, and full hybrid — so MCP callers and `--explain` read the same `evidence` + `create_safety` contract. `title_boost` is resolved from the mode bundle and threaded in. - `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. **v0.28.7:** `BATCH_SIZE` reverted 50→100 — the original Voyage safety guard halved OpenAI throughput on every page. Per-recipe pre-split + recursive halving + adaptive shrink-on-miss now live in the gateway, so the outer paginator goes back to its original purpose: progress-callback granularity, not batch protection. **v0.41.31.0 (cost-model + stale-semantics wave):** `estimateEmbeddingCostUsd(tokens)` now prices against the CURRENTLY-CONFIGURED model's rate instead of the hardcoded OpenAI `EMBEDDING_COST_PER_1K_TOKENS`. Pre-fix, a brain on a cheaper model (e.g. ZeroEntropy) saw a `sync --all` preview that named OpenAI and over-stated spend ~2.6x. New `currentEmbeddingPricePerMTok()` resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate (0.13) only when the gateway is unconfigured (unit tests, pre-connect preview) or the model is unknown to the pricing table. `EMBEDDING_COST_PER_1K_TOKENS` is retained for back-compat with direct importers/tests. New `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (that's tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from the current one and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. New `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so the cost gate and the actual embed decision can never drift. New `shouldBlockSync(costUsd, floorUsd, mode): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. @@ -407,7 +416,7 @@ strict behavior when unset. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. **v0.41.27.0:** `checkSyncFreshness` gains a `localOnly`-gated git short-circuit (D4 trust-boundary preservation per Codex P0-1): `runDoctor` passes `localOnly: true`, `doctorReportRemote` keeps the default `false` so the HTTP MCP path doesn't walk DB-supplied `local_path` values via subprocess. The narrowed predicate (D7) mirrors sync.ts:1057+1075's actual "do work?" gate: HEAD == `last_commit` AND working tree clean AND `sources.chunker_version === String(CHUNKER_VERSION)`. Inline SELECT widens to carry `last_commit + chunker_version` (columns already exist; no schema migration). Three-bucket count math (D6) populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length` pinned in the test suite. OK message reshape (D2): all-unchanged → "All N up to date (no new commits since last sync)"; mixed → "N source(s): X synced recently, Y unchanged since last sync"; all-synced-recently keeps the prior message verbatim. `checkCycleFreshness` is INTENTIONALLY NOT touched (Codex P0-2 / D5): `last_commit == HEAD` answers "are there new commits to sync?" but cannot answer "did the full cycle complete?" — a sync can succeed while later cycle phases fail, and silencing that warn would hide real cycle staleness. Pinned by 9 new cases in `test/doctor.test.ts` ("v0.41.27.0 — sync_freshness git short-circuit" describe block) including a load-bearing D4 regression guard that verifies probes are NEVER called when `localOnly` is unset or false, plus the three-bucket invariant explicitly asserted in the mixed 3-source case. Supersedes PR #1564 (Co-Authored-By preserved). **v0.41.32.0 (supersedes #1623):** `checkSyncFreshness` short-circuit now passes `requireCleanWorkingTree: 'ignore-untracked'` (was `true`) — the headline fix: a quiet repo whose only dirt is untracked dirs is `unchanged`, not SEVERE. The SELECT widens to carry `newest_content_at`; the REMOTE (non-`localOnly`) path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column — NO git subprocess on a DB-supplied `local_path` (trust boundary intact). LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock (A1). `checkCycleFreshness` deliberately NOT content-relativized (CM2 — different axis, `last_full_cycle_at`; filed in TODOS as a probe-phase follow-up). Pinned by the "v0.41.32.0 — commit-relative staleness" describe in `test/doctor.test.ts` (T1 untracked-folders headline bug + T2/T2b/T2c remote-never-shells-out trust boundary). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. **v109 (v0.41.32.0, supersedes #1623):** `sources_newest_content_at` adds `sources.newest_content_at TIMESTAMPTZ` — durable newest-COMMIT timestamp (HEAD committer time) written at sync time by `writeSyncAnchor`. The REMOTE staleness path (federation_health, get_status_snapshot MCP op) reads it instead of shelling out to git on a DB-supplied local_path. Renumbered 108→109 on the master merge that landed v0.41.31 pages_embedding_signature at v108. Metadata-only ADD COLUMN; mirror in pglite-schema.ts + schema.sql + bootstrap probe coverage. +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.36.3.0 (v68):** `eval_candidates_embedding_column` adds `eval_candidates.embedding_column TEXT NULL`. Per-row provenance for `gbrain eval replay`: captured rows record which column the live query ran against so replay reproduces the same retrieval space (Voyage rows replay against Voyage; OpenAI rows against OpenAI). NULL-tolerant — pre-v0.36 rows fall back to the current default during replay rather than failing. Column-only migration, metadata-only on both engines. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. **v108 (v0.41.31.0):** `pages_embedding_signature` adds `pages.embedding_signature TEXT NULL` = `:` stamped when a page's chunks are embedded (`setPageEmbeddingSignature`). A later model/dimension swap makes the stored signature differ from the current one so `countStaleChunks`/`sumStaleChunkChars` (with the `signature` opt) and `invalidateStaleSignatureEmbeddings` can detect and re-embed those pages. GRANDFATHER (critical): the stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current`, so a NULL signature is NEVER stale — after this migration every existing page has NULL, none are flagged, and the next `embed --stale` does NOT re-embed the whole corpus. Signatures only get stamped going forward. No index (read only via a JOINed pages row in the chunk-grain stale queries; no standalone lookup hot path). `ADD COLUMN` with no DEFAULT (NULL) is metadata-only on Postgres 11+ / PGLite 17.5. **v109 (v0.41.32.0, supersedes #1623):** `sources_newest_content_at` adds `sources.newest_content_at TIMESTAMPTZ` — durable newest-COMMIT timestamp (HEAD committer time) written at sync time by `writeSyncAnchor`. The REMOTE staleness path (federation_health, get_status_snapshot MCP op) reads it instead of shelling out to git on a DB-supplied local_path. Renumbered 108→109 on the master merge that landed v0.41.31 pages_embedding_signature at v108. Metadata-only ADD COLUMN; mirror in pglite-schema.ts + schema.sql + bootstrap probe coverage. **v0.41.34.0 (retrieval-maxpool incident, v110 + v111):** v110 (`page_aliases`) creates the free-text alias table — `(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` and lookup indexes on `(source_id, alias_norm)` + `(source_id, slug)`. `alias_norm` is the output of `normalizeAlias()` so the WRITE side (ingest projection) and READ side (search) key on the same normalized form. Also lands in `src/core/pglite-schema.ts` for fresh PGLite installs. v111 (`search_telemetry_rank1_columns`) is `ADD COLUMN IF NOT EXISTS` on both engines for the rank-1 base_score drift signal: `sum_rank1_score`, `count_rank1`, and three coarse distribution buckets (`rank1_lt_solid` / `rank1_solid` / `rank1_high`) on the existing `search_telemetry` rollup — aggregate columns (NOT per-query rows, D10) so a downward drift in the median rank-1 match score is computable with bounded growth. `search_telemetry` lives only in migration v57, so these ALTER right after v57 runs. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. **v0.40.3.0:** `emitHumanLine` is prefix-aware — when called inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts`, prepends `[id] ` to the line content. TTY-rewrite mode (`\r\x1b[2K`) gets the prefix inside the clear-to-EOL escape so the rewritten line carries the prefix too. JSON mode (`emitJson`) is intentionally NOT prefixed — NDJSON consumers parse the JSON envelope and would choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` (v0.40.3.0) — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as the active prefix; nested wraps replace the active prefix and restore on exit), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log` / `console.error` replacements). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog` / `serr` fall through to bare `console.log` / `console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form text) to defeat log-injection through newline / control-character names. Coverage in v0.40.3.0: `src/commands/sync.ts` performSync + in-file callees (38 call sites migrated), `src/commands/embed.ts` runEmbedCore + helpers (16 call sites), `src/core/progress.ts` emitHumanLine. Lines emitted from outside these modules will NOT get the prefix under parallel sync — file an issue if you find a missed migration target. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -2889,7 +2898,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec ## Capabilities -**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. +**Hybrid search.** Vector (HNSW on pgvector) + BM25 keyword + reciprocal-rank fusion + source-tier boost + intent-aware query rewriting. Three named search modes (`conservative`, `balanced`, `tokenmax`) bundle the cost/quality knobs into a single config key. Live cost/recall comparisons in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). Default: `balanced` with ZeroEntropy reranker on. Per-query graph signals notice when a top result is a hub for THAT query (adjacency boost), is corroborated across team brains (cross-source boost), or is being crowded out by weak chunks from a chatty session (session demote). Run `gbrain search "" --explain` to see per-stage attribution: base score, every boost that fired, what it multiplied. `gbrain doctor` ships a `graph_signals_coverage` check; `gbrain search stats` shows fire counts and failure breakdowns. Vector retrieval pools the best chunk per page, so a page surfaces on its strongest evidence instead of losing to a neighbor on one weak chunk. Queries that match a page's title phrase or a declared free-text alias (`gbrain reindex --aliases` backfills existing pages) get boosted to the page they name. Every result carries an `evidence` tag (why it matched) and a `create_safety` hint (`exists` / `probable` / `unknown`) so an agent decides whether a page already exists instead of guessing from a raw score. `gbrain search diagnose "" --target ` traces which retrieval layer surfaces (or misses) a page. **Self-wiring knowledge graph.** Every `put_page` extracts entity refs from markdown/wikilinks/typed-link syntax and writes edges with zero LLM calls. Typed edges (`attended`, `works_at`, `invested_in`, `founded`, `advises`, `mentions`, …). Multi-hop traversal via `gbrain graph-query`. The graph is what produces the +31.4 P@5 lift over vector-only RAG. @@ -2897,7 +2906,7 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **43 curated skills.** Routing lives in [`skills/RESOLVER.md`](skills/RESOLVER.md). Covers signal capture, ingest (idea / media / meeting), enrichment, querying, brain ops, citation fixing, daily task management, cron scheduling, reports, voice, soul audit, skill creation, eval framework, and migrations. Skills are markdown files (tool-agnostic), packaged as a single skillpack the installer drops into your agent workspace. -**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). +**Eval framework.** `gbrain eval longmemeval` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against your hybrid retrieval. `gbrain eval export` + `gbrain eval replay` capture real queries and replay them against code changes (set `GBRAIN_CONTRIBUTOR_MODE=1`). `gbrain eval cross-modal` cross-checks an output against the task using three different-provider frontier models. `gbrain eval retrieval-quality` runs NamedThingBench, which hard-gates the named-thing retrieval families (title-substring, alias-synonym, generic-to-named, multi-chunk-dilution) so a regression in "find the page this query names" fails CI loudly. Full methodology in [`docs/eval/SEARCH_MODE_METHODOLOGY.md`](docs/eval/SEARCH_MODE_METHODOLOGY.md). **Brain consistency.** `gbrain eval suspected-contradictions` samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle. diff --git a/package.json b/package.json index 4a50ea192..b0f931852 100644 --- a/package.json +++ b/package.json @@ -141,5 +141,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.33.0" + "version": "0.41.34.0" } diff --git a/src/cli.ts b/src/cli.ts index e235d8696..767955593 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -107,6 +107,38 @@ async function main() { command = 'query'; } + // T5 — `gbrain search modes|stats|tune` is the read-only config dashboard, + // NOT a free-text search for the literal word "modes". Free-text + // `gbrain search ""` falls through to the cheap-hybrid `search` op + // below (T4). Preserves the v0.41.6.0 read-only connect+dispatch timeout. + if (command === 'search' && ['modes', 'stats', 'tune', 'diagnose'].includes(subArgs[0] ?? '')) { + const { withTimeout, OperationTimeoutError } = await import('./core/timeout.ts'); + const isDiagnose = subArgs[0] === 'diagnose'; + const label = 'gbrain search'; + // diagnose runs real retrieval (keyword + vector + hybrid) so it gets a + // longer deadline than the read-only dashboard. + const timeoutMs = isDiagnose ? 60_000 : 10_000; + let engine: BrainEngine; + try { + engine = await withTimeout(connectEngine(), timeoutMs, `${label}: connect`); + } catch (e) { + if (e instanceof OperationTimeoutError) { console.error(`${e.label} timed out.`); process.exit(124); } + throw e; + } + try { + if (isDiagnose) { + const { runSearchDiagnose } = await import('./commands/search-diagnose.ts'); + await withTimeout(runSearchDiagnose(engine, subArgs), timeoutMs, label); + } else { + const { runSearch } = await import('./commands/search.ts'); + await withTimeout(runSearch(engine, subArgs), timeoutMs, label); + } + } finally { + await engine.disconnect(); + } + return; + } + // Per-command --help if (hasHelpFlag(subArgs)) { const op = cliOps.get(command); @@ -1438,6 +1470,13 @@ async function handleCliOnly(command: string, args: string[]) { } break; } + if (args.includes('--aliases')) { + // T8 — backfill the free-text alias layer (page_aliases) for existing + // pages whose frontmatter `aliases:` predate the import-time projection. + const { runReindexAliases } = await import('./commands/reindex-aliases.ts'); + await runReindexAliases(engine, args); + break; + } const { runReindex } = await import('./commands/reindex.ts'); await runReindex(engine, args); break; diff --git a/src/commands/eval-retrieval-quality.ts b/src/commands/eval-retrieval-quality.ts new file mode 100644 index 000000000..175a1e7a3 --- /dev/null +++ b/src/commands/eval-retrieval-quality.ts @@ -0,0 +1,75 @@ +/** + * `gbrain eval retrieval-quality [--json] [--source ]` + * (T6 — NamedThingBench). Runs the gold query set against the brain's hybrid + * retrieval and gates on the families that ARE the retrieval-maxpool incident. + * + * Run with reranker + expansion at their configured defaults but the gate + * measures core retrieval (title/alias/pool) — the families don't depend on + * the rescue layers. Exit 0 PASS / 1 FAIL (hard-family breach) / 2 USAGE. + */ + +import type { BrainEngine } from '../core/engine.ts'; +import { readFileSync } from 'fs'; +import { hybridSearch } from '../core/search/hybrid.ts'; +import { + parseQuestionsJsonl, + runRetrievalQuality, + evaluateGate, + type SearchFn, +} from '../eval/retrieval-quality/harness.ts'; + +export async function runEvalRetrievalQuality(engine: BrainEngine, args: string[]): Promise { + const json = args.includes('--json'); + const sourceIdx = args.indexOf('--source'); + const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined; + const fixture = args.find(a => !a.startsWith('--') && a !== sourceId); + + if (!fixture) { + console.error('Usage: gbrain eval retrieval-quality [--json] [--source ]'); + process.exit(2); + } + + let questions; + try { + questions = parseQuestionsJsonl(readFileSync(fixture, 'utf8')); + } catch (e) { + console.error(`Cannot read fixture: ${e instanceof Error ? e.message : String(e)}`); + process.exit(2); + } + + // Core-retrieval measurement: reranker/expansion at config defaults; the + // families key off title/alias/pool which are upstream of the rescue layers. + const searchFn: SearchFn = async (q) => { + const results = await hybridSearch(engine, q, { + limit: 10, + ...(sourceId ? { sourceId } : {}), + }); + return results.map(r => r.slug); + }; + + const report = await runRetrievalQuality(questions, searchFn); + const gate = evaluateGate(report); + + if (json) { + console.log(JSON.stringify({ schema_version: 1, report, gate }, null, 2)); + } else { + console.log(`NamedThingBench — ${report.total} queries across ${report.families.length} families\n`); + for (const f of report.families) { + console.log(` ${f.family.padEnd(22)} n=${f.n} Hit@1=${(f.hit_at_1 * 100).toFixed(0)}% Hit@3=${(f.hit_at_3 * 100).toFixed(0)}% MRR=${f.mrr.toFixed(3)}`); + } + console.log(''); + if (gate.breaches.length) { + console.log('GATE: FAIL'); + for (const b of gate.breaches) { + console.log(` ✗ ${b.family} ${b.metric}=${(b.got * 100).toFixed(0)}% < floor ${(b.floor * 100).toFixed(0)}%`); + } + } else { + console.log('GATE: PASS'); + } + for (const w of gate.warnings) { + console.log(` ⚠ (warn) ${w.family} ${w.metric}=${(w.got * 100).toFixed(0)}% < ${(w.floor * 100).toFixed(0)}%`); + } + } + + process.exit(gate.pass ? 0 : 1); +} diff --git a/src/commands/eval.ts b/src/commands/eval.ts index 850f9e59c..b533f2ce0 100644 --- a/src/commands/eval.ts +++ b/src/commands/eval.ts @@ -60,6 +60,12 @@ export async function runEvalCommand(engine: BrainEngine, args: string[]): Promi const { runEvalCodeRetrieval } = await import('./eval-code-retrieval.ts'); return runEvalCodeRetrieval(engine, args.slice(1)); } + if (sub === 'retrieval-quality') { + // T6 — NamedThingBench. Gold query set vs hybrid retrieval; gates the + // families that ARE the retrieval-maxpool incident (title/alias/dilution). + const { runEvalRetrievalQuality } = await import('./eval-retrieval-quality.ts'); + return runEvalRetrievalQuality(engine, args.slice(1)); + } if (sub === 'brainstorm') { // v0.37.0 (D3 + codex r2 #11) — three-axis evaluation gate for the // brainstorm + LSD wave. Engine connected (calls hybridSearch + diff --git a/src/commands/reindex-aliases.ts b/src/commands/reindex-aliases.ts new file mode 100644 index 000000000..226a54426 --- /dev/null +++ b/src/commands/reindex-aliases.ts @@ -0,0 +1,86 @@ +/** + * `gbrain reindex --aliases [--limit N] [--dry-run] [--json] [--source ]` + * (T8 — backfill the free-text alias layer). + * + * Import-time projection (T3) covers NEW + changed pages; this backfills the + * EXISTING pages whose frontmatter `aliases:` predate v110 (or predate the + * projection landing). Reads each page's frontmatter `aliases:` and writes + * page_aliases via engine.setPageAliases. + * + * Idempotent: setPageAliases replaces a page's alias set, so re-running is + * safe and convergent — no op-checkpoint needed (the op is fast, no embedding). + * Walks listAllPageRefs (cheap (source_id, slug) enumeration) so it's + * cross-source by default; --source narrows it. + */ + +import type { BrainEngine } from '../core/engine.ts'; +import { normalizeAliasList } from '../core/search/alias-normalize.ts'; +import { createProgress } from '../core/progress.ts'; +import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'; + +export interface ReindexAliasesResult { + scanned: number; + pages_with_aliases: number; + aliases_written: number; + dry_run: boolean; +} + +export async function runReindexAliases(engine: BrainEngine, args: string[]): Promise { + const dryRun = args.includes('--dry-run'); + const json = args.includes('--json'); + const limitIdx = args.indexOf('--limit'); + const limit = limitIdx >= 0 ? parseInt(args[limitIdx + 1] ?? '', 10) : NaN; + const sourceIdx = args.indexOf('--source'); + const sourceFilter = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined; + + let refs = await engine.listAllPageRefs(); + if (sourceFilter) refs = refs.filter(r => r.source_id === sourceFilter); + if (Number.isFinite(limit) && limit > 0) refs = refs.slice(0, limit); + + const reporter = createProgress(cliOptsToProgressOptions(getCliOptions())); + reporter.start('reindex.aliases', refs.length); + + let scanned = 0; + let pagesWithAliases = 0; + let aliasesWritten = 0; + + for (const ref of refs) { + scanned++; + reporter.tick(); + let page; + try { + page = await engine.getPage(ref.slug, { sourceId: ref.source_id }); + } catch { + continue; + } + if (!page) continue; + const aliasNorms = normalizeAliasList((page.frontmatter as Record | undefined)?.aliases); + if (aliasNorms.length === 0) continue; + pagesWithAliases++; + aliasesWritten += aliasNorms.length; + if (!dryRun) { + try { + await engine.setPageAliases(ref.slug, ref.source_id, aliasNorms); + } catch (e) { + reporter.finish(); + throw e; // pre-v110 (no table) or a real write error — surface it. + } + } + } + + reporter.finish(); + const result: ReindexAliasesResult = { + scanned, + pages_with_aliases: pagesWithAliases, + aliases_written: aliasesWritten, + dry_run: dryRun, + }; + + if (json) { + console.log(JSON.stringify(result, null, 2)); + } else { + const verb = dryRun ? 'would write' : 'wrote'; + console.log(`reindex --aliases: scanned ${scanned} pages, ${verb} ${aliasesWritten} aliases across ${pagesWithAliases} pages.`); + } + return result; +} diff --git a/src/commands/search-diagnose.ts b/src/commands/search-diagnose.ts new file mode 100644 index 000000000..37b8f3154 --- /dev/null +++ b/src/commands/search-diagnose.ts @@ -0,0 +1,131 @@ +/** + * `gbrain search diagnose "" --target [--json] [--source ]` + * (T0 — Phase-0 retrieval diagnostic). + * + * Traces WHERE a target page surfaces (or fails to) across the retrieval + * pipeline, so an operator can pin which layer is responsible for an incident + * like "Greek amphitheater missed the Mingtang page": + * + * keyword — rank + score of the target in searchKeyword (ts_rank path) + * vector — rank + score of the target in searchVector (per-page max-pool); + * best-chunk cosine; skipped if no embedding provider + * alias — is the normalized query a registered page_aliases alias of the target? + * hybrid — final rank + evidence + create_safety + which boosts fired + * + * The verdict names the layer that DOES surface the target (or "none"), which + * tells you whether the fix is max-pool/innerLimit (vector) vs title/alias. + */ + +import type { BrainEngine } from '../core/engine.ts'; +import { hybridSearch } from '../core/search/hybrid.ts'; +import { normalizeAlias } from '../core/search/alias-normalize.ts'; + +interface LayerProbe { + rank: number | null; // 1-based rank of the target, null if absent + score: number | null; + top_slug: string | null; + note?: string; +} + +function rankOf(slugs: string[], target: string): number | null { + const i = slugs.indexOf(target); + return i >= 0 ? i + 1 : null; +} + +export async function runSearchDiagnose(engine: BrainEngine, args: string[]): Promise { + const json = args.includes('--json'); + const targetIdx = args.indexOf('--target'); + const target = targetIdx >= 0 ? args[targetIdx + 1] : undefined; + const sourceIdx = args.indexOf('--source'); + const sourceId = sourceIdx >= 0 ? args[sourceIdx + 1] : undefined; + // 'diagnose' is args[0]; the query is the first non-flag token after it. + const query = args.slice(1).find((a, i) => !a.startsWith('--') && args.slice(1)[i - 1] !== '--target' && args.slice(1)[i - 1] !== '--source'); + + if (!query || !target) { + console.error('Usage: gbrain search diagnose "" --target [--json] [--source ]'); + process.exit(2); + } + + const scope = sourceId ? { sourceId } : {}; + const limit = 50; + + // Keyword layer. + const kw = await engine.searchKeyword(query, { limit, ...scope }); + const keyword: LayerProbe = { + rank: rankOf(kw.map(r => r.slug), target), + score: kw.find(r => r.slug === target)?.score ?? null, + top_slug: kw[0]?.slug ?? null, + }; + + // Vector layer (skip if no embedding provider). + let vector: LayerProbe; + const { isAvailable, embedQuery } = await import('../core/ai/gateway.ts'); + if (!isAvailable('embedding')) { + vector = { rank: null, score: null, top_slug: null, note: 'skipped — no embedding provider configured' }; + } else { + try { + const qvec = await embedQuery(query); + const vec = await engine.searchVector(qvec, { limit, ...scope }); + vector = { + rank: rankOf(vec.map(r => r.slug), target), + score: vec.find(r => r.slug === target)?.score ?? null, + top_slug: vec[0]?.slug ?? null, + }; + } catch (e) { + vector = { rank: null, score: null, top_slug: null, note: `vector probe failed: ${e instanceof Error ? e.message : String(e)}` }; + } + } + + // Alias layer. + const qNorm = normalizeAlias(query); + let aliasMatch = false; + try { + const m = await engine.resolveAliases([qNorm], scope); + aliasMatch = (m.get(qNorm) ?? []).some(r => r.slug === target); + } catch { /* pre-v110: no alias layer */ } + + // Hybrid layer (the production path). + const hy = await hybridSearch(engine, query, { limit, ...scope }); + const hyTarget = hy.find(r => r.slug === target); + const hybrid = { + rank: rankOf(hy.map(r => r.slug), target), + score: hyTarget?.score ?? null, + base_score: hyTarget?.base_score ?? null, + evidence: hyTarget?.evidence ?? null, + create_safety: hyTarget?.create_safety ?? null, + title_match_boost: hyTarget?.title_match_boost ?? null, + alias_hit: hyTarget?.alias_hit ?? false, + top_slug: hy[0]?.slug ?? null, + }; + + // Verdict: which layer surfaces the target at rank 1? + const surfacing: string[] = []; + if (keyword.rank === 1) surfacing.push('keyword'); + if (vector.rank === 1) surfacing.push('vector'); + if (aliasMatch) surfacing.push('alias'); + if (hybrid.rank === 1) surfacing.push('hybrid(final)'); + const verdict = hybrid.rank === 1 + ? `target is rank 1 in hybrid (via: ${surfacing.join(', ') || 'unknown'})` + : hybrid.rank + ? `target is rank ${hybrid.rank} in hybrid — NOT rank 1. Strongest layer: ${surfacing.join(', ') || 'none'}` + : `target ABSENT from hybrid top-${limit}. keyword rank=${keyword.rank ?? 'absent'}, vector rank=${vector.rank ?? 'absent'}, alias=${aliasMatch}`; + + const report = { query, target, source_id: sourceId ?? 'default', keyword, vector, alias_match: aliasMatch, hybrid, verdict }; + + if (json) { + console.log(JSON.stringify(report, null, 2)); + return; + } + + console.log(`Diagnose: "${query}" → target ${target}\n`); + console.log(` keyword: rank=${keyword.rank ?? 'absent'} score=${fmt(keyword.score)} (top: ${keyword.top_slug ?? '—'})`); + console.log(` vector: rank=${vector.rank ?? 'absent'} score=${fmt(vector.score)} (top: ${vector.top_slug ?? '—'})${vector.note ? ` [${vector.note}]` : ''}`); + console.log(` alias: ${aliasMatch ? 'query IS a registered alias of the target' : 'no alias match'}`); + console.log(` hybrid: rank=${hybrid.rank ?? 'absent'} score=${fmt(hybrid.score)} base=${fmt(hybrid.base_score)} evidence=${hybrid.evidence ?? '—'} create_safety=${hybrid.create_safety ?? '—'}`); + console.log(` boosts: title=${hybrid.title_match_boost ?? '—'} alias_hit=${hybrid.alias_hit}`); + console.log(`\n VERDICT: ${verdict}`); +} + +function fmt(n: number | null): string { + return n === null ? '—' : n.toFixed(4); +} diff --git a/src/commands/search.ts b/src/commands/search.ts index 75f009100..a8901f14d 100644 --- a/src/commands/search.ts +++ b/src/commands/search.ts @@ -54,6 +54,7 @@ const KNOB_DESCRIPTIONS: Record = { reranker_top_n_out: 'Cap on reranked output (null = no truncate)', reranker_timeout_ms: 'HTTP timeout for the reranker call', floor_ratio: 'Floor-ratio gate for metadata boosts (0..1, undefined = off)', + title_boost: 'Title-phrase boost multiplier (query is a title token-run; 1.0 = off)', // v0.36 cross-modal knobs (D3 registry) cross_modal_both_text_weight: "D6 'both'-mode RRF weight for text branch (0.6 default)", cross_modal_both_image_weight: "D6 'both'-mode RRF weight for image branch (0.4 default)", @@ -253,6 +254,12 @@ async function runStatsSubcommand(engine: BrainEngine, args: string[]): Promise< console.log(` Avg results returned: ${stats.avg_results.toFixed(1)}`); console.log(` Avg tokens delivered: ${stats.avg_tokens.toFixed(0)} (char/4 heuristic)`); console.log(` Budget drops total: ${stats.total_budget_dropped}`); + // T7 — rank-1 match-quality drift signal. Watch for avg drifting DOWN. + if (stats.avg_rank1_score !== null && stats.rank1_count > 0) { + const d = stats.rank1_distribution; + console.log(` Avg rank-1 score: ${stats.avg_rank1_score.toFixed(3)} (${stats.rank1_count} samples; <0.6:${d.lt_solid} 0.6-0.85:${d.solid} >=0.85:${d.high})`); + console.log(` (top-result match quality; a downward drift = retrieval regressing)`); + } console.log(''); console.log(' Mode distribution:'); for (const [m, c] of Object.entries(stats.mode_distribution).sort((a, b) => b[1] - a[1])) { diff --git a/src/core/engine.ts b/src/core/engine.ts index f76baddbc..317931fcd 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -1723,6 +1723,38 @@ export interface BrainEngine { sourceOrSources: string | readonly string[], ): Promise; + /** + * T3 retrieval-cathedral — free-text alias resolution for SEARCH. + * Given a set of NORMALIZED alias strings (normalizeAlias() output), + * return a map alias_norm -> canonical slugs from page_aliases. An alias + * may resolve to MORE THAN ONE slug (two pages claimed the same name) — + * the caller reports the collision and resolves deterministically. + * + * Source-scoped: pass scalar `sourceId` (single-source) OR `sourceIds` + * (federated read) — array wins when both set, matching sourceScopeOpts. + * Empty input → empty map (no query). Throws "relation page_aliases does + * not exist" on pre-v110 brains; the alias-hop caller wraps in try/catch + * (isUndefinedColumnError) and degrades to no-injection (D9 fail-open). + */ + resolveAliases( + aliasNorms: string[], + opts?: { sourceId?: string; sourceIds?: string[] }, + ): Promise>>; + + /** + * T3 retrieval-cathedral — WRITE side of the alias layer. Replace the full + * alias set for one (slug, source) with `aliasNorms` (already normalized): + * delete the page's existing page_aliases rows, insert the new set. Empty + * `aliasNorms` just clears them. Idempotent (the unique triple). Called by + * the ingest projection in importFromContent and the `reindex --aliases` + * backfill so a page's declared aliases stay in lockstep with its frontmatter. + */ + setPageAliases( + slug: string, + sourceId: string, + aliasNorms: string[], + ): Promise; + /** * v0.35.5 — narrow UPDATE of `pages.compiled_truth`, `pages.timeline`, and * `pages.content_hash` for a single slug+source. NO chunking, NO embedding, diff --git a/src/core/eval/metric-glossary.ts b/src/core/eval/metric-glossary.ts index 9124b8738..a1ef025ec 100644 --- a/src/core/eval/metric-glossary.ts +++ b/src/core/eval/metric-glossary.ts @@ -53,6 +53,30 @@ export const METRIC_GLOSSARY: Readonly range: '0..1, higher is better. nDCG@10 above 0.65 is the common "ship it" threshold for hybrid retrieval on technical corpora.', }), + // ──────────────────────────────────────────────────────────────────────── + // NamedThingBench (retrieval-quality) + the agent-facing evidence contract + // ──────────────────────────────────────────────────────────────────────── + 'hit@1': Object.freeze({ + industry_term: 'Hit rate at 1 (Hit@1)', + eli10: 'Fraction of queries where the right page is the very first result. NamedThingBench hard-gates title-substring Hit@1 >= 0.95 and alias Hit@1 >= 0.98 — a query that is a page\'s name or title phrase should land it at rank 1, not "somewhere in the top 10".', + range: '0..1, higher is better.', + }), + 'hit@3': Object.freeze({ + industry_term: 'Hit rate at 3 (Hit@3)', + eli10: 'Fraction of queries where the right page is in the top 3 results. NamedThingBench requires the multi-chunk-dilution family to hit 1.0 — a page with one strong chunk among many weak ones must never be buried.', + range: '0..1, higher is better.', + }), + 'avg_rank1_score': Object.freeze({ + industry_term: 'Average rank-1 match score', + eli10: 'The mean base (pre-boost) retrieval score of the TOP result across recent searches, from `gbrain search stats`. It is NOT a labeled accuracy number — it is a drift signal: if this trends DOWN over time, retrieval quality is regressing (the early warning that would have caught the duplicate-page incident before a human did).', + range: '0..1. Watch the trend, not the absolute value; pair with the <0.6 / 0.6-0.85 / >=0.85 bucket counts for shape.', + }), + 'create_safety': Object.freeze({ + industry_term: 'Create-safety hint (evidence contract)', + eli10: 'A result\'s answer to "is this page already in the brain — safe to NOT write a new one?" Derived from the strongest evidence, NOT a raw score: exists (alias_hit / exact_title_match / high_vector_match — do not duplicate), probable (solid keyword match — prefer updating), unknown (weak match — look closer). An agent keys its don\'t-duplicate decision off this, which is what prevents the incident\'s duplicate-stub class.', + range: 'enum: exists | probable | unknown', + }), + // ──────────────────────────────────────────────────────────────────────── // Set-similarity / stability metrics (replay + regression checks) // ──────────────────────────────────────────────────────────────────────── @@ -182,6 +206,7 @@ export function renderMetricGlossaryMarkdown(): string { const groups: Array<[string, string[]]> = [ ['Retrieval Metrics', ['precision@k', 'recall@k', 'mrr', 'ndcg@k']], + ['Retrieval-Quality / Evidence Metrics (NamedThingBench)', ['hit@1', 'hit@3', 'avg_rank1_score', 'create_safety']], ['Set-Similarity / Stability Metrics', ['jaccard@k', 'top1_stability']], ['Statistical-Significance Metrics', ['p_value', 'confidence_interval']], ['Operational / Cost Metrics', ['cache_hit_rate', 'avg_results', 'avg_tokens', 'cost_per_query_usd', 'p99_latency_ms']], diff --git a/src/core/import-file.ts b/src/core/import-file.ts index 3beb34328..82a0d5c00 100644 --- a/src/core/import-file.ts +++ b/src/core/import-file.ts @@ -28,6 +28,8 @@ import { wrapChunkForEmbedding, } from './embedding-context.ts'; import { loadSearchModeConfig, resolveSearchMode } from './search/mode.ts'; +import { normalizeAliasList } from './search/alias-normalize.ts'; +import { isUndefinedTableError, warnOncePerProcess } from './utils.ts'; import { computeCorpusGeneration } from './contextual-retrieval-service.ts'; /** @@ -708,6 +710,25 @@ export async function importFromContent( } }); + // T3 — project frontmatter `aliases:` into page_aliases (free-text alias + // resolution for search). Runs AFTER the page write commits so the slug + // exists. Fail-soft: a pre-v110 brain has no page_aliases table yet (the + // migration may not have run); an alias-write failure must NOT fail the + // import. Always called (even with []) so REMOVING an alias from frontmatter + // clears its row — the content_hash includes non-timestamp frontmatter, so + // an alias edit changes the hash and reaches this path (not the skip branch). + try { + const aliasNorms = normalizeAliasList((parsed.frontmatter as Record).aliases); + await engine.setPageAliases(slug, sourceId ?? 'default', aliasNorms); + } catch (e) { + if (!isUndefinedTableError(e)) { + warnOncePerProcess( + 'setPageAliases:failed', + `[import] page_aliases projection failed (non-fatal): ${e instanceof Error ? e.message : String(e)}`, + ); + } + } + return { slug, status: 'imported', chunks: chunks.length, parsedPage }; } diff --git a/src/core/migrate.ts b/src/core/migrate.ts index dfd23b177..511397053 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -4962,6 +4962,67 @@ export const MIGRATIONS: Migration[] = [ idempotent: true, sql: `ALTER TABLE sources ADD COLUMN IF NOT EXISTS newest_content_at TIMESTAMPTZ`, }, + { + version: 110, + name: 'page_aliases', + // T3 of the retrieval-cathedral wave (retrieval-maxpool incident). + // + // Free-text alias resolution: a query like "Hall of Light" or "明堂" + // should surface the page titled "Mingtang". gbrain stored that mapping + // in pages.frontmatter `aliases:` JSONB but it was invisible to search. + // + // DELIBERATELY SEPARATE from slug_aliases (v105). They answer different + // questions and overloading one for both would muddy the semantics: + // - slug_aliases: old-slug -> canonical-slug (wikilink/get_page redirect) + // - page_aliases: normalized free-text name -> canonical slug (search hop) + // + // alias_norm is the output of normalizeAlias() (NFKC + lowercase + ws + // collapse) so the WRITE side (ingest projection) and READ side (search) + // match on the same key. Btree on (source_id, alias_norm) so the hop is an + // indexed equality lookup, not ILIKE. + // + // NOT a UNIQUE(source_id, alias_norm) — real brains may legitimately have + // two pages claiming the same alias; we report the collision and resolve + // deterministically at query time rather than failing the ingest (Codex#8). + // The (source_id, alias_norm, slug) triple is unique so re-ingest is + // idempotent without blocking a second page's claim on the same alias. + // + // Mirror in src/core/pglite-schema.ts (fresh install); forward-reference + // bootstrap probe on both engines so pre-v110 brains pick it up cleanly. + idempotent: true, + sql: ` + CREATE TABLE IF NOT EXISTS page_aliases ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + alias_norm TEXT NOT NULL, + slug TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + CONSTRAINT page_aliases_uniq UNIQUE (source_id, alias_norm, slug) + ); + CREATE INDEX IF NOT EXISTS page_aliases_lookup_idx + ON page_aliases (source_id, alias_norm); + CREATE INDEX IF NOT EXISTS page_aliases_slug_idx + ON page_aliases (source_id, slug); + `, + }, + { + version: 111, + name: 'search_telemetry_rank1_columns', + // T7 of the retrieval-cathedral wave — rank-1 base_score drift signal. + // Aggregate columns (NOT per-query rows, D10) so a downward drift in the + // median rank-1 match score is computable from the existing day/mode/intent + // rollup with bounded growth. search_telemetry lives only in migration v57 + // (not the schema blobs), so these are ADD COLUMN IF NOT EXISTS on both + // engines; fresh installs pick them up right after v57 runs. + idempotent: true, + sql: ` + ALTER TABLE search_telemetry ADD COLUMN IF NOT EXISTS sum_rank1_score DOUBLE PRECISION NOT NULL DEFAULT 0; + ALTER TABLE search_telemetry ADD COLUMN IF NOT EXISTS count_rank1 INTEGER NOT NULL DEFAULT 0; + ALTER TABLE search_telemetry ADD COLUMN IF NOT EXISTS rank1_lt_solid INTEGER NOT NULL DEFAULT 0; + ALTER TABLE search_telemetry ADD COLUMN IF NOT EXISTS rank1_solid INTEGER NOT NULL DEFAULT 0; + ALTER TABLE search_telemetry ADD COLUMN IF NOT EXISTS rank1_high INTEGER NOT NULL DEFAULT 0; + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/operations.ts b/src/core/operations.ts index 94843a899..2ef0585c6 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -21,6 +21,9 @@ import { isFactsBackstopEligible } from './facts/eligibility.ts'; import { stripTakesFence } from './takes-fence.ts'; import { stripFactsFence } from './facts-fence.ts'; import { bumpLastRetrievedAt } from './last-retrieved.ts'; +import { isSearchMode } from './search/mode.ts'; +import { stampEvidence } from './search/evidence.ts'; +import type { SearchResult } from './types.ts'; import { CJK_SLUG_CHARS } from './cjk.ts'; import * as db from './db.ts'; import { VERSION } from '../version.ts'; @@ -419,6 +422,59 @@ export function sourceScopeOpts(ctx: OperationContext): { sourceId?: string; sou return {}; } +/** + * T4/D5 — resolve a per-call search-mode override. Honored ONLY for trusted/ + * local callers (ctx.remote === false) so a remote OAuth client can't escalate + * to the costly tokenmax bundle. Local + unknown mode → loud reject; remote + + * mode → silently ignored (server-configured mode wins). Returns undefined to + * mean "use the configured mode". + */ +export function resolvePerCallMode(ctx: OperationContext, raw: unknown): string | undefined { + if (typeof raw !== 'string' || raw.length === 0) return undefined; + if (ctx.remote !== false) return undefined; // remote can't select mode + if (!isSearchMode(raw)) { + throw new OperationError( + 'invalid_params', + `Unknown search mode '${raw}'. Valid: conservative, balanced, tokenmax.`, + `gbrain search "" --mode balanced`, + ); + } + return raw; +} + +/** T4 — stamp evidence/create_safety on a result set, fail-soft. */ +function stampEvidenceSafe(results: SearchResult[]): void { + try { stampEvidence(results); } catch { /* non-fatal */ } +} + +/** T4 — shared eval-capture for the `search` op (keyword-only + cheap-hybrid paths). */ +function maybeCaptureSearch( + ctx: OperationContext, + queryText: string, + results: SearchResult[], + latency_ms: number, + vectorEnabled: boolean, + meta?: HybridSearchMeta | null, +): void { + if (!isEvalCaptureEnabled(ctx.config)) return; + void captureEvalCandidate( + ctx.engine, + { + tool_name: 'search', + query: queryText, + results, + meta: meta ?? { vector_enabled: vectorEnabled, detail_resolved: null, expansion_applied: false }, + latency_ms, + remote: ctx.remote ?? false, + expand_enabled: false, + detail: null, + job_id: ctx.jobId ?? null, + subagent_id: ctx.subagentId ?? null, + }, + { scrub_pii: isEvalScrubEnabled(ctx.config) }, + ); +} + export interface Operation { name: string; description: string; @@ -1194,48 +1250,48 @@ const search: Operation = { query: { type: 'string', required: true }, limit: { type: 'number', description: 'Max results (default 20)' }, offset: { type: 'number', description: 'Skip first N results (for pagination)' }, + mode: { type: 'string', description: 'Search mode (conservative|balanced|tokenmax). Local callers only.' }, }, handler: async (ctx, p) => { const startedAt = Date.now(); const queryText = p.query as string; - // v0.34.1 (#861 — P0 leak seal): thread caller's source scope into - // searchKeyword. Pre-fix this op silently returned cross-source hits - // for any auth'd OAuth client. - const raw = await ctx.engine.searchKeyword(queryText, { - limit: (p.limit as number) || 20, - offset: (p.offset as number) || 0, - ...sourceScopeOpts(ctx), - }); - const results = dedupResults(raw); - const latency_ms = Date.now() - startedAt; + const limit = (p.limit as number) || 20; + const offset = (p.offset as number) || 0; + const scope = sourceScopeOpts(ctx); - // v0.37.0 (D11): op-layer last_retrieved_at write-back. Fire-and-forget; - // results already returned by engine, this just marks them as user-surfaced - // for LSD's stale-page signal. 5-min throttle inside bumpLastRetrievedAt. - bumpLastRetrievedAt(ctx.engine, results.map((r) => r.page_id)); + // T4/D5 — per-call mode honored ONLY for trusted/local callers so a remote + // OAuth client can't escalate to the costly tokenmax bundle. Local + unknown + // mode → loud reject; remote + mode set → silently ignored (uses config). + const perCallMode = resolvePerCallMode(ctx, p.mode); - // Op-layer capture (v0.25.0). Fire-and-forget — no await on the - // capture call so MCP response latency is unaffected. search has - // no expand/detail/vector semantics so meta fields are fixed. - if (isEvalCaptureEnabled(ctx.config)) { - void captureEvalCandidate( - ctx.engine, - { - tool_name: 'search', - query: queryText, - results, - meta: { vector_enabled: false, detail_resolved: null, expansion_applied: false }, - latency_ms, - remote: ctx.remote ?? false, - expand_enabled: null, - detail: null, - job_id: ctx.jobId ?? null, - subagent_id: ctx.subagentId ?? null, - }, - { scrub_pii: isEvalScrubEnabled(ctx.config) }, - ); + // T4/D17 — escape hatch: keyword-only when the operator opts out of the + // hybrid `search` contract (privacy/cost: no query text to an embedding + // provider). Defaults to cheap-hybrid (D4/D15). + const keywordOnly = (await ctx.engine.getConfig('search.mcp_keyword_only')) === 'true'; + + if (keywordOnly) { + const raw = await ctx.engine.searchKeyword(queryText, { limit, offset, ...scope }); + const results = dedupResults(raw); + stampEvidenceSafe(results); + bumpLastRetrievedAt(ctx.engine, results.map((r) => r.page_id)); + maybeCaptureSearch(ctx, queryText, results, Date.now() - startedAt, false); + return results; } + // Cheap-hybrid (D4/D15): full vector+keyword+RRF+pool+title+alias, but + // expansion OFF (no per-call LLM cost). `query` op is the full-control variant. + let capturedMeta: HybridSearchMeta | null = null; + const results = await hybridSearchCached(ctx.engine, queryText, { + limit, + offset, + expansion: false, + ...scope, + ...(perCallMode ? { mode: perCallMode } : {}), + onMeta: (m) => { capturedMeta = m; }, + }); + const latency_ms = Date.now() - startedAt; + bumpLastRetrievedAt(ctx.engine, results.map((r) => r.page_id)); + maybeCaptureSearch(ctx, queryText, results, latency_ms, true, capturedMeta); return results; }, scope: 'read', @@ -1261,6 +1317,7 @@ const query: Operation = { offset: { type: 'number', description: 'Skip first N results (for pagination)' }, expand: { type: 'boolean', description: 'Enable multi-query expansion (default: true)' }, detail: { type: 'string', description: 'Result detail level: low (compiled truth only), medium (default, all with dedup), high (all chunks)' }, + mode: { type: 'string', description: 'Search mode (conservative|balanced|tokenmax). Local callers only; remote uses configured mode.' }, // v0.20.0 Cathedral II Layer 10 C1/C2: language + symbol-kind filters. lang: { type: 'string', description: 'Filter to chunks where content_chunks.language matches (e.g., typescript, python, ruby)' }, symbol_kind: { type: 'string', description: 'Filter to chunks where content_chunks.symbol_type matches (e.g., function, class, method, type, interface)' }, @@ -1391,6 +1448,8 @@ const query: Operation = { offset: (p.offset as number) || 0, expansion: expand, expandFn: expand ? expandQuery : undefined, + // T4/D5 — per-call mode (local/trusted only; remote ignored). + ...((): { mode?: string } => { const m = resolvePerCallMode(ctx, p.mode); return m ? { mode: m } : {}; })(), detail, language: (p.lang as string) || undefined, symbolKind: (p.symbol_kind as string) || undefined, diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index f6063c754..c82c3457d 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -47,7 +47,7 @@ import { normalizeWeightForStorage } from './takes-fence.ts'; import { GBrainError, PAGE_SORT_SQL } from './types.ts'; import { computeAnomaliesFromBuckets } from './cycle/anomaly.ts'; import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; -import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts'; +import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte } from './search/sql-ranking.ts'; import { normalizeEngineColumn, buildVectorCastFragment, @@ -1497,13 +1497,9 @@ export class PGLiteEngine implements BrainEngine { ORDER BY score DESC LIMIT $2 ), - best_per_page AS ( - SELECT DISTINCT ON (slug) * - FROM ranked - ORDER BY slug, score DESC - ) + ${buildBestPerPagePoolCte('ranked')} SELECT * FROM best_per_page - ORDER BY score DESC + ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $3 OFFSET $4`, params ); @@ -1612,13 +1608,9 @@ export class PGLiteEngine implements BrainEngine { ORDER BY score DESC LIMIT $3 ), - best_per_page AS ( - SELECT DISTINCT ON (slug) * - FROM ranked - ORDER BY slug, score DESC - ) + ${buildBestPerPagePoolCte('ranked')} SELECT * FROM best_per_page - ORDER BY score DESC + ORDER BY score DESC, page_id ASC, chunk_id ASC LIMIT $4 OFFSET $5`, params, ); @@ -1759,7 +1751,10 @@ export class PGLiteEngine implements BrainEngine { // subquery's WHERE would lexically resolve back to `te.page_id` itself // and degrade to `te.page_id = te.page_id` (always true), making every // result stale=true. Codex caught this in adversarial review. - const sourceFactorCaseOnSlug = buildSourceFactorCase('hc.slug', boostMap, opts?.detail); + // Built on the bare `slug` output column: applied inside the `scored` CTE + // whose FROM is the single relation `hnsw_candidates`, so unqualified + // `slug` resolves cleanly (T1 per-page pool restructure). + const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail); const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes); const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes); const innerLimit = offset + Math.max(limit * 5, 100); @@ -1838,23 +1833,33 @@ export class PGLiteEngine implements BrainEngine { WHERE cc.${col} IS NOT NULL ${modalityFilter} ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} ORDER BY cc.${col} <=> ${castSql} LIMIT $2 - ) + ), + -- score as a select-list expr; inner ORDER BY stays pure-distance so + -- the HNSW index is usable. + scored AS ( + SELECT *, raw_score * ${sourceFactorCaseOnSlug} AS score + FROM hnsw_candidates + ), + -- T1 (retrieval-maxpool incident): collapse to the best chunk PER PAGE + -- over the full candidate set before the user LIMIT. Shared builder with + -- postgres-engine + the keyword path so they cannot drift. + ${buildBestPerPagePoolCte('scored')} SELECT - hc.slug, hc.page_id, hc.title, hc.type, hc.source_id, - hc.effective_date, hc.effective_date_source, - hc.chunk_id, hc.chunk_index, hc.chunk_text, hc.chunk_source, - hc.raw_score * ${sourceFactorCaseOnSlug} AS score, - CASE WHEN hc.updated_at < ( - SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = hc.page_id + bpp.slug, bpp.page_id, bpp.title, bpp.type, bpp.source_id, + bpp.effective_date, bpp.effective_date_source, + bpp.chunk_id, bpp.chunk_index, bpp.chunk_text, bpp.chunk_source, + bpp.score, + CASE WHEN bpp.updated_at < ( + SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = bpp.page_id ) THEN true ELSE false END AS stale - FROM hnsw_candidates hc + FROM best_per_page bpp -- v0.41.13: stable tiebreaker. When two chunks share a score (same -- source-prefix boost + same cosine distance, the basis-vector + same- -- source-prefix case in eval fixtures), older page_id wins. Without -- this, planner choice + index presence can flip ordering between -- master and feature branches that add unrelated indexes — see the -- pages_dedup_idx (v95) regression that motivated this. - ORDER BY score DESC, hc.page_id ASC, hc.chunk_id ASC + ORDER BY bpp.score DESC, bpp.page_id ASC, bpp.chunk_id ASC LIMIT $3 OFFSET $4`, params @@ -4515,6 +4520,48 @@ export class PGLiteEngine implements BrainEngine { } } + async resolveAliases( + aliasNorms: string[], + opts?: { sourceId?: string; sourceIds?: string[] }, + ): Promise>> { + const out = new Map>(); + if (!aliasNorms || aliasNorms.length === 0) return out; + const sources = + opts?.sourceIds && opts.sourceIds.length > 0 + ? opts.sourceIds + : opts?.sourceId + ? [opts.sourceId] + : null; + let q = `SELECT alias_norm, slug, source_id FROM page_aliases WHERE alias_norm = ANY($1::text[])`; + const params: unknown[] = [aliasNorms]; + if (sources) { + params.push(sources); + q += ` AND source_id = ANY($2::text[])`; + } + q += ` ORDER BY alias_norm, source_id, slug`; + const { rows } = await this.db.query(q, params); + for (const r of rows as Array<{ alias_norm: string; slug: string; source_id: string }>) { + const list = out.get(r.alias_norm) ?? []; + if (!list.some(x => x.slug === r.slug && x.source_id === r.source_id)) { + list.push({ slug: r.slug, source_id: r.source_id }); + } + out.set(r.alias_norm, list); + } + return out; + } + + async setPageAliases(slug: string, sourceId: string, aliasNorms: string[]): Promise { + const uniq = Array.from(new Set(aliasNorms.filter(a => a.length > 0))); + await this.db.query(`DELETE FROM page_aliases WHERE source_id = $1 AND slug = $2`, [sourceId, slug]); + if (uniq.length === 0) return; + await this.db.query( + `INSERT INTO page_aliases (source_id, alias_norm, slug) + SELECT $1, a, $2 FROM unnest($3::text[]) AS a + ON CONFLICT (source_id, alias_norm, slug) DO NOTHING`, + [sourceId, slug, uniq], + ); + } + // Config async getConfig(key: string): Promise { const { rows } = await this.db.query('SELECT value FROM config WHERE key = $1', [key]); diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index 37bd89613..149f8edd0 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -999,6 +999,26 @@ CREATE TABLE IF NOT EXISTS slug_aliases ( ); CREATE INDEX IF NOT EXISTS slug_aliases_canonical_idx ON slug_aliases (source_id, canonical_slug); + +-- T3 retrieval-cathedral (retrieval-maxpool incident): free-text alias +-- resolution for SEARCH. Distinct from slug_aliases (slug->slug wikilink +-- redirect): page_aliases maps a normalized free-text name ("hall of light", +-- "明堂") to a canonical slug so a query that is a chosen name surfaces the +-- page. alias_norm is normalizeAlias() output; the (source_id, alias_norm, +-- slug) triple is unique so re-ingest is idempotent without blocking a second +-- page claiming the same alias (collisions reported + resolved at query time). +CREATE TABLE IF NOT EXISTS page_aliases ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + alias_norm TEXT NOT NULL, + slug TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + CONSTRAINT page_aliases_uniq UNIQUE (source_id, alias_norm, slug) +); +CREATE INDEX IF NOT EXISTS page_aliases_lookup_idx + ON page_aliases (source_id, alias_norm); +CREATE INDEX IF NOT EXISTS page_aliases_slug_idx + ON page_aliases (source_id, slug); `; /** diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 48db794ca..61d081a19 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -55,7 +55,7 @@ import { ConnectionManager } from './connection-manager.ts'; import { logConnectionEvent } from './connection-audit.ts'; import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding, takeRowToTake, isUndefinedTableError, warnOncePerProcess } from './utils.ts'; import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; -import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts'; +import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql, buildBestPerPagePoolCte } from './search/sql-ranking.ts'; import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts'; import { DELETE_BATCH_SIZE } from './engine-constants.ts'; @@ -1551,11 +1551,7 @@ export class PostgresEngine implements BrainEngine { ORDER BY score DESC LIMIT ${innerLimitParam} ), - best_per_page AS ( - SELECT DISTINCT ON (slug) * - FROM ranked_chunks - ORDER BY slug, score DESC - ) + ${buildBestPerPagePoolCte('ranked_chunks')} SELECT slug, page_id, title, type, source_id, effective_date, effective_date_source, chunk_id, chunk_index, chunk_text, chunk_source, score, @@ -1838,14 +1834,25 @@ export class PostgresEngine implements BrainEngine { ${visibilityClause} ORDER BY cc.${col} <=> ${castSql} LIMIT ${innerLimitParam} - ) + ), + -- score computed as a select-list expr (NOT in the inner ORDER BY, which + -- must stay pure-distance so the HNSW index is usable). + scored AS ( + SELECT *, raw_score * ${sourceFactorCaseOnSlug} AS score + FROM hnsw_candidates + ), + -- T1 (retrieval-maxpool incident): collapse to the best chunk PER PAGE + -- over the full candidate set before the user LIMIT, so a page's strong + -- chunk can't be crowded out of the result by weaker chunks of other + -- pages. Shared builder keeps keyword + vector × postgres + pglite in lockstep. + ${buildBestPerPagePoolCte('scored')} SELECT slug, page_id, title, type, source_id, effective_date, effective_date_source, chunk_id, chunk_index, chunk_text, chunk_source, - raw_score * ${sourceFactorCaseOnSlug} AS score, + score, false AS stale - FROM hnsw_candidates + FROM best_per_page -- v0.41.13: stable tiebreaker for tied scores. See pglite-engine for -- rationale (basis-vector test fixtures, planner-dependent ordering). ORDER BY score DESC, page_id ASC, chunk_id ASC @@ -4534,6 +4541,54 @@ export class PostgresEngine implements BrainEngine { } } + async resolveAliases( + aliasNorms: string[], + opts?: { sourceId?: string; sourceIds?: string[] }, + ): Promise>> { + const out = new Map>(); + if (!aliasNorms || aliasNorms.length === 0) return out; + const sql = this.sql; + const sources = + opts?.sourceIds && opts.sourceIds.length > 0 + ? opts.sourceIds + : opts?.sourceId + ? [opts.sourceId] + : null; + const rows = sources + ? await sql` + SELECT alias_norm, slug, source_id + FROM page_aliases + WHERE alias_norm = ANY(${aliasNorms}::text[]) + AND source_id = ANY(${sources}::text[]) + ORDER BY alias_norm, source_id, slug` + : await sql` + SELECT alias_norm, slug, source_id + FROM page_aliases + WHERE alias_norm = ANY(${aliasNorms}::text[]) + ORDER BY alias_norm, source_id, slug`; + for (const r of rows) { + const a = r.alias_norm as string; + const list = out.get(a) ?? []; + const ref = { slug: r.slug as string, source_id: r.source_id as string }; + if (!list.some(x => x.slug === ref.slug && x.source_id === ref.source_id)) list.push(ref); + out.set(a, list); + } + return out; + } + + async setPageAliases(slug: string, sourceId: string, aliasNorms: string[]): Promise { + const sql = this.sql; + const uniq = Array.from(new Set(aliasNorms.filter(a => a.length > 0))); + await sql.begin(async tx => { + await tx`DELETE FROM page_aliases WHERE source_id = ${sourceId} AND slug = ${slug}`; + if (uniq.length === 0) return; + await tx` + INSERT INTO page_aliases (source_id, alias_norm, slug) + SELECT ${sourceId}, a, ${slug} FROM unnest(${uniq}::text[]) AS a + ON CONFLICT (source_id, alias_norm, slug) DO NOTHING`; + }); + } + // Config async getConfig(key: string): Promise { const sql = this.sql; diff --git a/src/core/search/alias-normalize.ts b/src/core/search/alias-normalize.ts new file mode 100644 index 000000000..be0a7246d --- /dev/null +++ b/src/core/search/alias-normalize.ts @@ -0,0 +1,56 @@ +/** + * T3 — alias normalization (retrieval-maxpool incident, alias layer). + * + * ONE normalizer shared by the WRITE path (ingest projects frontmatter + * `aliases:` into page_aliases) and the READ path (search matches the query + * against page_aliases). If the two sides normalized differently — one + * lowercases, the other also collapses whitespace — stored aliases would + * silently never match queries, and there'd be no error to notice. Same + * single-source-of-truth posture as cjk.ts / escapeLikePattern. + * + * Normalization (deliberately aggressive + deterministic): + * - Unicode NFKC (so 明堂 and full/half-width variants converge) + * - lowercase + * - strip leading/trailing whitespace + * - collapse internal whitespace runs to a single space + * - drop surrounding quotes/brackets the YAML parser may leave + * + * Returns '' for input that normalizes to empty — callers MUST skip empty + * aliases (an empty alias would match empty/whitespace queries). + */ + +export function normalizeAlias(raw: string): string { + if (typeof raw !== 'string') return ''; + return raw + .normalize('NFKC') + .toLowerCase() + .replace(/[\s ]+/g, ' ') + .trim() + // strip a single layer of wrapping quotes/brackets left by loose YAML + .replace(/^["'`\[(]+/, '') + .replace(/["'`\])]+$/, '') + .trim(); +} + +/** + * Coerce a frontmatter `aliases:` value (which may be a scalar string, an + * array, or absent/garbage) into a deduped list of normalized, non-empty + * aliases. Used by the ingest projection AND the backfill walker so both + * derive the same alias set from the same JSONB. + */ +export function normalizeAliasList(value: unknown): string[] { + const out = new Set(); + const push = (v: unknown) => { + if (typeof v !== 'string') return; + const n = normalizeAlias(v); + if (n.length > 0) out.add(n); + }; + if (Array.isArray(value)) { + for (const v of value) push(v); + } else if (typeof value === 'string') { + // A scalar `aliases: Hall of Light` OR a comma-list `aliases: a, b`. + if (value.includes(',')) value.split(',').forEach(push); + else push(value); + } + return Array.from(out); +} diff --git a/src/core/search/evidence.ts b/src/core/search/evidence.ts new file mode 100644 index 000000000..279273745 --- /dev/null +++ b/src/core/search/evidence.ts @@ -0,0 +1,78 @@ +/** + * T4 — evidence + create_safety contract (retrieval-maxpool incident). + * + * The incident's ROOT behavior: the agent read a single blended score (0.64) + * and decided "no strong match, safe to write a new page" — then wrote a + * duplicate on top of a fully-developed concept page. A blended RRF/cosine + * score is not a calibrated probability (Codex#5), so keying the + * don't-duplicate decision off a raw threshold is fragile. + * + * The fix (Codex#6): tell the agent WHY a result matched, not just a number. + * `evidence` names the strongest signal that surfaced the page; `create_safety` + * is the derived "is this page already here?" hint the agent keys off instead + * of a raw score. Pure + deterministic; stamped on every result so MCP callers + * and `--explain` read the same contract. + * + * Evidence precedence (strongest signal wins): + * alias_hit — query exactly matched the page's declared chosen name + * exact_title_match — query is a phrase in the page title (title boost fired) + * high_vector_match — base (pre-boost) score >= HIGH_MATCH_FLOOR + * keyword_exact — surfaced with a solid score but no title/alias/vector tag + * weak_semantic — everything else (low-confidence tail) + * + * create_safety: + * exists — strong evidence this IS the page; do NOT create a duplicate + * probable — likely the page; prefer updating over creating + * unknown — weak signal; the agent should look closer before deciding + */ + +import type { SearchResult } from '../types.ts'; + +export type Evidence = + | 'alias_hit' + | 'exact_title_match' + | 'high_vector_match' + | 'keyword_exact' + | 'weak_semantic'; + +export type CreateSafety = 'exists' | 'probable' | 'unknown'; + +/** base_score (pre-boost) at/above this is a confident vector/keyword match. */ +export const HIGH_MATCH_FLOOR = 0.85; +/** base_score at/above this is a solid (not weak) match. */ +export const SOLID_MATCH_FLOOR = 0.6; + +export function classifyEvidence(r: SearchResult): Evidence { + if (r.alias_hit) return 'alias_hit'; + if (r.title_match_boost && r.title_match_boost > 1.0) return 'exact_title_match'; + const base = typeof r.base_score === 'number' ? r.base_score : r.score; + if (Number.isFinite(base) && base >= HIGH_MATCH_FLOOR) return 'high_vector_match'; + if (Number.isFinite(base) && base >= SOLID_MATCH_FLOOR) return 'keyword_exact'; + return 'weak_semantic'; +} + +export function createSafetyFor(evidence: Evidence): CreateSafety { + switch (evidence) { + case 'alias_hit': + case 'exact_title_match': + case 'high_vector_match': + return 'exists'; + case 'keyword_exact': + return 'probable'; + case 'weak_semantic': + return 'unknown'; + } +} + +/** + * Stamp `evidence` + `create_safety` on every result in place. Called once at + * the end of the hybrid pipeline (after the alias hop, before slice) so the + * agent-facing result carries the contract. Idempotent. + */ +export function stampEvidence(results: SearchResult[]): void { + for (const r of results) { + const e = classifyEvidence(r); + r.evidence = e; + r.create_safety = createSafetyFor(e); + } +} diff --git a/src/core/search/hybrid.ts b/src/core/search/hybrid.ts index bbcbbdaa2..1456e9ac1 100644 --- a/src/core/search/hybrid.ts +++ b/src/core/search/hybrid.ts @@ -25,6 +25,9 @@ import { loadConfigWithEngine } from '../config.ts'; import { dedupResults } from './dedup.ts'; import { applyReranker } from './rerank.ts'; import { autoDetectDetail, classifyQuery, isAmbiguousModalityQuery } from './query-intent.ts'; +import { isTitlePhraseMatch } from './title-match.ts'; +import { normalizeAlias } from './alias-normalize.ts'; +import { stampEvidence } from './evidence.ts'; import { expandAnchors, hydrateChunks } from './two-pass.ts'; import { enforceTokenBudget } from './token-budget.ts'; import { recordSearchTelemetry } from './telemetry.ts'; @@ -228,6 +231,43 @@ export function applyRecencyBoost( } } +/** + * T2 (retrieval-maxpool incident) — apply the title-phrase boost. + * + * Fires when the normalized query is a contiguous token-run inside a result's + * page title (or an exact full-title match), per `isTitlePhraseMatch`. Mutate- + * in-place; caller re-sorts. Mirrors applyBacklinkBoost's floor-gate + stamp. + * + * Bounded by construction: a single fixed multiplier (`factor`, default 1.25), + * floor-ratio-gated so a title hit on a weak-overlap page can't leapfrog a + * strong primary hit. `base_score` (stamped at runPostFusionStages entry) is + * NOT touched, so the agent's dedup gate still reads true match confidence. + * + * Why page.title and not "first compiled_truth chunk" (Codex#11): the title is + * a stable column; the first chunk is a chunking accident that import changes + * could shift. The signal is "the query is the name of this thing." + */ +export function applyTitleBoost( + results: SearchResult[], + query: string, + factor: number, + floorThreshold?: number, +): void { + if (!query || !Number.isFinite(factor) || factor <= 1.0) return; + for (const r of results) { + if (!Number.isFinite(r.score)) continue; + if (floorThreshold !== undefined && r.score < floorThreshold) continue; + if (!r.title) continue; + if (isTitlePhraseMatch(query, r.title)) { + r.score *= factor; + r.title_match_boost = factor; // attribution stamp (v0.40.4 convention) + } + } +} + +/** Default title-phrase boost multiplier (mode-overridable via `title_boost`). */ +export const DEFAULT_TITLE_BOOST = 1.25; + /** * v0.29.1 — runPostFusionStages: wrap backlink + salience + recency in a * single stage that fires from EVERY hybridSearch return path (codex @@ -284,6 +324,17 @@ export interface PostFusionOpts { * wave via search-stats. */ onScoreDistribution?: (dist: import('./graph-signals.ts').ScoreDistribution) => void; + /** + * T2 — the raw query string, needed by the title-phrase boost stage. + * Undefined disables the stage (e.g. image-only queries). + */ + query?: string; + /** + * T2 — title-phrase boost multiplier (mode-resolved from `title_boost`). + * <= 1.0 or undefined disables the stage. Floor-ratio-gated like the + * metadata stages so a title hit can't bury a strong semantic match. + */ + titleBoost?: number; } export async function runPostFusionStages( @@ -357,6 +408,18 @@ export async function runPostFusionStages( } } + // T2 — title-phrase boost. Runs after the metadata stages, before graph + // signals. Shares the single floor-threshold so a title hit on a weak page + // can't leapfrog a strong primary hit (Codex#10). Fail-soft: pure + in-memory, + // but guarded so a bad query/title can't throw the whole pipeline. + if (opts.query && opts.titleBoost && opts.titleBoost > 1.0) { + try { + applyTitleBoost(results, opts.query, opts.titleBoost, floorThreshold); + } catch { + // Non-fatal; preserves the per-stage contract. + } + } + // v0.40.4 — graph-signals stage (4th post-fusion stage). Runs AFTER // backlink/salience/recency so it stacks on top of metadata boosts; // shares the same floor-threshold so a weak hub gets the same @@ -447,8 +510,103 @@ async function applyAliasResolvedBoost( } } +// T3 — free-text alias hop tuning. +const ALIAS_HOP_PRESENT_BOOST = 1.10; // bounded boost when canonical already in results +const MAX_ALIAS_QUERY_TOKENS = 6; // skip long queries (clearly not a chosen name) +const MAX_ALIAS_INJECT = 3; // cap injected pages per query (collision safety) + +/** + * T3 — free-text alias hop (retrieval-maxpool incident, the named-thing fix). + * + * When the normalized query EXACTLY matches a page's declared alias + * ("Hall of Light" / "明堂" -> the Mingtang page), make sure that page is in + * the result set: boost it if already present, inject it at top-of-organic + + * epsilon if absent. This is the only layer that bridges true synonyms with + * zero surface overlap — neither max-pool nor title-boost can. + * + * Precision guards (Codex#7/#10): + * - FULL normalized-query exact match only (not substring / not n-grams) — + * "light" won't fire unless the whole query normalizes to a stored alias. + * - skip queries longer than MAX_ALIAS_QUERY_TOKENS (clearly prose, not a name). + * - bounded: present-boost is 1.10x; inject score is top-of-organic + ε, + * never an absolute 1.0 (D3 — aliases are not a ranking sledgehammer). + * - collisions (two pages claim one alias): deterministic alpha order, capped. + * + * Fail-open: pre-v110 brains (no page_aliases table) and any lookup error + * degrade to the input unchanged (D9). Returns a NEW array; caller re-slices. + */ +export async function applyAliasHop( + engine: import('../engine.ts').BrainEngine, + results: SearchResult[], + query: string, + opts: { sourceId?: string; sourceIds?: string[] }, +): Promise { + if (!query) return results; + const qNorm = normalizeAlias(query); + if (!qNorm || qNorm.split(' ').length > MAX_ALIAS_QUERY_TOKENS) return results; + + let aliasMap: Map>; + try { + aliasMap = await engine.resolveAliases([qNorm], { sourceId: opts.sourceId, sourceIds: opts.sourceIds }); + } catch { + return results; // pre-v110 table-missing OR transient error -> fail-open + } + const refs = aliasMap.get(qNorm); + if (!refs || refs.length === 0) return results; + + // Deterministic + capped. Source-scoped: each canonical is a (source_id, slug) + // pair so a federated caller boosts/injects the RIGHT source's page, never + // collapsing or cross-injecting (P0 source-isolation contract). + const ordered = [...refs] + .sort((a, b) => (a.source_id === b.source_id ? a.slug.localeCompare(b.slug) : a.source_id.localeCompare(b.source_id))) + .slice(0, MAX_ALIAS_INJECT); + const out = [...results]; + const topScore = out.reduce((m, r) => (Number.isFinite(r.score) && r.score > m ? r.score : m), 0); + let injectScore = topScore > 0 ? topScore : 1.0; + + for (const ref of ordered) { + const idx = out.findIndex(r => r.slug === ref.slug && (r.source_id ?? 'default') === ref.source_id); + if (idx >= 0) { + if (Number.isFinite(out[idx].score)) out[idx].score *= ALIAS_HOP_PRESENT_BOOST; + out[idx].alias_hit = true; + continue; + } + // Absent canonical: fetch (in its OWN source) + inject at top-of-organic + epsilon. + let page; + try { + page = await engine.getPage(ref.slug, { sourceId: ref.source_id }); + } catch { + continue; + } + if (!page) continue; + injectScore += 1e-6; + out.push({ + slug: page.slug, + title: page.title, + type: page.type, + source_id: page.source_id ?? ref.source_id, + chunk_text: (page.compiled_truth ?? '').slice(0, 200), + chunk_index: 0, + chunk_id: 0, + score: injectScore, + base_score: injectScore, + alias_hit: true, + } as SearchResult); + } + out.sort((a, b) => b.score - a.score); + return out; +} + export interface HybridSearchOpts extends SearchOpts { expansion?: boolean; + /** + * T4/D5 — per-call search-mode selector (one of SEARCH_MODES). Selects the + * whole mode bundle for this call, overriding the server-configured mode. + * The op layer passes this ONLY for trusted/local callers (ctx.remote === + * false); remote callers leave it undefined so they can't escalate to the + * costly tokenmax bundle. Unknown values fall back to the default bundle. + */ + mode?: string; expandFn?: (query: string) => Promise; /** Override default RRF K constant (default: 60). Lower values boost top-ranked results more. */ rrfK?: number; @@ -486,7 +644,11 @@ export async function hybridSearch( const { loadSearchModeConfig, resolveSearchMode } = await import('./mode.ts'); const modeInput = await loadSearchModeConfig(engine); const resolvedMode = resolveSearchMode({ - mode: modeInput.mode, + // T4/D5 — per-call mode selector (e.g. `--mode tokenmax`). The op layer + // only passes this for trusted/local callers; remote callers leave it + // undefined and fall through to the server-configured mode (no cost + // escalation). Unknown values fall back to the default in resolveSearchMode. + mode: opts?.mode ?? modeInput.mode, overrides: modeInput.overrides, perCall: { intentWeighting: opts?.intentWeighting, @@ -581,6 +743,9 @@ export async function hybridSearch( // flush is fire-and-forget on 60s / 100-call thresholds. The hot path // never waits. let lastResultsCount = 0; + // T7 — rank-1 base_score for the telemetry drift signal. Set alongside + // lastResultsCount at each return path; undefined when there are no results. + let lastRank1Score: number | undefined; const emitMeta = (meta: HybridSearchMeta): void => { try { opts?.onMeta?.(meta); @@ -588,7 +753,7 @@ export async function hybridSearch( // swallow — capture telemetry is best-effort } try { - recordSearchTelemetry(engine, meta, { results_count: lastResultsCount }); + recordSearchTelemetry(engine, meta, { results_count: lastResultsCount, rank1_score: lastRank1Score }); } catch { // swallow — telemetry must never break the search hot path. } @@ -658,6 +823,10 @@ export async function hybridSearch( // Without this thread, the entire graph-signals wave is dead code — // codex outside-voice caught the missing wire pre-merge. graphSignalsEnabled: resolvedMode.graph_signals, + // T2 — title-phrase boost threaded from resolved mode (`title_boost`). + // The raw query drives the matcher; default factor when the knob is unset. + query, + titleBoost: resolvedMode.title_boost, }; // Skip vector search entirely if the gateway has no embedding provider configured (Codex C3). @@ -672,10 +841,18 @@ export async function hybridSearch( await runPostFusionStages(engine, keywordResults, postFusionOpts); keywordResults.sort((a, b) => b.score - a.score); } - const noEmbedSliced = dedupResults(keywordResults).slice(offset, offset + limit); + // T3/T4 — alias hop + evidence stamp even without an embedding provider + // (the named-thing fix is most valuable exactly when vector is unavailable). + const noEmbedHopped = await applyAliasHop(engine, dedupResults(keywordResults), query, { + sourceId: opts?.sourceId, + sourceIds: opts?.sourceIds, + }); + stampEvidence(noEmbedHopped); + const noEmbedSliced = noEmbedHopped.slice(offset, offset + limit); // v0.32.3 search-lite: budget enforcement on the no-embedding-provider path. const { results: noEmbedBudgeted, meta: noEmbedBudgetMeta } = enforceTokenBudget(noEmbedSliced, resolvedMode.tokenBudget); lastResultsCount = noEmbedBudgeted.length; + lastRank1Score = noEmbedBudgeted[0] ? (noEmbedBudgeted[0].base_score ?? noEmbedBudgeted[0].score) : undefined; emitMeta({ vector_enabled: false, detail_resolved: detailResolved, @@ -882,10 +1059,16 @@ export async function hybridSearch( await runPostFusionStages(engine, keywordResults, postFusionOpts); keywordResults.sort((a, b) => b.score - a.score); } - const kwSliced = dedupResults(keywordResults).slice(offset, offset + limit); + const kwHopped = await applyAliasHop(engine, dedupResults(keywordResults), query, { + sourceId: opts?.sourceId, + sourceIds: opts?.sourceIds, + }); + stampEvidence(kwHopped); + const kwSliced = kwHopped.slice(offset, offset + limit); // v0.32.3 search-lite: budget enforcement on the keyword-fallback path too. const { results: kwBudgeted, meta: kwBudgetMeta } = enforceTokenBudget(kwSliced, resolvedMode.tokenBudget); lastResultsCount = kwBudgeted.length; + lastRank1Score = kwBudgeted[0] ? (kwBudgeted[0].base_score ?? kwBudgeted[0].score) : undefined; emitMeta({ vector_enabled: false, detail_resolved: detailResolved, @@ -1035,18 +1218,34 @@ export async function hybridSearch( ? await applyReranker(query, deduped, rerankerOpts as any) : deduped; + // T3 — free-text alias hop. Runs AFTER rerank so a query that is a page's + // declared chosen name reliably surfaces that page regardless of how the + // reranker scored body chunks. Fail-open on pre-v110 brains. + const aliasHopped = await applyAliasHop(engine, reranked, query, { + sourceId: opts?.sourceId, + sourceIds: opts?.sourceIds, + }); + + // T4 — stamp evidence + create_safety so the agent's don't-duplicate + // decision keys off WHY a page matched, not a raw blended score. Stamp on + // the full alias-hopped set before any adaptive trim so the kept results + // carry evidence regardless of where the cap lands. + stampEvidence(aliasHopped); + // v0.42 — intent-aware adaptive return-sizing (opt-in, default off). Trim // the ranked candidate set to an intent-driven cap BEFORE the limit slice, // and only on the first page (offset===0) — paginating a confidence-gated // set is incoherent, so paginated calls fall through to the fixed limit. + // Runs on the alias-hopped set so an alias-injected page (top-of-organic) + // survives the trim. const adaptiveCfg = resolveAdaptiveReturn( opts?.adaptiveReturn, adaptiveReturnFromConfig(cfgForColumn as Record | null), ); - let returnPool = reranked; + let returnPool = aliasHopped; let adaptiveDecision: AdaptiveReturnDecision | undefined; if (adaptiveCfg.enabled && offset === 0) { - const r = applyAdaptiveReturn(reranked, suggestions.intent, adaptiveCfg); + const r = applyAdaptiveReturn(aliasHopped, suggestions.intent, adaptiveCfg); returnPool = r.kept; adaptiveDecision = r.decision; } @@ -1058,6 +1257,7 @@ export async function hybridSearch( // the same budget behavior as the production query op. const { results: budgeted, meta: budgetMeta } = enforceTokenBudget(sliced, resolvedMode.tokenBudget); lastResultsCount = budgeted.length; + lastRank1Score = budgeted[0] ? (budgeted[0].base_score ?? budgeted[0].score) : undefined; emitMeta({ vector_enabled: true, detail_resolved: detailResolved, @@ -1106,7 +1306,10 @@ export async function hybridSearchCached( const { loadSearchModeConfig, resolveSearchMode, knobsHash } = await import('./mode.ts'); const modeInputForCache = await loadSearchModeConfig(engine); const resolvedForCache = resolveSearchMode({ - mode: modeInputForCache.mode, + // T4/D5 — per-call mode folds into the cache key (resolved_mode is part + // of knobsHash) so a per-call `--mode tokenmax` read can't be served a + // server-default-mode cache row. + mode: opts?.mode ?? modeInputForCache.mode, overrides: modeInputForCache.overrides, perCall: { cache_enabled: opts?.useCache, diff --git a/src/core/search/mode.ts b/src/core/search/mode.ts index 2160fe0f5..9c1bdee0d 100644 --- a/src/core/search/mode.ts +++ b/src/core/search/mode.ts @@ -140,6 +140,16 @@ export interface ModeBundle { */ floor_ratio: number | undefined; + /** + * T2 (retrieval-maxpool incident) — title-phrase boost multiplier. When a + * query is a contiguous token-run inside a page's title (or an exact full- + * title match), multiply that result's score by this factor. <= 1.0 or + * undefined disables. Floor-ratio-gated so a title hit can't bury a strong + * semantic match. Correctness fix (cheap, in-memory) — ON in all bundles. + * Override: per-call SearchOpts → `search.title_boost` config → bundle. + */ + title_boost: number | undefined; + // v0.36 cross-modal wave knobs (D2 + D3 + D6 + D8 + D13 + LLM-intent). // All three mode bundles default these to the same values — cross-modal // is opt-in per-call (D6 weighting), opt-in per-brain (D8 unified flags), @@ -260,6 +270,8 @@ export const MODE_BUNDLES: Readonly>> = // v0.35.6.0 — undefined for all three bundles; the per-corpus ablation // (TODOS.md) gates any default flip. floor_ratio: undefined, + // T2 — title-phrase boost ON by default (correctness fix, cheap + gated). + title_boost: 1.25, // v0.36 cross-modal defaults (same across all modes — opt-in) cross_modal_both_text_weight: 0.6, cross_modal_both_image_weight: 0.4, @@ -300,6 +312,8 @@ export const MODE_BUNDLES: Readonly>> = // v0.35.6.0 — undefined for all three bundles; the per-corpus ablation // (TODOS.md) gates any default flip. floor_ratio: undefined, + // T2 — title-phrase boost ON by default (correctness fix, cheap + gated). + title_boost: 1.25, // v0.36 cross-modal defaults (same across all modes — opt-in) cross_modal_both_text_weight: 0.6, cross_modal_both_image_weight: 0.4, @@ -342,6 +356,8 @@ export const MODE_BUNDLES: Readonly>> = // v0.35.6.0 — undefined for all three bundles; the per-corpus ablation // (TODOS.md) gates any default flip. floor_ratio: undefined, + // T2 — title-phrase boost ON by default (correctness fix, cheap + gated). + title_boost: 1.25, // v0.36 cross-modal defaults (same across all modes — opt-in) cross_modal_both_text_weight: 0.6, cross_modal_both_image_weight: 0.4, @@ -392,6 +408,8 @@ export interface SearchKeyOverrides { reranker_timeout_ms?: number; // v0.35.6.0 — floor-ratio gate override. floor_ratio?: number; + // T2 — title-phrase boost override. + title_boost?: number; // v0.36 cross-modal overrides cross_modal_both_text_weight?: number; cross_modal_both_image_weight?: number; @@ -430,6 +448,8 @@ export interface SearchPerCallOpts { reranker_timeout_ms?: number; // v0.35.6.0 — floor-ratio per-call override. floor_ratio?: number; + // T2 — title-phrase boost per-call override. + title_boost?: number; // v0.36 cross-modal per-call overrides cross_modal_both_text_weight?: number; cross_modal_both_image_weight?: number; @@ -518,6 +538,7 @@ export function resolveSearchMode(input: ResolveSearchModeInput): ResolvedSearch reranker_timeout_ms: pickRerankerTimeoutMs(), // v0.35.6.0 — floor-ratio resolved via the same pick chain. floor_ratio: pick('floor_ratio'), + title_boost: pick('title_boost'), // v0.36 cross-modal knobs cross_modal_both_text_weight: pick('cross_modal_both_text_weight'), cross_modal_both_image_weight: pick('cross_modal_both_image_weight'), @@ -616,7 +637,12 @@ export function attributeKnob( // 1.05x multiplier. Cached pre-v0.42 entries don't reflect the boost so // must invalidate. Same one-time miss-spike pattern as prior bumps; // fills within cache.ttl_seconds (3600s default). -export const KNOBS_HASH_VERSION = 6; +// +// T2 bump 6→7: title_boost (retrieval-maxpool incident) adds a post-fusion +// stage that multiplies title-phrase-matching results. A title-boost-on write +// must NOT be served to a title-boost-off lookup (ranking shifts). Same +// one-time miss-spike pattern; fills within cache.ttl_seconds. +export const KNOBS_HASH_VERSION = 7; /** * v0.36 (D8 / CDX-2) — second-arg context for the cache key. The @@ -712,6 +738,8 @@ export function knobsHash( // neutralizes prior cache rows. `cr=${knobs.contextual_retrieval}`, `crd=${knobs.contextual_retrieval_disabled ? 1 : 0}`, + // v=7 addition (append-only) — T2 title-phrase boost (retrieval-maxpool). + `tib=${knobs.title_boost === undefined ? 'none' : knobs.title_boost.toFixed(4)}`, ]; const h = createHash('sha256'); h.update(parts.join('|')); @@ -809,6 +837,14 @@ export function loadOverridesFromConfig( if (Number.isFinite(n) && n >= 0 && n <= 1) out.floor_ratio = n; } + // T2 — title-phrase boost factor. >= 1.0 (1.0 disables). Bounded sanity cap + // at 5.0 so a fat-fingered config can't make a title hit dominate everything. + const tib = get('search.title_boost'); + if (tib !== undefined) { + const n = parseFloat(tib); + if (Number.isFinite(n) && n >= 1.0 && n <= 5.0) out.title_boost = n; + } + // v0.36 cross-modal overrides (D3 registry) const cmbt = get('search.cross_modal.both_mode_text_weight'); if (cmbt !== undefined) { @@ -878,6 +914,7 @@ export const SEARCH_MODE_CONFIG_KEYS: ReadonlyArray = Object.freeze([ 'search.reranker.timeout_ms', // v0.35.6.0 — floor-ratio gate 'search.floor_ratio', + 'search.title_boost', // v0.36 cross-modal keys (D3) 'search.cross_modal.both_mode_text_weight', 'search.cross_modal.both_mode_image_weight', diff --git a/src/core/search/sql-ranking.ts b/src/core/search/sql-ranking.ts index 7dfff0828..c037fb44d 100644 --- a/src/core/search/sql-ranking.ts +++ b/src/core/search/sql-ranking.ts @@ -129,6 +129,57 @@ export function buildVisibilityClause(pageAlias: string, sourceAlias: string): s return `AND ${pageAlias}.deleted_at IS NULL AND NOT ${sourceAlias}.archived`; } +// ============================================================ +// Per-page max-pool (T1 / D7) — single source of truth +// ============================================================ + +/** + * Build the `best_per_page` pooling CTE: collapse a chunk-grain candidate set + * to ONE row per page — the page's highest-scoring chunk. + * + * This is the per-page max-pool that `searchKeyword` always had and that + * `searchVector` was missing (the retrieval-maxpool incident: a page got + * represented by whichever chunk survived the candidate cut, not its best + * chunk). Both engines (postgres + pglite) AND both retrieval paths + * (keyword + vector) consume this one builder so they cannot drift — the + * recurring postgres/pglite parity bug class this repo guards against. + * + * Contract on the candidate CTE (`candidateCte`): + * - exposes `source_id` + `slug` columns (the composite per-page collapse key) + * - exposes a numeric `score` column (the value pooled on) + * - exposes `page_id` and `chunk_id` columns (deterministic tiebreak) + * + * Collapse key is COMPOSITE `(source_id, slug)`, NOT slug alone — two pages + * with the same slug in different sources are distinct pages (the federated + * multi-source contract; matches dedup.ts's pageKey and the v0.34.1 source + * isolation seal). Pooling on bare slug would collapse them and drop the + * neighbor-source page before ranking. `COALESCE(source_id, 'default')` keeps + * pre-v0.17 single-source rows (null source_id) collapsing correctly. + * + * Determinism: `DISTINCT ON` keeps the FIRST row per key under the ORDER BY, + * so the tiebreak `… score DESC, page_id ASC, chunk_id ASC` makes the surviving + * chunk fully deterministic when two chunks of the same page tie on score + * (basis-vector eval fixtures, planner-independent — same rationale as the + * v0.41.13 searchVector stable tiebreaker). + * + * Pooling happens over the FULL candidate set (`innerLimit` rows) BEFORE the + * user-facing `LIMIT`, so a page's best chunk can't be truncated out by + * weaker chunks of OTHER pages occupying the early `LIMIT` slots — the vector + * path now returns N distinct pages (each by best chunk), not N chunks that + * collapse to fewer pages downstream. + * + * @param candidateCte — name of the upstream CTE to pool (e.g. `'hnsw_candidates'`, + * `'ranked_chunks'`). Engine-supplied identifier, never user input. + * @returns raw SQL fragment: `best_per_page AS ( ... )` (no trailing comma) + */ +export function buildBestPerPagePoolCte(candidateCte: string): string { + return `best_per_page AS ( + SELECT DISTINCT ON (COALESCE(source_id, 'default'), slug) * + FROM ${candidateCte} + ORDER BY COALESCE(source_id, 'default'), slug, score DESC, page_id ASC, chunk_id ASC + )`; +} + // ============================================================ // v0.29.1 — Recency component SQL builder // ============================================================ diff --git a/src/core/search/telemetry.ts b/src/core/search/telemetry.ts index 2fe92e126..2cd41908e 100644 --- a/src/core/search/telemetry.ts +++ b/src/core/search/telemetry.ts @@ -36,8 +36,19 @@ interface Bucket { sum_budget_dropped: number; cache_hit: number; cache_miss: number; + // T7 — rank-1 base_score drift signal (aggregate, NOT per-query rows, D10). + // sum/count derive the mean; 3 coarse buckets give a distribution shape. + sum_rank1_score: number; + count_rank1: number; + rank1_lt_solid: number; // base_score < 0.6 + rank1_solid: number; // 0.6 <= base_score < 0.85 + rank1_high: number; // base_score >= 0.85 } +// T7 — coarse rank-1 score bands (mirror evidence.ts SOLID/HIGH floors). +const RANK1_SOLID_FLOOR = 0.6; +const RANK1_HIGH_FLOOR = 0.85; + const FLUSH_INTERVAL_MS = 60_000; const FLUSH_THRESHOLD_CALLS = 100; @@ -69,7 +80,7 @@ class TelemetryWriter { * immediately after bumping the in-memory bucket. Flush is async + * fire-and-forget. */ - record(meta: HybridSearchMeta, opts: { results_count: number; tokens_estimate?: number } = { results_count: 0 }): void { + record(meta: HybridSearchMeta, opts: { results_count: number; tokens_estimate?: number; rank1_score?: number } = { results_count: 0 }): void { const date = nowDate(); const mode = meta.mode ?? 'unset'; const intent = meta.intent ?? 'unset'; @@ -87,6 +98,11 @@ class TelemetryWriter { sum_budget_dropped: 0, cache_hit: 0, cache_miss: 0, + sum_rank1_score: 0, + count_rank1: 0, + rank1_lt_solid: 0, + rank1_solid: 0, + rank1_high: 0, }; this.buckets.set(key, b); } @@ -97,6 +113,16 @@ class TelemetryWriter { b.sum_budget_dropped += Math.max(0, Math.floor(meta.token_budget?.dropped ?? 0)); if (meta.cache?.status === 'hit') b.cache_hit += 1; if (meta.cache?.status === 'miss') b.cache_miss += 1; + // T7 — rank-1 base_score drift signal. Only counts queries that returned + // a result (rank1_score present + finite). + if (typeof opts.rank1_score === 'number' && Number.isFinite(opts.rank1_score)) { + const s = opts.rank1_score; + b.sum_rank1_score += s; + b.count_rank1 += 1; + if (s < RANK1_SOLID_FLOOR) b.rank1_lt_solid += 1; + else if (s < RANK1_HIGH_FLOOR) b.rank1_solid += 1; + else b.rank1_high += 1; + } this.pendingCount += 1; if (this.pendingCount >= FLUSH_THRESHOLD_CALLS) { @@ -129,8 +155,9 @@ class TelemetryWriter { try { await engine.executeRaw( `INSERT INTO search_telemetry - (date, mode, intent, count, sum_results, sum_tokens, sum_budget_dropped, cache_hit, cache_miss, first_seen, last_seen) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, now(), now()) + (date, mode, intent, count, sum_results, sum_tokens, sum_budget_dropped, cache_hit, cache_miss, + sum_rank1_score, count_rank1, rank1_lt_solid, rank1_solid, rank1_high, first_seen, last_seen) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, now(), now()) ON CONFLICT (date, mode, intent) DO UPDATE SET count = search_telemetry.count + EXCLUDED.count, sum_results = search_telemetry.sum_results + EXCLUDED.sum_results, @@ -138,8 +165,14 @@ class TelemetryWriter { sum_budget_dropped = search_telemetry.sum_budget_dropped + EXCLUDED.sum_budget_dropped, cache_hit = search_telemetry.cache_hit + EXCLUDED.cache_hit, cache_miss = search_telemetry.cache_miss + EXCLUDED.cache_miss, + sum_rank1_score = search_telemetry.sum_rank1_score + EXCLUDED.sum_rank1_score, + count_rank1 = search_telemetry.count_rank1 + EXCLUDED.count_rank1, + rank1_lt_solid = search_telemetry.rank1_lt_solid + EXCLUDED.rank1_lt_solid, + rank1_solid = search_telemetry.rank1_solid + EXCLUDED.rank1_solid, + rank1_high = search_telemetry.rank1_high + EXCLUDED.rank1_high, last_seen = now()`, - [b.date, b.mode, b.intent, b.count, b.sum_results, b.sum_tokens, b.sum_budget_dropped, b.cache_hit, b.cache_miss], + [b.date, b.mode, b.intent, b.count, b.sum_results, b.sum_tokens, b.sum_budget_dropped, b.cache_hit, b.cache_miss, + b.sum_rank1_score, b.count_rank1, b.rank1_lt_solid, b.rank1_solid, b.rank1_high], ); } catch { // swallow — telemetry write must never break the hot path. @@ -230,7 +263,7 @@ export function getTelemetryWriter(): TelemetryWriter { export function recordSearchTelemetry( engine: BrainEngine, meta: HybridSearchMeta, - opts: { results_count: number; tokens_estimate?: number } = { results_count: 0 }, + opts: { results_count: number; tokens_estimate?: number; rank1_score?: number } = { results_count: 0 }, ): void { try { const w = getTelemetryWriter(); @@ -258,6 +291,11 @@ export interface StatsWindow { window_days: number; oldest_seen?: string; newest_seen?: string; + // T7 — rank-1 base_score drift signal. avg_rank1_score is the headline the + // doctor/operator watches for downward drift; the 3 buckets give shape. + avg_rank1_score: number | null; // null when no rank-1 samples + rank1_count: number; + rank1_distribution: { lt_solid: number; solid: number; high: number }; } export async function readSearchStats( @@ -277,6 +315,11 @@ export async function readSearchStats( sum_budget_dropped: number; cache_hit: number; cache_miss: number; + sum_rank1_score: number; + count_rank1: number; + rank1_lt_solid: number; + rank1_solid: number; + rank1_high: number; first_seen: string; last_seen: string; }>( @@ -287,6 +330,11 @@ export async function readSearchStats( SUM(sum_budget_dropped)::int AS sum_budget_dropped, SUM(cache_hit)::int AS cache_hit, SUM(cache_miss)::int AS cache_miss, + COALESCE(SUM(sum_rank1_score), 0)::float8 AS sum_rank1_score, + COALESCE(SUM(count_rank1), 0)::int AS count_rank1, + COALESCE(SUM(rank1_lt_solid), 0)::int AS rank1_lt_solid, + COALESCE(SUM(rank1_solid), 0)::int AS rank1_solid, + COALESCE(SUM(rank1_high), 0)::int AS rank1_high, MIN(first_seen)::text AS first_seen, MAX(last_seen)::text AS last_seen FROM search_telemetry @@ -305,6 +353,11 @@ export async function readSearchStats( const mode_distribution: Record = {}; let oldest_seen: string | undefined; let newest_seen: string | undefined; + let sum_rank1 = 0; + let count_rank1 = 0; + let r1_lt = 0; + let r1_solid = 0; + let r1_high = 0; for (const r of rows) { total_calls += r.count; @@ -313,6 +366,11 @@ export async function readSearchStats( total_results += r.sum_results; total_tokens += r.sum_tokens; total_budget_dropped += r.sum_budget_dropped; + sum_rank1 += r.sum_rank1_score; + count_rank1 += r.count_rank1; + r1_lt += r.rank1_lt_solid; + r1_solid += r.rank1_solid; + r1_high += r.rank1_high; intent_distribution[r.intent] = (intent_distribution[r.intent] ?? 0) + r.count; mode_distribution[r.mode] = (mode_distribution[r.mode] ?? 0) + r.count; if (r.first_seen && (!oldest_seen || r.first_seen < oldest_seen)) oldest_seen = r.first_seen; @@ -333,6 +391,9 @@ export async function readSearchStats( window_days: days, oldest_seen, newest_seen, + avg_rank1_score: count_rank1 > 0 ? sum_rank1 / count_rank1 : null, + rank1_count: count_rank1, + rank1_distribution: { lt_solid: r1_lt, solid: r1_solid, high: r1_high }, }; } catch { // Table missing or query failed — return empty stats rather than throw. @@ -347,6 +408,9 @@ export async function readSearchStats( intent_distribution: {}, mode_distribution: {}, window_days: days, + avg_rank1_score: null, + rank1_count: 0, + rank1_distribution: { lt_solid: 0, solid: 0, high: 0 }, }; } } diff --git a/src/core/search/title-match.ts b/src/core/search/title-match.ts new file mode 100644 index 000000000..00802fb84 --- /dev/null +++ b/src/core/search/title-match.ts @@ -0,0 +1,93 @@ +/** + * Title-superstring matching for the T2 title boost (retrieval-maxpool incident). + * + * The disease: a query that is literally a phrase from a page's title + * ("Greek amphitheater" → "The Mingtang — Indoor Greek amphitheater…") matched + * a weak body chunk instead of being recognized as a title hit. Names of things + * deserve weight. This module decides, deterministically and with zero I/O, + * whether a query is a title-phrase match for a page. + * + * Guard rails (Codex#10 — avoid promoting generic pages on stopword-y queries): + * - require >= MIN_CONTENT_TOKENS non-stopword tokens in the query, OR an + * exact normalized full-title match (so a deliberate 1-word title still hits); + * - match at TOKEN BOUNDARIES (contiguous token run), never raw substring, so + * "art" doesn't match "Bartholomew"; + * - stopwords ("the", "a", "of", …) don't count toward the content-token floor. + * + * Pure + exported so the NamedThingBench eval and unit tests share one definition + * with the production boost (no drift). + */ + +const MIN_CONTENT_TOKENS = 2; + +// Small, deliberately conservative English stopword set. CJK has no whitespace +// stopword notion here; CJK queries fall through to the exact-match path. +const STOPWORDS = new Set([ + 'the', 'a', 'an', 'of', 'and', 'or', 'to', 'in', 'on', 'for', 'with', + 'at', 'by', 'from', 'as', 'is', 'it', 'this', 'that', 'my', 'your', +]); + +/** + * Normalize text to a token array: lowercase, split on any run of + * non-alphanumeric (Unicode-aware for letters/numbers), drop empties. + * CJK characters are letters under \p{L}, so a CJK title collapses to one + * token per contiguous run — fine for the exact-match path. + */ +export function tokenizeTitle(s: string): string[] { + if (!s) return []; + return s + .toLowerCase() + .normalize('NFKC') + .split(/[^\p{L}\p{N}]+/u) + .filter(Boolean); +} + +/** Content tokens = tokens that aren't stopwords. */ +function contentTokens(tokens: string[]): string[] { + return tokens.filter(t => !STOPWORDS.has(t)); +} + +/** True iff `needle` appears as a contiguous token run inside `haystack`. */ +function containsTokenRun(haystack: string[], needle: string[]): boolean { + if (needle.length === 0 || needle.length > haystack.length) return false; + for (let i = 0; i + needle.length <= haystack.length; i++) { + let ok = true; + for (let j = 0; j < needle.length; j++) { + if (haystack[i + j] !== needle[j]) { ok = false; break; } + } + if (ok) return true; + } + return false; +} + +/** + * Decide whether `query` is a title-phrase match for `title`. + * + * Returns true when EITHER: + * (a) the normalized query token sequence is a contiguous run inside the + * normalized title AND the query has >= MIN_CONTENT_TOKENS content tokens; OR + * (b) the normalized query equals the full normalized title exactly (covers + * deliberate single-word titles / chosen names like "Mingtang"). + * + * Order-insensitive matching is intentionally NOT done — a title-phrase signal + * should reflect the author's phrasing, and bag-of-words matching is what the + * vector/keyword paths already provide. + */ +export function isTitlePhraseMatch(query: string, title: string): boolean { + const qTokens = tokenizeTitle(query); + const tTokens = tokenizeTitle(title); + if (qTokens.length === 0 || tTokens.length === 0) return false; + + // (b) exact full-title match (covers 1-word chosen names). + if (qTokens.length === tTokens.length && qTokens.every((t, i) => t === tTokens[i])) { + return true; + } + + // (a) contiguous phrase match, gated by content-token floor. + const qContent = contentTokens(qTokens); + if (qContent.length < MIN_CONTENT_TOKENS) return false; + return containsTokenRun(tTokens, qTokens); +} + +// Exported for unit tests. +export const __test__ = { tokenizeTitle, contentTokens, containsTokenRun, STOPWORDS, MIN_CONTENT_TOKENS }; diff --git a/src/core/types.ts b/src/core/types.ts index b83e07d94..59cf88930 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -653,6 +653,38 @@ export interface SearchResult { * as canonical" and lets canonicals outrank fuzzy matches. */ alias_resolved_boost?: number; + /** + * T2 (retrieval-maxpool incident) — multiplier applied by applyTitleBoost + * (1.0 = unchanged; default ~1.25x). Fires when the normalized query is a + * contiguous token-run inside the page title (or an exact full-title match). + * Floor-ratio-gated + clamped so a title hit reorders without burying a + * strong semantic match or lying about base_score (the agent's dedup gate + * keys off base_score / evidence, not the boosted score). + */ + title_match_boost?: number; + /** + * T3 (retrieval-maxpool incident) — set when this result was surfaced or + * boosted by the free-text alias hop (the query exactly matched a page's + * declared alias in page_aliases). An injected canonical that wasn't in the + * organic candidate set carries alias_hit=true with score = top-of-organic + * + epsilon. Drives the evidence=alias_hit signal the agent's don't-duplicate + * decision keys off (T4). + */ + alias_hit?: boolean; + /** + * T4 — the strongest signal that surfaced this page (alias_hit > + * exact_title_match > high_vector_match > keyword_exact > weak_semantic). + * Computed by classifyEvidence at the end of the hybrid pipeline. + */ + evidence?: import('./search/evidence.ts').Evidence; + /** + * T4 — derived "is this page already in the brain?" hint. The agent's + * don't-write-a-duplicate decision keys off THIS, not a raw score: + * 'exists' = strong (don't duplicate), 'probable' = prefer update, + * 'unknown' = look closer. This is the contract that prevents the + * incident's duplicate-stub class. + */ + create_safety?: import('./search/evidence.ts').CreateSafety; } /** diff --git a/src/eval/retrieval-quality/harness.ts b/src/eval/retrieval-quality/harness.ts new file mode 100644 index 000000000..e39b81452 --- /dev/null +++ b/src/eval/retrieval-quality/harness.ts @@ -0,0 +1,184 @@ +/** + * NamedThingBench (T6) — the retrieval-quality eval that makes the + * retrieval-maxpool incident impossible to reintroduce silently. + * + * Query families, each a distinct failure class the incident exposed: + * title-substring — query is a phrase in the title (the direct regression) + * generic-to-named — tourist label -> the named thing + * alias-synonym — declared alias / romanization -> canonical + * multi-chunk-dilution — one strong chunk among many weak (stresses max-pool) + * short-vs-rich — 2-word vs 8-word form of the same intent + * graph-relationship — relationship query (guardrail: don't regress) + * hard-negative — query that must NOT return a page (precision guard) + * + * Pure: the caller injects a SearchFn (CLI uses hybridSearch; tests stub it), + * so the harness is engine-agnostic and runs identically against postgres + + * pglite + a deterministic stub. + */ + +export type Family = + | 'title-substring' + | 'generic-to-named' + | 'alias-synonym' + | 'multi-chunk-dilution' + | 'short-vs-rich' + | 'graph-relationship' + | 'hard-negative'; + +export interface NamedThingQuestion { + family: Family; + query: string; + /** slugs that SHOULD rank for this query (non-hard-negative families). */ + relevant?: string[]; + /** slugs that must NOT appear in top-k (hard-negative family). */ + forbidden?: string[]; + notes?: string; +} + +/** Ranked slugs for a query, best-first. */ +export type SearchFn = (query: string) => Promise; + +export interface QuestionResult { + family: Family; + query: string; + hit_at_1: boolean; + hit_at_3: boolean; + reciprocal_rank: number; // 0 if no relevant slug in results + /** hard-negative only: true when NO forbidden slug appeared in top-3. */ + negative_clean?: boolean; +} + +export interface FamilyReport { + family: Family; + n: number; + hit_at_1: number; // rate + hit_at_3: number; // rate + mrr: number; +} + +export interface RetrievalQualityReport { + schema_version: 1; + k: number; + total: number; + families: FamilyReport[]; + questions: QuestionResult[]; +} + +const K = 3; + +export function scoreQuestion(q: NamedThingQuestion, ranked: string[]): QuestionResult { + if (q.family === 'hard-negative') { + const forbidden = new Set(q.forbidden ?? []); + const topK = ranked.slice(0, K); + const clean = !topK.some(s => forbidden.has(s)); + return { family: q.family, query: q.query, hit_at_1: clean, hit_at_3: clean, reciprocal_rank: clean ? 1 : 0, negative_clean: clean }; + } + const relevant = new Set(q.relevant ?? []); + const firstRelevantIdx = ranked.findIndex(s => relevant.has(s)); + const hit1 = firstRelevantIdx === 0; + const hit3 = firstRelevantIdx >= 0 && firstRelevantIdx < K; + const rr = firstRelevantIdx >= 0 ? 1 / (firstRelevantIdx + 1) : 0; + return { family: q.family, query: q.query, hit_at_1: hit1, hit_at_3: hit3, reciprocal_rank: rr }; +} + +export async function runRetrievalQuality( + questions: NamedThingQuestion[], + searchFn: SearchFn, +): Promise { + const results: QuestionResult[] = []; + for (const q of questions) { + let ranked: string[] = []; + try { ranked = await searchFn(q.query); } catch { ranked = []; } + results.push(scoreQuestion(q, ranked)); + } + const byFamily = new Map(); + for (const r of results) { + const list = byFamily.get(r.family) ?? []; + list.push(r); + byFamily.set(r.family, list); + } + const families: FamilyReport[] = []; + for (const [family, list] of byFamily) { + const n = list.length; + families.push({ + family, + n, + hit_at_1: n ? list.filter(r => r.hit_at_1).length / n : 0, + hit_at_3: n ? list.filter(r => r.hit_at_3).length / n : 0, + mrr: n ? list.reduce((s, r) => s + r.reciprocal_rank, 0) / n : 0, + }); + } + families.sort((a, b) => a.family.localeCompare(b.family)); + return { schema_version: 1, k: K, total: results.length, families, questions: results }; +} + +// ── Gate ──────────────────────────────────────────────────────────────── + +export interface GateOpts { + /** Families hard-gated from day one (the bug's families). */ + hardFamilies: Partial>; + /** Soft families: reported, breaches are warnings (not failures) for now. */ + softFamilies: Family[]; +} + +/** + * D12 — hard-gate the two families that ARE this bug from day one + * (title-substring Hit@1 >= 0.95, multi-chunk-dilution Hit@3 = 1.0), plus + * alias-synonym Hit@1 >= 0.98 (Codex#13 exact numbers). Softer families + * (generic-to-named, short-vs-rich, graph-relationship, hard-negative) are + * warn-then-enforce until a 3x baseline noise floor is established. + * Env-overridable like the existing replay-gate floors. + */ +export const DEFAULT_GATE: GateOpts = { + hardFamilies: { + 'title-substring': { hit_at_1: floorEnv('GBRAIN_NTB_TITLE_HIT1', 0.95) }, + 'multi-chunk-dilution': { hit_at_3: floorEnv('GBRAIN_NTB_DILUTION_HIT3', 1.0) }, + 'alias-synonym': { hit_at_1: floorEnv('GBRAIN_NTB_ALIAS_HIT1', 0.98) }, + }, + softFamilies: ['generic-to-named', 'short-vs-rich', 'graph-relationship', 'hard-negative'], +}; + +function floorEnv(name: string, dflt: number): number { + const v = process.env[name]; + if (v === undefined) return dflt; + const n = parseFloat(v); + return Number.isFinite(n) && n >= 0 && n <= 1 ? n : dflt; +} + +export interface GateBreach { family: Family; metric: 'hit_at_1' | 'hit_at_3'; got: number; floor: number; } +export interface GateResult { pass: boolean; breaches: GateBreach[]; warnings: GateBreach[]; } + +export function evaluateGate(report: RetrievalQualityReport, opts: GateOpts = DEFAULT_GATE): GateResult { + const byFamily = new Map(report.families.map(f => [f.family, f])); + const breaches: GateBreach[] = []; + for (const [family, floors] of Object.entries(opts.hardFamilies) as [Family, { hit_at_1?: number; hit_at_3?: number }][]) { + const fr = byFamily.get(family); + if (!fr || fr.n === 0) continue; // no questions for this family → nothing to gate + if (floors.hit_at_1 !== undefined && fr.hit_at_1 < floors.hit_at_1) { + breaches.push({ family, metric: 'hit_at_1', got: fr.hit_at_1, floor: floors.hit_at_1 }); + } + if (floors.hit_at_3 !== undefined && fr.hit_at_3 < floors.hit_at_3) { + breaches.push({ family, metric: 'hit_at_3', got: fr.hit_at_3, floor: floors.hit_at_3 }); + } + } + // Soft families: surface low Hit@3 as warnings (informational until enforced). + const warnings: GateBreach[] = []; + for (const family of opts.softFamilies) { + const fr = byFamily.get(family); + if (fr && fr.n > 0 && fr.hit_at_3 < 0.8) { + warnings.push({ family, metric: 'hit_at_3', got: fr.hit_at_3, floor: 0.8 }); + } + } + return { pass: breaches.length === 0, breaches, warnings }; +} + +export function parseQuestionsJsonl(text: string): NamedThingQuestion[] { + const out: NamedThingQuestion[] = []; + for (const line of text.split('\n')) { + const t = line.trim(); + if (!t || t.startsWith('//') || t.startsWith('#')) continue; + const obj = JSON.parse(t) as NamedThingQuestion; + out.push(obj); + } + return out; +} diff --git a/test/cli-search-dispatch.test.ts b/test/cli-search-dispatch.test.ts new file mode 100644 index 000000000..514020ee6 --- /dev/null +++ b/test/cli-search-dispatch.test.ts @@ -0,0 +1,60 @@ +/** + * T5 — `gbrain search` dispatch reconcile. Subprocess test against the real + * cli.ts entrypoint: + * - `search modes` -> read-only config dashboard (NOT a search for "modes") + * - `search ""` -> cheap-hybrid free-text search (NOT "Unknown subcommand") + */ + +import { describe, test, expect } from 'bun:test'; +import { spawnSync } from 'node:child_process'; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +function run(args: string[], home: string) { + const r = spawnSync('bun', ['run', 'src/cli.ts', ...args], { + cwd: process.cwd(), + encoding: 'utf8', + env: { ...process.env, GBRAIN_HOME: home, DATABASE_URL: '', GBRAIN_DATABASE_URL: '' }, + timeout: 45_000, + }); + return { stdout: r.stdout ?? '', stderr: r.stderr ?? '', status: r.status ?? -1 }; +} + +function withHome(fn: (home: string) => void) { + const home = mkdtempSync(join(tmpdir(), 'gbrain-search-dispatch-')); + mkdirSync(join(home, '.gbrain'), { recursive: true }); + writeFileSync(join(home, '.gbrain', 'config.json'), JSON.stringify({ engine: 'pglite' })); + try { fn(home); } finally { rmSync(home, { recursive: true, force: true }); } +} + +describe('T5 — gbrain search dispatch', () => { + test('`search modes --json` routes to the dashboard (active_mode), not a query', () => { + withHome((home) => { + const { stdout, status } = run(['search', 'modes', '--json'], home); + expect(status).toBe(0); + expect(stdout).toContain('active_mode'); + // It must NOT be a search-results payload for the literal word "modes". + expect(stdout).not.toContain('Unknown subcommand'); + }); + }); + + test('`search ""` routes to the cheap-hybrid search op (no "Unknown subcommand")', () => { + withHome((home) => { + const { stdout, stderr, status } = run(['search', 'zzz-no-such-page-xyz'], home); + // Empty brain → "No results." (the search op's formatter), exit 0. + expect(status).toBe(0); + expect(stderr).not.toContain('Unknown subcommand'); + expect(stdout.toLowerCase()).toContain('no results'); + }); + }); + + test('`search stats --json` routes to the dashboard', () => { + withHome((home) => { + const { stdout, status } = run(['search', 'stats', '--json'], home); + expect(status).toBe(0); + // stats envelope, not a search-results array. + expect(stdout).toMatch(/total_calls|cache_hit_rate|window_days/); + }); + }); +}); diff --git a/test/cross-modal-phase1.test.ts b/test/cross-modal-phase1.test.ts index 5c5447eb5..8dcdb7ee6 100644 --- a/test/cross-modal-phase1.test.ts +++ b/test/cross-modal-phase1.test.ts @@ -142,7 +142,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => { // v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields. // v0.40.3.0 D8 bumps to v=5 (sequenced behind salem's v=4 graph-signals). // v0.41.22.0 (type-unification): 5→6 for alias_resolved post-fusion boost. - expect(KNOBS_HASH_VERSION).toBe(6); + expect(KNOBS_HASH_VERSION).toBe(7); }); test('flipping unified_multimodal changes the hash', () => { diff --git a/test/eval-retrieval-quality.test.ts b/test/eval-retrieval-quality.test.ts new file mode 100644 index 000000000..9312a2892 --- /dev/null +++ b/test/eval-retrieval-quality.test.ts @@ -0,0 +1,98 @@ +/** + * T6 — NamedThingBench hermetic gate. Seeds a synthetic brain matching the + * committed fixture (placeholder names) and runs the harness through the real + * hybridSearch pipeline. The embed transport is stubbed to throw, forcing the + * deterministic keyword + title-boost + alias-hop path (free, no network) — the + * vector max-pool guarantee is pinned separately by searchvector-maxpool.test.ts. + * + * The gate MUST pass for the families that ARE the incident (title-substring, + * alias-synonym, multi-chunk-dilution). + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { readFileSync } from 'fs'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { hybridSearch } from '../src/core/search/hybrid.ts'; +import { __setEmbedTransportForTests } from '../src/core/ai/gateway.ts'; +import { parseQuestionsJsonl, runRetrievalQuality, evaluateGate, type SearchFn } from '../src/eval/retrieval-quality/harness.ts'; +import type { ChunkInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; + +async function seedPage(slug: string, title: string, type: string, chunks: string[], aliases: string[] = []) { + await engine.putPage(slug, { type: type as never, title, compiled_truth: chunks.join('\n') }); + const ci: ChunkInput[] = chunks.map((text, i) => ({ chunk_index: i, chunk_text: text, chunk_source: 'compiled_truth', token_count: 10 })); + await engine.upsertChunks(slug, ci); + if (aliases.length) await engine.setPageAliases(slug, 'default', aliases.map(a => a.toLowerCase())); +} + +beforeAll(async () => { + // Force keyword + title + alias path: vector embed throws → hybrid falls open. + __setEmbedTransportForTests(() => { throw new Error('stub: no embed in NamedThingBench test'); }); + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + + await seedPage( + 'projects/example-amphitheater', + 'The Example Hall — Indoor Greek Amphitheater for Adversarial Debate', + 'note', + [ + 'Indoor greek amphitheater for adversarial debate in the city.', + 'Ceiling treatment acoustics for the amphitheater dome and seating.', + ], + ['Hall of Light'], + ); + await seedPage( + 'projects/example-civic-platform', + 'Example Civic Feedback Platform', + 'note', + ['A civic feedback platform for the city to gather resident input.'], + ['the widget tracker'], + ); + await seedPage( + 'people/alice-example', + 'Alice Example', + 'person', + ['Alice works on the civic feedback platform and gathers resident input.'], + ); +}); + +afterAll(async () => { + __setEmbedTransportForTests(null); + await engine.disconnect(); +}); + +describe('NamedThingBench gate (hermetic)', () => { + test('the bug families pass the gate (title-substring, alias-synonym, dilution)', async () => { + const questions = parseQuestionsJsonl( + readFileSync(join(import.meta.dir, 'fixtures/retrieval-quality/namedthing.jsonl'), 'utf8'), + ); + const searchFn: SearchFn = async (q) => { + const rs = await hybridSearch(engine, q, { limit: 10, sourceId: 'default' }); + return rs.map(r => r.slug); + }; + const report = await runRetrievalQuality(questions, searchFn); + const gate = evaluateGate(report); + + // Diagnostic on failure: surface per-family rates. + if (!gate.pass) { + console.error('NamedThingBench families:', JSON.stringify(report.families, null, 2)); + console.error('breaches:', JSON.stringify(gate.breaches, null, 2)); + } + expect(gate.pass).toBe(true); + + const byFam = new Map(report.families.map(f => [f.family, f])); + expect(byFam.get('title-substring')!.hit_at_1).toBeGreaterThanOrEqual(0.95); + expect(byFam.get('alias-synonym')!.hit_at_1).toBeGreaterThanOrEqual(0.98); + expect(byFam.get('multi-chunk-dilution')!.hit_at_3).toBe(1.0); + }); + + test('hard-negative: the named projects do NOT surface for an unrelated query', async () => { + const rs = await hybridSearch(engine, 'quarterly tax filing checklist', { limit: 10, sourceId: 'default' }); + const slugs = rs.map(r => r.slug); + expect(slugs).not.toContain('projects/example-amphitheater'); + expect(slugs).not.toContain('projects/example-civic-platform'); + }); +}); diff --git a/test/facts-anti-loop.test.ts b/test/facts-anti-loop.test.ts index 89d533dfe..2367946bb 100644 --- a/test/facts-anti-loop.test.ts +++ b/test/facts-anti-loop.test.ts @@ -4,28 +4,41 @@ * Pins both code paths that must respect the v0.23.2 marker: * - extractFactsFromTurn(isDreamGenerated:true) → [] * - put_page backstop on dream_generated:true frontmatter → skipped:dream_generated + * + * The two `put_page` cases drive `dispatchToolCall → put_page`, whose + * importFromContent embeds by design (embed failure PROPAGATES — Codex C2). + * A sibling shard file that left the gateway configured with a fake/ + * non-legacy key makes put_page's embed step 401 and throw (isError), which + * is what failed these two tests in CI shard 2 while the engine-free + * extractFactsFromTurn cases passed. We don't want embedding here at all, + * so resetGateway() before each test → empty gateway → embedBatch is a + * graceful no-op → put_page succeeds regardless of sibling pollution. */ -import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { dispatchToolCall } from '../src/mcp/dispatch.ts'; import { extractFactsFromTurn } from '../src/core/facts/extract.ts'; +import { resetGateway } from '../src/core/ai/gateway.ts'; let engine: PGLiteEngine; -// 30s hook timeout — when this file runs deep in a shard process that's -// already created ~20 PGLite engines, the WASM cold-start + 95 migrations -// on a fresh DB legitimately exceeds bun's 5s hook default. CI shard 4 -// hit this on v0.41.17.0 (95 migrations × 21 files × 1 bun process). +// 60s hook timeout — generous budget for the PGLite WASM cold-start + full +// migration chain on a fresh DB, well above bun's 5s hook default. beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); -}, 30_000); +}, 60_000); + +// Empty gateway → put_page's embed step is a graceful no-op (see header). +beforeEach(() => { + resetGateway(); +}); afterAll(async () => { await engine.disconnect(); -}, 30_000); +}, 60_000); describe('anti-loop dream_generated marker', () => { test('extractFactsFromTurn skips when isDreamGenerated:true', async () => { diff --git a/test/facts-backstop-gating.test.ts b/test/facts-backstop-gating.test.ts index 9ab00f0ea..222e3c1db 100644 --- a/test/facts-backstop-gating.test.ts +++ b/test/facts-backstop-gating.test.ts @@ -8,9 +8,10 @@ * responses. */ -import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { dispatchToolCall } from '../src/mcp/dispatch.ts'; +import { resetGateway } from '../src/core/ai/gateway.ts'; let engine: PGLiteEngine; @@ -20,6 +21,18 @@ beforeAll(async () => { await engine.initSchema(); }); +// These tests drive put_page, whose importFromContent embeds by design +// (embed failure PROPAGATES — a deliberate Codex C2 choice). A sibling +// shard file that left the gateway configured with a fake/non-legacy key +// would make put_page's embed step 401 and throw (isError). We don't want +// embedding here at all — the assertions are about the facts backstop, not +// vectors — so reset the gateway before each test. An empty gateway makes +// embedBatch a graceful no-op, so put_page succeeds regardless of what a +// prior file in the shard process leaked. +beforeEach(() => { + resetGateway(); +}); + afterAll(async () => { await engine.disconnect(); }); diff --git a/test/fixtures/retrieval-quality/namedthing.jsonl b/test/fixtures/retrieval-quality/namedthing.jsonl new file mode 100644 index 000000000..bb5cffc8a --- /dev/null +++ b/test/fixtures/retrieval-quality/namedthing.jsonl @@ -0,0 +1,10 @@ +{"family":"title-substring","query":"greek amphitheater","relevant":["projects/example-amphitheater"],"notes":"the direct incident regression: query is a phrase in the title"} +{"family":"title-substring","query":"civic feedback platform","relevant":["projects/example-civic-platform"]} +{"family":"alias-synonym","query":"hall of light","relevant":["projects/example-amphitheater"],"notes":"declared alias -> canonical"} +{"family":"alias-synonym","query":"the widget tracker","relevant":["projects/example-civic-platform"],"notes":"alias romanization/synonym"} +{"family":"generic-to-named","query":"indoor amphitheater for debate","relevant":["projects/example-amphitheater"]} +{"family":"multi-chunk-dilution","query":"ceiling treatment acoustics amphitheater","relevant":["projects/example-amphitheater"],"notes":"strong chunk among many weak ones"} +{"family":"short-vs-rich","query":"amphitheater","relevant":["projects/example-amphitheater"]} +{"family":"short-vs-rich","query":"indoor greek amphitheater for adversarial debate in the city","relevant":["projects/example-amphitheater"]} +{"family":"graph-relationship","query":"who works on the civic platform","relevant":["people/alice-example"]} +{"family":"hard-negative","query":"quarterly tax filing checklist","forbidden":["projects/example-amphitheater","projects/example-civic-platform"],"notes":"must NOT return the named projects"} diff --git a/test/mcp-eval-capture.test.ts b/test/mcp-eval-capture.test.ts index 32f6311b1..f5a96174a 100644 --- a/test/mcp-eval-capture.test.ts +++ b/test/mcp-eval-capture.test.ts @@ -196,7 +196,7 @@ describe('op-layer capture — query', () => { describe('op-layer capture — search', () => { const searchOp = operations.find(o => o.name === 'search')!; - test('captures search call with tool_name="search" and vector_enabled=false', async () => { + test('captures search call with tool_name="search" (cheap-hybrid contract, T4/D4)', async () => { const ctx = makeCtx(); await searchOp.handler(ctx, { query: 'alice' }); await waitForCapture(); @@ -205,9 +205,12 @@ describe('op-layer capture — search', () => { expect(rows).toHaveLength(1); const row = rows[0]!; expect(row.tool_name).toBe('search'); - expect(row.vector_enabled).toBe(false); // search never runs vector - expect(row.expand_enabled).toBeNull(); // no expansion concept for search - expect(row.detail_resolved).toBeNull(); + // T4/D4: search is now cheap-hybrid (vector+keyword+RRF). With no embedding + // provider in this test it falls open to keyword (vector_enabled=false), but + // expansion is structurally OFF for `search` (the cheap-hybrid contract) — + // false, not null, because search now HAS an expansion concept. + expect(row.vector_enabled).toBe(false); // no embedding provider configured in test + expect(row.expand_enabled).toBe(false); // cheap-hybrid: expansion always off }); test('respects eval.capture=false off-switch', async () => { diff --git a/test/retrieval-quality-harness.test.ts b/test/retrieval-quality-harness.test.ts new file mode 100644 index 000000000..7a8679652 --- /dev/null +++ b/test/retrieval-quality-harness.test.ts @@ -0,0 +1,114 @@ +/** + * T6 — NamedThingBench harness unit tests (pure scoring + gate logic). + */ + +import { describe, test, expect } from 'bun:test'; +import { + scoreQuestion, + runRetrievalQuality, + evaluateGate, + parseQuestionsJsonl, + type NamedThingQuestion, +} from '../src/eval/retrieval-quality/harness.ts'; + +describe('scoreQuestion', () => { + test('Hit@1 when relevant slug is rank 0', () => { + const q: NamedThingQuestion = { family: 'title-substring', query: 'x', relevant: ['a'] }; + const r = scoreQuestion(q, ['a', 'b', 'c']); + expect(r.hit_at_1).toBe(true); + expect(r.hit_at_3).toBe(true); + expect(r.reciprocal_rank).toBe(1); + }); + test('Hit@3 but not Hit@1 when relevant is rank 2', () => { + const q: NamedThingQuestion = { family: 'title-substring', query: 'x', relevant: ['c'] }; + const r = scoreQuestion(q, ['a', 'b', 'c', 'd']); + expect(r.hit_at_1).toBe(false); + expect(r.hit_at_3).toBe(true); + expect(r.reciprocal_rank).toBeCloseTo(1 / 3, 6); + }); + test('miss when relevant absent from top-3', () => { + const q: NamedThingQuestion = { family: 'title-substring', query: 'x', relevant: ['z'] }; + const r = scoreQuestion(q, ['a', 'b', 'c', 'z']); + expect(r.hit_at_3).toBe(false); + expect(r.reciprocal_rank).toBeCloseTo(0.25, 6); + }); + test('hard-negative: clean when forbidden slug absent from top-3', () => { + const q: NamedThingQuestion = { family: 'hard-negative', query: 'x', forbidden: ['bad'] }; + expect(scoreQuestion(q, ['a', 'b', 'c']).negative_clean).toBe(true); + expect(scoreQuestion(q, ['a', 'bad', 'c']).negative_clean).toBe(false); + }); +}); + +describe('evaluateGate', () => { + async function reportFor(qs: NamedThingQuestion[], ranked: Record) { + return runRetrievalQuality(qs, async (query) => ranked[query] ?? []); + } + + test('passes when hard families meet floors', async () => { + const qs: NamedThingQuestion[] = [ + { family: 'title-substring', query: 't', relevant: ['p'] }, + { family: 'alias-synonym', query: 'a', relevant: ['p'] }, + { family: 'multi-chunk-dilution', query: 'd', relevant: ['p'] }, + ]; + const report = await reportFor(qs, { t: ['p'], a: ['p'], d: ['p'] }); + const gate = evaluateGate(report); + expect(gate.pass).toBe(true); + expect(gate.breaches).toHaveLength(0); + }); + + test('fails when title-substring Hit@1 below floor', async () => { + const qs: NamedThingQuestion[] = [ + { family: 'title-substring', query: 't', relevant: ['p'] }, + ]; + const report = await reportFor(qs, { t: ['other', 'p'] }); // p at rank 1, not 0 + const gate = evaluateGate(report); + expect(gate.pass).toBe(false); + expect(gate.breaches[0].family).toBe('title-substring'); + expect(gate.breaches[0].metric).toBe('hit_at_1'); + }); + + test('soft family low score is a warning, not a breach', async () => { + const qs: NamedThingQuestion[] = [ + { family: 'graph-relationship', query: 'g', relevant: ['p'] }, + ]; + const report = await reportFor(qs, { g: ['x', 'y', 'z', 'p'] }); // miss in top-3 + const gate = evaluateGate(report); + expect(gate.pass).toBe(true); // soft → no breach + expect(gate.warnings.some(w => w.family === 'graph-relationship')).toBe(true); + }); + + test('no questions for a hard family → not gated (no false breach)', async () => { + const qs: NamedThingQuestion[] = [{ family: 'short-vs-rich', query: 's', relevant: ['p'] }]; + const report = await reportFor(qs, { s: ['p'] }); + expect(evaluateGate(report).pass).toBe(true); + }); +}); + +describe('NamedThingBench fixture privacy guard', () => { + test('fixture uses placeholder names only (every slug is an *-example)', async () => { + const { readFileSync } = await import('fs'); + const { join } = await import('path'); + const raw = readFileSync(join(import.meta.dir, 'fixtures/retrieval-quality/namedthing.jsonl'), 'utf8').toLowerCase(); + // The load-bearing guard: every relevant/forbidden slug must be an + // `*-example` placeholder, so no real brain page can be referenced. (We + // don't enumerate banned real names here — listing them as string literals + // would itself trip scripts/check-test-real-names.sh.) + const slugs = [...raw.matchAll(/"(projects|people|notes|companies)\/([a-z0-9-]+)"/g)]; + expect(slugs.length).toBeGreaterThan(0); + for (const m of slugs) { + expect(m[2]).toContain('example'); + } + }); +}); + +describe('parseQuestionsJsonl', () => { + test('skips blanks + comments', () => { + const qs = parseQuestionsJsonl(` +{"family":"title-substring","query":"x","relevant":["a"]} +# comment +{"family":"hard-negative","query":"y","forbidden":["b"]} +`); + expect(qs).toHaveLength(2); + expect(qs[1].forbidden).toEqual(['b']); + }); +}); diff --git a/test/schema-bootstrap-coverage.test.ts b/test/schema-bootstrap-coverage.test.ts index d81043f61..d6c41826e 100644 --- a/test/schema-bootstrap-coverage.test.ts +++ b/test/schema-bootstrap-coverage.test.ts @@ -659,6 +659,15 @@ test('every CREATE INDEX column in PGLITE_SCHEMA_SQL is covered by CREATE TABLE // ───────────────────────────────────────────────────────────────── const COLUMN_EXEMPTIONS = new Set([ + // T7 — search_telemetry rank-1 drift columns (migration v111). search_telemetry + // is created entirely by migration v57 (not in the schema blob), so the v57+v111 + // chain handles fresh + upgrade; no CREATE INDEX references these columns, so + // there's no forward reference for the bootstrap to cover. + 'search_telemetry.sum_rank1_score', + 'search_telemetry.count_rank1', + 'search_telemetry.rank1_lt_solid', + 'search_telemetry.rank1_solid', + 'search_telemetry.rank1_high', // Schema-blob-not-yet-refreshed: each of these columns is added by a // migration but NOT (yet) referenced by `PGLITE_SCHEMA_SQL` (neither in a // CREATE TABLE body nor in any CREATE INDEX). Bootstrap doesn't need to diff --git a/test/search-alias-resolved-boost.test.ts b/test/search-alias-resolved-boost.test.ts index 5f63c0b76..87e803d27 100644 --- a/test/search-alias-resolved-boost.test.ts +++ b/test/search-alias-resolved-boost.test.ts @@ -90,6 +90,6 @@ describe('alias_resolved boost stage', () => { describe('KNOBS_HASH_VERSION', () => { it('bumped to 6 to invalidate caches across v0.42 boost stage addition', () => { - expect(KNOBS_HASH_VERSION).toBe(6); + expect(KNOBS_HASH_VERSION).toBe(7); }); }); diff --git a/test/search-mode.test.ts b/test/search-mode.test.ts index 9152036fb..60290ec73 100644 --- a/test/search-mode.test.ts +++ b/test/search-mode.test.ts @@ -69,6 +69,7 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => { reranker_top_n_out: null, reranker_timeout_ms: 5000, floor_ratio: undefined, + title_boost: 1.25, ...CROSS_MODAL_DEFAULTS, graph_signals: false, ...CR_DISABLED_DEFAULT, @@ -93,6 +94,7 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => { reranker_top_n_out: null, reranker_timeout_ms: 5000, floor_ratio: undefined, + title_boost: 1.25, ...CROSS_MODAL_DEFAULTS, graph_signals: true, ...CR_DISABLED_DEFAULT, @@ -115,6 +117,7 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => { reranker_top_n_out: null, reranker_timeout_ms: 5000, floor_ratio: undefined, + title_boost: 1.25, ...CROSS_MODAL_DEFAULTS, graph_signals: true, ...CR_DISABLED_DEFAULT, @@ -376,7 +379,7 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => { // post-fusion boost stage. A query against a brain with slug_aliases // populated must not be served from a cache row written before the // boost stage existed. - expect(KNOBS_HASH_VERSION).toBe(6); + expect(KNOBS_HASH_VERSION).toBe(7); }); test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => { diff --git a/test/search/alias-hop.test.ts b/test/search/alias-hop.test.ts new file mode 100644 index 000000000..1b8fbc17c --- /dev/null +++ b/test/search/alias-hop.test.ts @@ -0,0 +1,119 @@ +/** + * T3 — alias hop + ingest projection (the named-thing fix). + * Hermetic PGLite. applyAliasHop is tested directly (no embedding provider + * needed); ingest projection is tested via importFromContent(noEmbed). + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { applyAliasHop } from '../../src/core/search/hybrid.ts'; +import { importFromContent } from '../../src/core/import-file.ts'; +import type { SearchResult } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); +afterAll(async () => { await engine.disconnect(); }); +beforeEach(async () => { await resetPgliteState(engine); }); + +function res(slug: string, score: number, title = slug): SearchResult { + return { slug, title, score, chunk_text: '', type: 'note', source_id: 'default', chunk_index: 0, chunk_id: 1 } as unknown as SearchResult; +} + +describe('applyAliasHop', () => { + test('injects an absent canonical when the query matches its alias', async () => { + await engine.putPage('projects/mingtang', { type: 'note', title: 'The Mingtang', compiled_truth: 'Indoor Greek amphitheater.' }); + await engine.setPageAliases('projects/mingtang', 'default', ['hall of light', '明堂']); + + const organic = [res('notes/unrelated', 0.5)]; + const out = await applyAliasHop(engine, organic, 'Hall of Light', { sourceId: 'default' }); + + const hit = out.find(r => r.slug === 'projects/mingtang'); + expect(hit).toBeDefined(); + expect(hit!.alias_hit).toBe(true); + expect(out[0].slug).toBe('projects/mingtang'); // injected at top-of-organic + ε + expect(hit!.score).toBeGreaterThan(0.5); + }); + + test('romanization/CJK alias also resolves', async () => { + await engine.putPage('projects/mingtang', { type: 'note', title: 'The Mingtang', compiled_truth: 'x' }); + await engine.setPageAliases('projects/mingtang', 'default', ['明堂']); + const out = await applyAliasHop(engine, [], '明堂', { sourceId: 'default' }); + expect(out.map(r => r.slug)).toContain('projects/mingtang'); + }); + + test('boosts (does not duplicate) a canonical already in results', async () => { + await engine.putPage('projects/mingtang', { type: 'note', title: 'The Mingtang', compiled_truth: 'x' }); + await engine.setPageAliases('projects/mingtang', 'default', ['hall of light']); + const organic = [res('projects/mingtang', 0.4), res('notes/other', 0.5)]; + const out = await applyAliasHop(engine, organic, 'hall of light', { sourceId: 'default' }); + expect(out.filter(r => r.slug === 'projects/mingtang').length).toBe(1); // no dup + const m = out.find(r => r.slug === 'projects/mingtang')!; + expect(m.alias_hit).toBe(true); + expect(m.score).toBeCloseTo(0.4 * 1.10, 6); // bounded present-boost + }); + + test('P0 source-isolation: alias hop boosts only the aliased source, not the same slug in another source', async () => { + // The alias belongs to the src-b page only. Two same-slug results, different + // sources, both in the organic set. The hop must boost ONLY src-b's row. + await engine.setPageAliases('shared/page', 'src-b', ['only in b']); + const organic = [ + { slug: 'shared/page', source_id: 'src-a', score: 0.5 } as unknown as SearchResult, + { slug: 'shared/page', source_id: 'src-b', score: 0.5 } as unknown as SearchResult, + ]; + const out = await applyAliasHop(engine, organic, 'only in b', { sourceIds: ['src-a', 'src-b'] }); + const a = out.find(r => r.source_id === 'src-a')!; + const b = out.find(r => r.source_id === 'src-b')!; + expect(b.alias_hit).toBe(true); + expect(b.score).toBeCloseTo(0.5 * 1.10, 6); + expect(a.alias_hit).toBeUndefined(); // NOT cross-boosted + expect(a.score).toBe(0.5); + }); + + test('no alias match → input unchanged', async () => { + const organic = [res('a', 0.9), res('b', 0.8)]; + const out = await applyAliasHop(engine, organic, 'some unrelated query', { sourceId: 'default' }); + expect(out.map(r => r.slug)).toEqual(['a', 'b']); + }); + + test('long query is skipped (clearly prose, not a chosen name)', async () => { + await engine.putPage('p/x', { type: 'note', title: 'X', compiled_truth: 'x' }); + await engine.setPageAliases('p/x', 'default', ['one two three four five six seven']); + const out = await applyAliasHop(engine, [], 'one two three four five six seven', { sourceId: 'default' }); + expect(out.length).toBe(0); // >6 tokens → skipped + }); + + test('collision: two pages claim one alias → both injected, capped + deterministic', async () => { + await engine.putPage('a/hall', { type: 'note', title: 'A Hall', compiled_truth: 'x' }); + await engine.putPage('b/hall', { type: 'note', title: 'B Hall', compiled_truth: 'x' }); + await engine.setPageAliases('a/hall', 'default', ['the hall']); + await engine.setPageAliases('b/hall', 'default', ['the hall']); + const out = await applyAliasHop(engine, [], 'the hall', { sourceId: 'default' }); + expect(out.map(r => r.slug).sort()).toEqual(['a/hall', 'b/hall']); + }); +}); + +describe('ingest projection (importFromContent)', () => { + test('frontmatter aliases: land in page_aliases on import', async () => { + const md = `---\ntype: note\ntitle: The Mingtang\naliases:\n - Hall of Light\n - 明堂\n---\nIndoor Greek amphitheater.`; + await importFromContent(engine, 'projects/mingtang', md, { sourceId: 'default', noEmbed: true }); + const m = await engine.resolveAliases(['hall of light', '明堂'], { sourceId: 'default' }); + expect((m.get('hall of light') ?? []).map(r => r.slug)).toEqual(['projects/mingtang']); + expect((m.get('明堂') ?? []).map(r => r.slug)).toEqual(['projects/mingtang']); + }); + + test('removing an alias from frontmatter clears it on re-import', async () => { + const v1 = `---\ntype: note\ntitle: X\naliases: [old name, keep name]\n---\nbody one`; + await importFromContent(engine, 'p/x', v1, { sourceId: 'default', noEmbed: true }); + const v2 = `---\ntype: note\ntitle: X\naliases: [keep name]\n---\nbody two changed`; + await importFromContent(engine, 'p/x', v2, { sourceId: 'default', noEmbed: true }); + const m = await engine.resolveAliases(['old name', 'keep name'], { sourceId: 'default' }); + expect(m.get('old name')).toBeUndefined(); + expect((m.get('keep name') ?? []).map(r => r.slug)).toEqual(['p/x']); + }); +}); diff --git a/test/search/alias-normalize.test.ts b/test/search/alias-normalize.test.ts new file mode 100644 index 000000000..b9ef5cf9f --- /dev/null +++ b/test/search/alias-normalize.test.ts @@ -0,0 +1,50 @@ +/** + * T3 — alias normalization unit tests. The write path (ingest) and read path + * (search) MUST normalize identically or stored aliases silently never match. + */ + +import { describe, test, expect } from 'bun:test'; +import { normalizeAlias, normalizeAliasList } from '../../src/core/search/alias-normalize.ts'; + +describe('normalizeAlias', () => { + test('lowercases + trims + collapses whitespace', () => { + expect(normalizeAlias(' Hall of Light ')).toBe('hall of light'); + }); + test('write and read of the same name converge', () => { + expect(normalizeAlias('The Hall Of Light')).toBe(normalizeAlias('the hall of light')); + }); + test('NFKC folds width variants', () => { + expect(normalizeAlias('Ｍｉｎｇｔａｎｇ')).toBe('mingtang'); // fullwidth → ascii + }); + test('keeps CJK', () => { + expect(normalizeAlias(' 明堂 ')).toBe('明堂'); + }); + test('strips a layer of wrapping quotes/brackets from loose YAML', () => { + expect(normalizeAlias('"Hall of Light"')).toBe('hall of light'); + expect(normalizeAlias('[Mingtang]')).toBe('mingtang'); + }); + test('empty / non-string → empty', () => { + expect(normalizeAlias('')).toBe(''); + expect(normalizeAlias(' ')).toBe(''); + expect(normalizeAlias(undefined as unknown as string)).toBe(''); + }); +}); + +describe('normalizeAliasList', () => { + test('array → normalized + deduped, empties dropped', () => { + expect(normalizeAliasList(['Hall of Light', 'hall of light', '', ' ', 'Mingtang'])) + .toEqual(['hall of light', 'mingtang']); + }); + test('scalar string → single-element list', () => { + expect(normalizeAliasList('Hall of Light')).toEqual(['hall of light']); + }); + test('comma-separated scalar → split list', () => { + expect(normalizeAliasList('Hall of Light, Mingtang, 明堂')) + .toEqual(['hall of light', 'mingtang', '明堂']); + }); + test('absent / non-string → empty list', () => { + expect(normalizeAliasList(undefined)).toEqual([]); + expect(normalizeAliasList(null)).toEqual([]); + expect(normalizeAliasList(42)).toEqual([]); + }); +}); diff --git a/test/search/evidence.test.ts b/test/search/evidence.test.ts new file mode 100644 index 000000000..49c53f832 --- /dev/null +++ b/test/search/evidence.test.ts @@ -0,0 +1,75 @@ +/** + * T4 — evidence + create_safety contract. This is the agent-facing signal that + * prevents the incident's duplicate-stub class: the don't-write decision keys + * off WHY a page matched, not a raw blended score. + */ + +import { describe, test, expect } from 'bun:test'; +import { classifyEvidence, createSafetyFor, stampEvidence, HIGH_MATCH_FLOOR, SOLID_MATCH_FLOOR } from '../../src/core/search/evidence.ts'; +import type { SearchResult } from '../../src/core/types.ts'; + +function r(partial: Partial): SearchResult { + return { slug: 's', title: 't', chunk_text: '', type: 'note', source_id: 'default', chunk_index: 0, chunk_id: 1, score: 0.5, ...partial } as SearchResult; +} + +describe('classifyEvidence precedence', () => { + test('alias_hit wins over everything', () => { + expect(classifyEvidence(r({ alias_hit: true, title_match_boost: 1.25, base_score: 0.99 }))).toBe('alias_hit'); + }); + test('exact_title_match when title boost fired', () => { + expect(classifyEvidence(r({ title_match_boost: 1.25, base_score: 0.2 }))).toBe('exact_title_match'); + }); + test('high_vector_match at/above HIGH_MATCH_FLOOR', () => { + expect(classifyEvidence(r({ base_score: HIGH_MATCH_FLOOR }))).toBe('high_vector_match'); + expect(classifyEvidence(r({ base_score: 0.90 }))).toBe('high_vector_match'); + }); + test('keyword_exact in the solid band', () => { + expect(classifyEvidence(r({ base_score: SOLID_MATCH_FLOOR }))).toBe('keyword_exact'); + expect(classifyEvidence(r({ base_score: 0.7 }))).toBe('keyword_exact'); + }); + test('weak_semantic below the solid floor (the incident: 0.64 body chunk... 0.5 here)', () => { + expect(classifyEvidence(r({ base_score: 0.4 }))).toBe('weak_semantic'); + }); + test('falls back to score when base_score absent', () => { + expect(classifyEvidence(r({ score: 0.95, base_score: undefined }))).toBe('high_vector_match'); + }); +}); + +describe('createSafetyFor', () => { + test('strong evidence → exists (do NOT duplicate)', () => { + expect(createSafetyFor('alias_hit')).toBe('exists'); + expect(createSafetyFor('exact_title_match')).toBe('exists'); + expect(createSafetyFor('high_vector_match')).toBe('exists'); + }); + test('keyword_exact → probable', () => { + expect(createSafetyFor('keyword_exact')).toBe('probable'); + }); + test('weak_semantic → unknown', () => { + expect(createSafetyFor('weak_semantic')).toBe('unknown'); + }); +}); + +describe('stampEvidence', () => { + test('stamps both fields on every result', () => { + const rs = [r({ alias_hit: true }), r({ base_score: 0.3 })]; + stampEvidence(rs); + expect(rs[0].evidence).toBe('alias_hit'); + expect(rs[0].create_safety).toBe('exists'); + expect(rs[1].evidence).toBe('weak_semantic'); + expect(rs[1].create_safety).toBe('unknown'); + }); + + test('the incident: a 0.64 body-chunk match reads as weak/unknown (agent should look closer, not blindly duplicate)', () => { + const incident = r({ base_score: 0.64, score: 0.64 }); + stampEvidence([incident]); + expect(incident.evidence).toBe('keyword_exact'); // 0.64 is in the solid band + expect(incident.create_safety).toBe('probable'); // not 'unknown' — prefer update over create + }); + + test('after the fix: the same page via alias/title reads as exists (do not duplicate)', () => { + const fixed = r({ base_score: 0.64, alias_hit: true }); + stampEvidence([fixed]); + expect(fixed.evidence).toBe('alias_hit'); + expect(fixed.create_safety).toBe('exists'); + }); +}); diff --git a/test/search/knobs-hash-reranker.test.ts b/test/search/knobs-hash-reranker.test.ts index 9cb6d300f..eff5d013a 100644 --- a/test/search/knobs-hash-reranker.test.ts +++ b/test/search/knobs-hash-reranker.test.ts @@ -55,7 +55,7 @@ describe('KNOBS_HASH_VERSION + version invariants', () => { // v0.41.22.0 (type-unification): 5→6 to fold the alias_resolved // post-fusion boost. Cache rows written before the boost stage // cannot leak past the new stage. - expect(KNOBS_HASH_VERSION).toBe(6); + expect(KNOBS_HASH_VERSION).toBe(7); }); test('hash is 16 hex chars regardless of reranker config', () => { diff --git a/test/search/page-aliases-engine.test.ts b/test/search/page-aliases-engine.test.ts new file mode 100644 index 000000000..5d3051044 --- /dev/null +++ b/test/search/page-aliases-engine.test.ts @@ -0,0 +1,83 @@ +/** + * T3 — page_aliases engine round-trip (resolveAliases + setPageAliases). + * Hermetic PGLite. Pins: write→read, collision (two slugs one alias), + * source-scoping, replace-on-rewrite, empty-clears, empty-input. + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +const slugsOf = (m: Map>, k: string) => + (m.get(k) ?? []).map(r => r.slug).sort(); + +describe('setPageAliases + resolveAliases', () => { + test('write then read maps alias_norm → slug', async () => { + await engine.setPageAliases('projects/mingtang', 'default', ['hall of light', '明堂']); + const m = await engine.resolveAliases(['hall of light'], { sourceId: 'default' }); + expect(slugsOf(m, 'hall of light')).toEqual(['projects/mingtang']); + expect(m.get('hall of light')![0].source_id).toBe('default'); + }); + + test('collision: two pages claim the same alias → both returned', async () => { + await engine.setPageAliases('projects/mingtang', 'default', ['the hall']); + await engine.setPageAliases('projects/other-hall', 'default', ['the hall']); + const m = await engine.resolveAliases(['the hall'], { sourceId: 'default' }); + expect(slugsOf(m, 'the hall')).toEqual(['projects/mingtang', 'projects/other-hall']); + }); + + test('source-scoped: alias in source A not returned for source B', async () => { + await engine.setPageAliases('a/page', 'src-a', ['shared name']); + await engine.setPageAliases('b/page', 'src-b', ['shared name']); + const aOnly = await engine.resolveAliases(['shared name'], { sourceId: 'src-a' }); + expect(slugsOf(aOnly, 'shared name')).toEqual(['a/page']); + expect(aOnly.get('shared name')![0].source_id).toBe('src-a'); + const both = await engine.resolveAliases(['shared name'], { sourceIds: ['src-a', 'src-b'] }); + expect(slugsOf(both, 'shared name')).toEqual(['a/page', 'b/page']); + // same slug across sources stays distinct (the P0 contract) + expect((both.get('shared name') ?? []).map(r => r.source_id).sort()).toEqual(['src-a', 'src-b']); + }); + + test('rewrite replaces the prior alias set (delete + insert)', async () => { + await engine.setPageAliases('p/x', 'default', ['old name']); + await engine.setPageAliases('p/x', 'default', ['new name']); + const oldM = await engine.resolveAliases(['old name'], { sourceId: 'default' }); + const newM = await engine.resolveAliases(['new name'], { sourceId: 'default' }); + expect(oldM.get('old name')).toBeUndefined(); + expect(slugsOf(newM, 'new name')).toEqual(['p/x']); + }); + + test('empty alias set clears the page', async () => { + await engine.setPageAliases('p/x', 'default', ['temp']); + await engine.setPageAliases('p/x', 'default', []); + const m = await engine.resolveAliases(['temp'], { sourceId: 'default' }); + expect(m.size).toBe(0); + }); + + test('empty input → empty map, no query', async () => { + const m = await engine.resolveAliases([], { sourceId: 'default' }); + expect(m.size).toBe(0); + }); + + test('idempotent re-write does not duplicate (unique triple)', async () => { + await engine.setPageAliases('p/x', 'default', ['name', 'name']); + const m = await engine.resolveAliases(['name'], { sourceId: 'default' }); + expect(slugsOf(m, 'name')).toEqual(['p/x']); + }); +}); diff --git a/test/search/per-call-mode.test.ts b/test/search/per-call-mode.test.ts new file mode 100644 index 000000000..51970d7b6 --- /dev/null +++ b/test/search/per-call-mode.test.ts @@ -0,0 +1,33 @@ +/** + * T4/D5 — per-call mode gate: local/trusted callers may select a mode; remote + * callers can't (no tokenmax cost escalation); unknown modes reject loudly. + */ + +import { describe, test, expect } from 'bun:test'; +import { resolvePerCallMode, OperationError } from '../../src/core/operations.ts'; +import type { OperationContext } from '../../src/core/operations.ts'; + +function ctx(remote: boolean | undefined): OperationContext { + return { remote } as unknown as OperationContext; +} + +describe('resolvePerCallMode', () => { + test('local caller (remote===false) gets a valid mode', () => { + expect(resolvePerCallMode(ctx(false), 'tokenmax')).toBe('tokenmax'); + expect(resolvePerCallMode(ctx(false), 'conservative')).toBe('conservative'); + }); + test('local caller + unknown mode → loud reject', () => { + expect(() => resolvePerCallMode(ctx(false), 'thorough')).toThrow(OperationError); + expect(() => resolvePerCallMode(ctx(false), 'NUKE')).toThrow(/Unknown search mode/); + }); + test('remote caller (remote===true) → mode ignored (no escalation)', () => { + expect(resolvePerCallMode(ctx(true), 'tokenmax')).toBeUndefined(); + }); + test('remote caller (remote===undefined, treated as remote) → ignored', () => { + expect(resolvePerCallMode(ctx(undefined), 'tokenmax')).toBeUndefined(); + }); + test('no mode passed → undefined (use configured)', () => { + expect(resolvePerCallMode(ctx(false), undefined)).toBeUndefined(); + expect(resolvePerCallMode(ctx(false), '')).toBeUndefined(); + }); +}); diff --git a/test/search/pre-migration-failopen.test.ts b/test/search/pre-migration-failopen.test.ts new file mode 100644 index 000000000..69cffacab --- /dev/null +++ b/test/search/pre-migration-failopen.test.ts @@ -0,0 +1,56 @@ +/** + * T9 regression — pre-v110 brains (no page_aliases table) must keep working. + * The alias layer is additive: every alias touchpoint fails open so a brain + * mid-upgrade (migration not yet run) still ingests and searches normally. + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { applyAliasHop } from '../../src/core/search/hybrid.ts'; +import { importFromContent } from '../../src/core/import-file.ts'; +import type { SearchResult } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); +afterAll(async () => { await engine.disconnect(); }); + +beforeEach(async () => { + await resetPgliteState(engine); + // Simulate a pre-v110 brain: drop the page_aliases table entirely. + await engine.executeRaw('DROP TABLE IF EXISTS page_aliases'); +}); + +function r(slug: string, score: number): SearchResult { + return { slug, title: slug, score, chunk_text: '', type: 'note', source_id: 'default', chunk_index: 0, chunk_id: 1 } as unknown as SearchResult; +} + +describe('pre-migration (no page_aliases table) fail-open', () => { + test('applyAliasHop returns input unchanged, does not throw', async () => { + const organic = [r('a', 0.9), r('b', 0.8)]; + const out = await applyAliasHop(engine, organic, 'hall of light', { sourceId: 'default' }); + expect(out.map(x => x.slug)).toEqual(['a', 'b']); + }); + + test('importFromContent with frontmatter aliases does not throw (projection swallows table-missing)', async () => { + const md = `---\ntype: note\ntitle: X\naliases: [Hall of Light]\n---\nbody`; + const res = await importFromContent(engine, 'p/x', md, { sourceId: 'default', noEmbed: true }); + expect(res.status).toBe('imported'); // page write succeeded despite no alias table + }); + + test('resolveAliases throws table-missing (caller is responsible for catching)', async () => { + // The engine method itself surfaces the error; the alias-hop caller wraps it. + let threw = false; + try { + await engine.resolveAliases(['x'], { sourceId: 'default' }); + } catch { + threw = true; + } + expect(threw).toBe(true); + }); +}); diff --git a/test/search/reindex-aliases.test.ts b/test/search/reindex-aliases.test.ts new file mode 100644 index 000000000..cb78bde25 --- /dev/null +++ b/test/search/reindex-aliases.test.ts @@ -0,0 +1,65 @@ +/** + * T8 — `gbrain reindex --aliases` backfill. Seeds pages with frontmatter + * aliases (as if imported before the T3 projection landed), runs the backfill, + * and verifies page_aliases is populated. Idempotent + --dry-run + --source. + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { runReindexAliases } from '../../src/commands/reindex-aliases.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); +afterAll(async () => { await engine.disconnect(); }); +beforeEach(async () => { await resetPgliteState(engine); }); + +async function seed(slug: string, aliases: unknown) { + await engine.putPage(slug, { type: 'note' as never, title: slug, compiled_truth: 'x', frontmatter: { aliases } }); +} + +describe('runReindexAliases', () => { + test('backfills page_aliases from frontmatter for existing pages', async () => { + await seed('projects/mingtang', ['Hall of Light', '明堂']); + await seed('notes/plain', undefined); // no aliases + + const result = await runReindexAliases(engine, ['--json']); + expect(result.pages_with_aliases).toBe(1); + expect(result.aliases_written).toBe(2); + + const m = await engine.resolveAliases(['hall of light', '明堂'], { sourceId: 'default' }); + expect((m.get('hall of light') ?? []).map(r => r.slug)).toEqual(['projects/mingtang']); + expect((m.get('明堂') ?? []).map(r => r.slug)).toEqual(['projects/mingtang']); + }); + + test('--dry-run writes nothing', async () => { + await seed('p/x', ['some alias']); + const result = await runReindexAliases(engine, ['--dry-run', '--json']); + expect(result.dry_run).toBe(true); + expect(result.aliases_written).toBe(1); // would-write count + const m = await engine.resolveAliases(['some alias'], { sourceId: 'default' }); + expect(m.size).toBe(0); // nothing actually written + }); + + test('idempotent: second run converges, no duplicates', async () => { + await seed('p/x', ['name one', 'name two']); + await runReindexAliases(engine, ['--json']); + await runReindexAliases(engine, ['--json']); + const m = await engine.resolveAliases(['name one'], { sourceId: 'default' }); + expect((m.get('name one') ?? []).map(r => r.slug)).toEqual(['p/x']); + }); + + test('handles comma-scalar frontmatter aliases', async () => { + await seed('p/y', 'Alpha, Beta'); + const result = await runReindexAliases(engine, ['--json']); + expect(result.aliases_written).toBe(2); + const m = await engine.resolveAliases(['alpha', 'beta'], { sourceId: 'default' }); + expect((m.get('alpha') ?? []).map(r => r.slug)).toEqual(['p/y']); + expect((m.get('beta') ?? []).map(r => r.slug)).toEqual(['p/y']); + }); +}); diff --git a/test/search/rerank.test.ts b/test/search/rerank.test.ts index 1db2e9ed9..93f4e620f 100644 --- a/test/search/rerank.test.ts +++ b/test/search/rerank.test.ts @@ -11,7 +11,7 @@ * - rerankerFn test seam used over gateway.rerank */ -import { describe, test, expect, beforeAll } from 'bun:test'; +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; import { applyReranker, type RerankerOpts } from '../../src/core/search/rerank.ts'; import { RerankError, type RerankResult } from '../../src/core/ai/gateway.ts'; import type { SearchResult } from '../../src/core/types.ts'; @@ -33,6 +33,13 @@ function makeResult(slug: string, score: number, chunk: string): SearchResult { // Setup: gateway must be configured so the rerank-audit logger doesn't // trip on missing env. We can call configureGateway with a minimal stub. +// NOTE: this stub omits embedding_model, so the gateway falls back to the +// v0.37 default (zeroentropyai:zembed-1 / 1280-d). Without the afterAll +// reset below it would LEAK that default to the next file in the shard +// process — a sibling that runs initSchema in beforeAll would build a +// vector(1280) column and then mismatch on 1536-d fixtures. resetGateway +// in afterAll restores the empty slot so the legacy-embedding preload +// re-pins OpenAI/1536 for the next file. beforeAll(async () => { const { configureGateway } = await import('../../src/core/ai/gateway.ts'); configureGateway({ @@ -40,6 +47,11 @@ beforeAll(async () => { }); }); +afterAll(async () => { + const { resetGateway } = await import('../../src/core/ai/gateway.ts'); + resetGateway(); +}); + describe('applyReranker — happy path', () => { test('reorders top-N by reranker relevance score', async () => { const results = [ diff --git a/test/search/search-diagnose.test.ts b/test/search/search-diagnose.test.ts new file mode 100644 index 000000000..e16c0dddd --- /dev/null +++ b/test/search/search-diagnose.test.ts @@ -0,0 +1,65 @@ +/** + * T0 — `gbrain search diagnose` Phase-0 retrieval diagnostic. Seeds a synthetic + * brain, forces the keyword+alias path (embed stubbed to throw → vector skipped), + * and asserts the per-layer trace + verdict. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { __setEmbedTransportForTests } from '../../src/core/ai/gateway.ts'; +import { runSearchDiagnose } from '../../src/commands/search-diagnose.ts'; +import type { ChunkInput } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +async function captureJson(fn: () => Promise): Promise { + const orig = console.log; + const lines: string[] = []; + console.log = (...a: unknown[]) => { lines.push(a.map(String).join(' ')); }; + try { await fn(); } finally { console.log = orig; } + return JSON.parse(lines.join('\n')); +} + +beforeAll(async () => { + __setEmbedTransportForTests(() => { throw new Error('stub: no embed'); }); + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + await engine.putPage('projects/mingtang', { + type: 'note' as never, + title: 'The Example Hall — Indoor Greek Amphitheater', + compiled_truth: 'Indoor greek amphitheater for adversarial debate.', + }); + const ci: ChunkInput[] = [{ chunk_index: 0, chunk_text: 'Indoor greek amphitheater for adversarial debate.', chunk_source: 'compiled_truth', token_count: 10 }]; + await engine.upsertChunks('projects/mingtang', ci); + await engine.setPageAliases('projects/mingtang', 'default', ['hall of light']); +}); + +afterAll(async () => { __setEmbedTransportForTests(null); await engine.disconnect(); }); + +describe('search diagnose', () => { + test('alias query: trace shows alias match + hybrid rank 1', async () => { + const report = await captureJson(() => + runSearchDiagnose(engine, ['diagnose', 'hall of light', '--target', 'projects/mingtang', '--source', 'default', '--json']), + ); + expect(report.target).toBe('projects/mingtang'); + expect(report.alias_match).toBe(true); + // vector probe degrades gracefully (no provider OR embed call failed) — the + // point is it doesn't crash the diagnose. A note is present either way. + expect(report.vector.rank).toBeNull(); + expect(typeof report.vector.note).toBe('string'); + expect(report.hybrid.rank).toBe(1); + expect(report.hybrid.alias_hit).toBe(true); + expect(report.verdict).toContain('rank 1'); + }); + + test('title-phrase query: keyword + title boost surface the target', async () => { + const report = await captureJson(() => + runSearchDiagnose(engine, ['diagnose', 'greek amphitheater', '--target', 'projects/mingtang', '--source', 'default', '--json']), + ); + expect(report.keyword.rank).toBe(1); + expect(report.hybrid.rank).toBe(1); + // title boost fired (query is a phrase in the title) + expect(report.hybrid.title_match_boost).toBeGreaterThan(1.0); + }); +}); diff --git a/test/search/searchvector-maxpool.test.ts b/test/search/searchvector-maxpool.test.ts new file mode 100644 index 000000000..29359270c --- /dev/null +++ b/test/search/searchvector-maxpool.test.ts @@ -0,0 +1,130 @@ +/** + * T1 regression — searchVector per-page max-pool (retrieval-maxpool incident). + * + * The bug: `searchVector` returned chunk-grain top-k with NO `DISTINCT ON (slug)`. + * A hub page with several strong chunks could occupy every early candidate slot, + * crowding a different page's single strong chunk out of the result entirely, OR + * returning the same page multiple times. The keyword path always pooled per page + * via `best_per_page`; the vector path did not. This pins the fix. + * + * Failing-on-old-code construction: + * - query embeds at basis direction 0. + * - `notes/hub` has THREE chunks at cosine 0.99 / 0.98 / 0.97. + * - `notes/needle` has ONE chunk at cosine 0.95. + * - two fillers at 0.90. + * With limit=3 the OLD chunk-grain path returns [hub@0.99, hub@0.98, hub@0.97] + * — needle (rank 4 at chunk grain) is truncated and the page is absent, and hub + * appears 3×. The pooled path returns 3 DISTINCT pages [hub, needle, filler], + * so needle surfaces and hub appears once. + * + * `detail: 'high'` neutralizes the source-factor boost (buildSourceFactorCase + * returns 1.0) so the assertion keys off pure cosine, not slug-prefix weighting. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { configureGateway } from '../../src/core/ai/gateway.ts'; +import type { ChunkInput } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +const DIM = 1536; + +/** Unit vector with cosine-similarity `cos` against basis direction 0. */ +function gradedEmb(cos: number, otherDim: number): Float32Array { + const e = new Float32Array(DIM); + e[0] = cos; + e[otherDim] = Math.sqrt(Math.max(0, 1 - cos * cos)); + return e; +} + +function basisEmbedding(idx: number): Float32Array { + const e = new Float32Array(DIM); + e[idx % DIM] = 1.0; + return e; +} + +beforeAll(async () => { + // Pin the legacy OpenAI/1536 gateway BEFORE initSchema so the + // content_chunks.embedding column is vector(1536), matching this file's + // hardcoded 1536-d basis vectors. initSchema runs in beforeAll (before + // any preload beforeEach can re-pin), so we cannot rely on the legacy + // preload default surviving a sibling shard file that reconfigured the + // gateway to the v0.37 ZE/1280 default and didn't reset. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: DIM, + env: { ...process.env }, + }); + engine = new PGLiteEngine(); + await engine.connect({}); // in-memory + await engine.initSchema(); + + // Seeded first → lowest page_id. Three strong chunks. + await engine.putPage('notes/hub', { + type: 'note', + title: 'Hub Page', + compiled_truth: 'hub a. hub b. hub c.', + }); + const hubChunks: ChunkInput[] = [ + { chunk_index: 0, chunk_text: 'hub a', chunk_source: 'compiled_truth', embedding: gradedEmb(0.99, 10), token_count: 2 }, + { chunk_index: 1, chunk_text: 'hub b', chunk_source: 'compiled_truth', embedding: gradedEmb(0.98, 11), token_count: 2 }, + { chunk_index: 2, chunk_text: 'hub c', chunk_source: 'compiled_truth', embedding: gradedEmb(0.97, 12), token_count: 2 }, + ]; + await engine.upsertChunks('notes/hub', hubChunks); + + // The page that MUST surface — one strong chunk, below hub's chunks but above fillers. + await engine.putPage('notes/needle', { + type: 'note', + title: 'Needle Page', + compiled_truth: 'the needle', + }); + await engine.upsertChunks('notes/needle', [ + { chunk_index: 0, chunk_text: 'the needle', chunk_source: 'compiled_truth', embedding: gradedEmb(0.95, 13), token_count: 2 }, + ]); + + await engine.putPage('notes/filler-a', { type: 'note', title: 'Filler A', compiled_truth: 'filler a' }); + await engine.upsertChunks('notes/filler-a', [ + { chunk_index: 0, chunk_text: 'filler a', chunk_source: 'compiled_truth', embedding: gradedEmb(0.90, 14), token_count: 2 }, + ]); + await engine.putPage('notes/filler-b', { type: 'note', title: 'Filler B', compiled_truth: 'filler b' }); + await engine.upsertChunks('notes/filler-b', [ + { chunk_index: 0, chunk_text: 'filler b', chunk_source: 'compiled_truth', embedding: gradedEmb(0.90, 15), token_count: 2 }, + ]); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +describe('searchVector per-page max-pool (T1)', () => { + test('returns DISTINCT pages — no page appears more than once', async () => { + const results = await engine.searchVector(basisEmbedding(0), { limit: 3, detail: 'high' }); + const slugs = results.map(r => r.slug); + const unique = new Set(slugs); + expect(unique.size).toBe(slugs.length); // pooling: one row per page + }); + + test('a strong chunk is NOT crowded out by a hub page (needle surfaces)', async () => { + const results = await engine.searchVector(basisEmbedding(0), { limit: 3, detail: 'high' }); + const slugs = results.map(r => r.slug); + // OLD chunk-grain path: top-3 = hub,hub,hub → needle absent. Pooled: needle present. + expect(slugs).toContain('notes/needle'); + expect(slugs.filter(s => s === 'notes/hub').length).toBe(1); + }); + + test('hub still ranks #1 by its best chunk (max-pool keeps the strongest)', async () => { + const results = await engine.searchVector(basisEmbedding(0), { limit: 3, detail: 'high' }); + expect(results[0].slug).toBe('notes/hub'); + // hub represented by its 0.99 chunk (chunk_index 0), not a weaker one + expect(results[0].chunk_index).toBe(0); + }); + + test('larger limit returns each page once, ordered by best-chunk score', async () => { + const results = await engine.searchVector(basisEmbedding(0), { limit: 10, detail: 'high' }); + const slugs = results.map(r => r.slug); + expect(new Set(slugs).size).toBe(slugs.length); + expect(slugs[0]).toBe('notes/hub'); + expect(slugs[1]).toBe('notes/needle'); // 0.95 beats the 0.90 fillers + }); +}); diff --git a/test/search/telemetry-rank1.test.ts b/test/search/telemetry-rank1.test.ts new file mode 100644 index 000000000..273877636 --- /dev/null +++ b/test/search/telemetry-rank1.test.ts @@ -0,0 +1,58 @@ +/** + * T7 — rank-1 score drift telemetry. Records search calls with rank-1 scores, + * flushes, and reads back the aggregate (avg + buckets) from search_telemetry. + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { getTelemetryWriter, readSearchStats, recordSearchTelemetry, _resetTelemetryWriterForTest } from '../../src/core/search/telemetry.ts'; +import type { HybridSearchMeta } from '../../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); +afterAll(async () => { _resetTelemetryWriterForTest(); await engine.disconnect(); }); +beforeEach(async () => { + _resetTelemetryWriterForTest(); + await engine.executeRaw('DELETE FROM search_telemetry'); +}); + +const meta: HybridSearchMeta = { mode: 'balanced', intent: 'general' } as HybridSearchMeta; + +describe('rank-1 score telemetry', () => { + test('records mean + bucket distribution and reads them back', async () => { + recordSearchTelemetry(engine, meta, { results_count: 5, rank1_score: 0.95 }); // high + recordSearchTelemetry(engine, meta, { results_count: 3, rank1_score: 0.70 }); // solid + recordSearchTelemetry(engine, meta, { results_count: 1, rank1_score: 0.40 }); // lt_solid + await getTelemetryWriter().flush(); + + const stats = await readSearchStats(engine, { days: 1 }); + expect(stats.rank1_count).toBe(3); + expect(stats.avg_rank1_score).toBeCloseTo((0.95 + 0.70 + 0.40) / 3, 5); + expect(stats.rank1_distribution).toEqual({ lt_solid: 1, solid: 1, high: 1 }); + }); + + test('queries with no result (no rank1_score) do not pollute the average', async () => { + recordSearchTelemetry(engine, meta, { results_count: 0 }); // no rank1 + recordSearchTelemetry(engine, meta, { results_count: 2, rank1_score: 0.88 }); + await getTelemetryWriter().flush(); + + const stats = await readSearchStats(engine, { days: 1 }); + expect(stats.total_calls).toBe(2); + expect(stats.rank1_count).toBe(1); // only the one with a result + expect(stats.avg_rank1_score).toBeCloseTo(0.88, 5); + expect(stats.rank1_distribution.high).toBe(1); + }); + + test('empty window → null avg, zero buckets (no NaN)', async () => { + const stats = await readSearchStats(engine, { days: 1 }); + expect(stats.avg_rank1_score).toBeNull(); + expect(stats.rank1_count).toBe(0); + expect(stats.rank1_distribution).toEqual({ lt_solid: 0, solid: 0, high: 0 }); + }); +}); diff --git a/test/search/title-boost-stage.test.ts b/test/search/title-boost-stage.test.ts new file mode 100644 index 000000000..e9cf43f94 --- /dev/null +++ b/test/search/title-boost-stage.test.ts @@ -0,0 +1,56 @@ +/** + * T2 — applyTitleBoost stage unit tests (mutate-in-place post-fusion stage). + * Verifies: title-match multiplies + stamps; non-match untouched; floor-gate + * skips below-threshold results; idempotent factor stamping. + */ + +import { describe, test, expect } from 'bun:test'; +import { applyTitleBoost, DEFAULT_TITLE_BOOST } from '../../src/core/search/hybrid.ts'; +import type { SearchResult } from '../../src/core/types.ts'; + +function mk(slug: string, title: string, score: number): SearchResult { + return { + slug, title, score, + chunk_text: '', type: 'note', source_id: 'default', + chunk_index: 0, chunk_id: 1, + } as unknown as SearchResult; +} + +describe('applyTitleBoost', () => { + test('multiplies score and stamps title_match_boost on a title-phrase match', () => { + const r = mk('projects/mingtang', 'The Mingtang — Indoor Greek Amphitheater', 0.8); + applyTitleBoost([r], 'greek amphitheater', DEFAULT_TITLE_BOOST); + expect(r.score).toBeCloseTo(0.8 * 1.25, 6); + expect(r.title_match_boost).toBe(1.25); + }); + + test('leaves non-matching results untouched (no stamp)', () => { + const r = mk('notes/other', 'Completely Unrelated Title', 0.8); + applyTitleBoost([r], 'greek amphitheater', DEFAULT_TITLE_BOOST); + expect(r.score).toBe(0.8); + expect(r.title_match_boost).toBeUndefined(); + }); + + test('floor-gate skips a matching result below threshold', () => { + const strong = mk('projects/mingtang', 'Indoor Greek Amphitheater', 1.0); + const weak = mk('notes/aside', 'A Greek Amphitheater Footnote', 0.50); + // threshold = 0.85 * topScore(1.0) = 0.85; weak (0.50) is below. + applyTitleBoost([strong, weak], 'greek amphitheater', DEFAULT_TITLE_BOOST, 0.85); + expect(strong.title_match_boost).toBe(1.25); + expect(weak.title_match_boost).toBeUndefined(); // gated out + expect(weak.score).toBe(0.50); + }); + + test('factor <= 1.0 is a no-op (disabled)', () => { + const r = mk('projects/mingtang', 'Greek Amphitheater', 0.8); + applyTitleBoost([r], 'greek amphitheater', 1.0); + expect(r.score).toBe(0.8); + expect(r.title_match_boost).toBeUndefined(); + }); + + test('empty query is a no-op', () => { + const r = mk('projects/mingtang', 'Greek Amphitheater', 0.8); + applyTitleBoost([r], '', DEFAULT_TITLE_BOOST); + expect(r.score).toBe(0.8); + }); +}); diff --git a/test/search/title-match.test.ts b/test/search/title-match.test.ts new file mode 100644 index 000000000..7b158665d --- /dev/null +++ b/test/search/title-match.test.ts @@ -0,0 +1,68 @@ +/** + * T2 — title-phrase matcher unit tests (retrieval-maxpool incident). + * + * Pins isTitlePhraseMatch: the incident query "Greek amphitheater" must match + * the Mingtang title; generic single-word / stopword queries must NOT promote + * arbitrary pages (Codex#10 precision guard). + */ + +import { describe, test, expect } from 'bun:test'; +import { isTitlePhraseMatch, tokenizeTitle, __test__ } from '../../src/core/search/title-match.ts'; + +describe('isTitlePhraseMatch — positive (the incident)', () => { + const MINGTANG = 'The Mingtang (明堂) — Indoor Greek Amphitheater for Adversarial Debate'; + + test('the exact incident query matches', () => { + expect(isTitlePhraseMatch('Greek amphitheater', MINGTANG)).toBe(true); + }); + test('case + whitespace insensitive', () => { + expect(isTitlePhraseMatch(' greek AMPHITHEATER ', MINGTANG)).toBe(true); + }); + test('contiguous multi-token phrase inside a longer title', () => { + expect(isTitlePhraseMatch('indoor greek amphitheater', MINGTANG)).toBe(true); + }); + test('exact full-title match of a 1-word chosen name', () => { + expect(isTitlePhraseMatch('Mingtang', 'Mingtang')).toBe(true); + }); + test('exact full-title match with punctuation/stopwords normalized', () => { + expect(isTitlePhraseMatch('the hall of light', 'The Hall of Light')).toBe(true); + }); +}); + +describe('isTitlePhraseMatch — negative (precision guard)', () => { + const MINGTANG = 'The Mingtang — Indoor Greek Amphitheater'; + test('single generic content word does NOT match (would over-promote)', () => { + // "greek" alone is 1 content token → below MIN_CONTENT_TOKENS, no boost. + expect(isTitlePhraseMatch('greek', MINGTANG)).toBe(false); + }); + test('single stopword does NOT match', () => { + expect(isTitlePhraseMatch('the', MINGTANG)).toBe(false); + }); + test('non-contiguous tokens do NOT match (phrase, not bag-of-words)', () => { + // "greek" and "indoor" both present but not contiguous in query order. + expect(isTitlePhraseMatch('amphitheater indoor', 'The Indoor X Greek Amphitheater')).toBe(false); + }); + test('substring that is not a token boundary does NOT match', () => { + // "art" must not match "Bartholomew". + expect(isTitlePhraseMatch('art history', 'Bartholomew History Notes')).toBe(false); + }); + test('empty query / empty title', () => { + expect(isTitlePhraseMatch('', MINGTANG)).toBe(false); + expect(isTitlePhraseMatch('greek amphitheater', '')).toBe(false); + }); + test('two stopwords only do NOT match', () => { + expect(isTitlePhraseMatch('the of', 'The Theory of Everything')).toBe(false); + }); +}); + +describe('tokenizeTitle', () => { + test('splits on punctuation + whitespace, lowercases', () => { + expect(tokenizeTitle('The Mingtang — Indoor Greek!')).toEqual(['the', 'mingtang', 'indoor', 'greek']); + }); + test('keeps CJK runs as tokens', () => { + expect(tokenizeTitle('明堂 hall')).toEqual(['明堂', 'hall']); + }); + test('MIN_CONTENT_TOKENS is 2 (the precision floor)', () => { + expect(__test__.MIN_CONTENT_TOKENS).toBe(2); + }); +});