From 736e8de1ecd8aeeee6ce11fd6788c39d818f0ae4 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 1 May 2026 09:02:43 -0700 Subject: [PATCH] v0.25.0 feat: BrainBench-Real session capture + public-exports contract test (#437) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(v0.22.0): eval_candidates + eval_capture_failures schema (Lane 1A) R1 substrate for BrainBench-Real, replayed onto master after Cathedral II landed. Migration v30 (slotted after master's v25-v29 Cathedral II wave) creates two tables: eval_candidates: per-call capture of MCP/CLI/subagent query+search traffic. Column set lets gbrain-evals replay with full fidelity — source_ids from v0.18 multi-source, vector_enabled/detail_resolved/ expansion_applied so replay knows what hybridSearch actually did, remote + job_id + subagent_id so rows are traceable to their origin. query is CHECK-capped at 50KB; PII scrubber (Lane 1B) runs before insert. eval_capture_failures: cross-process audit trail. In-process counters don't work because `gbrain doctor` runs in a separate process from the MCP server. Persistent rows let doctor query capture health via COUNT(*) GROUP BY reason over the last 24h. Both tables get RLS on Postgres gated on BYPASSRLS (matches v24/v29 posture). PGLite ignores RLS; sqlFor split carries only DDL. 5 new BrainEngine methods (breaking-interface addition, drives v0.22.0 minor bump): logEvalCandidate, listEvalCandidates, deleteEvalCandidatesBefore, logEvalCaptureFailure, listEvalCaptureFailures. listEvalCandidates uses ORDER BY created_at DESC, id DESC so `gbrain eval export` is deterministic across same-millisecond inserts. Also adds HybridSearchMeta type for the side-channel callback used by Lane 1C's op-layer capture (no change to hybridSearch return shape — that respects Cathedral II's existing SearchResult[] contract). Tests: 14 PGLite round-trip cases + 8 v30 structural assertions. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(v0.22.0): PII scrubber + op-layer capture module (Lane 1B) Replayed onto master post-Cathedral II. Same semantics as the original v0.21.0 work — only adjusted to import HybridSearchMeta from types.ts (canonical home) instead of redeclaring it locally. src/core/eval-capture-scrub.ts — pure-function regex scrubber with 6 pattern families: emails, phones (US + E.164), SSN (year-aware), Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. Zero deps. Adversarial-input safe. src/core/eval-capture.ts — op-layer hook helper: - buildEvalCandidateInput(ctx, {scrub_pii}) — pure row builder - classifyCaptureFailure(err) — Postgres SQLSTATE → reason tag - captureEvalCandidate(engine, ctx, opts) — best-effort, never throws - isEvalCaptureEnabled / isEvalScrubEnabled — file-plane config checks GBrainConfig gains `eval?: {capture?, scrub_pii?}`. Both default ON. File-plane only — `gbrain config set` writes the DB plane, doesn't control capture. Tests: 17 scrubber + 21 capture-module cases. Zero regressions. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(v0.22.0): hybridSearch onMeta callback + op-layer capture (Lane 1C) Replayed onto master. Adapted from the original v0.21.0 work to keep Cathedral II's contract intact: hybridSearch's return stays `Promise` (unchanged), and meta surfaces via an optional `onMeta?: (meta: HybridSearchMeta) => void` callback in HybridSearchOpts. Cathedral II callers leave onMeta undefined and pay no cost. The op-layer capture wrapper passes a closure that threads meta into the captured row so gbrain-evals can distinguish: - "with OPENAI_API_KEY" vs "keyword-only fallback" (vector_enabled) - "expansion fired" vs "expansion requested + silently fell back" (expansion_applied) - what hybridSearch actually used after auto-detect (detail_resolved) Op-layer capture wired into both `query` and `search` op handlers in src/core/operations.ts. Single hook site catches MCP dispatch + CLI + subagent tool-bridge from the same place. Fire-and-forget, never throws, respects ctx.config.eval.capture off-switch. Tests: - test/hybrid-meta.test.ts (8 cases) — onMeta accuracy across the 4 return paths in hybridSearch + verification that omitting onMeta leaves Cathedral II callers unchanged. - test/mcp-eval-capture.test.ts (10 cases) — query/search ops capture correctly with MCP/CLI/subagent contexts, scrub on/off, capture=false off-switch, non-captured ops (list_pages, get_page), F1 failure isolation. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(v0.22.0): gbrain eval export/prune + doctor eval_capture check (Lane 1D) Replayed onto master. Same semantics as the original v0.21.0 work. CLI: gbrain eval export [--since DUR] [--limit N] [--tool query|search] NDJSON to stdout, every row prefixed with "schema_version":1 per docs/eval-capture.md contract. EPIPE-safe streaming, stderr heartbeats, deterministic ordering (created_at DESC, id DESC). gbrain eval prune --older-than DUR [--dry-run] Explicit retention cleanup. Requires --older-than (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s. Legacy bare `gbrain eval --qrels …` still works via sub-subcommand fall-through. gbrain doctor gains an eval_capture check between markdown_body_completeness and queue_health: reads eval_capture_failures for the last 24h, groups by reason, warns when non-zero. Pre-v30 brains get "Skipped (table unavailable)" — non-fatal. docs/eval-capture.md ships the stable NDJSON schema reference for gbrain-evals consumers. Tests: 9 export cases + 5 prune cases. Doctor check covered by existing doctor tests on master. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(v0.22.0): public-exports contract test + CI count guard (Lane 2 / R2) Master locks 17 public subpath exports as gbrain's stable third-party contract. Zero enforcement existed. This PR locks the surface in two layers: 1. test/public-exports.test.ts — runtime contract test. Reads package.json "exports" at startup. For each subpath, imports via the package name ("gbrain/engine"), NOT the relative filesystem path — that's the difference between exercising the actual resolver and bypassing it. Every subpath gets a canary symbol pinned (e.g. gbrain/search/hybrid must export hybridSearch + rrfFusion) so a refactor that renames or removes one fails CI before downstream consumers (gbrain-evals) silently break. 2. scripts/check-exports-count.sh — CI structural guard. Wired into `bun test` after check-jsonb-pattern.sh + check-progress-to-stdout.sh + check-wasm-embedded.sh per master's precedent. EXPECTED_COUNT=17 baseline — shrinks fail loudly, growth also fails so the new canary must be pinned in the runtime test deliberately. Co-Authored-By: Claude Opus 4.7 (1M context) * docs+e2e(v0.22.0): VERSION/CHANGELOG/CLAUDE/README + Postgres E2E (Lane 3) Bump VERSION + package.json to 0.22.0 (next free slot after master's v0.21.0 Code Cathedral II minor). CHANGELOG.md v0.22.0 entry follows the Garry voice template: - Bold 2-line headline - Lead paragraph contextualizing v0.20 + v0.21 + v0.22 progression - Numbers-that-matter table comparing v0.21.0 → v0.22.0 - "What this means for you" sectioned by audience - "## To take advantage of v0.22.0" operator runbook - Itemized changes CLAUDE.md updates: - Key files: 8 new module entries (eval-capture*, eval-export, eval-prune, docs/eval-capture.md, public-exports test). hybrid.ts entry rewritten to reflect the additive `onMeta` callback (return shape unchanged). - Key commands: new v0.22.0 section for `gbrain eval export`, `gbrain eval prune`, and the doctor `eval_capture` check, with the file-plane vs DB-plane config gotcha called out. README.md: one-paragraph pointer after the BrainBench blurb so anyone reading the landing page sees the new session-capture feature. llms.txt + llms-full.txt regenerated to pick up the doc additions. test/e2e/eval-capture.test.ts (Postgres-only E1 spec): - CHECK violation surfaces as Postgres SQLSTATE 23514 on oversize input - RLS is actually enabled on both eval_candidates + eval_capture_failures - 50 concurrent logEvalCandidate calls — no deadlock, all distinct IDs Skips gracefully when DATABASE_URL is unset. Co-Authored-By: Claude Opus 4.7 (1M context) * docs(todos): P0 — PGLite test-runner concurrency flake Pre-existing on master, surfaces ~27 false failures when bun test runs all 174 files together. Each failing file passes in isolation. Tracked for a dedicated investigation branch. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(v0.22.0): adversarial review post-fixes (doctor RLS, onMeta safety) Two surgical fixes from /ship adversarial review, plus 6 follow-ups TODO'd into v0.22.1: - doctor.ts: distinguish pre-v30 missing-table (42P01, ok skip) from RLS-denied SELECT (42501, warn) and other DB errors (warn). The check exists specifically to surface capture-failure misconfigs cross-process, so silently reporting "ok / skipped" on the most diagnostic class defeated the purpose. - hybrid.ts: wrap onMeta invocation in try/catch via small emitMeta helper. The callback is part of the public gbrain/search/hybrid contract; a throwing user-supplied closure must never break the search hot path. - TODOS.md: 6 P1 follow-ups (eval prune real COUNT, scrubber CC false positives, dead 'scrubber_exception' enum value, id-cursor for cross-window dedup, public-export canary pinning, EXPECTED_COUNT dedup). - TODOS.md: P0 entry for the pre-existing PGLite test-runner concurrency flake (~27 false failures in full bun test on master). - CHANGELOG.md: 2 bullets noting the doctor + onMeta hardening. Co-Authored-By: Claude Opus 4.7 (1M context) * chore(version): bump v0.22.0 → v0.25.0 (queue-aware version pick) Master is at v0.21.0. Open PRs claim v0.21.1 (#432) and v0.24.0 (#387). v0.25 is the first uncontested slot, so this branch claims it. Pure rename across VERSION, package.json, CHANGELOG header, and every "v0.22.0" reference in CLAUDE.md / README.md / TODOS.md / docs/eval-capture.md / src/ / test/ files. CHANGELOG date bumped to 2026-04-26. llms.txt + llms-full.txt regenerated. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(v0.25.0): gbrain eval replay + contributor doc + CONTRIBUTING link Closes the gap between "session capture works" (this PR's core) and "contributors actually use it before merging." Three artifacts: - src/commands/eval-replay.ts (~340 LOC) — reads NDJSON from `gbrain eval export`, re-runs each captured query/search against the current brain, computes set-Jaccard@k, top-1 stability, and latency delta. Stable JSON shape (schema_version:1) for CI gating; human mode prints a regression table sorted worst-first. Pure Bun, zero new deps. Stub-engine tests cover Jaccard math, NDJSON parser (including v2 forward-compat rejection + line-numbered errors), --limit, --verbose, --json, and graceful per-row error handling. 16/16 passing. - docs/eval-bench.md (~80 lines) — contributor guide. The 4-command loop (export → change → replay → diff), metric definitions with healthy ranges (Jaccard ≥0.85, top-1 ≥85%, latency Δ within ±50ms), trigger paths, CI integration snippet, hand-crafted NDJSON corpus path for fresh installs, and the off-switch. Pairs with the existing docs/eval-capture.md which is the consumer-facing wire format. - CONTRIBUTING.md gains a "Running real-world eval benchmarks (touching retrieval code)" section with the trigger paths and a link to docs/eval-bench.md. Reviewers now have a one-line ask: "did you run replay?" CLAUDE.md key files updated. CHANGELOG bullets added. llms.txt regenerated. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(v0.25.0): CONTRIBUTOR_MODE flag — capture off by default for users Eval capture was on for everyone in the v0.25.0 draft. Privacy footgun: end users had retrieval traffic accumulate in their brain DB without asking, even with PII scrubbing. Flips to off by default + explicit opt-in for contributors who actually use the replay loop. Resolution order in isEvalCaptureEnabled(): 1. config.eval.capture === true → on 2. config.eval.capture === false → off 3. process.env.GBRAIN_CONTRIBUTOR_MODE === '1' → on 4. otherwise → off The env var is the contributor-facing toggle (one line in .zshrc, no JSON edit). Explicit config wins both directions for users who want to override per-brain. PII scrubbing gate stays independent — default true regardless of CONTRIBUTOR_MODE — so any brain that does capture still scrubs. Tests rewritten: env var hygiene per-test (origMode preserved + restored in finally). 9/9 pass; total v0.25.0 suite is 198/198. Docs: - README.md gains a Contributing-section pointer to the env var. - CONTRIBUTING.md gains a "CONTRIBUTOR_MODE — turn on the dev loop" section with verification commands and resolution-order table. - docs/eval-bench.md leads with the prerequisite (must set the env var for the rest of the doc to be useful). - docs/eval-capture.md "Config" section split into Path A (env var) + Path B (config) with explicit resolution-order rules. - CHANGELOG v0.25.0 entry corrected ("on by default" was wrong) plus a new top itemized bullet calling out the gate change. - CLAUDE.md eval-capture entry annotated with the new gate logic. Co-Authored-By: Claude Opus 4.7 (1M context) * docs: post-ship documentation pass for v0.25.0 Cross-references every doc against the final state of the branch (CONTRIBUTOR_MODE flag, eval replay tool, off-by-default capture): - README.md: top callout rewritten — was implying capture-on-by-default contradicting the gate landed in 7a80ce25. Now leads with "contributor opt-in" and links docs/eval-bench.md alongside docs/eval-capture.md. - AGENTS.md: new "Eval retrieval changes" task entry with the CONTRIBUTOR_MODE+replay one-liner so non-Claude agents (Codex, Cursor, Aider) have the same path. - CLAUDE.md: "Key commands added in v0.25.0" gains the replay command and a CONTRIBUTOR_MODE bullet covering the resolution order. - CHANGELOG.md: headline rewritten to match the actual feature ("benchmark retrieval changes against real captured queries before merging" — was "every real query is captured"). Stale "v0.22 ships the substrate" → v0.25. Test count corrected 82 → 144 (added 16 replay + 9 CONTRIBUTOR_MODE + 8 v31-shape tests since the original count). Two metric rows added to the numbers table: default-off posture, in-tree replay tooling. "To take advantage" block split into user vs contributor branches with shell-rc instructions. - TODOS.md: v0.22.1 follow-up reference corrected to v0.25.1. llms.txt + llms-full.txt regenerated. Typecheck clean. 198/198 v0.25.0 tests still green. Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Claude Opus 4.7 (1M context) --- AGENTS.md | 5 + CHANGELOG.md | 94 +++++++ CLAUDE.md | 19 +- CONTRIBUTING.md | 78 ++++++ README.md | 4 + TODOS.md | 35 +++ VERSION | 2 +- docs/eval-bench.md | 224 +++++++++++++++++ docs/eval-capture.md | 160 ++++++++++++ llms-full.txt | 28 ++- package.json | 5 +- scripts/check-exports-count.sh | 48 ++++ src/commands/doctor.ts | 52 ++++ src/commands/eval-export.ts | 159 ++++++++++++ src/commands/eval-prune.ts | 115 +++++++++ src/commands/eval-replay.ts | 424 ++++++++++++++++++++++++++++++++ src/commands/eval.ts | 17 ++ src/core/config.ts | 12 + src/core/engine.ts | 18 ++ src/core/eval-capture-scrub.ts | 104 ++++++++ src/core/eval-capture.ts | 208 ++++++++++++++++ src/core/migrate.ts | 100 ++++++++ src/core/operations.ts | 72 +++++- src/core/pglite-engine.ts | 78 ++++++ src/core/pglite-schema.ts | 29 +++ src/core/postgres-engine.ts | 70 ++++++ src/core/schema-embedded.ts | 40 ++- src/core/search/hybrid.ts | 38 ++- src/core/types.ts | 63 +++++ src/schema.sql | 38 +++ test/e2e/eval-capture.test.ts | 144 +++++++++++ test/eval-candidates.test.ts | 201 +++++++++++++++ test/eval-capture-scrub.test.ts | 118 +++++++++ test/eval-capture.test.ts | 311 +++++++++++++++++++++++ test/eval-export.test.ts | 178 ++++++++++++++ test/eval-prune.test.ts | 134 ++++++++++ test/eval-replay.test.ts | 421 +++++++++++++++++++++++++++++++ test/hybrid-meta.test.ts | 102 ++++++++ test/mcp-eval-capture.test.ts | 245 ++++++++++++++++++ test/migrate.test.ts | 66 +++++ test/public-exports.test.ts | 102 ++++++++ 41 files changed, 4350 insertions(+), 11 deletions(-) create mode 100644 docs/eval-bench.md create mode 100644 docs/eval-capture.md create mode 100755 scripts/check-exports-count.sh create mode 100644 src/commands/eval-export.ts create mode 100644 src/commands/eval-prune.ts create mode 100644 src/commands/eval-replay.ts create mode 100644 src/core/eval-capture-scrub.ts create mode 100644 src/core/eval-capture.ts create mode 100644 test/e2e/eval-capture.test.ts create mode 100644 test/eval-candidates.test.ts create mode 100644 test/eval-capture-scrub.test.ts create mode 100644 test/eval-capture.test.ts create mode 100644 test/eval-export.test.ts create mode 100644 test/eval-prune.test.ts create mode 100644 test/eval-replay.test.ts create mode 100644 test/hybrid-meta.test.ts create mode 100644 test/mcp-eval-capture.test.ts create mode 100644 test/public-exports.test.ts diff --git a/AGENTS.md b/AGENTS.md index 1fd6278d5..4a6f08bc0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,6 +37,11 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont [`docs/guides/minions-fix.md`](./docs/guides/minions-fix.md), `gbrain doctor --fix`. - **Migrate:** [`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md), [`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations`. +- **Eval retrieval changes:** capture is off by default. To benchmark a + retrieval change against real captured queries, set + `GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson` + and `gbrain eval replay --against base.ndjson`. Full guide: + [`docs/eval-bench.md`](./docs/eval-bench.md). - **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map. [`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for single-fetch ingestion. diff --git a/CHANGELOG.md b/CHANGELOG.md index 0006ac366..01aead804 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,100 @@ All notable changes to GBrain will be documented in this file. +## [0.25.0] - 2026-04-26 + +## **Contributors can now benchmark retrieval changes against real captured queries before merging.** +## **`GBRAIN_CONTRIBUTOR_MODE=1` turns on capture; `gbrain eval replay` is the dev loop.** + +v0.20 (gbrain-evals extraction) and v0.21 (Cathedral II) gave gbrain its install surface back and turned code into a first-class graph. The remaining gap was data: `amara-life`, the fictional 418-item corpus over in gbrain-evals, is great for reproducibility but not for catching regressions against the queries your agents *actually* serve. v0.25 ships the substrate: with `GBRAIN_CONTRIBUTOR_MODE=1` set, every `query` and `search` call through MCP, CLI, or the subagent tool-bridge records into a new `eval_candidates` table. `gbrain eval export` streams the rows as NDJSON. `gbrain eval replay --against ` re-runs each captured query against your current build and prints three numbers: mean Jaccard@k, top-1 stability, and latency Δ. Point gbrain-evals at the stream and you have BrainBench-Real on every release. Run replay locally and you have a regression gate on every retrieval PR. + +Capture is **off by default**. Production users get a quiet brain — no surprise data accumulation, no privacy footgun. Contributors flip it on with one shell rc line: `export GBRAIN_CONTRIBUTOR_MODE=1`. From that shell forward, every query/search lands in `eval_candidates`. PII is scrubbed at write time (emails, phones, SSN, Luhn-verified credit cards, JWTs, bearer tokens). Queries over 50KB get rejected. RLS matches the v0.18.1 / v0.21.0 posture ... new tables get enabled on Postgres, gated on BYPASSRLS so it never locks an operator out of their own data. `gbrain doctor` surfaces silent capture failures cross-process so if something stops working you see it in health checks, not three weeks later when the replay numbers look weird. + +Cathedral II (v0.21.0) callers are unaffected. `hybridSearch` still returns `Promise` ... meta arrives via an optional `onMeta` callback in `HybridSearchOpts`, used only by the op-layer capture wrapper to record what hybridSearch *actually* did (vector ran or fell back, expansion fired or didn't, post-auto-detect detail). + +### The numbers that matter + +Measured on this branch's diff against v0.21.0: + +| Metric | v0.21.0 | v0.25.0 | Δ | +|---|---|---|---| +| Real-query capture | none | every MCP/CLI/subagent `query` + `search` | **the whole feature** | +| PII classes redacted at write | 0 | 6 (email, phone, SSN, CC+Luhn, JWT, bearer) | ... | +| Schema columns per captured row | ... | 16 (tool, query, slugs, chunks, source_ids, expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied, latency, remote, job_id, subagent_id, created_at, id) | ... | +| `hybridSearch` return shape | `Promise` | `Promise` (unchanged) | 0 break | +| `hybridSearch` opts surface | existing | + `onMeta?: (m: HybridSearchMeta) => void` | additive | +| BrainEngine methods | shipped Cathedral II surface | +5 eval-capture | **BREAKING for custom engines** | +| Public subpath exports | 17 | 17 (now contract-tested) | 0 drift | +| Default capture posture | n/a | OFF for users, ON via `GBRAIN_CONTRIBUTOR_MODE=1` | privacy-positive default | +| Dev-loop tooling for retrieval PRs | none in-tree | `gbrain eval replay --against ` | the regression gate | +| New tests | ... | 144 (14 engine round-trip + 17 scrubber + 27 capture module + 8 hybrid meta + 10 op-layer capture + 9 export + 5 prune + 30 public-exports + 16 replay + 8 v31-migrate) | ... | + +### What this means for you + +**Brain operators (the 99%):** run `gbrain upgrade`. Nothing changes — capture stays off, your brain stays quiet. The substrate is in place if you ever want to turn it on (e.g. to debug a retrieval issue), but you don't have to. + +**Contributors / maintainers:** add `export GBRAIN_CONTRIBUTOR_MODE=1` to your shell rc. Every `query`/`search` from then on writes to `eval_candidates`. After a week of dogfooding, snapshot with `gbrain eval export --since 7d > real.ndjson` and `gbrain eval replay --against real.ndjson` against your branch to gate retrieval changes. To opt out per-brain regardless of the env var: write `{"eval":{"capture":false}}` to `~/.gbrain/config.json`. + +**Anyone calling `hybridSearch` directly:** no change required. The return type is still `Promise`. If you want the new meta side-channel, pass `onMeta: (m) => { ... }` in opts — otherwise leave it undefined and pay no cost. + +**Downstream TypeScript consumers implementing their own BrainEngine:** five new methods need implementations ... `logEvalCandidate`, `listEvalCandidates`, `deleteEvalCandidatesBefore`, `logEvalCaptureFailure`, `listEvalCaptureFailures`. Return types are in `src/core/types.ts`. This is why v0.25.0 is a minor bump. + +## To take advantage of v0.25.0 + +**If you're a regular gbrain user:** `gbrain upgrade` is enough. Capture stays off, your brain stays quiet, nothing else changes. The substrate exists if you ever want to opt in (write `{"eval":{"capture":true}}` to `~/.gbrain/config.json`), but you don't have to. + +**If you're a contributor or maintainer working on retrieval:** + +1. **Apply the migration** (idempotent; `gbrain upgrade` does this automatically): + ```bash + gbrain apply-migrations --yes + ``` + +2. **Turn on capture** by adding one line to your `~/.zshrc` or `~/.bashrc`: + ```bash + export GBRAIN_CONTRIBUTOR_MODE=1 + ``` + Reload your shell (`source ~/.zshrc`) or open a new terminal. + +3. **Verify the tables exist + capture is healthy:** + ```bash + gbrain query "any test query" >/dev/null + psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # > 0 means capture is running + gbrain doctor # should show "eval_capture: No capture failures in the last 24h" + ``` + +4. **Dogfood for a week, then run the dev loop:** + ```bash + gbrain eval export --since 7d > baseline.ndjson + # ... make a retrieval change ... + gbrain eval replay --against baseline.ndjson # mean Jaccard@k, top-1 stability, latency Δ + ``` + + See [`docs/eval-bench.md`](https://github.com/garrytan/gbrain/blob/master/docs/eval-bench.md) for the full guide and CI integration snippet. + +5. **If something looks wrong:** `gbrain doctor` names the `eval_capture_failures` breakdown by reason. File an issue with the breakdown + your config shape at https://github.com/garrytan/gbrain/issues. + +### Itemized changes + +- **`GBRAIN_CONTRIBUTOR_MODE=1` env var gates capture.** Default flipped from on-for-everyone to off-unless-opted-in. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Production users get a quiet brain by default; contributors flip the env var in `.zshrc` and unlock the full export → replay loop. README + CONTRIBUTING document the flag prominently. +- v31 migration: `eval_candidates` + `eval_capture_failures` on both Postgres + PGLite, RLS gated on BYPASSRLS, CHECK constraints + indexes +- Op-layer capture wrapper in `src/core/operations.ts` (covers MCP + CLI + subagent tool-bridge from one site) +- PII scrubber in `src/core/eval-capture-scrub.ts` (6 regex families, adversarial-input safe) +- Cross-process audit via `eval_capture_failures` + `gbrain doctor` 24h breakdown +- `gbrain eval export` (NDJSON, schema_version:1, EPIPE-safe) + `gbrain eval prune` (explicit retention) +- `gbrain eval replay --against ` — contributor-facing dev loop. Reads a captured snapshot, re-runs every `query` / `search` against the current brain, prints mean Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. JSON mode (`schema_version: 1`) for CI gating, top-regressions table for human eyeballs. Closes the gap between "data captured" and "data used to gate a PR." +- `docs/eval-bench.md` — contributor guide that walks the 4-command loop (export → change → replay → diff), defines metrics (Jaccard@k, top-1 stability, latency Δ) with healthy ranges, lists the source paths that should trigger a re-run, and shows how to wire it into CI. Linked from CONTRIBUTING.md. +- `CONTRIBUTING.md` adds a "Running real-world eval benchmarks (touching retrieval code)" section so PRs that change retrieval have an obvious replay path. Maintainer reviews can ask "did you run replay?" instead of writing a custom benchmark per PR. +- `hybridSearch` adds `onMeta?: (m) => void` to opts (Cathedral II callers unaffected) +- `BrainEngine` gains 5 methods (breaking-interface for custom engines, drives v0.25.0 minor bump) +- `test/public-exports.test.ts` + `scripts/check-exports-count.sh` lock the 17-subpath public surface +- Config gains `eval: {capture?, scrub_pii?}` (file-plane only); env var `GBRAIN_CONTRIBUTOR_MODE=1` is the contributor-facing toggle that doesn't require editing JSON +- `listEvalCandidates` orders `created_at DESC, id DESC` (deterministic export windows) +- `docs/eval-capture.md` — stable NDJSON schema reference for gbrain-evals +- `gbrain doctor` `eval_capture` check now distinguishes pre-v31 missing-table (status: ok / skipped) from RLS-denied SELECT or transient DB error (status: warn) — the diagnostic that surfaces a misconfigured RLS role no longer goes silent +- `hybridSearch.onMeta` callback wrapped in try/catch — a throwing user-supplied callback can't break the search hot path (defensive across the public `gbrain/search/hybrid` surface) +- +144 unit tests across 9 new files (eval-candidates 14, eval-capture-scrub 17, eval-capture 27, eval-export 9, eval-prune 5, eval-replay 16, hybrid-meta 8, mcp-eval-capture 10, public-exports 30) plus 8 v31-shape checks added to `test/migrate.test.ts`. 0 regressions across the full suite (198 v0.25.0-related tests pass cleanly). + ## [0.24.0] - 2026-04-26 ## **The skillify loop stops lying. Privacy guard runs, `--llm` is honest, ghost rows go away.** diff --git a/CLAUDE.md b/CLAUDE.md index 26a1e685a..e32f0f802 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -49,7 +49,16 @@ strict behavior when unset. - `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator - `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. - `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. -- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison +- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. +- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows. +- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count. +- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow. +- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". +- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. +- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. +- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. +- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. +- `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff - `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`. - `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic. @@ -212,6 +221,14 @@ Key commands added for Minions (job queue): - `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall. - `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only) +Key commands added in v0.25.0: +- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay. +- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s. +- `gbrain eval replay --against FILE.ndjson [--limit N] [--top-regressions K] [--json] [--verbose]` — contributor-facing dev loop. Reads a captured NDJSON snapshot, re-runs each `query` / `search` op against the current brain, computes mean set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. JSON mode (`schema_version: 1`) for CI gating; human mode prints a regression table sorted worst-first. Closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow. +- `gbrain doctor` gains an `eval_capture` check: reads `eval_capture_failures` for the last 24h, groups by reason, warns when non-zero. Cross-process visibility (doctor runs in a separate process from MCP). Pre-v31 brains get `Skipped (table unavailable)` — non-fatal. +- Config addition: `eval: { capture?: boolean, scrub_pii?: boolean }` in `~/.gbrain/config.json`. **File-plane only** — `gbrain config set` writes the DB plane and does NOT control capture. +- **`GBRAIN_CONTRIBUTOR_MODE=1` env var** is the contributor-facing toggle. Capture is **off by default** as of v0.25.0; production users get a quiet brain. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Documented in README.md, CONTRIBUTING.md, and `docs/eval-bench.md`. + Key commands added in v0.12.2: - `gbrain repair-jsonb [--dry-run] [--json]` — repair double-encoded JSONB rows left over from v0.12.0-and-earlier Postgres writes. Idempotent; PGLite no-ops. The `v0_12_2` migration runs this automatically on `gbrain upgrade`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ad4121f2b..750b215a1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -124,6 +124,84 @@ See `docs/ENGINES.md` for the full guide. In short: The SQLite engine is designed and ready for implementation. See `docs/SQLITE_ENGINE.md`. +## CONTRIBUTOR_MODE — turn on the dev loop + +gbrain captures retrieval traffic so you can replay real queries against +your code changes before merging. **This is off by default** (production +users get a quiet brain, no surprise data accumulation). Contributors turn +it on with one shell rc line: + +```bash +# In ~/.zshrc or ~/.bashrc: +export GBRAIN_CONTRIBUTOR_MODE=1 +``` + +That's it. Every `query` / `search` you (or agents pointed at your dev +brain) run from that shell now writes a row to `eval_candidates`, and the +[replay tool](#running-real-world-eval-benchmarks-touching-retrieval-code) +has data to work against. + +What CONTRIBUTOR_MODE actually does: + +- Turns on `query`/`search` capture into the local `eval_candidates` table. + Without it the gate is closed and capture is a no-op. +- That's all. PII scrubbing, retention, and replay are independent. + +Resolution order (most explicit wins): + +1. `eval.capture: true` in `~/.gbrain/config.json` → on +2. `eval.capture: false` in `~/.gbrain/config.json` → off +3. `GBRAIN_CONTRIBUTOR_MODE=1` → on +4. otherwise → off + +Quick check that capture is actually running: + +```bash +gbrain query "anything" >/dev/null +psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' +# (or `gbrain doctor` — surfaces silent capture failures cross-process) +``` + +To disable capture even with the env var set, write +`{"eval": {"capture": false}}` to `~/.gbrain/config.json` — explicit config +beats the env var both directions. + +## Running real-world eval benchmarks (touching retrieval code) + +If your PR touches retrieval — search ranking, RRF fusion, embeddings, +intent classification, query expansion, source boost, or the `query` / +`search` op handlers — run `gbrain eval replay` against a snapshot of +real traffic before merging. Requires `CONTRIBUTOR_MODE` (above) so you +have captured rows to replay against. + +Quick loop: + +```bash +gbrain eval export --since 7d > baseline.ndjson # snapshot before your change +# ... make your change ... +gbrain eval replay --against baseline.ndjson # diff retrieval, get Jaccard@k +``` + +Three numbers come back: mean Jaccard@k between captured and current slug +sets, top-1 stability, and mean latency Δ. The replay tool flags the worst +regressions so you can eyeball whether the change is hurting real queries. + +Trigger paths (rerun if your diff touches any of these): + +- `src/core/search/hybrid.ts` +- `src/core/search/source-boost.ts`, `sql-ranking.ts` +- `src/core/search/intent.ts`, `expansion.ts`, `dedup.ts` +- `src/core/embedding.ts` +- `src/core/operations.ts` (query / search handlers) +- `src/core/postgres-engine.ts` / `pglite-engine.ts` (searchKeyword / + searchVector SQL) + +See [`docs/eval-bench.md`](./docs/eval-bench.md) for the full guide +including CI integration, hand-crafted NDJSON corpora (so a fresh checkout +without captured data can still replay), and cost considerations. The +NDJSON wire format is documented in +[`docs/eval-capture.md`](./docs/eval-capture.md). + ## Welcome PRs - SQLite engine implementation diff --git a/README.md b/README.md index 46d20cb9f..b4cb0867e 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,8 @@ The brain wires itself. Every page write extracts entity references and creates GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours. +**New in v0.25.0 — BrainBench-Real (session capture, contributor opt-in):** with `GBRAIN_CONTRIBUTOR_MODE=1` set in your shell, every real `query` + `search` call through MCP, CLI, or the subagent tool-bridge gets captured (PII-scrubbed) into an `eval_candidates` table. Snapshot with `gbrain eval export`, replay against your code change with `gbrain eval replay`. Three numbers come back: mean Jaccard@k between captured and current retrieved slugs, top-1 stability, and latency Δ. **Off by default** for production users — no surprise data accumulation. Walkthrough: [docs/eval-bench.md](docs/eval-bench.md). NDJSON wire format: [docs/eval-capture.md](docs/eval-capture.md). + > **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys. > **LLMs:** fetch [`llms.txt`](llms.txt) for the documentation map, or [`llms-full.txt`](llms-full.txt) for the same map with core docs inlined in one fetch. **Agents:** start with [`AGENTS.md`](AGENTS.md) (or [`CLAUDE.md`](CLAUDE.md) if you're Claude Code). @@ -744,6 +746,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration. +If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in. + PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`. ## License diff --git a/TODOS.md b/TODOS.md index aa7854e4a..ee340c3d7 100644 --- a/TODOS.md +++ b/TODOS.md @@ -687,6 +687,21 @@ iteration's residuals. ## P0 +### PGLite test-runner concurrency flake (~27 false failures in full `bun test`) +**What:** Fix the concurrent-PGLite-init flake that surfaces ~27 `error: PGLite not connected. Call connect() first.` failures when `bun test` runs all 174 unit-test files together. Each failing file passes in isolation; failures only appear under full-suite parallelism. + +**Why:** The failures are masking real signal. /ship and any solo dev running `bun test` has to manually triage 27 results every time. Today they're all in `test/cathedral-ii-pglite.test.ts`, `test/cathedral-ii-brainbench.test.ts` (Layer 5/6/7/8 + parent_scope_coverage + call_graph_recall), `test/sync.test.ts` (4 dry-run cases), `test/reindex-code.test.ts` (Layer 13 E2). All exist on master and date back to v0.12.3-v0.21.0 — pre-existing, not caused by any one branch. + +**Context:** Confirmed pre-existing on master via `git diff origin/master...HEAD --stat -- ` returning empty. Tests pass cleanly in 1-3-file batches. Wall clock for the full suite is 596s. Likely root causes: (a) PGLite has a singleton or shared OPFS-like state that races under parallel `PGlite.create()` calls, (b) `test/cathedral-ii-pglite.test.ts` "fresh-install schema" tests assume exclusive PGLite access, (c) bun test concurrency exceeds what PGLite's WASM init can handle. + +**Pros:** Green suite signal. Faster shipping. Stops eroding trust in `bun test`. + +**Cons:** Likely needs PGLite engine-per-test isolation (each test gets its own dedicated engine instance via tmpdir) or a `bun test --concurrency=N` cap. Both touch test infra used by 50+ files. + +**Effort:** M (human: 1 day to root-cause + implement / CC: ~2-3 hours via /investigate). + +**Discovered:** v0.25.0 ship, 2026-04-25. + ### Fix `bun build --compile` WASM embedding for PGLite **What:** Submit PR to oven-sh/bun fixing WASM file embedding in `bun build --compile` (issue oven-sh/bun#15032). @@ -726,6 +741,26 @@ iteration's residuals. **Depends on:** v0.10.0 GStackBrain skill layer (shipped). +## P1 (new from v0.25.0 — eval-capture adversarial review) + +### v0.25.0 eval-capture follow-ups (6 surgical hardenings) +**Priority:** P1 + +**What:** Six targeted hardenings on the v0.25.0 eval-capture surface, all surfaced by the /ship adversarial review and triaged out of the v0.25.0 PR to keep scope tight: + +1. `gbrain eval prune --dry-run`: replace the `listEvalCandidates(limit:100k) + filter` count with a real `engine.countEvalCandidatesBefore(date)` method. Today the warning at `eval-prune.ts:107-109` honestly tells the user the count may be undercounted, but a brain with > 100k rows + old data could still confuse a careful operator. New `BrainEngine` method on both engines, ~30 LOC, lifts the floor count to a true count. +2. PII scrubber CC false-positive rate: 16-digit Luhn-valid order IDs / invoice numbers get redacted as `[REDACTED]`. Either require a contextual prefix (`card`, `cc`, `credit`) within N chars, or document the tradeoff explicitly in `docs/eval-capture.md`. The two approaches differ in coverage so list them as alternatives. +3. `eval_capture_failures.reason` enum: `'scrubber_exception'` is dead telemetry — no realistic path emits it (the scrubber is regex-only and never throws). Either remove the value from the schema CHECK + enum, OR wrap `scrubPii` in a try-catch inside `buildEvalCandidateInput` so the value is actually reachable. +4. `id DESC` tiebreaker docs: CLAUDE.md says "stable id-desc tiebreaker so `--since` windows never dupe/miss rows". This is true within a single call but doesn't prevent dupe/miss across overlapping windows when LIMIT < total. Either add a real `id`-cursor (`WHERE id < $cursor`) for export, or scope the doc claim to "within a single export call". +5. Public-exports canaries: 6 of 17 subpaths (`gbrain` root, `/minions`, `/engine-factory`, `/transcription`, `/backoff`, `/extract`) have `canary: []` — the test only checks the import resolves, so a barrel module accidentally losing its named exports would still pass. Pin one stable canary symbol per subpath. +6. `EXPECTED_COUNT` duplication: `scripts/check-exports-count.sh` and `test/public-exports.test.ts` both hardcode `17`. Drift risk. Make one read the other (or both compute from `package.json`). + +**Why:** All 6 are real (some informational, some footgun-class) but each is small and surgical. Bundling into one v0.25.1 follow-up PR keeps the v0.25.0 ship clean and lets the fixes land with their own dedicated tests + CHANGELOG entry. + +**Effort:** S total (human: ~half day / CC: ~1.5 hours). + +**Discovered:** v0.25.0 ship adversarial review, 2026-04-25. + ## P1 (new from v0.7.0) ### ~~Constrained health_check DSL for third-party recipes~~ diff --git a/VERSION b/VERSION index 2094a100c..d21d277be 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.24.0 +0.25.0 diff --git a/docs/eval-bench.md b/docs/eval-bench.md new file mode 100644 index 000000000..5399e90de --- /dev/null +++ b/docs/eval-bench.md @@ -0,0 +1,224 @@ +# Running real-world eval benchmarks against your gbrain changes + +Audience: gbrain maintainers and contributors. If you're touching retrieval +(search, ranking, embeddings, intent classification, query expansion, source +boost, hybrid fusion), this is the doc. + +For the **NDJSON wire format** consumed by gbrain-evals, see +[`eval-capture.md`](./eval-capture.md). This doc is the human dev loop +that lives on top of that format. + +## Prerequisite: turn on contributor mode + +Capture is **off by default** for production users (privacy-positive — no +surprise data accumulation). Contributors flip it on with one line: + +```bash +# In ~/.zshrc or ~/.bashrc: +export GBRAIN_CONTRIBUTOR_MODE=1 +``` + +Verify: + +```bash +gbrain query "anything" >/dev/null +psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0 +``` + +To override (force on/off regardless of env var), edit `~/.gbrain/config.json`: + +```json +{"eval": {"capture": true}} // force on +{"eval": {"capture": false}} // force off +``` + +Explicit config beats the env var both directions. + +## The 4-command loop + +```bash +# ① Capture: writes to eval_candidates whenever CONTRIBUTOR_MODE is set. +# Inspect what's been collected: +gbrain doctor # surfaces capture failures +psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' + +# ② Snapshot: freeze a baseline before your code change. +gbrain eval export --since 7d > baseline.ndjson + +# ③ Code change: do whatever you want — tune RRF_K, swap embed model, edit +# hybrid.ts, add a new boost source, change the intent classifier. + +# ④ Replay: re-run every captured query against the current build. +gbrain eval replay --against baseline.ndjson +``` + +Output: + +``` +Replaying 247 captured queries… + ...25/247 + ...50/247 + ... +Replayed 247 of 247 captured queries (0 skipped, 0 errored) +Mean Jaccard@k: 0.927 +Top-1 stability: 91.5% +Mean latency Δ: +14ms (current vs captured) + +Top 5 regression(s): + jaccard=0.20 captured=12 current=3 "find every reference to widget-co" + jaccard=0.43 captured=14 current=8 "show me everything tagged for review" + jaccard=0.50 captured=8 current=4 "what did alice say about the spec" + ... +``` + +Three numbers tell you whether the change is safe to land: + +| Metric | What it means | Healthy range | +|---|---|---| +| **Mean Jaccard@k** | Average overlap between captured retrieved slugs and current run's slugs. 1.0 = identical sets. | ≥0.85 for "neutral" changes. <0.7 means major retrieval shift. | +| **Top-1 stability** | Fraction of queries whose #1 result didn't change. | ≥85% for tuning passes. <70% means top-of-funnel broke. | +| **Mean latency Δ** | Current minus captured. Positive = slower now. | Within ±50ms of captured. >2× anywhere = regression alarm. | + +## What it actually does + +`gbrain eval replay` reads your NDJSON snapshot and, for each row: + +1. Re-executes the same op (`searchKeyword` for `tool_name='search'`, + `hybridSearch` for `tool_name='query'`) with the captured `detail` and + `expand_enabled` values threaded back in. +2. Captures the current `retrieved_slugs` (deduped, in result order). +3. Computes set-Jaccard between captured and current slug sets. +4. Records top-1 match (was the #1 result the same slug?). +5. Records latency delta vs captured `latency_ms`. + +It does NOT compute MRR or nDCG — those need ground-truth relevance labels, +not a baseline comparison. For metric-against-truth eval, use +`gbrain eval --qrels ` (the legacy IR-eval path, still supported). The +replay tool answers a different question: "did my code change move +retrieval, and which queries did it move most?" + +## Best-effort by design + +Replay is not pure. Three things can drift between capture and replay: + +1. **Brain state** — your brain probably has more pages now than when the + snapshot was taken. Unless you explicitly seed a fixed corpus, mean + Jaccard will drop simply because new pages are eligible. +2. **Embedding source** — if you changed `OPENAI_API_KEY` between capture + and replay (or the embedding model rotated), vector-path results drift + even with identical code. +3. **Capture cap** — captured `retrieved_slugs` is a deduped set; it doesn't + preserve internal ranking metadata. Two tools can return the same slug + set with different scores — Jaccard will say 1.0, but a downstream + consumer that orders by score may behave differently. + +The metrics are **regression alarms on real queries**, not a hash check. +Pair them with manual inspection of the top regressions. + +## Cost + +Every `query` row in the snapshot embeds the query string via OpenAI to run +the vector half of `hybridSearch`. Cost is identical to a normal `gbrain +query` invocation — text-embedding-3-large at OpenAI list price, batched +inside a single replay row. + +If you're iterating locally and don't want to pay per change, use +`--limit 50` to cap rows replayed. The 50 most recent rows are usually +enough to catch direction; expand for the final pre-merge run. + +```bash +# Iteration mode — 50 most recent queries +gbrain eval replay --against baseline.ndjson --limit 50 + +# Pre-merge — full snapshot +gbrain eval replay --against baseline.ndjson --top-regressions 20 +``` + +## CI integration + +```bash +gbrain eval replay --against baseline.ndjson --json > replay.json +jq -e '.summary.mean_jaccard >= 0.85' replay.json || exit 1 +jq -e '.summary.top1_stability_rate >= 0.85' replay.json || exit 1 +``` + +Stable JSON shape (schema_version: 1): + +```json +{ + "schema_version": 1, + "summary": { + "rows_total": 247, + "rows_replayed": 247, + "rows_skipped": 0, + "rows_errored": 0, + "mean_jaccard": 0.927, + "top1_stability_rate": 0.915, + "mean_latency_delta_ms": 14, + "rows_over_2x_latency": 0 + } +} +``` + +`--verbose` adds a `results: [...]` array with one entry per replayed row +(useful for piping into jq or a notebook for deeper analysis). + +## When to run this + +Before merging anything that touches: + +- `src/core/search/hybrid.ts` (RRF, fusion, dedup, two-pass retrieval) +- `src/core/search/source-boost.ts` / `sql-ranking.ts` (per-source ranking) +- `src/core/search/intent.ts` (auto-detail classification) +- `src/core/search/expansion.ts` (Haiku query expansion) +- `src/core/search/dedup.ts` (cross-page result collapse) +- `src/core/embedding.ts` or any embedding model swap +- `src/core/operations.ts` `query` or `search` op handlers (capture surface) +- `src/core/postgres-engine.ts` / `pglite-engine.ts` `searchKeyword` / + `searchVector` SQL + +Skip for: schema-only migrations, doc changes, tests-only PRs, CLI ergonomics +that don't touch retrieval. + +## Building your own corpus + +If you don't have captured traffic yet (fresh install, can't dogfood for a +week before merging), you can hand-author an NDJSON file: + +```jsonl +{"schema_version":1,"id":1,"tool_name":"query","query":"who is alice","retrieved_slugs":["people/alice","people/alice-bio"],"expand_enabled":false,"detail":null,"latency_ms":0,"remote":false} +{"schema_version":1,"id":2,"tool_name":"search","query":"acme deal","retrieved_slugs":["deals/acme-seed","companies/acme"],"latency_ms":0,"remote":false} +``` + +Then run `gbrain eval replay --against handcrafted.ndjson` to confirm the +authoritative slugs come back. This is the seam between the BrainBench-Real +pipeline (replay against live captures) and the BrainBench fixed-fixture +pipeline (`gbrain eval --qrels` with the sibling +[gbrain-evals](https://github.com/garrytan/gbrain-evals) corpus). + +## Off-switch + +Two ways to disable capture: + +```bash +unset GBRAIN_CONTRIBUTOR_MODE # easy: just unset the env var +``` + +Or force off regardless of the env var via `~/.gbrain/config.json`: + +```json +{"eval": {"capture": false}} +``` + +Existing `eval_candidates` rows stay until you `gbrain eval prune +--older-than 0d` (or just drop the table). + +## Failure modes + +| What you see | What it means | +|---|---| +| `Mean Jaccard@k: 0.4`, top regressions all in one source dir | Source boost or hard-exclude regression on that prefix | +| `Top-1 stability: 30%`, mean Jaccard still high | RRF tuning shifted the rank order without changing the set — re-tune `rrfK` | +| `Mean latency Δ: +500ms`, jaccard high | Vector path got slower; check embedding API or HNSW probes | +| `rows_errored > 0` | One or more queries threw. Inspect first 3 in human output, or `--json` to see all `error_message` fields | +| Many `skipped: empty query` | Capture ran on rows where someone passed empty `query` — check why those were captured | diff --git a/docs/eval-capture.md b/docs/eval-capture.md new file mode 100644 index 000000000..2e671d9d2 --- /dev/null +++ b/docs/eval-capture.md @@ -0,0 +1,160 @@ +# Eval capture — NDJSON schema reference + +**Status:** stable from v0.21.0. Schema versioning via `schema_version` +on every row; additive changes increment the minor version; removals +are breaking-schema-v2. + +**Audience:** downstream consumers (primarily the sibling +[gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) that +replay captured real-world queries as a BrainBench-Real fixture. + +## The pipeline + +``` +MCP / CLI / subagent tool-bridge caller + │ + ▼ +src/core/operations.ts — query + search op handlers + │ + │ (hybridSearch or searchKeyword) + │ + ▼ +{results, meta: HybridSearchMeta} ┌── captureEvalCandidate + │ │ (fire-and-forget) + ▼ │ +return to caller ▼ + scrubPii(query) ←── src/core/eval-capture-scrub.ts + │ + ▼ + buildEvalCandidateInput + │ + ▼ + engine.logEvalCandidate + │ + ┌──────────────┴──────────────┐ + │ success │ fail + ▼ ▼ + INSERT into eval_candidates engine.logEvalCaptureFailure + (reason: db_down | rls_reject | + check_violation | + scrubber_exception | other) +``` + +## `gbrain eval export` — the consumer contract + +```sh +gbrain eval export [--since DUR] [--limit N] [--tool query|search] +``` + +Emits NDJSON to **stdout**. One JSON object per `\n`-terminated line. +stderr receives progress heartbeats. Every line starts with +`"schema_version": 1` so a forward-compat parser can fail loudly on +schema v2 instead of silently misparsing. + +Typical usage from gbrain-evals: + +```sh +# Snapshot the last week of real traffic for replay +gbrain eval export --since 7d > brainbench-real.ndjson +``` + +```sh +# Stream through jq for ad-hoc analysis +gbrain eval export --tool query | jq -c 'select(.latency_ms > 500)' +``` + +## Row schema (v1) + +Every exported row has this shape. Field order in JSON output is not +guaranteed; consumers MUST key by name, not position. + +| Field | Type | Notes | +|---|---|---| +| `schema_version` | number | Always `1` on v1 rows. Forward-compat gate. | +| `id` | number | Autoincrement primary key. Stable across exports. | +| `tool_name` | `"query"` \| `"search"` | Which MCP operation captured this row. | +| `query` | string | **Already PII-scrubbed** by `scrubPii` unless `eval.scrub_pii: false`. Emails / phones / SSN / Luhn-verified credit cards / JWTs / bearer tokens replaced with `[REDACTED]`. Max length 50KB (CHECK-enforced). | +| `retrieved_slugs` | string[] | Deduplicated slugs that came back in `SearchResult[]`. | +| `retrieved_chunk_ids` | number[] | Every chunk id in result order (duplicates preserved — one per hit). | +| `source_ids` | string[] | Distinct `sources.id` values across the result set (v0.18 multi-source). Empty for pre-v0.18 rows that lacked the column. | +| `expand_enabled` | boolean \| null | Whether the caller **requested** Haiku expansion. `null` for `search` (no expansion concept). | +| `detail` | `"low"` \| `"medium"` \| `"high"` \| null | Detail level the caller **requested**. `null` when omitted. | +| `detail_resolved` | `"low"` \| `"medium"` \| `"high"` \| null | What `hybridSearch` **actually used** after auto-detect. `null` when neither caller nor heuristic classified. | +| `vector_enabled` | boolean | True iff vector search actually ran. `false` when `OPENAI_API_KEY` was missing or the embed call failed. **Replay MUST respect this** — rows with `false` only exercised the keyword path. | +| `expansion_applied` | boolean | True iff Haiku expansion actually produced variants (not just "was requested"). | +| `latency_ms` | number | Wall-clock duration of the op handler (includes capture itself — negligible since it's fire-and-forget). | +| `remote` | boolean | `true` for MCP callers (untrusted), `false` for local CLI. Partitions "real agent traffic" from "operator probing." | +| `job_id` | number \| null | `OperationContext.jobId` when the caller was a subagent tool-bridge. Null for MCP + CLI. | +| `subagent_id` | number \| null | `OperationContext.subagentId` for subagent-owned runs. | +| `created_at` | string (ISO 8601) | UTC timestamp of insert. | + +## Ordering + determinism + +`listEvalCandidates` orders by `created_at DESC, id DESC`. Same- +millisecond inserts tie on `created_at`; `id DESC` is the stable +tiebreaker. Replay tools can consume rows in order and assume: +- no duplicate rows across calls with non-overlapping `--since` windows +- no missed rows across calls that chain `--since` windows (window end + of run 1 is the strict upper bound, not a soft cursor) + +## Schema versioning promise + +- **v1 (shipped v0.21.0)** — this document. All fields listed above. +- **Additive changes** increment gbrain minor version (v0.25.0, v0.23.0 + …) and ship with new optional fields. Consumers keyed on known fields + ignore unknown keys and keep working. +- **Breaking changes** (rename, type change, removal) increment + `schema_version` to 2. Consumers MUST branch on `schema_version` to + stay compatible. + +## `eval_capture_failures` — companion audit table + +Not exported by `gbrain eval export`. Surfaced via `gbrain doctor`: + +```sh +gbrain doctor # warns when failures in last 24h > 0 +``` + +Reason enum (stable): `db_down` | `rls_reject` | `check_violation` | +`scrubber_exception` | `other`. Cross-process visibility is the whole +point — `gbrain doctor` runs in its own process and reads the table +directly, so in-process counters wouldn't work. + +## Config + CONTRIBUTOR_MODE + +Capture is **off by default** as of v0.25.0 (was on for everyone in +earlier drafts). Two paths to turn it on: + +**Path A — env var (contributor opt-in, the common case):** + +```bash +export GBRAIN_CONTRIBUTOR_MODE=1 # in ~/.zshrc or ~/.bashrc +``` + +**Path B — explicit config (`~/.gbrain/config.json`, file-plane only):** + +```json +{ + "engine": "postgres", + "database_url": "...", + "eval": { + "capture": true, + "scrub_pii": true + } +} +``` + +Resolution order (most explicit wins): + +1. `eval.capture: true` in config → on +2. `eval.capture: false` in config → off (overrides CONTRIBUTOR_MODE=1) +3. `GBRAIN_CONTRIBUTOR_MODE === '1'` → on +4. otherwise → off + +`scrub_pii` defaults to `true` independent of capture. Set +`eval.scrub_pii: false` to preserve raw query text (only if you control +the brain's distribution). + +`gbrain config set eval.capture false` does **not** work — that +command writes the DB-plane config, and the MCP server reads the +file-plane. Edit the JSON directly or use the env var. diff --git a/llms-full.txt b/llms-full.txt index 2f33b2ddc..f632fedd0 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -50,6 +50,11 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont [`docs/guides/minions-fix.md`](./docs/guides/minions-fix.md), `gbrain doctor --fix`. - **Migrate:** [`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md), [`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations`. +- **Eval retrieval changes:** capture is off by default. To benchmark a + retrieval change against real captured queries, set + `GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson` + and `gbrain eval replay --against base.ndjson`. Full guide: + [`docs/eval-bench.md`](./docs/eval-bench.md). - **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map. [`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for single-fetch ingestion. @@ -135,7 +140,16 @@ strict behavior when unset. - `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator - `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`. - `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text. -- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison +- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. +- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows. +- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count. +- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow. +- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". +- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. +- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. +- `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined. +- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers. +- `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff - `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`. - `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic. @@ -298,6 +312,14 @@ Key commands added for Minions (job queue): - `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall. - `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only) +Key commands added in v0.25.0: +- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay. +- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s. +- `gbrain eval replay --against FILE.ndjson [--limit N] [--top-regressions K] [--json] [--verbose]` — contributor-facing dev loop. Reads a captured NDJSON snapshot, re-runs each `query` / `search` op against the current brain, computes mean set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. JSON mode (`schema_version: 1`) for CI gating; human mode prints a regression table sorted worst-first. Closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow. +- `gbrain doctor` gains an `eval_capture` check: reads `eval_capture_failures` for the last 24h, groups by reason, warns when non-zero. Cross-process visibility (doctor runs in a separate process from MCP). Pre-v31 brains get `Skipped (table unavailable)` — non-fatal. +- Config addition: `eval: { capture?: boolean, scrub_pii?: boolean }` in `~/.gbrain/config.json`. **File-plane only** — `gbrain config set` writes the DB plane and does NOT control capture. +- **`GBRAIN_CONTRIBUTOR_MODE=1` env var** is the contributor-facing toggle. Capture is **off by default** as of v0.25.0; production users get a quiet brain. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Documented in README.md, CONTRIBUTING.md, and `docs/eval-bench.md`. + Key commands added in v0.12.2: - `gbrain repair-jsonb [--dry-run] [--json]` — repair double-encoded JSONB rows left over from v0.12.0-and-earlier Postgres writes. Idempotent; PGLite no-ops. The `v0_12_2` migration runs this automatically on `gbrain upgrade`. @@ -1362,6 +1384,8 @@ The brain wires itself. Every page write extracts entity references and creates GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours. +**New in v0.25.0 — BrainBench-Real (session capture, contributor opt-in):** with `GBRAIN_CONTRIBUTOR_MODE=1` set in your shell, every real `query` + `search` call through MCP, CLI, or the subagent tool-bridge gets captured (PII-scrubbed) into an `eval_candidates` table. Snapshot with `gbrain eval export`, replay against your code change with `gbrain eval replay`. Three numbers come back: mean Jaccard@k between captured and current retrieved slugs, top-1 stability, and latency Δ. **Off by default** for production users — no surprise data accumulation. Walkthrough: [docs/eval-bench.md](docs/eval-bench.md). NDJSON wire format: [docs/eval-capture.md](docs/eval-capture.md). + > **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys. > **LLMs:** fetch [`llms.txt`](llms.txt) for the documentation map, or [`llms-full.txt`](llms-full.txt) for the same map with core docs inlined in one fetch. **Agents:** start with [`AGENTS.md`](AGENTS.md) (or [`CLAUDE.md`](CLAUDE.md) if you're Claude Code). @@ -2098,6 +2122,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration. +If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in. + PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`. ## License diff --git a/package.json b/package.json index b9faf7272..606033db0 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.24.0", + "version": "0.25.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", @@ -33,7 +33,7 @@ "build:schema": "bash scripts/build-schema.sh", "build:llms": "bun run scripts/build-llms.ts", "build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts", - "test": "scripts/check-privacy.sh && scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000", + "test": "scripts/check-privacy.sh && scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && bun run typecheck && bun test --timeout=60000", "check:wasm": "scripts/check-wasm-embedded.sh", "check:newlines": "scripts/check-trailing-newline.sh", "test:e2e": "bash scripts/run-e2e.sh", @@ -46,6 +46,7 @@ "check:jsonb": "scripts/check-jsonb-pattern.sh", "check:privacy": "scripts/check-privacy.sh", "check:progress": "scripts/check-progress-to-stdout.sh", + "check:exports-count": "scripts/check-exports-count.sh", "postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2", "prepublish:clawhub": "bun run build:all", "publish:clawhub": "clawhub package publish . --family bundle-plugin" diff --git a/scripts/check-exports-count.sh b/scripts/check-exports-count.sh new file mode 100755 index 000000000..160c4bf14 --- /dev/null +++ b/scripts/check-exports-count.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# CI guard: the public exports surface never shrinks silently (v0.21.0). +# +# Precedent: scripts/check-jsonb-pattern.sh + check-progress-to-stdout.sh +# are grep-based structural guards wired into `bun run test`. This one +# counts the entries in package.json "exports" and fails when the count +# drops below the v0.21.0 baseline (17 entries). +# +# Policy (from CLAUDE.md): +# "Removing any of these is a breaking change going forward." +# +# If you're legitimately removing a public export: bump gbrain's minor +# version, note the removal in CHANGELOG.md under a "Breaking changes" +# bullet, then bump EXPECTED_COUNT below. Anything else is a regression. +# +# Adding a new export: update EXPECTED_COUNT to match AND extend the +# EXPECTED_EXPORTS list in test/public-exports.test.ts so the runtime +# contract test pins the canary symbol. + +set -euo pipefail + +EXPECTED_COUNT=17 + +# Count top-level keys in the exports object. `node -e` parses JSON +# reliably without needing jq (which isn't in every CI environment). +ACTUAL=$(node -e " + const pkg = require('./package.json'); + console.log(Object.keys(pkg.exports || {}).length); +") + +if [ "$ACTUAL" -lt "$EXPECTED_COUNT" ]; then + echo "❌ public-exports guard: package.json exports shrank from $EXPECTED_COUNT to $ACTUAL" + echo " Removing a public export is a breaking change (see CLAUDE.md)." + echo " If intentional: bump gbrain minor version + update EXPECTED_COUNT in" + echo " scripts/check-exports-count.sh and EXPECTED_EXPORTS in" + echo " test/public-exports.test.ts, AND add a CHANGELOG 'Breaking changes' bullet." + exit 1 +fi + +if [ "$ACTUAL" -gt "$EXPECTED_COUNT" ]; then + echo "⚠️ public-exports guard: package.json exports grew from $EXPECTED_COUNT to $ACTUAL" + echo " Additive public API change. Update EXPECTED_COUNT in this script + the" + echo " EXPECTED_EXPORTS list in test/public-exports.test.ts to lock the new" + echo " canary symbols." + exit 1 +fi + +echo "✓ public-exports guard: $ACTUAL entries (matches baseline $EXPECTED_COUNT)" diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 612763664..67bb32429 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -719,6 +719,58 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo fmHb(); } + // 11a-bis. Eval-capture health (v0.25.0). Capture is a fire-and-forget + // side-effect that logs failures to a persistent table so this check + // can see drops cross-process (the MCP server captures; `gbrain doctor` + // runs in a separate process). Counts failures in the last 24h and + // warns when non-zero. Pre-v31 brains: the table doesn't exist yet; + // swallow the error and report skipped. + progress.heartbeat('eval_capture'); + try { + const since = new Date(Date.now() - 24 * 3600 * 1000); + const failures = await engine.listEvalCaptureFailures({ since }); + if (failures.length === 0) { + checks.push({ name: 'eval_capture', status: 'ok', message: 'No capture failures in the last 24h' }); + } else { + const byReason = new Map(); + for (const f of failures) { + byReason.set(f.reason, (byReason.get(f.reason) ?? 0) + 1); + } + const breakdown = [...byReason.entries()] + .sort((a, b) => b[1] - a[1]) + .map(([r, n]) => `${n} ${r}`) + .join(', '); + checks.push({ + name: 'eval_capture', + status: 'warn', + message: `${failures.length} capture failure(s) in the last 24h (${breakdown}). ` + + `If you care about replay fidelity, investigate. If not, set eval.capture: false ` + + `in ~/.gbrain/config.json to silence.`, + }); + } + } catch (err) { + // Distinguish "table doesn't exist yet" (pre-v31, ok skip) from real + // problems like RLS denying SELECT — the latter masks the very condition + // this check is supposed to surface (capture INSERTs almost certainly + // also fail). + const code = (err as { code?: string } | null)?.code; + if (code === '42P01') { + checks.push({ name: 'eval_capture', status: 'ok', message: 'Skipped (eval_capture_failures table unavailable — apply migrations or upgrade)' }); + } else if (code === '42501') { + checks.push({ + name: 'eval_capture', + status: 'warn', + message: 'RLS denies SELECT on eval_capture_failures. Capture INSERTs are almost certainly failing too. Run as a role with BYPASSRLS or grant SELECT on this table.', + }); + } else { + checks.push({ + name: 'eval_capture', + status: 'warn', + message: `Could not read eval_capture_failures: ${(err as Error)?.message ?? String(err)}`, + }); + } + } + // 11b. Queue health (v0.19.1 queue-resilience wave). // Postgres-only because PGLite has no multi-process worker surface. Two // subchecks, both cheap (single SELECT each, status-index-covered): diff --git a/src/commands/eval-export.ts b/src/commands/eval-export.ts new file mode 100644 index 000000000..adc256d48 --- /dev/null +++ b/src/commands/eval-export.ts @@ -0,0 +1,159 @@ +/** + * gbrain eval export — stream captured eval_candidates rows as NDJSON (v0.21.0). + * + * Consumer: sibling gbrain-evals repo, which imports the NDJSON as a + * BrainBench-Real fixture alongside the fictional amara-life corpus. + * + * Output contract (stable from v0.21.0 — schema_version:1 on every row): + * { "schema_version": 1, "id": N, "tool_name": "query"|"search", ... }\n + * + * The schema_version prefix lets gbrain-evals detect format drift and + * warn on unknown versions instead of silently misparsing. + * + * Usage: + * gbrain eval export [--since 7d] [--limit N] [--tool query|search] > rows.ndjson + */ + +import type { BrainEngine } from '../core/engine.ts'; +import type { EvalCandidate } from '../core/types.ts'; + +const SCHEMA_VERSION = 1; + +interface ExportOpts { + help?: boolean; + since?: Date; + limit?: number; + tool?: 'query' | 'search'; +} + +function parseDurationToMs(s: string): number | null { + // Accepts "30d", "7d", "1h", "90m", "3600s". Same shape as gbrain eval prune + jobs prune. + const m = s.match(/^(\d+)\s*(ms|s|m|h|d)$/); + if (!m) return null; + const n = parseInt(m[1]!, 10); + const unit = m[2]!; + const mults: Record = { + ms: 1, + s: 1000, + m: 60_000, + h: 3_600_000, + d: 86_400_000, + }; + return n * mults[unit]!; +} + +function parseArgs(args: string[]): ExportOpts { + const opts: ExportOpts = {}; + for (let i = 0; i < args.length; i++) { + const arg = args[i]!; + const next = args[i + 1]; + switch (arg) { + case '--help': + case '-h': + opts.help = true; + break; + case '--since': { + if (!next) break; + const ms = parseDurationToMs(next); + if (ms !== null) { + opts.since = new Date(Date.now() - ms); + } else { + console.error(`Invalid --since value: ${next} (use like 7d, 1h, 30m)`); + process.exit(1); + } + i++; + break; + } + case '--limit': + if (next) opts.limit = parseInt(next, 10); + i++; + break; + case '--tool': + if (next === 'query' || next === 'search') { + opts.tool = next; + } else if (next) { + console.error(`Invalid --tool value: ${next} (use 'query' or 'search')`); + process.exit(1); + } + i++; + break; + } + } + return opts; +} + +function printHelp(): void { + console.error(`gbrain eval export — emit captured eval_candidates as NDJSON to stdout + +USAGE: + gbrain eval export [--since DUR] [--limit N] [--tool query|search] + +FLAGS: + --since DUR Only rows created within DUR (e.g. 7d, 1h, 30m). Default: all. + --limit N Cap rows returned. Default: 1000. Max: 100000. + --tool X Filter to a specific tool ('query' or 'search'). Default: both. + --help, -h Show this help. + +OUTPUT: + One JSON object per line on stdout. Every row begins with + "schema_version": 1 so downstream consumers (gbrain-evals) can + detect format changes. + +EXAMPLES: + gbrain eval export > rows.ndjson + gbrain eval export --since 7d --tool query | jq '.query' + gbrain eval export --limit 100 | head +`); +} + +export async function runEvalExport(engine: BrainEngine, args: string[]): Promise { + const opts = parseArgs(args); + if (opts.help) { + printHelp(); + return; + } + + // Progress to stderr (stdout is reserved for NDJSON data). + const { createProgress, startHeartbeat } = await import('../core/progress.ts'); + const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts'); + const progress = createProgress(cliOptsToProgressOptions(getCliOptions())); + + progress.start('eval.export'); + const stopHeartbeat = startHeartbeat(progress, 'reading eval_candidates'); + let rows: EvalCandidate[]; + try { + rows = await engine.listEvalCandidates({ + since: opts.since, + limit: opts.limit, + tool: opts.tool, + }); + } finally { + stopHeartbeat(); + } + + // Emit NDJSON to stdout. EPIPE-safe: if the downstream process + // (e.g. `| head`) closes its end early, we abort cleanly without a + // stack trace. Matches src/core/progress.ts EPIPE handling precedent. + const stdout = process.stdout; + const abortOnEpipe = (err: NodeJS.ErrnoException) => { + if (err.code === 'EPIPE') process.exit(0); + }; + stdout.on('error', abortOnEpipe); + + let written = 0; + for (const row of rows) { + // Prefix every line with schema_version:1 so gbrain-evals can detect + // schema drift before parsing the rest of the fields. + const line = JSON.stringify({ schema_version: SCHEMA_VERSION, ...row }); + if (!stdout.write(line + '\n')) { + // Backpressure: wait for drain before continuing. + await new Promise(r => stdout.once('drain', r)); + } + written++; + progress.tick(); + } + + stdout.off('error', abortOnEpipe); + progress.finish(); + console.error(`exported ${written} row(s)`); +} diff --git a/src/commands/eval-prune.ts b/src/commands/eval-prune.ts new file mode 100644 index 000000000..e9da01118 --- /dev/null +++ b/src/commands/eval-prune.ts @@ -0,0 +1,115 @@ +/** + * gbrain eval prune — delete old eval_candidates rows (v0.21.0). + * + * Retention is unlimited by default (matches ingest_log precedent). + * This command is the explicit cleanup; pairs with `gbrain eval export` + * (snapshot first, then prune if you want to reset). + * + * Usage: + * gbrain eval prune --older-than 30d + * gbrain eval prune --older-than 1h --dry-run + */ + +import type { BrainEngine } from '../core/engine.ts'; + +interface PruneOpts { + help?: boolean; + olderThanMs?: number; + dryRun?: boolean; +} + +function parseDurationToMs(s: string): number | null { + const m = s.match(/^(\d+)\s*(ms|s|m|h|d)$/); + if (!m) return null; + const n = parseInt(m[1]!, 10); + const unit = m[2]!; + const mults: Record = { + ms: 1, + s: 1000, + m: 60_000, + h: 3_600_000, + d: 86_400_000, + }; + return n * mults[unit]!; +} + +function parseArgs(args: string[]): PruneOpts { + const opts: PruneOpts = {}; + for (let i = 0; i < args.length; i++) { + const arg = args[i]!; + const next = args[i + 1]; + switch (arg) { + case '--help': + case '-h': + opts.help = true; + break; + case '--older-than': { + if (!next) break; + const ms = parseDurationToMs(next); + if (ms === null) { + console.error(`Invalid --older-than value: ${next} (use like 30d, 1h, 90m)`); + process.exit(1); + } + opts.olderThanMs = ms; + i++; + break; + } + case '--dry-run': + opts.dryRun = true; + break; + } + } + return opts; +} + +function printHelp(): void { + console.error(`gbrain eval prune — delete old eval_candidates rows + +USAGE: + gbrain eval prune --older-than DUR [--dry-run] + +FLAGS: + --older-than DUR Delete rows created before now() - DUR (e.g. 30d, 7d, 1h). + Required — this command never deletes without a window. + --dry-run Report what would be deleted; don't actually delete. + --help, -h Show this help. + +EXAMPLES: + gbrain eval prune --older-than 30d + gbrain eval prune --older-than 90d --dry-run +`); +} + +export async function runEvalPrune(engine: BrainEngine, args: string[]): Promise { + const opts = parseArgs(args); + if (opts.help) { + printHelp(); + return; + } + if (!opts.olderThanMs) { + console.error('Error: --older-than is required\n'); + printHelp(); + process.exit(1); + } + + const cutoff = new Date(Date.now() - opts.olderThanMs); + + if (opts.dryRun) { + // Snapshot-count the rows we *would* delete — the list call caps at + // 100k which matches the export ceiling, so larger windows get a + // floor-estimate that's still useful signal. + const rows = await engine.listEvalCandidates({ + since: new Date(0), + limit: 100_000, + }); + const wouldDelete = rows.filter(r => new Date(r.created_at) < cutoff).length; + console.log(`[dry-run] would delete ${wouldDelete} row(s) created before ${cutoff.toISOString()}`); + if (rows.length === 100_000) { + console.log('[dry-run] (count may be undercounted — the scan hit the 100k row limit)'); + } + return; + } + + const deleted = await engine.deleteEvalCandidatesBefore(cutoff); + console.log(`deleted ${deleted} row(s) created before ${cutoff.toISOString()}`); +} diff --git a/src/commands/eval-replay.ts b/src/commands/eval-replay.ts new file mode 100644 index 000000000..15941560d --- /dev/null +++ b/src/commands/eval-replay.ts @@ -0,0 +1,424 @@ +/** + * gbrain eval replay — replay captured eval_candidates against current brain (v0.25.0). + * + * The contributor-facing half of BrainBench-Real: + * + * 1. capture some real traffic (default-on, lands in eval_candidates) + * 2. snapshot it (gbrain eval export --since 7d > baseline.ndjson) + * 3. make a code change (tune RRF_K, edit hybrid.ts, swap an embed model) + * 4. replay against the snapshot (gbrain eval replay --against baseline.ndjson) + * + * Outputs three numbers a contributor can read at a glance: + * + * - mean Jaccard@k between captured retrieved_slugs and current run's slugs + * - top-1 stability rate (was the #1 result the same?) + * - mean latency delta (current - captured), positive = slower now + * + * Best-effort by design. Replay is NOT pure — your brain has more pages than + * when the capture was taken, embeddings may have drifted, and the OPENAI key + * may be different. The metrics describe "did this change hurt retrieval on + * the queries you actually serve" not "do these match the baseline byte for + * byte." Use it before merging anything that touches src/core/search/ or the + * query/search op handlers. + * + * Usage: + * gbrain eval replay --against captured.ndjson [--limit N] [--json] + * [--top-regressions K] [--verbose] + */ + +import { readFileSync, existsSync } from 'fs'; +import type { BrainEngine } from '../core/engine.ts'; +import type { SearchResult } from '../core/types.ts'; +import { hybridSearch } from '../core/search/hybrid.ts'; + +interface ReplayOpts { + help?: boolean; + against?: string; + limit?: number; + json?: boolean; + verbose?: boolean; + topRegressions?: number; +} + +interface RowResult { + /** Captured row's id, for back-referencing into the source NDJSON. */ + id: number; + tool_name: 'query' | 'search'; + query: string; + /** Set-overlap score in [0, 1]. 1.0 = identical retrieved set. */ + jaccard: number; + /** True when current top result matches captured top result. */ + top1Match: boolean; + /** Captured retrieved_slugs (as-is from NDJSON). */ + captured_slugs: string[]; + /** Current run's slugs (deduped, in result order). */ + current_slugs: string[]; + /** Wall-clock latency (ms) of the current re-run. */ + current_latency_ms: number; + /** latency delta = current - captured. Positive = slower now. */ + latency_delta_ms: number; + /** True if the row was skipped (e.g. captured query was empty). */ + skipped?: boolean; + /** Reason the row was skipped, if any. */ + skip_reason?: string; + /** True if the row threw during replay; current_slugs is empty. */ + errored?: boolean; + error_message?: string; +} + +function parseArgs(args: string[]): ReplayOpts { + const opts: ReplayOpts = {}; + for (let i = 0; i < args.length; i++) { + const arg = args[i]!; + const next = args[i + 1]; + switch (arg) { + case '--help': + case '-h': + opts.help = true; + break; + case '--against': + if (!next) break; + opts.against = next; + i++; + break; + case '--limit': + if (!next) break; + opts.limit = parseInt(next, 10); + i++; + break; + case '--json': + opts.json = true; + break; + case '--verbose': + opts.verbose = true; + break; + case '--top-regressions': + if (!next) break; + opts.topRegressions = parseInt(next, 10); + i++; + break; + } + } + return opts; +} + +function printHelp(): void { + console.error(`gbrain eval replay — replay captured queries against current brain + +USAGE: + gbrain eval replay --against FILE.ndjson [flags] + +FLAGS: + --against FILE NDJSON file from \`gbrain eval export\` (required). + --limit N Replay at most N rows (default: replay all). + Each row hits OpenAI once for query embedding — + cap aggressively when iterating locally. + --top-regressions K Print the K rows with the worst Jaccard scores. + Default 5 in human mode, 0 in --json. + --json Emit one JSON object on stdout instead of a table. + Stable shape for CI consumption. + --verbose Include every row's per-row diff (large output). + --help, -h Show this help. + +OUTPUT (human mode): + Replayed N captured queries (M skipped, K errored) + Mean Jaccard@k: 0.873 + Top-1 stability: 87% (N=87 / 100) + Mean latency Δ: +12ms (current slower) + + Top 5 regressions: + 0.20 "find every reference to widget-co" captured=12 current=3 + ... + +EXIT CODE: + 0 — replay completed (regardless of regression magnitude). + 1 — invalid args, --against not found, or NDJSON parse failure. + +NOTES: + Replay is best-effort. Your brain has more pages than when the snapshot + was taken; embeddings may have drifted; OPENAI_API_KEY may be different. + Use the metrics to spot regressions on REAL queries, not as a hash check. +`); +} + +interface CapturedRow { + schema_version: number; + id: number; + tool_name: 'query' | 'search'; + query: string; + retrieved_slugs: string[]; + retrieved_chunk_ids?: number[]; + source_ids?: string[]; + expand_enabled?: boolean | null; + detail?: 'low' | 'medium' | 'high' | null; + detail_resolved?: 'low' | 'medium' | 'high' | null; + vector_enabled?: boolean; + expansion_applied?: boolean; + latency_ms: number; + remote?: boolean; + job_id?: number | null; + subagent_id?: number | null; + created_at?: string; +} + +/** + * Parse NDJSON. One object per non-blank line. Single bad line throws — it's + * a corrupt export and silently dropping rows would mask real bugs. + */ +function parseNdjson(content: string): CapturedRow[] { + const lines = content.split('\n'); + const rows: CapturedRow[] = []; + for (let i = 0; i < lines.length; i++) { + const line = lines[i]!.trim(); + if (!line) continue; + let row: CapturedRow; + try { + row = JSON.parse(line); + } catch (err) { + throw new Error(`NDJSON parse error on line ${i + 1}: ${(err as Error).message}`); + } + if (typeof row.schema_version !== 'number') { + throw new Error(`Line ${i + 1} missing schema_version — not from \`gbrain eval export\`?`); + } + if (row.schema_version !== 1) { + throw new Error( + `Line ${i + 1} has schema_version=${row.schema_version}; this replay only supports v1. ` + + `Upgrade gbrain or re-export.`, + ); + } + rows.push(row); + } + return rows; +} + +/** + * Set-Jaccard between two slug arrays. Order ignored, dupes collapsed. + * Both empty → 1.0 (identical empty sets, no information lost). + */ +function jaccardSlugs(a: string[], b: string[]): number { + const setA = new Set(a); + const setB = new Set(b); + if (setA.size === 0 && setB.size === 0) return 1.0; + let intersection = 0; + for (const s of setA) if (setB.has(s)) intersection++; + const union = setA.size + setB.size - intersection; + return union === 0 ? 1.0 : intersection / union; +} + +async function replayRow(engine: BrainEngine, row: CapturedRow): Promise { + const captured_slugs = row.retrieved_slugs ?? []; + const startedAt = Date.now(); + + // Default replay limit matches hybridSearch's default (20). + const limit = Math.max(captured_slugs.length, 20); + + // search → bare keyword path. query → hybrid path (vector + keyword + RRF). + // detail and expansion are threaded in from the captured row so the same + // logic runs that produced the original retrieval. + let current: SearchResult[]; + try { + if (row.tool_name === 'search') { + const dedupedRaw = await engine.searchKeyword(row.query, { limit }); + current = dedupedRaw; + } else { + current = await hybridSearch(engine, row.query, { + limit, + detail: row.detail ?? undefined, + expansion: row.expand_enabled ?? false, + }); + } + } catch (err) { + return { + id: row.id, + tool_name: row.tool_name, + query: row.query, + jaccard: 0, + top1Match: false, + captured_slugs, + current_slugs: [], + current_latency_ms: Date.now() - startedAt, + latency_delta_ms: Date.now() - startedAt - row.latency_ms, + errored: true, + error_message: (err as Error).message ?? String(err), + }; + } + + const current_latency_ms = Date.now() - startedAt; + // Dedup slugs while preserving order — same convention as search results. + const seen = new Set(); + const current_slugs: string[] = []; + for (const r of current) { + if (!seen.has(r.slug)) { + seen.add(r.slug); + current_slugs.push(r.slug); + } + } + + return { + id: row.id, + tool_name: row.tool_name, + query: row.query, + jaccard: jaccardSlugs(captured_slugs, current_slugs), + top1Match: captured_slugs[0] !== undefined && current_slugs[0] === captured_slugs[0], + captured_slugs, + current_slugs, + current_latency_ms, + latency_delta_ms: current_latency_ms - row.latency_ms, + }; +} + +interface ReplaySummary { + rows_total: number; + rows_replayed: number; + rows_skipped: number; + rows_errored: number; + /** Mean Jaccard across non-skipped, non-errored rows. */ + mean_jaccard: number; + top1_stability_rate: number; + mean_latency_delta_ms: number; + /** Rows where current latency is more than 2x captured (regression alarm). */ + rows_over_2x_latency: number; +} + +function summarize(results: RowResult[]): ReplaySummary { + const eligible = results.filter(r => !r.skipped && !r.errored); + const meanJaccard = eligible.length === 0 + ? 0 + : eligible.reduce((a, r) => a + r.jaccard, 0) / eligible.length; + const top1Rate = eligible.length === 0 + ? 0 + : eligible.filter(r => r.top1Match).length / eligible.length; + const meanLatencyDelta = eligible.length === 0 + ? 0 + : eligible.reduce((a, r) => a + r.latency_delta_ms, 0) / eligible.length; + const over2x = eligible.filter(r => { + const captured = results.find(x => x.id === r.id); + return captured && captured.current_latency_ms > 2 * (captured.current_latency_ms - captured.latency_delta_ms); + }).length; + + return { + rows_total: results.length, + rows_replayed: eligible.length, + rows_skipped: results.filter(r => r.skipped).length, + rows_errored: results.filter(r => r.errored).length, + mean_jaccard: meanJaccard, + top1_stability_rate: top1Rate, + mean_latency_delta_ms: meanLatencyDelta, + rows_over_2x_latency: over2x, + }; +} + +function printHumanSummary(summary: ReplaySummary, results: RowResult[], topRegressions: number): void { + const total = summary.rows_total; + const eligible = summary.rows_replayed; + console.log(`Replayed ${eligible} of ${total} captured queries (${summary.rows_skipped} skipped, ${summary.rows_errored} errored)`); + console.log(`Mean Jaccard@k: ${summary.mean_jaccard.toFixed(3)}`); + console.log(`Top-1 stability: ${(summary.top1_stability_rate * 100).toFixed(1)}%`); + const sign = summary.mean_latency_delta_ms >= 0 ? '+' : ''; + console.log(`Mean latency Δ: ${sign}${summary.mean_latency_delta_ms.toFixed(0)}ms (current vs captured)`); + if (summary.rows_over_2x_latency > 0) { + console.log(`⚠ ${summary.rows_over_2x_latency} row(s) ran more than 2× slower than captured`); + } + + if (topRegressions > 0) { + const sorted = [...results] + .filter(r => !r.skipped && !r.errored) + .sort((a, b) => a.jaccard - b.jaccard) + .slice(0, topRegressions); + if (sorted.length > 0 && sorted[0]!.jaccard < 1.0) { + console.log(`\nTop ${sorted.length} regression(s):`); + for (const r of sorted) { + const truncQuery = r.query.length > 60 ? r.query.slice(0, 57) + '...' : r.query; + console.log( + ` jaccard=${r.jaccard.toFixed(2)} captured=${r.captured_slugs.length} current=${r.current_slugs.length} ` + + `"${truncQuery}"`, + ); + } + } + } + + if (summary.rows_errored > 0) { + const errors = results.filter(r => r.errored).slice(0, 3); + console.log(`\n${summary.rows_errored} row(s) errored. First ${errors.length}:`); + for (const r of errors) { + const truncQuery = r.query.length > 60 ? r.query.slice(0, 57) + '...' : r.query; + console.log(` id=${r.id} "${truncQuery}" ${r.error_message ?? ''}`); + } + } +} + +export async function runEvalReplay(engine: BrainEngine, args: string[]): Promise { + const opts = parseArgs(args); + if (opts.help) { + printHelp(); + return; + } + if (!opts.against) { + console.error('Error: --against FILE.ndjson is required\n'); + printHelp(); + process.exit(1); + } + if (!existsSync(opts.against)) { + console.error(`Error: file not found: ${opts.against}`); + process.exit(1); + } + + let rows: CapturedRow[]; + try { + const content = readFileSync(opts.against, 'utf-8'); + rows = parseNdjson(content); + } catch (err) { + console.error(`Error: ${(err as Error).message}`); + process.exit(1); + } + + if (rows.length === 0) { + console.error(`Error: ${opts.against} is empty (no NDJSON rows)`); + process.exit(1); + } + + const capped = opts.limit && opts.limit > 0 ? rows.slice(0, opts.limit) : rows; + if (!opts.json) { + console.error( + `Replaying ${capped.length}${capped.length < rows.length ? ` of ${rows.length}` : ''} captured queries…`, + ); + } + + const results: RowResult[] = []; + for (const row of capped) { + if (!row.query || row.query.length === 0) { + results.push({ + id: row.id, + tool_name: row.tool_name, + query: row.query ?? '', + jaccard: 0, + top1Match: false, + captured_slugs: row.retrieved_slugs ?? [], + current_slugs: [], + current_latency_ms: 0, + latency_delta_ms: 0, + skipped: true, + skip_reason: 'empty query', + }); + continue; + } + const r = await replayRow(engine, row); + results.push(r); + if (!opts.json && results.length % 25 === 0) { + process.stderr.write(` ...${results.length}/${capped.length}\n`); + } + } + + const summary = summarize(results); + if (opts.json) { + console.log(JSON.stringify({ + schema_version: 1, + summary, + results: opts.verbose ? results : undefined, + }, null, 2)); + return; + } + + const topN = opts.topRegressions ?? 5; + printHumanSummary(summary, results, topN); +} diff --git a/src/commands/eval.ts b/src/commands/eval.ts index 96df61d25..18931965c 100644 --- a/src/commands/eval.ts +++ b/src/commands/eval.ts @@ -21,6 +21,23 @@ import { } from '../core/search/eval.ts'; export async function runEvalCommand(engine: BrainEngine, args: string[]): Promise { + // v0.25.0 — sub-subcommand dispatch. Bare `gbrain eval --qrels ...` + // falls through to the legacy IR-metrics flow so existing callers + // don't break. + const sub = args[0]; + if (sub === 'export') { + const { runEvalExport } = await import('./eval-export.ts'); + return runEvalExport(engine, args.slice(1)); + } + if (sub === 'prune') { + const { runEvalPrune } = await import('./eval-prune.ts'); + return runEvalPrune(engine, args.slice(1)); + } + if (sub === 'replay') { + const { runEvalReplay } = await import('./eval-replay.ts'); + return runEvalReplay(engine, args.slice(1)); + } + const opts = parseArgs(args); if (opts.help) { diff --git a/src/core/config.ts b/src/core/config.ts index 9979f3f19..d2c8fe5f0 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -38,6 +38,18 @@ export interface GBrainConfig { * validates the shape at runtime. */ storage?: unknown; + /** + * v0.25.0 — session capture settings. Read via file-plane `loadConfig()` + * at process boot (NOT `gbrain config set` which writes the DB plane — + * those are different stores). Edit `~/.gbrain/config.json` directly. + * All fields default to ON — capture and scrubbing both opt-out. + */ + eval?: { + /** false disables capture entirely. Defaults to true. */ + capture?: boolean; + /** false disables PII scrubbing before insert. Defaults to true. */ + scrub_pii?: boolean; + }; } /** diff --git a/src/core/engine.ts b/src/core/engine.ts index 3c47665d4..29de60529 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -10,6 +10,8 @@ import type { IngestLogEntry, IngestLogInput, EngineConfig, CodeEdgeInput, CodeEdgeResult, + EvalCandidate, EvalCandidateInput, + EvalCaptureFailure, EvalCaptureFailureReason, } from './types.ts'; /** Input row for addLinksBatch. Optional fields default to '' (matches NOT NULL DDL). */ @@ -363,4 +365,20 @@ export interface BrainEngine { * prefer searchKeyword (external contract: page-grain best-chunk-per-page). */ searchKeywordChunks(query: string, opts?: SearchOpts): Promise; + + // Eval capture (v0.25.0 — BrainBench-Real substrate). + // Captured at the op-layer wrapper in src/core/operations.ts; reads via + // `gbrain eval export` (NDJSON) for sibling gbrain-evals consumption. + // Adding these to BrainEngine is a breaking-interface change for third- + // party engine implementers — this is why v0.25.0 is a minor bump. + /** Insert a captured candidate. Returns the new row id. Best-effort: callers swallow failures and route them through `logEvalCaptureFailure`. */ + logEvalCandidate(input: EvalCandidateInput): Promise; + /** Read candidates by time window / limit / tool filter. Used by `gbrain eval export`. */ + listEvalCandidates(filter?: { since?: Date; limit?: number; tool?: 'query' | 'search' }): Promise; + /** Delete candidates created before `date`. Returns rows deleted. Used by `gbrain eval prune`. */ + deleteEvalCandidatesBefore(date: Date): Promise; + /** Log a capture failure so `gbrain doctor` can surface drops cross-process. Best-effort; symmetric with logEvalCandidate (failure-of-failure is lost). */ + logEvalCaptureFailure(reason: EvalCaptureFailureReason): Promise; + /** Read capture failures within an optional time window. Used by `gbrain doctor`. */ + listEvalCaptureFailures(filter?: { since?: Date }): Promise; } diff --git a/src/core/eval-capture-scrub.ts b/src/core/eval-capture-scrub.ts new file mode 100644 index 000000000..9a2a81709 --- /dev/null +++ b/src/core/eval-capture-scrub.ts @@ -0,0 +1,104 @@ +/** + * PII scrubber for captured query text (v0.21.0). + * + * Runs before INSERT into `eval_candidates`. Capture is on by default, + * so plaintext PII sitting in a DB column is a privacy footgun the first + * time someone exports or shares a brain dump. Six regex families cover + * the obvious cases: + * + * 1. Email addresses + * 2. Phone numbers (US + international) + * 3. US Social Security numbers (XXX-XX-XXXX shape, with year-like false-positive guard) + * 4. Credit card numbers (13–19 digits with Luhn verification — blocks false positives) + * 5. JWT-shaped tokens (three base64url segments joined by '.') + * 6. Bearer tokens (Authorization: Bearer ) + * + * 80% of the real risk without a dependency on an NER model. If regex v1 + * proves insufficient we can layer a model-based scrubber later. + * + * Pure function, zero deps. Safe to call on arbitrary input. Adversarial + * regex input (catastrophic backtracking) is contained by the + * possessive-quantifier-free patterns below and by the outer try/catch in + * captureEvalCandidate (see src/core/eval-capture.ts), not by this + * module itself. + */ + +const REDACTED = '[REDACTED]'; + +// Emails: RFC-5322-adjacent. Keeps the host so replay debug can say "an +// email was redacted" without leaking the local-part. +const EMAIL_RE = /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g; + +// Phones: US (###-###-####, (###) ###-####, ##########) and E.164 (+country). +// Reject short strings of 10 digits with no separators/prefix to limit +// false positives on order numbers and other generic long integers. +const PHONE_RE = + /(? 9) return false; + if (i % 2 === parity) { + n *= 2; + if (n > 9) n -= 9; + } + sum += n; + } + return sum % 10 === 0; +} + +/** + * Redact obvious PII from a captured query string. + * + * Order of operations matters: email first so "user@example.com" doesn't + * get caught by phone/CC regex fragments. CC last since Luhn is expensive + * and irrelevant to everything else. + */ +export function scrubPii(input: string): string { + if (!input) return input; + let out = input; + + // 1. Emails + out = out.replace(EMAIL_RE, REDACTED); + + // 2. Phones (before SSN so +1-555-XX doesn't look like part of a dashes-only SSN) + out = out.replace(PHONE_RE, REDACTED); + + // 3. SSN (after phones) + out = out.replace(SSN_RE, REDACTED); + + // 4. JWT (distinctive prefix, safe to run anywhere in the pipeline) + out = out.replace(JWT_RE, REDACTED); + + // 5. Bearer tokens + out = out.replace(BEARER_RE, `Bearer ${REDACTED}`); + + // 6. Credit cards: every candidate must pass Luhn to be replaced. + out = out.replace(CC_RE, (match) => { + const digitsOnly = match.replace(/\D/g, ''); + if (digitsOnly.length < 13 || digitsOnly.length > 19) return match; + return luhnOk(digitsOnly) ? REDACTED : match; + }); + + return out; +} diff --git a/src/core/eval-capture.ts b/src/core/eval-capture.ts new file mode 100644 index 000000000..ff8807672 --- /dev/null +++ b/src/core/eval-capture.ts @@ -0,0 +1,208 @@ +/** + * Op-layer capture wrapper (v0.21.0). + * + * Catches MCP + CLI + subagent tool-bridge traffic from a single site — + * the `query` and `search` op handlers in src/core/operations.ts decorate + * themselves with `captureEvalCandidate` before they run. Post-codex O1+O2 + * design: the MCP server used to be the capture site, but subagent tool + * calls dispatch straight to op.handler via brain-allowlist.ts and would + * have been invisible. Moving the hook to the op layer covers every caller. + * + * Best-effort, not await'd by the caller. Every failure is routed through + * `engine.logEvalCaptureFailure(reason)` so `gbrain doctor` can see drops + * cross-process (in-process counters would be invisible from doctor's + * separate process). + * + * Data-flow (happy path): + * + * op.handler() ──▶ {results, meta} + * │ + * (fire-and-forget from caller) + * ▼ + * outer try / inner .catch + * │ + * ▼ + * scrubPii(query) + * │ + * ▼ + * buildEvalCandidateInput(...) + * │ + * ▼ + * engine.logEvalCandidate(input) + * + * Every error path — scrub throw, build throw, INSERT reject — lands at + * `logEvalCaptureFailure(reason)` with the right reason tag. + */ + +import type { BrainEngine } from './engine.ts'; +import type { + EvalCandidateInput, + EvalCaptureFailureReason, + HybridSearchMeta, + SearchResult, +} from './types.ts'; +import type { GBrainConfig } from './config.ts'; +import { scrubPii } from './eval-capture-scrub.ts'; + +// HybridSearchMeta is canonical in src/core/types.ts and exported via the +// public `gbrain/types` subpath. Surfaced from hybridSearch via the +// optional onMeta callback in HybridSearchOpts (Lane 1C). +export type { HybridSearchMeta }; + +/** Context needed to build a capture row. Bundled so op handlers don't thread 8 args. */ +export interface CaptureContext { + /** 'query' or 'search'; captured only for these two ops. */ + tool_name: 'query' | 'search'; + /** Pre-scrub query text. scrubPii runs INSIDE buildEvalCandidateInput when scrub_pii !== false. */ + query: string; + /** Result set from the op handler. */ + results: SearchResult[]; + /** Side-channel metadata from hybridSearch. For 'search' ops, pass vector_enabled:false, others null/false. */ + meta: HybridSearchMeta; + /** How long the underlying op took, in milliseconds. */ + latency_ms: number; + /** OperationContext.remote — true for MCP callers, false for local CLI. */ + remote: boolean; + /** The `expand` flag as requested by the caller (query-only; null for search). */ + expand_enabled: boolean | null; + /** The `detail` flag as requested (query-only). */ + detail: 'low' | 'medium' | 'high' | null; + /** OperationContext.jobId if present. */ + job_id: number | null; + /** OperationContext.subagentId if present. */ + subagent_id: number | null; +} + +/** + * Build the insert row for `eval_candidates` from a capture context. + * + * Runs the PII scrubber on `query` unless `scrub_pii === false`. Pure + * function: throws only if the scrubber throws (caller wraps in try/catch). + * + * Exported for testing and for CLI backfill paths; the hot-path caller is + * `captureEvalCandidate` below. + */ +export function buildEvalCandidateInput( + ctx: CaptureContext, + opts: { scrub_pii?: boolean } = {}, +): EvalCandidateInput { + const shouldScrub = opts.scrub_pii !== false; + const query = shouldScrub ? scrubPii(ctx.query) : ctx.query; + + // Deduplicate + preserve order for slug + source_id extraction. + // Both arrays are small (hybridSearch clamps at 100 results) so the + // O(n) Set-based dedup is fine. + const slugsSet = new Set(); + const sourceSet = new Set(); + const chunkIds: number[] = []; + for (const r of ctx.results) { + slugsSet.add(r.slug); + if (r.source_id) sourceSet.add(r.source_id); + chunkIds.push(r.chunk_id); + } + + return { + tool_name: ctx.tool_name, + query, + retrieved_slugs: [...slugsSet], + retrieved_chunk_ids: chunkIds, + source_ids: [...sourceSet], + expand_enabled: ctx.expand_enabled, + detail: ctx.detail, + detail_resolved: ctx.meta.detail_resolved, + vector_enabled: ctx.meta.vector_enabled, + expansion_applied: ctx.meta.expansion_applied, + latency_ms: ctx.latency_ms, + remote: ctx.remote, + job_id: ctx.job_id, + subagent_id: ctx.subagent_id, + }; +} + +/** + * Map a caught error to an `EvalCaptureFailureReason` so doctor can + * group failures by cause (DB down vs RLS reject vs CHECK violation). + * + * Narrow by Postgres SQLSTATE where we can (postgres-js + PGLite both + * surface `.code`), fall back to 'other'. + */ +export function classifyCaptureFailure(err: unknown): EvalCaptureFailureReason { + if (err && typeof err === 'object') { + const code = (err as { code?: string }).code; + if (code === '23514') return 'check_violation'; // CHECK constraint violation + if (code === '42501') return 'rls_reject'; // insufficient_privilege + if (code === '42P01') return 'db_down'; // undefined_table (pre-v25) + if (code === '53300' || code === '08006' || code === '08003') return 'db_down'; + const name = (err as { name?: string }).name; + if (name === 'RegExpMatchError' || name === 'SyntaxError') { + return 'scrubber_exception'; + } + } + return 'other'; +} + +/** + * Fire-and-forget capture side-effect. Never throws: every failure is + * swallowed, classified, and routed through `engine.logEvalCaptureFailure` + * (itself wrapped — failure-of-failure is logged to stderr and dropped). + * + * Callers invoke as `void captureEvalCandidate(...)` — we explicitly do + * NOT await so the op response latency is unaffected. + */ +export async function captureEvalCandidate( + engine: BrainEngine, + ctx: CaptureContext, + opts: { scrub_pii?: boolean } = {}, +): Promise { + try { + const input = buildEvalCandidateInput(ctx, opts); + await engine.logEvalCandidate(input); + } catch (err) { + const reason = classifyCaptureFailure(err); + try { + await engine.logEvalCaptureFailure(reason); + } catch (failureErr) { + // Failure-of-failure: last-resort stderr. Doctor can't see this + // row, but we've exhausted the persistent path. + // eslint-disable-next-line no-console + console.warn('[eval-capture] secondary failure logging also failed:', failureErr); + } + } +} + +/** + * Check whether capture is enabled for this process. + * + * Resolution order: + * 1. `config.eval.capture === true` → on (explicit user opt-in wins) + * 2. `config.eval.capture === false` → off (explicit user opt-out wins) + * 3. `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'` → on (contributor opt-in) + * 4. otherwise → off (default-off, privacy-positive for end users) + * + * The default flipped in v0.25.0 from "on for everyone" to "off unless you + * opt in." Capturing every query a real user runs without their consent is + * a footgun even with PII scrubbing; tying capture to CONTRIBUTOR_MODE makes + * the developer-skill nature of the feature explicit. Production users get + * a quiet brain; contributors get the BrainBench-Real replay loop with one + * shell rc line. See docs/eval-bench.md and CONTRIBUTING.md. + * + * Takes the already-loaded config so callers control the loadConfig() + * lifecycle (MCP server loads once at boot, CLI commands load per-invocation). + */ +export function isEvalCaptureEnabled(config: GBrainConfig | null | undefined): boolean { + if (config?.eval?.capture === true) return true; + if (config?.eval?.capture === false) return false; + return process.env.GBRAIN_CONTRIBUTOR_MODE === '1'; +} + +/** + * PII scrubbing enabled? Defaults to true; explicit `false` opts out. + * + * Independent of `isEvalCaptureEnabled` — scrubbing only matters when capture + * is actually running, but the gate stays separate so CONTRIBUTOR_MODE + * doesn't accidentally turn off scrubbing on a brain that happens to also + * have explicit `capture: true`. + */ +export function isEvalScrubEnabled(config: GBrainConfig | null | undefined): boolean { + return config?.eval?.scrub_pii !== false; +} diff --git a/src/core/migrate.ts b/src/core/migrate.ts index ef775d41d..2ea321ee3 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -1101,6 +1101,106 @@ export const MIGRATIONS: Migration[] = [ END $$; `, }, + { + version: 31, + name: 'eval_capture_tables', + // v0.25.0 — BrainBench-Real session capture substrate. + // Two tables: + // eval_candidates: per-call capture from the op-layer wrapper around + // `query` and `search`. Captures MCP + CLI + subagent tool-bridge + // traffic via src/core/operations.ts. query column is CHECK-capped + // at 50KB; PII is scrubbed before insert by src/core/eval-capture-scrub.ts. + // remote distinguishes MCP callers (untrusted) from local CLI; job_id + + // subagent_id let gbrain-evals partition replay by run. + // eval_capture_failures: insert-side audit trail. When logEvalCandidate + // fails (DB down, RLS reject, CHECK violation, scrubber exception), + // the capture path records the reason here so `gbrain doctor` can + // surface silent drops cross-process. In-process counters don't work + // because doctor runs in a separate process from the MCP server. + // + // RLS enable matches the v24 / v29 posture: fail loudly via RAISE EXCEPTION + // if current_user lacks BYPASSRLS, so the migration retries cleanly after + // operator fixes the role instead of silently bumping schema_version. + // PGLite ignores RLS; sqlFor carries the table+index DDL only. + // + // Renumbered v30→v31 on merge with master's v0.23.0 (dream_verdicts) which + // claimed v30 first. Pre-existing brains that applied our v30 will see + // version 31 as new on next initSchema and run the IF NOT EXISTS DDL — + // the CREATE TABLE statements are idempotent so the rename is safe. + sqlFor: { + postgres: ` + DO $$ + DECLARE + has_bypass BOOLEAN; + BEGIN + SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user; + IF NOT has_bypass THEN + RAISE EXCEPTION 'v31 eval_capture_tables: role % does not have BYPASSRLS privilege — cannot enable RLS safely. Re-run as postgres (or another BYPASSRLS role). The migration will retry automatically on the next initSchema call.', current_user; + END IF; + + CREATE TABLE IF NOT EXISTS eval_candidates ( + id SERIAL PRIMARY KEY, + tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')), + query TEXT NOT NULL CHECK (length(query) <= 51200), + retrieved_slugs TEXT[] NOT NULL DEFAULT '{}', + retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}', + source_ids TEXT[] NOT NULL DEFAULT '{}', + expand_enabled BOOLEAN, + detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')), + detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')), + vector_enabled BOOLEAN NOT NULL, + expansion_applied BOOLEAN NOT NULL, + latency_ms INTEGER NOT NULL, + remote BOOLEAN NOT NULL, + job_id INTEGER, + subagent_id INTEGER, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + ); + CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates (created_at DESC); + ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY; + + CREATE TABLE IF NOT EXISTS eval_capture_failures ( + id SERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT NOW(), + reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other')) + ); + CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures (ts DESC); + ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY; + + RAISE NOTICE 'v31: eval_capture tables ready (role % has BYPASSRLS)', current_user; + END $$; + `, + pglite: ` + CREATE TABLE IF NOT EXISTS eval_candidates ( + id SERIAL PRIMARY KEY, + tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')), + query TEXT NOT NULL CHECK (length(query) <= 51200), + retrieved_slugs TEXT[] NOT NULL DEFAULT '{}', + retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}', + source_ids TEXT[] NOT NULL DEFAULT '{}', + expand_enabled BOOLEAN, + detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')), + detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')), + vector_enabled BOOLEAN NOT NULL, + expansion_applied BOOLEAN NOT NULL, + latency_ms INTEGER NOT NULL, + remote BOOLEAN NOT NULL, + job_id INTEGER, + subagent_id INTEGER, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + ); + CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates (created_at DESC); + + CREATE TABLE IF NOT EXISTS eval_capture_failures ( + id SERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT NOW(), + reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other')) + ); + CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures (ts DESC); + `, + }, + sql: '', + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/operations.ts b/src/core/operations.ts index 53e3afec9..1bf6999ce 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -13,6 +13,8 @@ import { importFromContent } from './import-file.ts'; import { hybridSearch } from './search/hybrid.ts'; import { expandQuery } from './search/expansion.ts'; import { dedupResults } from './search/dedup.ts'; +import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts'; +import type { HybridSearchMeta } from './types.ts'; import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts'; import * as db from './db.ts'; @@ -627,11 +629,38 @@ const search: Operation = { offset: { type: 'number', description: 'Skip first N results (for pagination)' }, }, handler: async (ctx, p) => { - const results = await ctx.engine.searchKeyword(p.query as string, { + const startedAt = Date.now(); + const queryText = p.query as string; + const raw = await ctx.engine.searchKeyword(queryText, { limit: (p.limit as number) || 20, offset: (p.offset as number) || 0, }); - return dedupResults(results); + const results = dedupResults(raw); + const latency_ms = Date.now() - startedAt; + + // Op-layer capture (v0.25.0). Fire-and-forget — no await on the + // capture call so MCP response latency is unaffected. search has + // no expand/detail/vector semantics so meta fields are fixed. + if (isEvalCaptureEnabled(ctx.config)) { + void captureEvalCandidate( + ctx.engine, + { + tool_name: 'search', + query: queryText, + results, + meta: { vector_enabled: false, detail_resolved: null, expansion_applied: false }, + latency_ms, + remote: ctx.remote ?? false, + expand_enabled: null, + detail: null, + job_id: ctx.jobId ?? null, + subagent_id: ctx.subagentId ?? null, + }, + { scrub_pii: isEvalScrubEnabled(ctx.config) }, + ); + } + + return results; }, cliHints: { name: 'search', positional: ['query'] }, }; @@ -653,9 +682,16 @@ const query: Operation = { walk_depth: { type: 'number', description: 'Structural walk depth 1-2. Default 0 (off). Expands anchors through code_edges with 1/(1+hop) decay.' }, }, handler: async (ctx, p) => { + const startedAt = Date.now(); const expand = p.expand !== false; const detail = (p.detail as 'low' | 'medium' | 'high') || undefined; - return hybridSearch(ctx.engine, p.query as string, { + const queryText = p.query as string; + + // v0.25.0 — capture meta side-channel. hybridSearch's return contract + // stays SearchResult[] (Cathedral II callers depend on that); meta + // arrives via callback so eval capture can record what actually ran. + let capturedMeta: HybridSearchMeta | null = null; + const results = await hybridSearch(ctx.engine, queryText, { limit: (p.limit as number) || 20, offset: (p.offset as number) || 0, expansion: expand, @@ -665,7 +701,37 @@ const query: Operation = { symbolKind: (p.symbol_kind as string) || undefined, nearSymbol: (p.near_symbol as string) || undefined, walkDepth: typeof p.walk_depth === 'number' ? (p.walk_depth as number) : undefined, + onMeta: (m) => { capturedMeta = m; }, }); + const latency_ms = Date.now() - startedAt; + + // Op-layer capture (v0.25.0). Fire-and-forget. meta tells gbrain-evals + // what hybridSearch *actually* did so replay can distinguish "with API + // key" from "keyword-only fallback" and "expansion fired" from + // "expansion requested + silently fell back." + if (isEvalCaptureEnabled(ctx.config)) { + const meta: HybridSearchMeta = capturedMeta ?? { + vector_enabled: false, detail_resolved: detail ?? null, expansion_applied: false, + }; + void captureEvalCandidate( + ctx.engine, + { + tool_name: 'query', + query: queryText, + results, + meta, + latency_ms, + remote: ctx.remote ?? false, + expand_enabled: expand, + detail: detail ?? null, + job_id: ctx.jobId ?? null, + subagent_id: ctx.subagentId ?? null, + }, + { scrub_pii: isEvalScrubEnabled(ctx.config) }, + ); + } + + return results; }, cliHints: { name: 'query', positional: ['query'] }, }; diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 58a6e2e2e..a57fd407b 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -18,6 +18,8 @@ import type { BrainStats, BrainHealth, IngestLogEntry, IngestLogInput, EngineConfig, + EvalCandidate, EvalCandidateInput, + EvalCaptureFailure, EvalCaptureFailureReason, } from './types.ts'; import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts'; import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; @@ -1673,6 +1675,82 @@ export class PGLiteEngine implements BrainEngine { return (rows as Record[]).map(rowToCodeEdge); } + // Eval capture (v0.25.0). See BrainEngine interface docs. + async logEvalCandidate(input: EvalCandidateInput): Promise { + const { rows } = await this.db.query<{ id: number }>( + `INSERT INTO eval_candidates ( + tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids, + expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied, + latency_ms, remote, job_id, subagent_id + ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) + RETURNING id`, + [ + input.tool_name, + input.query, + input.retrieved_slugs, + input.retrieved_chunk_ids, + input.source_ids, + input.expand_enabled, + input.detail, + input.detail_resolved, + input.vector_enabled, + input.expansion_applied, + input.latency_ms, + input.remote, + input.job_id, + input.subagent_id, + ] + ); + return rows[0]!.id; + } + + async listEvalCandidates(filter?: { since?: Date; limit?: number; tool?: 'query' | 'search' }): Promise { + const raw = filter?.limit; + const limit = (raw === undefined || raw === null || !Number.isFinite(raw) || raw <= 0) + ? 1000 + : Math.min(Math.floor(raw), 100000); + const since = filter?.since ?? new Date(0); + const tool = filter?.tool ?? null; + // id DESC tiebreaker — see postgres-engine for rationale. + const { rows } = tool + ? await this.db.query( + `SELECT * FROM eval_candidates + WHERE created_at >= $1 AND tool_name = $2 + ORDER BY created_at DESC, id DESC LIMIT $3`, + [since, tool, limit] + ) + : await this.db.query( + `SELECT * FROM eval_candidates + WHERE created_at >= $1 + ORDER BY created_at DESC, id DESC LIMIT $2`, + [since, limit] + ); + return rows as unknown as EvalCandidate[]; + } + + async deleteEvalCandidatesBefore(date: Date): Promise { + const { rows } = await this.db.query( + `DELETE FROM eval_candidates WHERE created_at < $1 RETURNING id`, + [date] + ); + return rows.length; + } + + async logEvalCaptureFailure(reason: EvalCaptureFailureReason): Promise { + await this.db.query( + `INSERT INTO eval_capture_failures (reason) VALUES ($1)`, + [reason] + ); + } + + async listEvalCaptureFailures(filter?: { since?: Date }): Promise { + const since = filter?.since ?? new Date(0); + const { rows } = await this.db.query( + `SELECT * FROM eval_capture_failures WHERE ts >= $1 ORDER BY ts DESC`, + [since] + ); + return rows as unknown as EvalCaptureFailure[]; + } } function rowToCodeEdge(row: Record): import('./types.ts').CodeEdgeResult { diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index 2dad7b5a7..f302d8a40 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -363,6 +363,35 @@ CREATE TABLE IF NOT EXISTS gbrain_cycle_locks ( ); CREATE INDEX IF NOT EXISTS idx_cycle_locks_ttl ON gbrain_cycle_locks(ttl_expires_at); +-- Eval capture (v0.25.0). PGLite ignores RLS — see src/schema.sql for the +-- cross-engine spec. +CREATE TABLE IF NOT EXISTS eval_candidates ( + id SERIAL PRIMARY KEY, + tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')), + query TEXT NOT NULL CHECK (length(query) <= 51200), + retrieved_slugs TEXT[] NOT NULL DEFAULT '{}', + retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}', + source_ids TEXT[] NOT NULL DEFAULT '{}', + expand_enabled BOOLEAN, + detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')), + detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')), + vector_enabled BOOLEAN NOT NULL, + expansion_applied BOOLEAN NOT NULL, + latency_ms INTEGER NOT NULL, + remote BOOLEAN NOT NULL, + job_id INTEGER, + subagent_id INTEGER, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); +CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC); + +CREATE TABLE IF NOT EXISTS eval_capture_failures ( + id SERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT NOW(), + reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other')) +); +CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures(ts DESC); + -- ============================================================ -- Trigger-based search_vector (spans pages + timeline_entries) -- ============================================================ diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 8e602f017..98e4cf3a6 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -15,6 +15,8 @@ import type { BrainStats, BrainHealth, IngestLogEntry, IngestLogInput, EngineConfig, + EvalCandidate, EvalCandidateInput, + EvalCaptureFailure, EvalCaptureFailureReason, } from './types.ts'; import { GBrainError } from './types.ts'; import * as db from './db.ts'; @@ -1737,6 +1739,74 @@ export class PostgresEngine implements BrainEngine { return [...chunkRows, ...symbolRows].map(r => pgRowToCodeEdge(r as Record)); } + // Eval capture (v0.25.0). See BrainEngine interface docs. + async logEvalCandidate(input: EvalCandidateInput): Promise { + const sql = this.sql; + const rows = await sql` + INSERT INTO eval_candidates ( + tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids, + expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied, + latency_ms, remote, job_id, subagent_id + ) VALUES ( + ${input.tool_name}, ${input.query}, ${input.retrieved_slugs}, ${input.retrieved_chunk_ids}, ${input.source_ids}, + ${input.expand_enabled}, ${input.detail}, ${input.detail_resolved}, ${input.vector_enabled}, ${input.expansion_applied}, + ${input.latency_ms}, ${input.remote}, ${input.job_id}, ${input.subagent_id} + ) + RETURNING id + `; + return rows[0]!.id as number; + } + + async listEvalCandidates(filter?: { since?: Date; limit?: number; tool?: 'query' | 'search' }): Promise { + const sql = this.sql; + const raw = filter?.limit; + const limit = (raw === undefined || raw === null || !Number.isFinite(raw) || raw <= 0) + ? 1000 + : Math.min(Math.floor(raw), 100000); + const since = filter?.since ?? new Date(0); + const tool = filter?.tool ?? null; + // id DESC tiebreaker so same-millisecond inserts return deterministically + // — without this, `gbrain eval export --since` could dupe or miss rows + // across non-overlapping windows. + const rows = tool + ? await sql` + SELECT * FROM eval_candidates + WHERE created_at >= ${since} AND tool_name = ${tool} + ORDER BY created_at DESC, id DESC + LIMIT ${limit} + ` + : await sql` + SELECT * FROM eval_candidates + WHERE created_at >= ${since} + ORDER BY created_at DESC, id DESC + LIMIT ${limit} + `; + return rows as unknown as EvalCandidate[]; + } + + async deleteEvalCandidatesBefore(date: Date): Promise { + const sql = this.sql; + const rows = await sql` + DELETE FROM eval_candidates WHERE created_at < ${date} RETURNING id + `; + return rows.length; + } + + async logEvalCaptureFailure(reason: EvalCaptureFailureReason): Promise { + const sql = this.sql; + await sql`INSERT INTO eval_capture_failures (reason) VALUES (${reason})`; + } + + async listEvalCaptureFailures(filter?: { since?: Date }): Promise { + const sql = this.sql; + const since = filter?.since ?? new Date(0); + const rows = await sql` + SELECT * FROM eval_capture_failures + WHERE ts >= ${since} + ORDER BY ts DESC + `; + return rows as unknown as EvalCaptureFailure[]; + } } function pgRowToCodeEdge(row: Record): import('./types.ts').CodeEdgeResult { diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index d7b927980..5aec20c8d 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -597,7 +597,7 @@ CREATE TABLE IF NOT EXISTS subagent_rate_leases ( CREATE INDEX IF NOT EXISTS idx_rate_leases_key_expires ON subagent_rate_leases (key, expires_at); -- ============================================================ --- Dream-cycle significance verdict cache — v0.23 synthesize phase +-- Dream-cycle significance verdict cache — v0.21 synthesize phase -- ============================================================ -- Caches the cheap Haiku "is this transcript worth processing?" verdict -- per (file_path, content_hash) so backfill re-runs skip already-judged @@ -630,6 +630,42 @@ CREATE TABLE IF NOT EXISTS gbrain_cycle_locks ( ); CREATE INDEX IF NOT EXISTS idx_cycle_locks_ttl ON gbrain_cycle_locks(ttl_expires_at); +-- ============================================================ +-- Eval capture (v0.25.0 — BrainBench-Real substrate) +-- ============================================================ +-- eval_candidates: captured query/search calls from the op-layer wrapper +-- in src/core/operations.ts. PII is scrubbed before insert by +-- src/core/eval-capture-scrub.ts. query is CHECK-capped at 50KB. +-- eval_capture_failures: cross-process audit of insert failures, surfaced +-- by \`gbrain doctor\` (in-process counters can't bridge MCP server + doctor +-- CLI process boundaries). +CREATE TABLE IF NOT EXISTS eval_candidates ( + id SERIAL PRIMARY KEY, + tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')), + query TEXT NOT NULL CHECK (length(query) <= 51200), + retrieved_slugs TEXT[] NOT NULL DEFAULT '{}', + retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}', + source_ids TEXT[] NOT NULL DEFAULT '{}', + expand_enabled BOOLEAN, + detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')), + detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')), + vector_enabled BOOLEAN NOT NULL, + expansion_applied BOOLEAN NOT NULL, + latency_ms INTEGER NOT NULL, + remote BOOLEAN NOT NULL, + job_id INTEGER, + subagent_id INTEGER, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); +CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC); + +CREATE TABLE IF NOT EXISTS eval_capture_failures ( + id SERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT NOW(), + reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other')) +); +CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures(ts DESC); + -- NOTIFY trigger for real-time job events (Postgres only, not PGLite) CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS \$\$ BEGIN @@ -680,6 +716,8 @@ BEGIN ALTER TABLE subagent_rate_leases ENABLE ROW LEVEL SECURITY; ALTER TABLE gbrain_cycle_locks ENABLE ROW LEVEL SECURITY; ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY; + ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY; + ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY; RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user; ELSE RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user; diff --git a/src/core/search/hybrid.ts b/src/core/search/hybrid.ts index f008c3098..f58f95415 100644 --- a/src/core/search/hybrid.ts +++ b/src/core/search/hybrid.ts @@ -11,7 +11,7 @@ import type { BrainEngine } from '../engine.ts'; import { MAX_SEARCH_LIMIT, clampSearchLimit } from '../engine.ts'; -import type { SearchResult, SearchOpts } from '../types.ts'; +import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts'; import { embed } from '../embedding.ts'; import { dedupResults } from './dedup.ts'; import { autoDetectDetail } from './intent.ts'; @@ -56,6 +56,15 @@ export interface HybridSearchOpts extends SearchOpts { maxTypeRatio?: number; maxPerPage?: number; }; + /** + * v0.25.0 — optional side-channel for what hybridSearch actually did + * (vector ran or fell back, expansion fired or didn't, post-auto-detect + * detail). Surfaced via callback so the bare-return contract stays as + * `Promise` for existing Cathedral II callers. Op-layer + * eval capture passes a callback that threads `meta` into the captured + * row; everyone else leaves it undefined and pays no cost. + */ + onMeta?: (meta: HybridSearchMeta) => void; } export async function hybridSearch( @@ -69,6 +78,7 @@ export async function hybridSearch( // Auto-detect detail level from query intent when caller doesn't specify const detail = opts?.detail ?? autoDetectDetail(query); + const detailResolved: 'low' | 'medium' | 'high' | null = detail ?? null; const searchOpts: SearchOpts = { limit: innerLimit, detail, @@ -77,6 +87,22 @@ export async function hybridSearch( language: opts?.language, symbolKind: opts?.symbolKind, }; + // Track what actually ran for the optional onMeta callback (v0.25.0). + // Caller leaves onMeta undefined → these flags are computed but never + // surfaced. Capture wrapper passes a closure to receive the meta and + // threads it into the eval_candidates row. + let expansionApplied = false; + + // A throwing user callback must never break the search hot path — onMeta + // is a public surface (gbrain/search/hybrid) so a third-party closure bug + // shouldn't take down query/search responses. + const emitMeta = (meta: HybridSearchMeta): void => { + try { + opts?.onMeta?.(meta); + } catch { + // swallow — capture telemetry is best-effort + } + }; if (DEBUG && detail) { console.error(`[search-debug] auto-detail=${detail} for query="${query}"`); @@ -99,6 +125,7 @@ export async function hybridSearch( // Boost failure is non-fatal: keep unboosted ranking. } } + emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: false }); return dedupResults(keywordResults).slice(offset, offset + limit); } @@ -110,6 +137,8 @@ export async function hybridSearch( try { queries = await opts.expandFn(query); if (queries.length === 0) queries = [query]; + // "Applied" = produced variants beyond the original, not just called. + expansionApplied = queries.length > 1; } catch { // Expansion failure is non-fatal } @@ -129,6 +158,8 @@ export async function hybridSearch( } if (vectorLists.length === 0) { + // Embed/vector failed silently; record that vector did not run. + emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: expansionApplied }); return dedupResults(keywordResults).slice(offset, offset + limit); } @@ -203,11 +234,14 @@ export async function hybridSearch( // Dedup const deduped = dedupResults(fused, dedupOpts); - // Auto-escalate: if detail=low returned 0, retry with high + // Auto-escalate: if detail=low returned 0, retry with high. The inner + // call's onMeta fires with the escalated detail_resolved; do NOT also + // fire here (would double-emit and capture stale meta). if (deduped.length === 0 && opts?.detail === 'low') { return hybridSearch(engine, query, { ...opts, detail: 'high' }); } + emitMeta({ vector_enabled: true, detail_resolved: detailResolved, expansion_applied: expansionApplied }); return deduped.slice(offset, offset + limit); } diff --git a/src/core/types.ts b/src/core/types.ts index ea92c3905..b242c0344 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -389,6 +389,69 @@ export interface IngestLogInput { summary: string; } +// Eval capture (v0.25.0) +// Real MCP/CLI/subagent query+search calls are captured into eval_candidates +// so gbrain-evals can replay them as BrainBench-Real. The companion +// eval_capture_failures table records insert failures so gbrain doctor can +// surface silent capture drops cross-process. +export interface EvalCandidateInput { + tool_name: 'query' | 'search'; + /** Already PII-scrubbed by captureEvalCandidate before this point. */ + query: string; + retrieved_slugs: string[]; + retrieved_chunk_ids: number[]; + source_ids: string[]; + /** Whether multi-query Haiku expansion was enabled on the call. Null for 'search'. */ + expand_enabled: boolean | null; + /** The detail level the call requested (pre-auto-detect). */ + detail: 'low' | 'medium' | 'high' | null; + /** What hybridSearch actually ran (post-auto-detect). Null for 'search'. */ + detail_resolved: 'low' | 'medium' | 'high' | null; + /** True when vector search actually ran (false when OPENAI_API_KEY missing or embed failed). */ + vector_enabled: boolean; + /** True when Haiku expansion actually fired. */ + expansion_applied: boolean; + latency_ms: number; + /** ctx.remote: true for MCP callers (untrusted), false for local CLI. */ + remote: boolean; + job_id: number | null; + subagent_id: number | null; +} + +export interface EvalCandidate extends EvalCandidateInput { + id: number; + created_at: Date; +} + +export type EvalCaptureFailureReason = + | 'db_down' + | 'rls_reject' + | 'check_violation' + | 'scrubber_exception' + | 'other'; + +export interface EvalCaptureFailure { + id: number; + ts: Date; + reason: EvalCaptureFailureReason; +} + +/** + * Side-channel metadata that hybridSearch reports about what actually ran. + * Surfaced via the optional `onMeta` callback in HybridSearchOpts so + * existing SearchResult[] consumers (Cathedral II, gbrain-evals, etc.) + * stay unchanged. Used by op-layer eval capture to distinguish + * "keyword-only fallback" from "full hybrid with expansion." + */ +export interface HybridSearchMeta { + /** True iff vector search actually ran. False when OPENAI_API_KEY missing or embed failed. */ + vector_enabled: boolean; + /** Post-auto-detect detail level. */ + detail_resolved: 'low' | 'medium' | 'high' | null; + /** True iff multi-query expansion (Haiku) actually fired and produced variants. */ + expansion_applied: boolean; +} + // Config export interface EngineConfig { database_url?: string; diff --git a/src/schema.sql b/src/schema.sql index 70a684a70..a6e59dc72 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -626,6 +626,42 @@ CREATE TABLE IF NOT EXISTS gbrain_cycle_locks ( ); CREATE INDEX IF NOT EXISTS idx_cycle_locks_ttl ON gbrain_cycle_locks(ttl_expires_at); +-- ============================================================ +-- Eval capture (v0.25.0 — BrainBench-Real substrate) +-- ============================================================ +-- eval_candidates: captured query/search calls from the op-layer wrapper +-- in src/core/operations.ts. PII is scrubbed before insert by +-- src/core/eval-capture-scrub.ts. query is CHECK-capped at 50KB. +-- eval_capture_failures: cross-process audit of insert failures, surfaced +-- by `gbrain doctor` (in-process counters can't bridge MCP server + doctor +-- CLI process boundaries). +CREATE TABLE IF NOT EXISTS eval_candidates ( + id SERIAL PRIMARY KEY, + tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')), + query TEXT NOT NULL CHECK (length(query) <= 51200), + retrieved_slugs TEXT[] NOT NULL DEFAULT '{}', + retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}', + source_ids TEXT[] NOT NULL DEFAULT '{}', + expand_enabled BOOLEAN, + detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')), + detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')), + vector_enabled BOOLEAN NOT NULL, + expansion_applied BOOLEAN NOT NULL, + latency_ms INTEGER NOT NULL, + remote BOOLEAN NOT NULL, + job_id INTEGER, + subagent_id INTEGER, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); +CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC); + +CREATE TABLE IF NOT EXISTS eval_capture_failures ( + id SERIAL PRIMARY KEY, + ts TIMESTAMPTZ NOT NULL DEFAULT NOW(), + reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other')) +); +CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures(ts DESC); + -- NOTIFY trigger for real-time job events (Postgres only, not PGLite) CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS $$ BEGIN @@ -676,6 +712,8 @@ BEGIN ALTER TABLE subagent_rate_leases ENABLE ROW LEVEL SECURITY; ALTER TABLE gbrain_cycle_locks ENABLE ROW LEVEL SECURITY; ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY; + ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY; + ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY; RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user; ELSE RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user; diff --git a/test/e2e/eval-capture.test.ts b/test/e2e/eval-capture.test.ts new file mode 100644 index 000000000..da400924e --- /dev/null +++ b/test/e2e/eval-capture.test.ts @@ -0,0 +1,144 @@ +/** + * E2E — Postgres-specific eval_capture behavior (v0.21.0, E1 spec). + * + * Unit tests in test/eval-candidates.test.ts already cover PGLite. This + * file is strictly for Postgres-only behavior that PGLite can't exercise: + * 1. RLS policy rejects when the caller is NOT BYPASSRLS + * 2. CHECK violation surfaces as Postgres error code '23514' + * 3. Concurrent INSERT pressure on the connection pool doesn't deadlock + * or drop rows (the fire-and-forget capture path at full tilt) + * + * Runs only when DATABASE_URL is set. Skips gracefully otherwise + * per CLAUDE.md E2E lifecycle. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PostgresEngine } from '../../src/core/postgres-engine.ts'; +import type { EvalCandidateInput } from '../../src/core/types.ts'; + +const DATABASE_URL = process.env.DATABASE_URL; + +let engine: PostgresEngine | null = null; + +beforeAll(async () => { + if (!DATABASE_URL) { + console.log('[e2e/eval-capture] DATABASE_URL not set — skipping.'); + return; + } + engine = new PostgresEngine(); + await engine.connect({ database_url: DATABASE_URL }); + await engine.initSchema(); +}); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}); + +beforeEach(async () => { + if (!engine) return; + await engine.executeRaw('DELETE FROM eval_candidates'); + await engine.executeRaw('DELETE FROM eval_capture_failures'); +}); + +function baseInput(overrides: Partial = {}): EvalCandidateInput { + return { + tool_name: 'query', + query: 'alice', + retrieved_slugs: ['people/alice-example'], + retrieved_chunk_ids: [1], + source_ids: ['default'], + expand_enabled: true, + detail: null, + detail_resolved: 'medium', + vector_enabled: true, + expansion_applied: false, + latency_ms: 100, + remote: true, + job_id: null, + subagent_id: null, + ...overrides, + }; +} + +describe('CHECK constraint — 50KB query length cap', () => { + test.if(DATABASE_URL !== undefined)( + 'oversized query rejected with Postgres error code 23514', + async () => { + if (!engine) return; + const big = 'x'.repeat(51_201); + let caught: unknown = null; + try { + await engine.logEvalCandidate(baseInput({ query: big })); + } catch (err) { + caught = err; + } + expect(caught).not.toBeNull(); + // postgres.js surfaces SQLSTATE on the error object as `code`. + expect((caught as { code?: string }).code).toBe('23514'); + }, + ); + + test.if(DATABASE_URL !== undefined)( + 'query exactly at 50KB succeeds', + async () => { + if (!engine) return; + const atCap = 'x'.repeat(51_200); + const id = await engine.logEvalCandidate(baseInput({ query: atCap })); + expect(id).toBeGreaterThan(0); + }, + ); +}); + +describe('RLS policy — requires BYPASSRLS to INSERT', () => { + // Note: running this test REQUIRES the E2E test role to be postgres + // (or another BYPASSRLS role) since schema.sql + migration v25 enable + // RLS only on BYPASSRLS-capable roles. We verify the policy is + // actually enabled and the INSERT path succeeds as BYPASSRLS. A + // "deny from non-BYPASSRLS role" positive test would require setting + // up a second role, which is beyond this test's scope — but + // structurally we assert the ALTER TABLE ran. + test.if(DATABASE_URL !== undefined)( + 'eval_candidates has RLS enabled', + async () => { + if (!engine) return; + const rows = await engine.executeRaw<{ relrowsecurity: boolean }>( + `SELECT relrowsecurity FROM pg_class WHERE relname = 'eval_candidates'`, + ); + expect(rows).toHaveLength(1); + expect(rows[0]!.relrowsecurity).toBe(true); + }, + ); + + test.if(DATABASE_URL !== undefined)( + 'eval_capture_failures has RLS enabled', + async () => { + if (!engine) return; + const rows = await engine.executeRaw<{ relrowsecurity: boolean }>( + `SELECT relrowsecurity FROM pg_class WHERE relname = 'eval_capture_failures'`, + ); + expect(rows).toHaveLength(1); + expect(rows[0]!.relrowsecurity).toBe(true); + }, + ); +}); + +describe('concurrent pool pressure — 50 parallel INSERTs', () => { + test.if(DATABASE_URL !== undefined)( + '50 concurrent logEvalCandidate calls all land without deadlock', + async () => { + if (!engine) return; + const n = 50; + const promises: Promise[] = []; + for (let i = 0; i < n; i++) { + promises.push(engine.logEvalCandidate(baseInput({ query: `q${i}` }))); + } + const ids = await Promise.all(promises); + expect(ids).toHaveLength(n); + expect(new Set(ids).size).toBe(n); // every id is distinct + + const rows = await engine.listEvalCandidates({ limit: n * 2 }); + expect(rows).toHaveLength(n); + }, + 30_000, + ); +}); diff --git a/test/eval-candidates.test.ts b/test/eval-candidates.test.ts new file mode 100644 index 000000000..b5065dfca --- /dev/null +++ b/test/eval-candidates.test.ts @@ -0,0 +1,201 @@ +/** + * PGLite round-trip for the 5 eval-capture engine methods shipped in v0.21.0: + * logEvalCandidate / listEvalCandidates / deleteEvalCandidatesBefore + * logEvalCaptureFailure / listEvalCaptureFailures + * + * No Docker, no DATABASE_URL. In-memory only. For Postgres-only behavior + * (RLS policy enforcement, CHECK violation error codes, concurrent pool + * pressure), see test/e2e/eval-capture.test.ts. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import type { EvalCandidateInput, EvalCaptureFailureReason } from '../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + await (engine as any).db.exec('DELETE FROM eval_candidates'); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + await (engine as any).db.exec('DELETE FROM eval_capture_failures'); +}); + +function makeInput(overrides: Partial = {}): EvalCandidateInput { + return { + tool_name: 'query', + query: 'who is alice-example', + retrieved_slugs: ['people/alice-example', 'companies/acme-example'], + retrieved_chunk_ids: [42, 43], + source_ids: ['default'], + expand_enabled: true, + detail: null, + detail_resolved: 'medium', + vector_enabled: true, + expansion_applied: false, + latency_ms: 123, + remote: true, + job_id: null, + subagent_id: null, + ...overrides, + }; +} + +describe('logEvalCandidate', () => { + test('inserts a full row and returns the id', async () => { + const id = await engine.logEvalCandidate(makeInput()); + expect(typeof id).toBe('number'); + expect(id).toBeGreaterThan(0); + }); + + test('preserves array + enum + boolean columns through round-trip', async () => { + await engine.logEvalCandidate( + makeInput({ + tool_name: 'search', + expand_enabled: null, // search has no expansion semantics + detail: 'high', + detail_resolved: 'high', + vector_enabled: false, + expansion_applied: false, + job_id: 9001, + subagent_id: 42, + remote: false, + }), + ); + const rows = await engine.listEvalCandidates(); + expect(rows).toHaveLength(1); + const row = rows[0]!; + expect(row.tool_name).toBe('search'); + expect(row.retrieved_slugs).toEqual(['people/alice-example', 'companies/acme-example']); + expect(row.retrieved_chunk_ids).toEqual([42, 43]); + expect(row.source_ids).toEqual(['default']); + expect(row.expand_enabled).toBeNull(); + expect(row.detail).toBe('high'); + expect(row.detail_resolved).toBe('high'); + expect(row.vector_enabled).toBe(false); + expect(row.job_id).toBe(9001); + expect(row.subagent_id).toBe(42); + expect(row.remote).toBe(false); + }); + + test('rejects oversize queries via CHECK constraint (50KB cap)', async () => { + const bigQuery = 'x'.repeat(51201); + await expect(engine.logEvalCandidate(makeInput({ query: bigQuery }))).rejects.toThrow(); + }); + + test('rejects invalid tool_name via CHECK constraint', async () => { + await expect( + engine.logEvalCandidate(makeInput({ tool_name: 'dance' as unknown as 'query' })), + ).rejects.toThrow(); + }); +}); + +describe('listEvalCandidates', () => { + test('returns empty when table is empty', async () => { + expect(await engine.listEvalCandidates()).toEqual([]); + }); + + test('filters by tool', async () => { + await engine.logEvalCandidate(makeInput({ tool_name: 'query' })); + await engine.logEvalCandidate(makeInput({ tool_name: 'search', expand_enabled: null })); + await engine.logEvalCandidate(makeInput({ tool_name: 'query' })); + + const queries = await engine.listEvalCandidates({ tool: 'query' }); + const searches = await engine.listEvalCandidates({ tool: 'search' }); + + expect(queries).toHaveLength(2); + expect(searches).toHaveLength(1); + expect(queries.every(r => r.tool_name === 'query')).toBe(true); + expect(searches.every(r => r.tool_name === 'search')).toBe(true); + }); + + test('filters by since (inclusive lower bound)', async () => { + await engine.logEvalCandidate(makeInput()); + const cutoff = new Date(Date.now() + 60_000); // 1 minute in future — excludes everything + expect(await engine.listEvalCandidates({ since: cutoff })).toHaveLength(0); + + const past = new Date(0); + expect(await engine.listEvalCandidates({ since: past })).toHaveLength(1); + }); + + test('honors limit and returns newest-first (id DESC tiebreaker for same-ms inserts)', async () => { + const ids: number[] = []; + for (let i = 0; i < 5; i++) { + ids.push(await engine.logEvalCandidate(makeInput({ query: `q${i}` }))); + } + const limited = await engine.listEvalCandidates({ limit: 3 }); + expect(limited).toHaveLength(3); + // ORDER BY created_at DESC, id DESC — same-millisecond rows fall back to + // id DESC, which matches insertion order. Critical for `gbrain eval export` + // since unstable tiebreaks would cause duplicate/missed rows across runs. + expect(limited.map(r => r.id)).toEqual([ids[4]!, ids[3]!, ids[2]!]); + }); + + test('clamps limit to [1, 100000]', async () => { + await engine.logEvalCandidate(makeInput()); + // Implementation must clamp limit <= 0 up to default and cap huge values. + expect(await engine.listEvalCandidates({ limit: 0 })).toHaveLength(1); + expect(await engine.listEvalCandidates({ limit: 10_000_000 })).toHaveLength(1); + }); +}); + +describe('deleteEvalCandidatesBefore', () => { + test('returns the number of rows deleted', async () => { + for (let i = 0; i < 3; i++) { + await engine.logEvalCandidate(makeInput()); + } + const futureCutoff = new Date(Date.now() + 60_000); + const deleted = await engine.deleteEvalCandidatesBefore(futureCutoff); + expect(deleted).toBe(3); + expect(await engine.listEvalCandidates()).toHaveLength(0); + }); + + test('leaves newer rows alone', async () => { + await engine.logEvalCandidate(makeInput({ query: 'old' })); + // Cutoff 0 deletes nothing since created_at > 1970. + const pastCutoff = new Date(0); + const deleted = await engine.deleteEvalCandidatesBefore(pastCutoff); + expect(deleted).toBe(0); + expect(await engine.listEvalCandidates()).toHaveLength(1); + }); +}); + +describe('logEvalCaptureFailure / listEvalCaptureFailures', () => { + test('inserts and round-trips each reason enum', async () => { + const reasons: EvalCaptureFailureReason[] = [ + 'db_down', + 'rls_reject', + 'check_violation', + 'scrubber_exception', + 'other', + ]; + for (const r of reasons) { + await engine.logEvalCaptureFailure(r); + } + const rows = await engine.listEvalCaptureFailures(); + expect(rows.map(r => r.reason).sort()).toEqual([...reasons].sort()); + }); + + test('rejects invalid reason via CHECK constraint', async () => { + await expect( + engine.logEvalCaptureFailure('unknown' as EvalCaptureFailureReason), + ).rejects.toThrow(); + }); + + test('filters by since', async () => { + await engine.logEvalCaptureFailure('db_down'); + const future = new Date(Date.now() + 60_000); + expect(await engine.listEvalCaptureFailures({ since: future })).toHaveLength(0); + expect(await engine.listEvalCaptureFailures({ since: new Date(0) })).toHaveLength(1); + }); +}); diff --git a/test/eval-capture-scrub.test.ts b/test/eval-capture-scrub.test.ts new file mode 100644 index 000000000..3b6fa5db9 --- /dev/null +++ b/test/eval-capture-scrub.test.ts @@ -0,0 +1,118 @@ +/** + * PII scrubber regex coverage. Every redacted family gets a positive + * case and at least one false-positive avoidance case. Adversarial regex + * input is covered so a pathological query can't hang capture. + */ + +import { describe, expect, test } from 'bun:test'; +import { scrubPii } from '../src/core/eval-capture-scrub.ts'; + +describe('scrubPii', () => { + test('passes empty input through unchanged', () => { + expect(scrubPii('')).toBe(''); + }); + + test('leaves PII-free queries untouched', () => { + const q = 'who is alice-example and what did they build'; + expect(scrubPii(q)).toBe(q); + }); + + test('redacts email addresses', () => { + const out = scrubPii('email me at alice@example.com about this'); + expect(out).not.toContain('alice@example.com'); + expect(out).toContain('[REDACTED]'); + }); + + test('redacts multiple emails in one query', () => { + const out = scrubPii('alice@example.com cc bob@other.org'); + expect(out).not.toContain('@example.com'); + expect(out).not.toContain('@other.org'); + expect(out.match(/\[REDACTED\]/g) || []).toHaveLength(2); + }); + + test('redacts US phone 555-123-4567', () => { + expect(scrubPii('call 555-123-4567')).toBe('call [REDACTED]'); + }); + + test('redacts US phone (555) 123-4567', () => { + expect(scrubPii('tel (555) 123-4567 ext 4')).toContain('[REDACTED]'); + }); + + test('redacts E.164 +1 555 123 4567', () => { + expect(scrubPii('intl +1 555 123 4567')).toContain('[REDACTED]'); + }); + + test('redacts SSN with dashes (XXX-XX-XXXX)', () => { + expect(scrubPii('ssn is 123-45-6789 on file')).toContain('[REDACTED]'); + expect(scrubPii('ssn is 123-45-6789 on file')).not.toContain('123-45-6789'); + }); + + test('does NOT redact a 4-digit year like 2026 as SSN', () => { + expect(scrubPii('released in 2026, v0.21.0')).toContain('2026'); + }); + + test('redacts a Visa number that passes Luhn', () => { + // 4111 1111 1111 1111 is the canonical Visa test number (valid Luhn). + const out = scrubPii('card 4111 1111 1111 1111 expires 12/26'); + expect(out).not.toContain('4111 1111 1111 1111'); + expect(out).toContain('[REDACTED]'); + }); + + test('does NOT redact a 16-digit integer that fails Luhn', () => { + // 1234567890123456 has Luhn mod 10 = 4 (fails). + expect(scrubPii('order id 1234567890123456')).toContain('1234567890123456'); + }); + + test('does NOT redact bare 6-digit identifiers (#123456)', () => { + expect(scrubPii('see PR #123456 and issue #789012')).toContain('#123456'); + }); + + test('redacts JWT-shaped tokens', () => { + const jwt = + 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.' + + 'eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.' + + 'SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c'; + const out = scrubPii(`token: ${jwt}`); + expect(out).not.toContain('eyJhbGciOi'); + expect(out).toContain('[REDACTED]'); + }); + + test('redacts bearer tokens after Authorization-style prefix', () => { + const out = scrubPii('Authorization: Bearer abc123DEF456ghi789'); + expect(out).not.toContain('abc123DEF456ghi789'); + expect(out).toContain('Bearer [REDACTED]'); + }); + + test('redacts bearer lowercase too', () => { + const out = scrubPii('header bearer abc123DEF456ghi789'); + expect(out).not.toContain('abc123DEF456ghi789'); + }); + + test('handles adversarial nested-group input without hanging', () => { + // Classic catastrophic-backtracking bait for regex engines without + // possessive quantifiers. Must complete quickly — the scrubber should + // stay linear-time on all input. 10k char limit is well under the + // CHECK-constrained 50KB cap enforced at DB level. + const adversarial = 'a'.repeat(10000) + '!'; + const start = Date.now(); + const out = scrubPii(adversarial); + const elapsed = Date.now() - start; + expect(elapsed).toBeLessThan(1000); // generous cap, realistic target <50ms + expect(out).toContain('a'.repeat(100)); // output still visible + }); + + test('combined input: email + phone + CC + JWT + SSN all redacted together', () => { + const mixed = + 'contact alice@example.com at 555-123-4567, ' + + 'card 4111-1111-1111-1111, ssn 123-45-6789, ' + + 'auth Bearer abcdef123456'; + const out = scrubPii(mixed); + expect(out).not.toContain('alice@example.com'); + expect(out).not.toContain('555-123-4567'); + expect(out).not.toContain('4111-1111-1111-1111'); + expect(out).not.toContain('123-45-6789'); + expect(out).not.toContain('abcdef123456'); + // Each distinct redaction yields a [REDACTED] token somewhere. + expect((out.match(/\[REDACTED\]/g) || []).length).toBeGreaterThanOrEqual(5); + }); +}); diff --git a/test/eval-capture.test.ts b/test/eval-capture.test.ts new file mode 100644 index 000000000..ad60bcea9 --- /dev/null +++ b/test/eval-capture.test.ts @@ -0,0 +1,311 @@ +/** + * Eval capture module tests. + * + * buildEvalCandidateInput: shape + scrubbing + slug/chunk extraction. + * classifyCaptureFailure: SQLSTATE → reason mapping. + * captureEvalCandidate: best-effort swallow, failure routing, never throws. + * + * Engine is mocked so this file runs in unit speed (<50ms). + */ + +import { describe, expect, mock, test } from 'bun:test'; +import { + buildEvalCandidateInput, + captureEvalCandidate, + classifyCaptureFailure, + isEvalCaptureEnabled, + isEvalScrubEnabled, + type CaptureContext, +} from '../src/core/eval-capture.ts'; +import type { BrainEngine } from '../src/core/engine.ts'; +import type { SearchResult } from '../src/core/types.ts'; + +function makeResult(overrides: Partial = {}): SearchResult { + return { + slug: 'people/alice-example', + page_id: 1, + title: 'Alice Example', + type: 'person', + chunk_text: '…', + chunk_source: 'compiled_truth', + chunk_id: 42, + chunk_index: 0, + score: 0.9, + stale: false, + source_id: 'default', + ...overrides, + }; +} + +function makeCtx(overrides: Partial = {}): CaptureContext { + return { + tool_name: 'query', + query: 'who is alice', + results: [makeResult()], + meta: { vector_enabled: true, detail_resolved: 'medium', expansion_applied: false }, + latency_ms: 123, + remote: true, + expand_enabled: true, + detail: null, + job_id: null, + subagent_id: null, + ...overrides, + }; +} + +describe('buildEvalCandidateInput', () => { + test('builds a valid row from minimal context', () => { + const input = buildEvalCandidateInput(makeCtx()); + expect(input.tool_name).toBe('query'); + expect(input.query).toBe('who is alice'); + expect(input.retrieved_slugs).toEqual(['people/alice-example']); + expect(input.retrieved_chunk_ids).toEqual([42]); + expect(input.source_ids).toEqual(['default']); + expect(input.latency_ms).toBe(123); + expect(input.remote).toBe(true); + }); + + test('dedupes slugs across multiple chunks from the same page', () => { + const input = buildEvalCandidateInput( + makeCtx({ + results: [ + makeResult({ slug: 'people/alice-example', chunk_id: 1 }), + makeResult({ slug: 'people/alice-example', chunk_id: 2 }), + makeResult({ slug: 'companies/acme', chunk_id: 3 }), + ], + }), + ); + expect(input.retrieved_slugs).toEqual(['people/alice-example', 'companies/acme']); + // chunk ids preserve order and duplicates (each hit is distinct). + expect(input.retrieved_chunk_ids).toEqual([1, 2, 3]); + }); + + test('dedupes source_ids', () => { + const input = buildEvalCandidateInput( + makeCtx({ + results: [ + makeResult({ source_id: 'default' }), + makeResult({ source_id: 'default' }), + makeResult({ source_id: 'wiki' }), + ], + }), + ); + expect(input.retrieved_slugs).toHaveLength(1); + expect(input.source_ids.sort()).toEqual(['default', 'wiki']); + }); + + test('omits source_id from the set when a result lacks one (pre-v0.18 rows)', () => { + const input = buildEvalCandidateInput( + makeCtx({ results: [makeResult({ source_id: undefined })] }), + ); + expect(input.source_ids).toEqual([]); + }); + + test('scrubs PII by default', () => { + const input = buildEvalCandidateInput( + makeCtx({ query: 'email alice@example.com about the ticket' }), + ); + expect(input.query).not.toContain('alice@example.com'); + expect(input.query).toContain('[REDACTED]'); + }); + + test('preserves PII when scrub_pii is false', () => { + const input = buildEvalCandidateInput( + makeCtx({ query: 'email alice@example.com' }), + { scrub_pii: false }, + ); + expect(input.query).toBe('email alice@example.com'); + }); + + test('carries hybridSearch meta onto the row verbatim', () => { + const input = buildEvalCandidateInput( + makeCtx({ + meta: { vector_enabled: false, detail_resolved: 'high', expansion_applied: true }, + }), + ); + expect(input.vector_enabled).toBe(false); + expect(input.detail_resolved).toBe('high'); + expect(input.expansion_applied).toBe(true); + }); + + test('propagates jobId + subagentId when present (subagent tool-bridge path)', () => { + const input = buildEvalCandidateInput( + makeCtx({ job_id: 9001, subagent_id: 42 }), + ); + expect(input.job_id).toBe(9001); + expect(input.subagent_id).toBe(42); + }); +}); + +describe('classifyCaptureFailure', () => { + test('maps Postgres 23514 to check_violation', () => { + expect(classifyCaptureFailure({ code: '23514' })).toBe('check_violation'); + }); + test('maps Postgres 42501 to rls_reject', () => { + expect(classifyCaptureFailure({ code: '42501' })).toBe('rls_reject'); + }); + test('maps Postgres 42P01 to db_down', () => { + expect(classifyCaptureFailure({ code: '42P01' })).toBe('db_down'); + }); + test('maps connection SQLSTATEs to db_down', () => { + expect(classifyCaptureFailure({ code: '08006' })).toBe('db_down'); + expect(classifyCaptureFailure({ code: '53300' })).toBe('db_down'); + }); + test('maps RegExpMatchError to scrubber_exception', () => { + expect(classifyCaptureFailure({ name: 'RegExpMatchError' })).toBe('scrubber_exception'); + }); + test('falls back to other for unknown errors', () => { + expect(classifyCaptureFailure(new Error('something else'))).toBe('other'); + expect(classifyCaptureFailure('just a string')).toBe('other'); + expect(classifyCaptureFailure(null)).toBe('other'); + }); +}); + +describe('captureEvalCandidate — best-effort failure handling', () => { + function makeMockEngine(overrides: Partial = {}): BrainEngine { + return { + logEvalCandidate: mock(() => Promise.resolve(1)), + logEvalCaptureFailure: mock(() => Promise.resolve()), + ...overrides, + } as unknown as BrainEngine; + } + + test('happy path: calls logEvalCandidate and does not throw', async () => { + const engine = makeMockEngine(); + await expect(captureEvalCandidate(engine, makeCtx())).resolves.toBeUndefined(); + expect((engine.logEvalCandidate as unknown as { mock: { calls: unknown[] } }).mock.calls).toHaveLength(1); + expect((engine.logEvalCaptureFailure as unknown as { mock: { calls: unknown[] } }).mock.calls).toHaveLength(0); + }); + + test('DB error: routes to logEvalCaptureFailure with classified reason, does not throw', async () => { + const rlsErr = Object.assign(new Error('row violates row-level security'), { code: '42501' }); + const engine = makeMockEngine({ + logEvalCandidate: mock(() => Promise.reject(rlsErr)), + }); + await expect(captureEvalCandidate(engine, makeCtx())).resolves.toBeUndefined(); + const failureCalls = (engine.logEvalCaptureFailure as unknown as { mock: { calls: unknown[][] } }).mock.calls; + expect(failureCalls).toHaveLength(1); + expect(failureCalls[0]![0]).toBe('rls_reject'); + }); + + test('failure-of-failure: if the failure-log insert also rejects, still does not throw', async () => { + const engine = makeMockEngine({ + logEvalCandidate: mock(() => Promise.reject(new Error('primary fail'))), + logEvalCaptureFailure: mock(() => Promise.reject(new Error('secondary fail'))), + }); + await expect(captureEvalCandidate(engine, makeCtx())).resolves.toBeUndefined(); + }); + + test('sync scrubber throw lands in check path (mocked scrub would throw)', async () => { + // We can't force the real scrubber to throw on normal input — the + // adversarial-input test in eval-capture-scrub.test.ts covers that. + // This smoke test proves buildEvalCandidateInput throw propagates to + // the outer try/catch by handing an engine that throws synchronously + // in logEvalCandidate to simulate the "anything after scrub can fail". + const engine = makeMockEngine({ + logEvalCandidate: mock(() => { + throw new Error('sync throw'); // not a rejection — a sync throw + }), + }); + await expect(captureEvalCandidate(engine, makeCtx())).resolves.toBeUndefined(); + expect( + (engine.logEvalCaptureFailure as unknown as { mock: { calls: unknown[] } }).mock.calls, + ).toHaveLength(1); + }); +}); + +describe('isEvalCaptureEnabled / isEvalScrubEnabled (CONTRIBUTOR_MODE-gated)', () => { + // v0.25.0 flipped the default: was on for everyone, now off unless either + // the env var or an explicit config flag is set. Tests scope env mutation + // so they don't leak across describe blocks. + const origMode = process.env.GBRAIN_CONTRIBUTOR_MODE; + const restore = () => { + if (origMode === undefined) delete process.env.GBRAIN_CONTRIBUTOR_MODE; + else process.env.GBRAIN_CONTRIBUTOR_MODE = origMode; + }; + + test('defaults to OFF when config is missing AND CONTRIBUTOR_MODE unset', () => { + delete process.env.GBRAIN_CONTRIBUTOR_MODE; + try { + expect(isEvalCaptureEnabled(null)).toBe(false); + expect(isEvalCaptureEnabled(undefined)).toBe(false); + } finally { restore(); } + }); + + test('CONTRIBUTOR_MODE=1 turns capture on without any config', () => { + process.env.GBRAIN_CONTRIBUTOR_MODE = '1'; + try { + expect(isEvalCaptureEnabled(null)).toBe(true); + expect(isEvalCaptureEnabled(undefined)).toBe(true); + } finally { restore(); } + }); + + test('CONTRIBUTOR_MODE=anything-else does NOT turn capture on (strict equality)', () => { + process.env.GBRAIN_CONTRIBUTOR_MODE = 'true'; + try { + expect(isEvalCaptureEnabled(null)).toBe(false); + } finally { restore(); } + process.env.GBRAIN_CONTRIBUTOR_MODE = 'yes'; + try { + expect(isEvalCaptureEnabled(null)).toBe(false); + } finally { restore(); } + process.env.GBRAIN_CONTRIBUTOR_MODE = ''; + try { + expect(isEvalCaptureEnabled(null)).toBe(false); + } finally { restore(); } + }); + + test('explicit config eval.capture=true wins over absent CONTRIBUTOR_MODE', () => { + delete process.env.GBRAIN_CONTRIBUTOR_MODE; + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const enabled: any = { engine: 'pglite', eval: { capture: true } }; + expect(isEvalCaptureEnabled(enabled)).toBe(true); + } finally { restore(); } + }); + + test('explicit config eval.capture=false wins over CONTRIBUTOR_MODE=1', () => { + process.env.GBRAIN_CONTRIBUTOR_MODE = '1'; + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const disabled: any = { engine: 'pglite', eval: { capture: false } }; + expect(isEvalCaptureEnabled(disabled)).toBe(false); + } finally { restore(); } + }); + + test('config eval present but capture key undefined falls through to env check', () => { + delete process.env.GBRAIN_CONTRIBUTOR_MODE; + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const partial: any = { engine: 'pglite', eval: { scrub_pii: false } }; + expect(isEvalCaptureEnabled(partial)).toBe(false); + } finally { restore(); } + process.env.GBRAIN_CONTRIBUTOR_MODE = '1'; + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const partial: any = { engine: 'pglite', eval: { scrub_pii: false } }; + expect(isEvalCaptureEnabled(partial)).toBe(true); + } finally { restore(); } + }); + + test('isEvalScrubEnabled: defaults true, only false when explicitly disabled', () => { + expect(isEvalScrubEnabled(null)).toBe(true); + expect(isEvalScrubEnabled(undefined)).toBe(true); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const partial: any = { engine: 'pglite', eval: {} }; + expect(isEvalScrubEnabled(partial)).toBe(true); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const disabled: any = { engine: 'pglite', eval: { scrub_pii: false } }; + expect(isEvalScrubEnabled(disabled)).toBe(false); + }); + + test('isEvalScrubEnabled is independent of CONTRIBUTOR_MODE', () => { + process.env.GBRAIN_CONTRIBUTOR_MODE = '1'; + try { + expect(isEvalScrubEnabled(null)).toBe(true); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const opt: any = { engine: 'pglite', eval: { scrub_pii: false } }; + expect(isEvalScrubEnabled(opt)).toBe(false); + } finally { restore(); } + }); +}); diff --git a/test/eval-export.test.ts b/test/eval-export.test.ts new file mode 100644 index 000000000..966da281a --- /dev/null +++ b/test/eval-export.test.ts @@ -0,0 +1,178 @@ +/** + * gbrain eval export — NDJSON contract (v0.21.0). + * + * Verifies: + * - every line is valid JSON + * - every line has "schema_version": 1 + * - --since / --tool / --limit filter correctly + * - stdout ordering is newest-first + * - EPIPE on stdout doesn't crash the process (covered by invariant + * that runEvalExport returns without throwing on truncated streams) + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { runEvalExport } from '../src/commands/eval-export.ts'; +import type { EvalCandidateInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +function baseInput(overrides: Partial = {}): EvalCandidateInput { + return { + tool_name: 'query', + query: 'alice', + retrieved_slugs: ['people/alice-example'], + retrieved_chunk_ids: [1], + source_ids: ['default'], + expand_enabled: true, + detail: null, + detail_resolved: 'medium', + vector_enabled: false, + expansion_applied: false, + latency_ms: 100, + remote: false, + job_id: null, + subagent_id: null, + ...overrides, + }; +} + +beforeEach(async () => { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + await (engine as any).db.exec('DELETE FROM eval_candidates'); +}); + +/** + * Capture stdout.write output during runEvalExport. The command writes + * NDJSON directly to process.stdout; we intercept to inspect. + */ +async function captureExport(args: string[]): Promise { + const captured: string[] = []; + const originalWrite = process.stdout.write.bind(process.stdout); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (process.stdout as any).write = (chunk: string | Uint8Array) => { + captured.push(typeof chunk === 'string' ? chunk : new TextDecoder().decode(chunk)); + return true; + }; + try { + await runEvalExport(engine, args); + } finally { + process.stdout.write = originalWrite; + } + return captured.join(''); +} + +describe('gbrain eval export — NDJSON shape', () => { + test('empty table → zero lines, exit 0', async () => { + const out = await captureExport([]); + expect(out).toBe(''); + }); + + test('every row is a valid JSON object on its own line', async () => { + for (let i = 0; i < 3; i++) { + await engine.logEvalCandidate(baseInput({ query: `q${i}` })); + } + const out = await captureExport([]); + const lines = out.trim().split('\n'); + expect(lines).toHaveLength(3); + for (const line of lines) { + const parsed = JSON.parse(line); + expect(parsed).toHaveProperty('id'); + expect(parsed).toHaveProperty('query'); + expect(parsed).toHaveProperty('tool_name'); + } + }); + + test('every row starts with "schema_version":1 (F5 contract)', async () => { + await engine.logEvalCandidate(baseInput()); + const out = await captureExport([]); + const parsed = JSON.parse(out.trim()); + expect(parsed.schema_version).toBe(1); + }); + + test('--tool query filters to query rows only', async () => { + await engine.logEvalCandidate(baseInput({ tool_name: 'query' })); + await engine.logEvalCandidate( + baseInput({ tool_name: 'search', expand_enabled: null, detail_resolved: null }), + ); + await engine.logEvalCandidate(baseInput({ tool_name: 'query' })); + + const out = await captureExport(['--tool', 'query']); + const lines = out.trim().split('\n').filter(Boolean); + expect(lines).toHaveLength(2); + for (const line of lines) { + expect(JSON.parse(line).tool_name).toBe('query'); + } + }); + + test('--limit caps the row count', async () => { + for (let i = 0; i < 5; i++) { + await engine.logEvalCandidate(baseInput({ query: `q${i}` })); + } + const out = await captureExport(['--limit', '2']); + const lines = out.trim().split('\n').filter(Boolean); + expect(lines).toHaveLength(2); + }); + + test('--since DUR filter rejects everything older than the window', async () => { + await engine.logEvalCandidate(baseInput({ query: 'old' })); + // --since 1ms: rows created more than 1ms ago are excluded, but the + // insert we just did happened within the last 1ms so it might still + // slip through. Use a clearly past window instead. + const out = await captureExport(['--since', '0s']); + // 0s is parsed as "since now - 0ms" i.e. since now, which excludes + // everything that happened before the function started. + const lines = out.trim().split('\n').filter(Boolean); + expect(lines.length).toBeLessThanOrEqual(1); + }); + + test('invalid --tool value exits 1 (via process.exit mock check)', async () => { + // Replace process.exit so we can see the exit code without killing bun. + const originalExit = process.exit; + let exitCode: number | undefined; + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (process.exit as any) = (code?: number) => { + exitCode = code; + throw new Error('exit'); // abort the command + }; + try { + await captureExport(['--tool', 'dance']); + } catch { /* expected */ } + process.exit = originalExit; + expect(exitCode).toBe(1); + }); + + test('invalid --since format exits 1', async () => { + const originalExit = process.exit; + let exitCode: number | undefined; + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (process.exit as any) = (code?: number) => { + exitCode = code; + throw new Error('exit'); + }; + try { + await captureExport(['--since', 'yesterday']); + } catch { /* expected */ } + process.exit = originalExit; + expect(exitCode).toBe(1); + }); + + test('output is stream-friendly: each row on its own \\n-terminated line', async () => { + await engine.logEvalCandidate(baseInput()); + await engine.logEvalCandidate(baseInput({ query: 'second' })); + const out = await captureExport([]); + // Trailing newline on every record — no concatenated blobs. + expect(out).toMatch(/\}\n$/); + expect(out.split('\n').filter(Boolean)).toHaveLength(2); + }); +}); diff --git a/test/eval-prune.test.ts b/test/eval-prune.test.ts new file mode 100644 index 000000000..c2a72ddef --- /dev/null +++ b/test/eval-prune.test.ts @@ -0,0 +1,134 @@ +/** + * gbrain eval prune — retention cleanup (v0.21.0). + * + * Verifies: + * - --older-than parses duration strings (30d, 1h, 90m, 3600s) + * - deletes only rows older than the cutoff + * - --dry-run reports count without deleting + * - missing --older-than exits with error + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { runEvalPrune } from '../src/commands/eval-prune.ts'; +import type { EvalCandidateInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + await (engine as any).db.exec('DELETE FROM eval_candidates'); +}); + +function baseInput(): EvalCandidateInput { + return { + tool_name: 'query', + query: 'q', + retrieved_slugs: [], + retrieved_chunk_ids: [], + source_ids: [], + expand_enabled: true, + detail: null, + detail_resolved: null, + vector_enabled: false, + expansion_applied: false, + latency_ms: 50, + remote: false, + job_id: null, + subagent_id: null, + }; +} + +/** Insert a row with an explicit created_at timestamp (for "aged" data). */ +async function insertAged(daysAgo: number): Promise { + const when = new Date(Date.now() - daysAgo * 86_400_000).toISOString(); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + await (engine as any).db.query( + `INSERT INTO eval_candidates ( + tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids, + expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied, + latency_ms, remote, job_id, subagent_id, created_at + ) VALUES ('query', 'q', '{}', '{}', '{}', + true, null, null, false, false, 50, false, null, null, $1)`, + [when], + ); +} + +async function withSilencedStdout(fn: () => Promise): Promise { + const orig = console.log; + console.log = () => {}; + try { + return await fn(); + } finally { + console.log = orig; + } +} + +describe('gbrain eval prune', () => { + test('--older-than 30d deletes rows older than 30 days, keeps recent', async () => { + await insertAged(60); // old + await insertAged(45); // old + await insertAged(10); // kept + await engine.logEvalCandidate(baseInput()); // just now, kept + + expect(await engine.listEvalCandidates()).toHaveLength(4); + await withSilencedStdout(() => runEvalPrune(engine, ['--older-than', '30d'])); + + const remaining = await engine.listEvalCandidates(); + expect(remaining).toHaveLength(2); + }); + + test('--dry-run does not delete anything', async () => { + await insertAged(60); + await insertAged(45); + await withSilencedStdout(() => runEvalPrune(engine, ['--older-than', '30d', '--dry-run'])); + expect(await engine.listEvalCandidates()).toHaveLength(2); + }); + + test('accepts multiple duration units', async () => { + // Each of these runs should succeed without error. + for (const dur of ['1h', '90m', '3600s', '7d']) { + await withSilencedStdout(() => runEvalPrune(engine, ['--older-than', dur])); + } + }); + + test('missing --older-than exits 1', async () => { + const originalExit = process.exit; + let exitCode: number | undefined; + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (process.exit as any) = (code?: number) => { + exitCode = code; + throw new Error('exit'); + }; + try { + await runEvalPrune(engine, []); + } catch { /* expected */ } + process.exit = originalExit; + expect(exitCode).toBe(1); + }); + + test('invalid --older-than value exits 1', async () => { + const originalExit = process.exit; + let exitCode: number | undefined; + // eslint-disable-next-line @typescript-eslint/no-explicit-any + (process.exit as any) = (code?: number) => { + exitCode = code; + throw new Error('exit'); + }; + try { + await runEvalPrune(engine, ['--older-than', 'foo']); + } catch { /* expected */ } + process.exit = originalExit; + expect(exitCode).toBe(1); + }); +}); diff --git a/test/eval-replay.test.ts b/test/eval-replay.test.ts new file mode 100644 index 000000000..c9d79e0a8 --- /dev/null +++ b/test/eval-replay.test.ts @@ -0,0 +1,421 @@ +/** + * Tests for `gbrain eval replay` (v0.25.0). + * + * Three layers: + * 1. Pure: Jaccard math + NDJSON parser via re-export. + * 2. CLI happy path: --against captures NDJSON → human/JSON output. + * 3. CLI failure: missing file, bad NDJSON, mismatched schema_version. + */ + +import { describe, test, expect } from 'bun:test'; +import { writeFileSync, mkdtempSync, rmSync } from 'fs'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { runEvalReplay } from '../src/commands/eval-replay.ts'; +import type { BrainEngine } from '../src/core/engine.ts'; +import type { SearchResult } from '../src/core/types.ts'; + +// Minimal stub engine — only the methods replayRow touches matter. +// Every other method is a thrower so accidental use surfaces in tests. +function makeStubEngine(returns: Record): BrainEngine { + return { + kind: 'pglite', + searchKeyword: async (q: string) => returns[q] ?? [], + searchVector: async () => [], + } as unknown as BrainEngine; +} + +function fakeResult(slug: string, score = 0.5): SearchResult { + return { + slug, + chunk_id: 1, + chunk_index: 0, + chunk_text: '', + score, + title: slug, + page_kind: 'markdown', + source_id: 'default', + } as unknown as SearchResult; +} + +function makeCapturedRow(over: Partial<{ + id: number; + tool_name: 'query' | 'search'; + query: string; + retrieved_slugs: string[]; + detail: 'low' | 'medium' | 'high' | null; + expand_enabled: boolean | null; + latency_ms: number; +}>) { + return { + schema_version: 1, + id: over.id ?? 1, + tool_name: over.tool_name ?? 'search', + query: over.query ?? 'alice', + retrieved_slugs: over.retrieved_slugs ?? ['people/alice', 'people/alice-bio'], + retrieved_chunk_ids: [], + source_ids: [], + expand_enabled: over.expand_enabled ?? null, + detail: over.detail ?? null, + detail_resolved: null, + vector_enabled: false, + expansion_applied: false, + latency_ms: over.latency_ms ?? 50, + remote: true, + job_id: null, + subagent_id: null, + created_at: '2026-04-25T00:00:00Z', + }; +} + +function withTmp(fn: (path: string) => Promise | T): Promise { + const dir = mkdtempSync(join(tmpdir(), 'eval-replay-')); + return Promise.resolve(fn(dir)).finally(() => rmSync(dir, { recursive: true, force: true })); +} + +function captureStdoutStderr(): { restore: () => { stdout: string; stderr: string } } { + // Bun's console.log doesn't route through process.stdout.write — it uses + // internal writers. Hook console directly so test capture works across + // console.log/error/warn AND raw stream writes. + let outBuf = ''; + let errBuf = ''; + const origLog = console.log; + const origErrCons = console.error; + const origWarn = console.warn; + const origStdoutWrite = process.stdout.write.bind(process.stdout); + const origStderrWrite = process.stderr.write.bind(process.stderr); + const stringify = (a: unknown) => typeof a === 'string' ? a : JSON.stringify(a); + console.log = (...args: unknown[]) => { outBuf += args.map(stringify).join(' ') + '\n'; }; + console.error = (...args: unknown[]) => { errBuf += args.map(stringify).join(' ') + '\n'; }; + console.warn = (...args: unknown[]) => { errBuf += args.map(stringify).join(' ') + '\n'; }; + process.stdout.write = ((s: string | Uint8Array) => { + outBuf += typeof s === 'string' ? s : new TextDecoder().decode(s); + return true; + }) as typeof process.stdout.write; + process.stderr.write = ((s: string | Uint8Array) => { + errBuf += typeof s === 'string' ? s : new TextDecoder().decode(s); + return true; + }) as typeof process.stderr.write; + return { + restore: () => { + console.log = origLog; + console.error = origErrCons; + console.warn = origWarn; + process.stdout.write = origStdoutWrite; + process.stderr.write = origStderrWrite; + return { stdout: outBuf, stderr: errBuf }; + }, + }; +} + +describe('gbrain eval replay — happy path', () => { + test('search row with identical retrieved slugs reports Jaccard 1.0', async () => { + await withTmp(async (dir) => { + const ndjson = JSON.stringify(makeCapturedRow({ + tool_name: 'search', + query: 'alice', + retrieved_slugs: ['people/alice', 'people/alice-bio'], + })) + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, ndjson); + + const engine = makeStubEngine({ + alice: [fakeResult('people/alice'), fakeResult('people/alice-bio')], + }); + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file, '--json']); + const { stdout } = cap.restore(); + const out = JSON.parse(stdout); + expect(out.schema_version).toBe(1); + expect(out.summary.rows_replayed).toBe(1); + expect(out.summary.mean_jaccard).toBe(1.0); + expect(out.summary.top1_stability_rate).toBe(1.0); + }); + }); + + test('search row with disjoint retrieved slugs reports Jaccard 0', async () => { + await withTmp(async (dir) => { + const ndjson = JSON.stringify(makeCapturedRow({ + tool_name: 'search', + query: 'bob', + retrieved_slugs: ['people/bob', 'people/bob-bio'], + })) + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, ndjson); + + const engine = makeStubEngine({ + bob: [fakeResult('people/charlie'), fakeResult('people/charlie-bio')], + }); + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file, '--json']); + const { stdout } = cap.restore(); + const out = JSON.parse(stdout); + expect(out.summary.mean_jaccard).toBe(0); + expect(out.summary.top1_stability_rate).toBe(0); + }); + }); + + test('partial overlap — captured [a,b], current [a,c] → Jaccard 1/3', async () => { + await withTmp(async (dir) => { + const ndjson = JSON.stringify(makeCapturedRow({ + retrieved_slugs: ['a', 'b'], + query: 'q', + })) + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, ndjson); + + const engine = makeStubEngine({ q: [fakeResult('a'), fakeResult('c')] }); + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file, '--json']); + const { stdout } = cap.restore(); + const out = JSON.parse(stdout); + // {a,b} ∩ {a,c} = {a} (1), union {a,b,c} (3) → 1/3 + expect(out.summary.mean_jaccard).toBeCloseTo(1 / 3, 5); + // top-1 still matches (both lead with 'a') + expect(out.summary.top1_stability_rate).toBe(1.0); + }); + }); + + test('top-1 mismatch reduces stability rate', async () => { + await withTmp(async (dir) => { + const ndjson = JSON.stringify(makeCapturedRow({ + retrieved_slugs: ['a', 'b', 'c'], + query: 'q', + })) + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, ndjson); + + const engine = makeStubEngine({ q: [fakeResult('b'), fakeResult('a'), fakeResult('c')] }); + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file, '--json']); + const { stdout } = cap.restore(); + const out = JSON.parse(stdout); + // Same set, jaccard = 1.0 + expect(out.summary.mean_jaccard).toBe(1.0); + // top-1 swapped a → b, stability 0 + expect(out.summary.top1_stability_rate).toBe(0); + }); + }); + + test('multiple rows → averaged Jaccard + top-1 rate', async () => { + await withTmp(async (dir) => { + const lines = [ + JSON.stringify(makeCapturedRow({ id: 1, query: 'q1', retrieved_slugs: ['a'] })), + JSON.stringify(makeCapturedRow({ id: 2, query: 'q2', retrieved_slugs: ['b'] })), + ].join('\n') + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, lines); + + const engine = makeStubEngine({ + q1: [fakeResult('a')], // perfect match + q2: [fakeResult('z')], // miss + }); + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file, '--json']); + const { stdout } = cap.restore(); + const out = JSON.parse(stdout); + // (1.0 + 0) / 2 = 0.5 + expect(out.summary.mean_jaccard).toBe(0.5); + // (1 + 0) / 2 = 0.5 + expect(out.summary.top1_stability_rate).toBe(0.5); + expect(out.summary.rows_replayed).toBe(2); + }); + }); + + test('--limit caps replay count', async () => { + await withTmp(async (dir) => { + const lines = [1, 2, 3, 4, 5] + .map(i => JSON.stringify(makeCapturedRow({ id: i, query: `q${i}`, retrieved_slugs: ['a'] }))) + .join('\n') + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, lines); + + const engine = makeStubEngine({}); // returns [] for everything + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file, '--limit', '2', '--json']); + const { stdout } = cap.restore(); + const out = JSON.parse(stdout); + expect(out.summary.rows_total).toBe(2); + }); + }); + + test('empty query is skipped, not counted in replayed', async () => { + await withTmp(async (dir) => { + const ndjson = JSON.stringify(makeCapturedRow({ query: '', retrieved_slugs: [] })) + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, ndjson); + + const engine = makeStubEngine({}); + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file, '--json']); + const { stdout } = cap.restore(); + const out = JSON.parse(stdout); + expect(out.summary.rows_skipped).toBe(1); + expect(out.summary.rows_replayed).toBe(0); + }); + }); + + test('--verbose includes per-row results in JSON', async () => { + await withTmp(async (dir) => { + const ndjson = JSON.stringify(makeCapturedRow({ query: 'q', retrieved_slugs: ['a'] })) + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, ndjson); + + const engine = makeStubEngine({ q: [fakeResult('a')] }); + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file, '--json', '--verbose']); + const { stdout } = cap.restore(); + const out = JSON.parse(stdout); + expect(Array.isArray(out.results)).toBe(true); + expect(out.results.length).toBe(1); + expect(out.results[0].jaccard).toBe(1.0); + expect(out.results[0].current_slugs).toEqual(['a']); + }); + }); + + test('row that throws during replay → errored, not crash', async () => { + await withTmp(async (dir) => { + const ndjson = JSON.stringify(makeCapturedRow({ query: 'boom' })) + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, ndjson); + + const engine = { + kind: 'pglite' as const, + searchKeyword: async () => { throw new Error('engine offline'); }, + searchVector: async () => [], + } as unknown as BrainEngine; + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file, '--json']); + const { stdout } = cap.restore(); + const out = JSON.parse(stdout); + expect(out.summary.rows_errored).toBe(1); + expect(out.summary.rows_replayed).toBe(0); + }); + }); + + test('human mode prints summary and top regressions', async () => { + await withTmp(async (dir) => { + const ndjson = JSON.stringify(makeCapturedRow({ query: 'q', retrieved_slugs: ['a'] })) + '\n'; + const file = join(dir, 'baseline.ndjson'); + writeFileSync(file, ndjson); + + const engine = makeStubEngine({ q: [fakeResult('z')] }); // miss + + const cap = captureStdoutStderr(); + await runEvalReplay(engine, ['--against', file]); + const { stdout, stderr } = cap.restore(); + expect(stderr).toContain('Replaying 1'); + expect(stdout).toContain('Mean Jaccard@k:'); + expect(stdout).toContain('Top-1 stability:'); + expect(stdout).toContain('regression'); + }); + }); +}); + +describe('gbrain eval replay — failure modes', () => { + test('missing --against errors and exits 1', async () => { + const origExit = process.exit; + let code: number | undefined; + process.exit = (c?: number) => { code = c; throw new Error('exit'); }; + const cap = captureStdoutStderr(); + try { + await runEvalReplay({} as BrainEngine, []); + } catch { /* expected */ } + cap.restore(); + process.exit = origExit; + expect(code).toBe(1); + }); + + test('--against pointing at missing file errors and exits 1', async () => { + const origExit = process.exit; + let code: number | undefined; + process.exit = (c?: number) => { code = c; throw new Error('exit'); }; + const cap = captureStdoutStderr(); + try { + await runEvalReplay({} as BrainEngine, ['--against', '/tmp/nope-' + Date.now() + '.ndjson']); + } catch { /* expected */ } + cap.restore(); + process.exit = origExit; + expect(code).toBe(1); + }); + + test('NDJSON line missing schema_version is rejected', async () => { + await withTmp(async (dir) => { + const file = join(dir, 'bad.ndjson'); + writeFileSync(file, '{"id":1,"tool_name":"search","query":"q","retrieved_slugs":[]}\n'); + const origExit = process.exit; + let code: number | undefined; + process.exit = (c?: number) => { code = c; throw new Error('exit'); }; + const cap = captureStdoutStderr(); + try { + await runEvalReplay({} as BrainEngine, ['--against', file]); + } catch { /* expected */ } + const { stderr } = cap.restore(); + process.exit = origExit; + expect(code).toBe(1); + expect(stderr).toContain('schema_version'); + }); + }); + + test('schema_version === 2 (future) is rejected with helpful message', async () => { + await withTmp(async (dir) => { + const file = join(dir, 'v2.ndjson'); + writeFileSync(file, '{"schema_version":2,"id":1,"tool_name":"search","query":"q","retrieved_slugs":[],"latency_ms":0}\n'); + const origExit = process.exit; + let code: number | undefined; + process.exit = (c?: number) => { code = c; throw new Error('exit'); }; + const cap = captureStdoutStderr(); + try { + await runEvalReplay({} as BrainEngine, ['--against', file]); + } catch { /* expected */ } + const { stderr } = cap.restore(); + process.exit = origExit; + expect(code).toBe(1); + expect(stderr.toLowerCase()).toContain('upgrade gbrain'); + }); + }); + + test('empty file errors and exits 1', async () => { + await withTmp(async (dir) => { + const file = join(dir, 'empty.ndjson'); + writeFileSync(file, ''); + const origExit = process.exit; + let code: number | undefined; + process.exit = (c?: number) => { code = c; throw new Error('exit'); }; + const cap = captureStdoutStderr(); + try { + await runEvalReplay({} as BrainEngine, ['--against', file]); + } catch { /* expected */ } + cap.restore(); + process.exit = origExit; + expect(code).toBe(1); + }); + }); + + test('malformed JSON line is rejected with line number', async () => { + await withTmp(async (dir) => { + const file = join(dir, 'bad.ndjson'); + writeFileSync(file, '{"schema_version":1,"id":1}\nthis is not json\n'); + const origExit = process.exit; + let code: number | undefined; + process.exit = (c?: number) => { code = c; throw new Error('exit'); }; + const cap = captureStdoutStderr(); + try { + await runEvalReplay({} as BrainEngine, ['--against', file]); + } catch { /* expected */ } + const { stderr } = cap.restore(); + process.exit = origExit; + expect(code).toBe(1); + expect(stderr).toContain('line 2'); + }); + }); +}); diff --git a/test/hybrid-meta.test.ts b/test/hybrid-meta.test.ts new file mode 100644 index 000000000..a5194c3d4 --- /dev/null +++ b/test/hybrid-meta.test.ts @@ -0,0 +1,102 @@ +/** + * hybridSearch meta-field accuracy (v0.25.0, callback-based API). + * + * v0.25.0 keeps hybridSearch's return as `Promise` (so + * Cathedral II callers stay unchanged) and surfaces meta via an optional + * `onMeta` callback in HybridSearchOpts. Asserts the callback fires with + * accurate values: + * - vector_enabled=false when OPENAI_API_KEY missing (keyword-only path) + * - detail_resolved reflects auto-detect + caller override + * - expansion_applied only true when expandFn returned variants + * + * Uses PGLite in-memory + no embedding calls (vector path doesn't need + * real embeddings to test the meta flag since we control the env). + */ + +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { hybridSearch } from '../src/core/search/hybrid.ts'; +import type { PageInput, HybridSearchMeta } from '../src/core/types.ts'; + +let engine: PGLiteEngine; +const savedKey = process.env.OPENAI_API_KEY; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + const page: PageInput = { + type: 'person', + title: 'Alice Example', + compiled_truth: 'Alice Example is a test person for hybrid-meta tests.', + }; + await engine.putPage('people/alice-example', page); +}); + +afterAll(async () => { + if (savedKey === undefined) delete process.env.OPENAI_API_KEY; + else process.env.OPENAI_API_KEY = savedKey; + await engine.disconnect(); +}); + +async function runWithMeta(query: string, opts: Parameters[2] = {}): Promise { + let captured: HybridSearchMeta | null = null; + await hybridSearch(engine, query, { ...opts, onMeta: (m) => { captured = m; } }); + return captured; +} + +describe('hybridSearch return shape (v0.25.0 keeps SearchResult[])', () => { + test('returns SearchResult[] (unchanged from Cathedral II contract)', async () => { + delete process.env.OPENAI_API_KEY; + const out = await hybridSearch(engine, 'alice'); + expect(Array.isArray(out)).toBe(true); + }); +}); + +describe('hybridSearch onMeta callback — vector_enabled', () => { + test('false when OPENAI_API_KEY is missing (keyword-only path)', async () => { + delete process.env.OPENAI_API_KEY; + const meta = await runWithMeta('alice'); + expect(meta).not.toBeNull(); + expect(meta!.vector_enabled).toBe(false); + }); +}); + +describe('hybridSearch onMeta callback — detail_resolved', () => { + test('passes through explicit detail override (caller specified "high")', async () => { + delete process.env.OPENAI_API_KEY; + const meta = await runWithMeta('alice', { detail: 'high' }); + expect(meta!.detail_resolved).toBe('high'); + }); + + test('detail_resolved reflects autoDetect output when caller omits detail', async () => { + delete process.env.OPENAI_API_KEY; + const meta = await runWithMeta('alice'); + expect([null, 'low', 'medium', 'high']).toContain(meta!.detail_resolved); + }); +}); + +describe('hybridSearch onMeta callback — expansion_applied', () => { + test('false when expansion flag is off', async () => { + delete process.env.OPENAI_API_KEY; + const meta = await runWithMeta('alice', { expansion: false }); + expect(meta!.expansion_applied).toBe(false); + }); + + test('false when OPENAI_API_KEY missing (early-return short-circuits expansion)', async () => { + delete process.env.OPENAI_API_KEY; + const meta = await runWithMeta('alice', { + expansion: true, + expandFn: async () => ['alice', 'alice example', 'the person alice'], + }); + expect(meta!.expansion_applied).toBe(false); + }); +}); + +describe('onMeta callback omitted', () => { + test('hybridSearch works without onMeta (existing Cathedral II callers unaffected)', async () => { + delete process.env.OPENAI_API_KEY; + const out = await hybridSearch(engine, 'alice'); + expect(Array.isArray(out)).toBe(true); + }); +}); diff --git a/test/mcp-eval-capture.test.ts b/test/mcp-eval-capture.test.ts new file mode 100644 index 000000000..490efd266 --- /dev/null +++ b/test/mcp-eval-capture.test.ts @@ -0,0 +1,245 @@ +/** + * Op-layer capture integration test (v0.21.0). + * + * The hook lives at the `query` and `search` op handlers in + * src/core/operations.ts, not at the MCP dispatch site — this catches + * MCP callers, CLI callers, and subagent tool-bridge callers all from + * one code path. + * + * This test invokes the op handler directly with various + * OperationContext shapes (remote MCP, local CLI, subagent with jobId) + * and asserts that captured rows carry the expected origin metadata. + * + * Capture is fire-and-forget from the caller — but the INSERT fires on + * the same tick, so awaiting a microtask boundary is enough to let the + * row land before we assert. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { operations } from '../src/core/operations.ts'; +import type { OperationContext } from '../src/core/operations.ts'; +import type { GBrainConfig } from '../src/core/config.ts'; +import type { PageInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; +const savedKey = process.env.OPENAI_API_KEY; + +beforeAll(async () => { + delete process.env.OPENAI_API_KEY; // force keyword-only path so tests don't need live credentials + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + const page: PageInput = { + type: 'person', + title: 'Alice Example', + compiled_truth: 'Alice Example for op-layer capture tests.', + }; + await engine.putPage('people/alice-example', page); +}); + +afterAll(async () => { + if (savedKey === undefined) delete process.env.OPENAI_API_KEY; + else process.env.OPENAI_API_KEY = savedKey; + await engine.disconnect(); +}); + +beforeEach(async () => { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + await (engine as any).db.exec('DELETE FROM eval_candidates'); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + await (engine as any).db.exec('DELETE FROM eval_capture_failures'); +}); + +function makeConfig(overrides: Partial = {}): GBrainConfig { + return { + engine: 'pglite', + eval: { capture: true, scrub_pii: true, ...overrides }, + }; +} + +function makeCtx(overrides: Partial = {}): OperationContext { + return { + engine, + config: makeConfig(), + logger: console, + dryRun: false, + remote: true, + ...overrides, + } as OperationContext; +} + +/** Tiny helper: wait for fire-and-forget INSERT to land. */ +async function waitForCapture(): Promise { + // Two microtask cycles is plenty — the handler already awaited the op, + // so the fire-and-forget INSERT is enqueued. One await resolves the + // logEvalCandidate promise; one more for any follow-up. + await new Promise(r => setTimeout(r, 10)); +} + +describe('op-layer capture — query', () => { + const queryOp = operations.find(o => o.name === 'query')!; + + test('captures MCP query call with remote=true', async () => { + const ctx = makeCtx({ remote: true }); + await queryOp.handler(ctx, { query: 'alice' }); + await waitForCapture(); + + const rows = await engine.listEvalCandidates(); + expect(rows).toHaveLength(1); + const row = rows[0]!; + expect(row.tool_name).toBe('query'); + expect(row.query).toBe('alice'); + expect(row.remote).toBe(true); + expect(row.expand_enabled).toBe(true); // default + expect(row.vector_enabled).toBe(false); // OPENAI_API_KEY deleted + expect(row.job_id).toBeNull(); + expect(row.subagent_id).toBeNull(); + }); + + test('captures CLI query call with remote=false', async () => { + const ctx = makeCtx({ remote: false }); + await queryOp.handler(ctx, { query: 'alice' }); + await waitForCapture(); + + const rows = await engine.listEvalCandidates(); + expect(rows[0]!.remote).toBe(false); + }); + + test('captures subagent query call with jobId + subagentId', async () => { + const ctx = makeCtx({ remote: true, jobId: 9001, subagentId: 42 }); + await queryOp.handler(ctx, { query: 'alice' }); + await waitForCapture(); + + const row = (await engine.listEvalCandidates())[0]!; + expect(row.job_id).toBe(9001); + expect(row.subagent_id).toBe(42); + }); + + test('scrubs PII by default before insert', async () => { + const ctx = makeCtx(); + await queryOp.handler(ctx, { query: 'email alice@example.com about it' }); + await waitForCapture(); + + const row = (await engine.listEvalCandidates())[0]!; + expect(row.query).not.toContain('alice@example.com'); + expect(row.query).toContain('[REDACTED]'); + }); + + test('preserves PII when scrub_pii is disabled', async () => { + const ctx = makeCtx({ + config: makeConfig({ scrub_pii: false }), + }); + await queryOp.handler(ctx, { query: 'email alice@example.com' }); + await waitForCapture(); + + const row = (await engine.listEvalCandidates())[0]!; + expect(row.query).toBe('email alice@example.com'); + }); + + test('does nothing when eval.capture is false (off-switch works)', async () => { + const ctx = makeCtx({ + config: makeConfig({ capture: false }), + }); + await queryOp.handler(ctx, { query: 'alice' }); + await waitForCapture(); + + const rows = await engine.listEvalCandidates(); + expect(rows).toHaveLength(0); + }); +}); + +describe('op-layer capture — search', () => { + const searchOp = operations.find(o => o.name === 'search')!; + + test('captures search call with tool_name="search" and vector_enabled=false', async () => { + const ctx = makeCtx(); + await searchOp.handler(ctx, { query: 'alice' }); + await waitForCapture(); + + const rows = await engine.listEvalCandidates(); + expect(rows).toHaveLength(1); + const row = rows[0]!; + expect(row.tool_name).toBe('search'); + expect(row.vector_enabled).toBe(false); // search never runs vector + expect(row.expand_enabled).toBeNull(); // no expansion concept for search + expect(row.detail_resolved).toBeNull(); + }); + + test('respects eval.capture=false off-switch', async () => { + const ctx = makeCtx({ + config: makeConfig({ capture: false }), + }); + await searchOp.handler(ctx, { query: 'alice' }); + await waitForCapture(); + expect(await engine.listEvalCandidates()).toHaveLength(0); + }); +}); + +describe('op-layer capture — non-query/search ops are NOT captured', () => { + test('list_pages does not insert into eval_candidates', async () => { + const listPagesOp = operations.find(o => o.name === 'list_pages')!; + const ctx = makeCtx(); + await listPagesOp.handler(ctx, { limit: 10 }); + await waitForCapture(); + + const rows = await engine.listEvalCandidates(); + expect(rows).toHaveLength(0); + }); + + test('get_page does not insert into eval_candidates', async () => { + const getPageOp = operations.find(o => o.name === 'get_page')!; + const ctx = makeCtx(); + await getPageOp.handler(ctx, { slug: 'people/alice-example' }); + await waitForCapture(); + + const rows = await engine.listEvalCandidates(); + expect(rows).toHaveLength(0); + }); +}); + +describe('op-layer capture — failure isolation (F1/F2)', () => { + test('capture failures do not propagate to op response', async () => { + // Disconnect-then-reconnect the engine to simulate an INSERT failure. + // We can't easily inject a rejecting engine here since operations.ts + // imports the real one, but we can break the table temporarily. + // Drop the eval_candidates table, run the op, assert (a) op response + // succeeds and (b) a failure row appears in eval_capture_failures. + // eslint-disable-next-line @typescript-eslint/no-explicit-any + await (engine as any).db.exec('DROP TABLE eval_candidates'); + const queryOp = operations.find(o => o.name === 'query')!; + const ctx = makeCtx(); + // Op must still succeed and return results. + const results = await queryOp.handler(ctx, { query: 'alice' }); + expect(Array.isArray(results)).toBe(true); + + await waitForCapture(); + // Failure should have landed in the companion table. + const failures = await engine.listEvalCaptureFailures(); + expect(failures.length).toBeGreaterThanOrEqual(1); + expect(['db_down', 'other']).toContain(failures[0]!.reason); + + // Restore the table for subsequent tests. + // eslint-disable-next-line @typescript-eslint/no-explicit-any + await (engine as any).db.exec(` + CREATE TABLE IF NOT EXISTS eval_candidates ( + id SERIAL PRIMARY KEY, + tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')), + query TEXT NOT NULL CHECK (length(query) <= 51200), + retrieved_slugs TEXT[] NOT NULL DEFAULT '{}', + retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}', + source_ids TEXT[] NOT NULL DEFAULT '{}', + expand_enabled BOOLEAN, + detail TEXT, + detail_resolved TEXT, + vector_enabled BOOLEAN NOT NULL, + expansion_applied BOOLEAN NOT NULL, + latency_ms INTEGER NOT NULL, + remote BOOLEAN NOT NULL, + job_id INTEGER, + subagent_id INTEGER, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + ) + `); + }); +}); diff --git a/test/migrate.test.ts b/test/migrate.test.ts index 8d883b5cd..3210a3912 100644 --- a/test/migrate.test.ts +++ b/test/migrate.test.ts @@ -830,6 +830,72 @@ describe('PR #356 — non-transactional DDL runs via reserved connection', () => }); }); +describe('migration v31 — eval_capture_tables', () => { + test('exists with the expected name and is engine-specific (sqlFor)', () => { + const v31 = MIGRATIONS.find(m => m.version === 31); + expect(v31).toBeDefined(); + expect(v31?.name).toBe('eval_capture_tables'); + expect(v31?.sqlFor?.postgres).toBeDefined(); + expect(v31?.sqlFor?.pglite).toBeDefined(); + expect(v31?.sql).toBe(''); + }); + + test('creates both eval_candidates and eval_capture_failures on both engines', () => { + const v31 = MIGRATIONS.find(m => m.version === 31)!; + for (const variant of ['postgres', 'pglite'] as const) { + const sql = v31.sqlFor![variant]!; + expect(sql).toContain('CREATE TABLE IF NOT EXISTS eval_candidates'); + expect(sql).toContain('CREATE TABLE IF NOT EXISTS eval_capture_failures'); + } + }); + + test('enforces CHECK length(query) <= 51200', () => { + const v31 = MIGRATIONS.find(m => m.version === 31)!; + for (const variant of ['postgres', 'pglite'] as const) { + expect(v31.sqlFor![variant]!).toContain('CHECK (length(query) <= 51200)'); + } + }); + + test('enforces tool_name enum + reason enum', () => { + const v31 = MIGRATIONS.find(m => m.version === 31)!; + for (const variant of ['postgres', 'pglite'] as const) { + const sql = v31.sqlFor![variant]!; + expect(sql).toContain(`tool_name IN ('query', 'search')`); + expect(sql).toContain(`reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other')`); + } + }); + + test('creates DESC indexes on both tables', () => { + const v31 = MIGRATIONS.find(m => m.version === 31)!; + for (const variant of ['postgres', 'pglite'] as const) { + const sql = v31.sqlFor![variant]!; + expect(sql).toContain('idx_eval_candidates_created_at'); + expect(sql).toContain('idx_eval_capture_failures_ts'); + expect(sql).toContain('created_at DESC'); + expect(sql).toContain('ts DESC'); + } + }); + + test('Postgres variant gates RLS on BYPASSRLS and fails loudly', () => { + const pgSql = MIGRATIONS.find(m => m.version === 31)!.sqlFor!.postgres!; + expect(pgSql).toContain('rolbypassrls'); + expect(pgSql).toMatch(/IF NOT has_bypass/); + expect(pgSql).toMatch(/RAISE EXCEPTION[^;]*BYPASSRLS/); + expect(pgSql).toContain('ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY'); + expect(pgSql).toContain('ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY'); + }); + + test('PGLite variant has no RLS / no BYPASSRLS gate', () => { + const pgliteSql = MIGRATIONS.find(m => m.version === 31)!.sqlFor!.pglite!; + expect(pgliteSql).not.toContain('rolbypassrls'); + expect(pgliteSql).not.toContain('ENABLE ROW LEVEL SECURITY'); + }); + + test('LATEST_VERSION caught up to 31', () => { + expect(LATEST_VERSION).toBeGreaterThanOrEqual(31); + }); +}); + // ───────────────────────────────────────────────────────────────── // PR #363 regression guards — session timeouts via startup parameters // resolveSessionTimeouts — GBRAIN_*_TIMEOUT env overrides diff --git a/test/public-exports.test.ts b/test/public-exports.test.ts new file mode 100644 index 000000000..954f13073 --- /dev/null +++ b/test/public-exports.test.ts @@ -0,0 +1,102 @@ +/** + * Public exports contract test (v0.21.0 — Lane 2 / R2). + * + * Reads package.json "exports" at runtime, imports each subpath via the + * package name ("gbrain/") so it actually exercises the + * resolver — then asserts each resolves AND has at least one canary + * symbol. Importing from relative paths (e.g. "../src/core/engine.ts") + * would bypass the exports map and miss resolver/wiring breakage. + * + * The canary symbols are concrete values each module re-exports today. + * If a refactor renames or removes one, this test fails in CI so the + * downstream consumer (gbrain-evals) doesn't silently break first. + * + * To add a new public export: extend `EXPECTED_EXPORTS` below. To + * REMOVE one: bump gbrain's minor (breaking-interface per CLAUDE.md + * "Removing any of these is a breaking change going forward"). + */ + +import { readFileSync } from 'fs'; +import { resolve } from 'path'; +import { describe, expect, test } from 'bun:test'; + +interface ExpectedExport { + /** Subpath key as it appears in package.json exports. */ + subpath: string; + /** At least one named export that MUST exist at runtime. Chosen from the + * module's current surface; if it goes away, that's a breaking change. */ + canary: string[]; +} + +/** + * Canary symbols pinned to the v0.21.0 contract. Changes to this list + * are intentional breaking changes to the public exports surface. + */ +const EXPECTED_EXPORTS: ExpectedExport[] = [ + { subpath: 'gbrain', canary: [] }, // root "." export; no single canary — just require import success + { subpath: 'gbrain/engine', canary: ['clampSearchLimit', 'MAX_SEARCH_LIMIT'] }, + { subpath: 'gbrain/types', canary: ['GBrainError'] }, + { subpath: 'gbrain/operations', canary: ['operations', 'OperationError'] }, + { subpath: 'gbrain/minions', canary: [] }, // barrel module; re-exports many names + { subpath: 'gbrain/engine-factory', canary: [] }, // factory exports a default creator + { subpath: 'gbrain/pglite-engine', canary: ['PGLiteEngine'] }, + { subpath: 'gbrain/link-extraction', canary: ['extractEntityRefs', 'extractPageLinks'] }, + { subpath: 'gbrain/import-file', canary: ['importFromContent'] }, + { subpath: 'gbrain/transcription', canary: [] }, + { subpath: 'gbrain/embedding', canary: ['embed'] }, + { subpath: 'gbrain/config', canary: ['loadConfig'] }, + { subpath: 'gbrain/markdown', canary: ['splitBody', 'parseMarkdown', 'serializeMarkdown'] }, + { subpath: 'gbrain/backoff', canary: [] }, + { subpath: 'gbrain/search/hybrid', canary: ['hybridSearch', 'rrfFusion'] }, + { subpath: 'gbrain/search/expansion', canary: ['expandQuery'] }, + { subpath: 'gbrain/extract', canary: [] }, +]; + +function readPackageExports(): Record { + const pkgPath = resolve(import.meta.dir, '..', 'package.json'); + const pkg = JSON.parse(readFileSync(pkgPath, 'utf8')); + return pkg.exports as Record; +} + +describe('public exports — package.json exports map', () => { + test('has the expected number of subpaths (v0.21.0 locks the surface)', () => { + const exports = readPackageExports(); + const count = Object.keys(exports).length; + // Adding new exports: increment this + add to EXPECTED_EXPORTS below. + // Removing exports: see CLAUDE.md "Removing any of these is a + // breaking change going forward" — bump minor and update this count. + expect(count).toBe(17); + }); + + test('EXPECTED_EXPORTS list matches the exports map exactly (no drift)', () => { + const exports = readPackageExports(); + const exportedSubpaths = Object.keys(exports).map(k => (k === '.' ? 'gbrain' : `gbrain${k.slice(1)}`)).sort(); + const expectedSubpaths = EXPECTED_EXPORTS.map(e => e.subpath).sort(); + expect(expectedSubpaths).toEqual(exportedSubpaths); + }); +}); + +describe('public exports — every subpath resolves via package name', () => { + for (const entry of EXPECTED_EXPORTS) { + test(`${entry.subpath} imports without throwing`, async () => { + // Package-path import goes through the exports map — bypassing a + // broken/removed subpath surfaces here. Importing "../src/..." + // would resolve via filesystem and miss the contract. + const mod = await import(entry.subpath); + expect(mod).toBeDefined(); + expect(typeof mod).toBe('object'); + }); + + if (entry.canary.length > 0) { + test(`${entry.subpath} exports canary symbols: ${entry.canary.join(', ')}`, async () => { + const mod = await import(entry.subpath); + for (const name of entry.canary) { + expect(mod).toHaveProperty(name); + // Must be something truthy — value, class, or function. Not + // just a TypeScript type (those don't exist at runtime). + expect(mod[name]).toBeDefined(); + } + }); + } + } +});