mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.25.0 feat: BrainBench-Real session capture + public-exports contract test (#437)
* feat(v0.22.0): eval_candidates + eval_capture_failures schema (Lane 1A)
R1 substrate for BrainBench-Real, replayed onto master after Cathedral II
landed. Migration v30 (slotted after master's v25-v29 Cathedral II wave)
creates two tables:
eval_candidates: per-call capture of MCP/CLI/subagent query+search
traffic. Column set lets gbrain-evals replay with full fidelity —
source_ids from v0.18 multi-source, vector_enabled/detail_resolved/
expansion_applied so replay knows what hybridSearch actually did,
remote + job_id + subagent_id so rows are traceable to their origin.
query is CHECK-capped at 50KB; PII scrubber (Lane 1B) runs before insert.
eval_capture_failures: cross-process audit trail. In-process counters
don't work because `gbrain doctor` runs in a separate process from
the MCP server. Persistent rows let doctor query capture health via
COUNT(*) GROUP BY reason over the last 24h.
Both tables get RLS on Postgres gated on BYPASSRLS (matches v24/v29
posture). PGLite ignores RLS; sqlFor split carries only DDL.
5 new BrainEngine methods (breaking-interface addition, drives v0.22.0
minor bump): logEvalCandidate, listEvalCandidates,
deleteEvalCandidatesBefore, logEvalCaptureFailure, listEvalCaptureFailures.
listEvalCandidates uses ORDER BY created_at DESC, id DESC so
`gbrain eval export` is deterministic across same-millisecond inserts.
Also adds HybridSearchMeta type for the side-channel callback used by
Lane 1C's op-layer capture (no change to hybridSearch return shape —
that respects Cathedral II's existing SearchResult[] contract).
Tests: 14 PGLite round-trip cases + 8 v30 structural assertions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.22.0): PII scrubber + op-layer capture module (Lane 1B)
Replayed onto master post-Cathedral II. Same semantics as the original
v0.21.0 work — only adjusted to import HybridSearchMeta from types.ts
(canonical home) instead of redeclaring it locally.
src/core/eval-capture-scrub.ts — pure-function regex scrubber with 6
pattern families: emails, phones (US + E.164), SSN (year-aware),
Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. Zero
deps. Adversarial-input safe.
src/core/eval-capture.ts — op-layer hook helper:
- buildEvalCandidateInput(ctx, {scrub_pii}) — pure row builder
- classifyCaptureFailure(err) — Postgres SQLSTATE → reason tag
- captureEvalCandidate(engine, ctx, opts) — best-effort, never throws
- isEvalCaptureEnabled / isEvalScrubEnabled — file-plane config checks
GBrainConfig gains `eval?: {capture?, scrub_pii?}`. Both default ON.
File-plane only — `gbrain config set` writes the DB plane, doesn't
control capture.
Tests: 17 scrubber + 21 capture-module cases. Zero regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.22.0): hybridSearch onMeta callback + op-layer capture (Lane 1C)
Replayed onto master. Adapted from the original v0.21.0 work to keep
Cathedral II's contract intact: hybridSearch's return stays
`Promise<SearchResult[]>` (unchanged), and meta surfaces via an optional
`onMeta?: (meta: HybridSearchMeta) => void` callback in HybridSearchOpts.
Cathedral II callers leave onMeta undefined and pay no cost. The
op-layer capture wrapper passes a closure that threads meta into the
captured row so gbrain-evals can distinguish:
- "with OPENAI_API_KEY" vs "keyword-only fallback" (vector_enabled)
- "expansion fired" vs "expansion requested + silently fell back" (expansion_applied)
- what hybridSearch actually used after auto-detect (detail_resolved)
Op-layer capture wired into both `query` and `search` op handlers in
src/core/operations.ts. Single hook site catches MCP dispatch + CLI +
subagent tool-bridge from the same place. Fire-and-forget, never throws,
respects ctx.config.eval.capture off-switch.
Tests:
- test/hybrid-meta.test.ts (8 cases) — onMeta accuracy across the 4
return paths in hybridSearch + verification that omitting onMeta
leaves Cathedral II callers unchanged.
- test/mcp-eval-capture.test.ts (10 cases) — query/search ops capture
correctly with MCP/CLI/subagent contexts, scrub on/off, capture=false
off-switch, non-captured ops (list_pages, get_page), F1 failure
isolation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.22.0): gbrain eval export/prune + doctor eval_capture check (Lane 1D)
Replayed onto master. Same semantics as the original v0.21.0 work.
CLI:
gbrain eval export [--since DUR] [--limit N] [--tool query|search]
NDJSON to stdout, every row prefixed with "schema_version":1 per
docs/eval-capture.md contract. EPIPE-safe streaming, stderr
heartbeats, deterministic ordering (created_at DESC, id DESC).
gbrain eval prune --older-than DUR [--dry-run]
Explicit retention cleanup. Requires --older-than (never deletes
without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
Legacy bare `gbrain eval --qrels …` still works via sub-subcommand
fall-through.
gbrain doctor gains an eval_capture check between markdown_body_completeness
and queue_health: reads eval_capture_failures for the last 24h, groups by
reason, warns when non-zero. Pre-v30 brains get "Skipped (table
unavailable)" — non-fatal.
docs/eval-capture.md ships the stable NDJSON schema reference for
gbrain-evals consumers.
Tests: 9 export cases + 5 prune cases. Doctor check covered by
existing doctor tests on master.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.22.0): public-exports contract test + CI count guard (Lane 2 / R2)
Master locks 17 public subpath exports as gbrain's stable third-party
contract. Zero enforcement existed. This PR locks the surface in two
layers:
1. test/public-exports.test.ts — runtime contract test.
Reads package.json "exports" at startup. For each subpath, imports
via the package name ("gbrain/engine"), NOT the relative filesystem
path — that's the difference between exercising the actual resolver
and bypassing it. Every subpath gets a canary symbol pinned (e.g.
gbrain/search/hybrid must export hybridSearch + rrfFusion) so a
refactor that renames or removes one fails CI before downstream
consumers (gbrain-evals) silently break.
2. scripts/check-exports-count.sh — CI structural guard.
Wired into `bun test` after check-jsonb-pattern.sh +
check-progress-to-stdout.sh + check-wasm-embedded.sh per master's
precedent. EXPECTED_COUNT=17 baseline — shrinks fail loudly,
growth also fails so the new canary must be pinned in the runtime
test deliberately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs+e2e(v0.22.0): VERSION/CHANGELOG/CLAUDE/README + Postgres E2E (Lane 3)
Bump VERSION + package.json to 0.22.0 (next free slot after master's
v0.21.0 Code Cathedral II minor).
CHANGELOG.md v0.22.0 entry follows the Garry voice template:
- Bold 2-line headline
- Lead paragraph contextualizing v0.20 + v0.21 + v0.22 progression
- Numbers-that-matter table comparing v0.21.0 → v0.22.0
- "What this means for you" sectioned by audience
- "## To take advantage of v0.22.0" operator runbook
- Itemized changes
CLAUDE.md updates:
- Key files: 8 new module entries (eval-capture*, eval-export,
eval-prune, docs/eval-capture.md, public-exports test).
hybrid.ts entry rewritten to reflect the additive `onMeta` callback
(return shape unchanged).
- Key commands: new v0.22.0 section for `gbrain eval export`,
`gbrain eval prune`, and the doctor `eval_capture` check, with the
file-plane vs DB-plane config gotcha called out.
README.md: one-paragraph pointer after the BrainBench blurb so anyone
reading the landing page sees the new session-capture feature.
llms.txt + llms-full.txt regenerated to pick up the doc additions.
test/e2e/eval-capture.test.ts (Postgres-only E1 spec):
- CHECK violation surfaces as Postgres SQLSTATE 23514 on oversize input
- RLS is actually enabled on both eval_candidates + eval_capture_failures
- 50 concurrent logEvalCandidate calls — no deadlock, all distinct IDs
Skips gracefully when DATABASE_URL is unset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(todos): P0 — PGLite test-runner concurrency flake
Pre-existing on master, surfaces ~27 false failures when bun test runs all
174 files together. Each failing file passes in isolation. Tracked for a
dedicated investigation branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v0.22.0): adversarial review post-fixes (doctor RLS, onMeta safety)
Two surgical fixes from /ship adversarial review, plus 6 follow-ups TODO'd
into v0.22.1:
- doctor.ts: distinguish pre-v30 missing-table (42P01, ok skip) from
RLS-denied SELECT (42501, warn) and other DB errors (warn). The check
exists specifically to surface capture-failure misconfigs cross-process,
so silently reporting "ok / skipped" on the most diagnostic class
defeated the purpose.
- hybrid.ts: wrap onMeta invocation in try/catch via small emitMeta
helper. The callback is part of the public gbrain/search/hybrid
contract; a throwing user-supplied closure must never break the search
hot path.
- TODOS.md: 6 P1 follow-ups (eval prune real COUNT, scrubber CC false
positives, dead 'scrubber_exception' enum value, id-cursor for
cross-window dedup, public-export canary pinning, EXPECTED_COUNT dedup).
- TODOS.md: P0 entry for the pre-existing PGLite test-runner concurrency
flake (~27 false failures in full bun test on master).
- CHANGELOG.md: 2 bullets noting the doctor + onMeta hardening.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(version): bump v0.22.0 → v0.25.0 (queue-aware version pick)
Master is at v0.21.0. Open PRs claim v0.21.1 (#432) and v0.24.0 (#387).
v0.25 is the first uncontested slot, so this branch claims it. Pure
rename across VERSION, package.json, CHANGELOG header, and every "v0.22.0"
reference in CLAUDE.md / README.md / TODOS.md / docs/eval-capture.md /
src/ / test/ files. CHANGELOG date bumped to 2026-04-26.
llms.txt + llms-full.txt regenerated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.25.0): gbrain eval replay + contributor doc + CONTRIBUTING link
Closes the gap between "session capture works" (this PR's core) and
"contributors actually use it before merging." Three artifacts:
- src/commands/eval-replay.ts (~340 LOC) — reads NDJSON from `gbrain eval
export`, re-runs each captured query/search against the current brain,
computes set-Jaccard@k, top-1 stability, and latency delta. Stable JSON
shape (schema_version:1) for CI gating; human mode prints a regression
table sorted worst-first. Pure Bun, zero new deps. Stub-engine tests
cover Jaccard math, NDJSON parser (including v2 forward-compat
rejection + line-numbered errors), --limit, --verbose, --json, and
graceful per-row error handling. 16/16 passing.
- docs/eval-bench.md (~80 lines) — contributor guide. The 4-command loop
(export → change → replay → diff), metric definitions with healthy
ranges (Jaccard ≥0.85, top-1 ≥85%, latency Δ within ±50ms), trigger
paths, CI integration snippet, hand-crafted NDJSON corpus path for
fresh installs, and the off-switch. Pairs with the existing
docs/eval-capture.md which is the consumer-facing wire format.
- CONTRIBUTING.md gains a "Running real-world eval benchmarks (touching
retrieval code)" section with the trigger paths and a link to
docs/eval-bench.md. Reviewers now have a one-line ask: "did you run
replay?"
CLAUDE.md key files updated. CHANGELOG bullets added. llms.txt
regenerated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.25.0): CONTRIBUTOR_MODE flag — capture off by default for users
Eval capture was on for everyone in the v0.25.0 draft. Privacy footgun:
end users had retrieval traffic accumulate in their brain DB without
asking, even with PII scrubbing. Flips to off by default + explicit
opt-in for contributors who actually use the replay loop.
Resolution order in isEvalCaptureEnabled():
1. config.eval.capture === true → on
2. config.eval.capture === false → off
3. process.env.GBRAIN_CONTRIBUTOR_MODE === '1' → on
4. otherwise → off
The env var is the contributor-facing toggle (one line in .zshrc, no
JSON edit). Explicit config wins both directions for users who want to
override per-brain.
PII scrubbing gate stays independent — default true regardless of
CONTRIBUTOR_MODE — so any brain that does capture still scrubs.
Tests rewritten: env var hygiene per-test (origMode preserved + restored
in finally). 9/9 pass; total v0.25.0 suite is 198/198.
Docs:
- README.md gains a Contributing-section pointer to the env var.
- CONTRIBUTING.md gains a "CONTRIBUTOR_MODE — turn on the dev loop"
section with verification commands and resolution-order table.
- docs/eval-bench.md leads with the prerequisite (must set the env var
for the rest of the doc to be useful).
- docs/eval-capture.md "Config" section split into Path A (env var) +
Path B (config) with explicit resolution-order rules.
- CHANGELOG v0.25.0 entry corrected ("on by default" was wrong) plus a
new top itemized bullet calling out the gate change.
- CLAUDE.md eval-capture entry annotated with the new gate logic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: post-ship documentation pass for v0.25.0
Cross-references every doc against the final state of the branch
(CONTRIBUTOR_MODE flag, eval replay tool, off-by-default capture):
- README.md: top callout rewritten — was implying capture-on-by-default
contradicting the gate landed in 7a80ce25. Now leads with
"contributor opt-in" and links docs/eval-bench.md alongside
docs/eval-capture.md.
- AGENTS.md: new "Eval retrieval changes" task entry with the
CONTRIBUTOR_MODE+replay one-liner so non-Claude agents (Codex, Cursor,
Aider) have the same path.
- CLAUDE.md: "Key commands added in v0.25.0" gains the replay command and
a CONTRIBUTOR_MODE bullet covering the resolution order.
- CHANGELOG.md: headline rewritten to match the actual feature ("benchmark
retrieval changes against real captured queries before merging" — was
"every real query is captured"). Stale "v0.22 ships the substrate"
→ v0.25. Test count corrected 82 → 144 (added 16 replay + 9
CONTRIBUTOR_MODE + 8 v31-shape tests since the original count). Two
metric rows added to the numbers table: default-off posture, in-tree
replay tooling. "To take advantage" block split into user vs
contributor branches with shell-rc instructions.
- TODOS.md: v0.22.1 follow-up reference corrected to v0.25.1.
llms.txt + llms-full.txt regenerated. Typecheck clean. 198/198 v0.25.0
tests still green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
4fc1246606
commit
736e8de1ec
@@ -37,6 +37,11 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
[`docs/guides/minions-fix.md`](./docs/guides/minions-fix.md), `gbrain doctor --fix`.
|
||||
- **Migrate:** [`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md),
|
||||
[`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations`.
|
||||
- **Eval retrieval changes:** capture is off by default. To benchmark a
|
||||
retrieval change against real captured queries, set
|
||||
`GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson`
|
||||
and `gbrain eval replay --against base.ndjson`. Full guide:
|
||||
[`docs/eval-bench.md`](./docs/eval-bench.md).
|
||||
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
|
||||
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
|
||||
single-fetch ingestion.
|
||||
|
||||
@@ -2,6 +2,100 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.25.0] - 2026-04-26
|
||||
|
||||
## **Contributors can now benchmark retrieval changes against real captured queries before merging.**
|
||||
## **`GBRAIN_CONTRIBUTOR_MODE=1` turns on capture; `gbrain eval replay` is the dev loop.**
|
||||
|
||||
v0.20 (gbrain-evals extraction) and v0.21 (Cathedral II) gave gbrain its install surface back and turned code into a first-class graph. The remaining gap was data: `amara-life`, the fictional 418-item corpus over in gbrain-evals, is great for reproducibility but not for catching regressions against the queries your agents *actually* serve. v0.25 ships the substrate: with `GBRAIN_CONTRIBUTOR_MODE=1` set, every `query` and `search` call through MCP, CLI, or the subagent tool-bridge records into a new `eval_candidates` table. `gbrain eval export` streams the rows as NDJSON. `gbrain eval replay --against <ndjson>` re-runs each captured query against your current build and prints three numbers: mean Jaccard@k, top-1 stability, and latency Δ. Point gbrain-evals at the stream and you have BrainBench-Real on every release. Run replay locally and you have a regression gate on every retrieval PR.
|
||||
|
||||
Capture is **off by default**. Production users get a quiet brain — no surprise data accumulation, no privacy footgun. Contributors flip it on with one shell rc line: `export GBRAIN_CONTRIBUTOR_MODE=1`. From that shell forward, every query/search lands in `eval_candidates`. PII is scrubbed at write time (emails, phones, SSN, Luhn-verified credit cards, JWTs, bearer tokens). Queries over 50KB get rejected. RLS matches the v0.18.1 / v0.21.0 posture ... new tables get enabled on Postgres, gated on BYPASSRLS so it never locks an operator out of their own data. `gbrain doctor` surfaces silent capture failures cross-process so if something stops working you see it in health checks, not three weeks later when the replay numbers look weird.
|
||||
|
||||
Cathedral II (v0.21.0) callers are unaffected. `hybridSearch` still returns `Promise<SearchResult[]>` ... meta arrives via an optional `onMeta` callback in `HybridSearchOpts`, used only by the op-layer capture wrapper to record what hybridSearch *actually* did (vector ran or fell back, expansion fired or didn't, post-auto-detect detail).
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured on this branch's diff against v0.21.0:
|
||||
|
||||
| Metric | v0.21.0 | v0.25.0 | Δ |
|
||||
|---|---|---|---|
|
||||
| Real-query capture | none | every MCP/CLI/subagent `query` + `search` | **the whole feature** |
|
||||
| PII classes redacted at write | 0 | 6 (email, phone, SSN, CC+Luhn, JWT, bearer) | ... |
|
||||
| Schema columns per captured row | ... | 16 (tool, query, slugs, chunks, source_ids, expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied, latency, remote, job_id, subagent_id, created_at, id) | ... |
|
||||
| `hybridSearch` return shape | `Promise<SearchResult[]>` | `Promise<SearchResult[]>` (unchanged) | 0 break |
|
||||
| `hybridSearch` opts surface | existing | + `onMeta?: (m: HybridSearchMeta) => void` | additive |
|
||||
| BrainEngine methods | shipped Cathedral II surface | +5 eval-capture | **BREAKING for custom engines** |
|
||||
| Public subpath exports | 17 | 17 (now contract-tested) | 0 drift |
|
||||
| Default capture posture | n/a | OFF for users, ON via `GBRAIN_CONTRIBUTOR_MODE=1` | privacy-positive default |
|
||||
| Dev-loop tooling for retrieval PRs | none in-tree | `gbrain eval replay --against <ndjson>` | the regression gate |
|
||||
| New tests | ... | 144 (14 engine round-trip + 17 scrubber + 27 capture module + 8 hybrid meta + 10 op-layer capture + 9 export + 5 prune + 30 public-exports + 16 replay + 8 v31-migrate) | ... |
|
||||
|
||||
### What this means for you
|
||||
|
||||
**Brain operators (the 99%):** run `gbrain upgrade`. Nothing changes — capture stays off, your brain stays quiet. The substrate is in place if you ever want to turn it on (e.g. to debug a retrieval issue), but you don't have to.
|
||||
|
||||
**Contributors / maintainers:** add `export GBRAIN_CONTRIBUTOR_MODE=1` to your shell rc. Every `query`/`search` from then on writes to `eval_candidates`. After a week of dogfooding, snapshot with `gbrain eval export --since 7d > real.ndjson` and `gbrain eval replay --against real.ndjson` against your branch to gate retrieval changes. To opt out per-brain regardless of the env var: write `{"eval":{"capture":false}}` to `~/.gbrain/config.json`.
|
||||
|
||||
**Anyone calling `hybridSearch` directly:** no change required. The return type is still `Promise<SearchResult[]>`. If you want the new meta side-channel, pass `onMeta: (m) => { ... }` in opts — otherwise leave it undefined and pay no cost.
|
||||
|
||||
**Downstream TypeScript consumers implementing their own BrainEngine:** five new methods need implementations ... `logEvalCandidate`, `listEvalCandidates`, `deleteEvalCandidatesBefore`, `logEvalCaptureFailure`, `listEvalCaptureFailures`. Return types are in `src/core/types.ts`. This is why v0.25.0 is a minor bump.
|
||||
|
||||
## To take advantage of v0.25.0
|
||||
|
||||
**If you're a regular gbrain user:** `gbrain upgrade` is enough. Capture stays off, your brain stays quiet, nothing else changes. The substrate exists if you ever want to opt in (write `{"eval":{"capture":true}}` to `~/.gbrain/config.json`), but you don't have to.
|
||||
|
||||
**If you're a contributor or maintainer working on retrieval:**
|
||||
|
||||
1. **Apply the migration** (idempotent; `gbrain upgrade` does this automatically):
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
|
||||
2. **Turn on capture** by adding one line to your `~/.zshrc` or `~/.bashrc`:
|
||||
```bash
|
||||
export GBRAIN_CONTRIBUTOR_MODE=1
|
||||
```
|
||||
Reload your shell (`source ~/.zshrc`) or open a new terminal.
|
||||
|
||||
3. **Verify the tables exist + capture is healthy:**
|
||||
```bash
|
||||
gbrain query "any test query" >/dev/null
|
||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # > 0 means capture is running
|
||||
gbrain doctor # should show "eval_capture: No capture failures in the last 24h"
|
||||
```
|
||||
|
||||
4. **Dogfood for a week, then run the dev loop:**
|
||||
```bash
|
||||
gbrain eval export --since 7d > baseline.ndjson
|
||||
# ... make a retrieval change ...
|
||||
gbrain eval replay --against baseline.ndjson # mean Jaccard@k, top-1 stability, latency Δ
|
||||
```
|
||||
|
||||
See [`docs/eval-bench.md`](https://github.com/garrytan/gbrain/blob/master/docs/eval-bench.md) for the full guide and CI integration snippet.
|
||||
|
||||
5. **If something looks wrong:** `gbrain doctor` names the `eval_capture_failures` breakdown by reason. File an issue with the breakdown + your config shape at https://github.com/garrytan/gbrain/issues.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- **`GBRAIN_CONTRIBUTOR_MODE=1` env var gates capture.** Default flipped from on-for-everyone to off-unless-opted-in. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Production users get a quiet brain by default; contributors flip the env var in `.zshrc` and unlock the full export → replay loop. README + CONTRIBUTING document the flag prominently.
|
||||
- v31 migration: `eval_candidates` + `eval_capture_failures` on both Postgres + PGLite, RLS gated on BYPASSRLS, CHECK constraints + indexes
|
||||
- Op-layer capture wrapper in `src/core/operations.ts` (covers MCP + CLI + subagent tool-bridge from one site)
|
||||
- PII scrubber in `src/core/eval-capture-scrub.ts` (6 regex families, adversarial-input safe)
|
||||
- Cross-process audit via `eval_capture_failures` + `gbrain doctor` 24h breakdown
|
||||
- `gbrain eval export` (NDJSON, schema_version:1, EPIPE-safe) + `gbrain eval prune` (explicit retention)
|
||||
- `gbrain eval replay --against <ndjson>` — contributor-facing dev loop. Reads a captured snapshot, re-runs every `query` / `search` against the current brain, prints mean Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. JSON mode (`schema_version: 1`) for CI gating, top-regressions table for human eyeballs. Closes the gap between "data captured" and "data used to gate a PR."
|
||||
- `docs/eval-bench.md` — contributor guide that walks the 4-command loop (export → change → replay → diff), defines metrics (Jaccard@k, top-1 stability, latency Δ) with healthy ranges, lists the source paths that should trigger a re-run, and shows how to wire it into CI. Linked from CONTRIBUTING.md.
|
||||
- `CONTRIBUTING.md` adds a "Running real-world eval benchmarks (touching retrieval code)" section so PRs that change retrieval have an obvious replay path. Maintainer reviews can ask "did you run replay?" instead of writing a custom benchmark per PR.
|
||||
- `hybridSearch` adds `onMeta?: (m) => void` to opts (Cathedral II callers unaffected)
|
||||
- `BrainEngine` gains 5 methods (breaking-interface for custom engines, drives v0.25.0 minor bump)
|
||||
- `test/public-exports.test.ts` + `scripts/check-exports-count.sh` lock the 17-subpath public surface
|
||||
- Config gains `eval: {capture?, scrub_pii?}` (file-plane only); env var `GBRAIN_CONTRIBUTOR_MODE=1` is the contributor-facing toggle that doesn't require editing JSON
|
||||
- `listEvalCandidates` orders `created_at DESC, id DESC` (deterministic export windows)
|
||||
- `docs/eval-capture.md` — stable NDJSON schema reference for gbrain-evals
|
||||
- `gbrain doctor` `eval_capture` check now distinguishes pre-v31 missing-table (status: ok / skipped) from RLS-denied SELECT or transient DB error (status: warn) — the diagnostic that surfaces a misconfigured RLS role no longer goes silent
|
||||
- `hybridSearch.onMeta` callback wrapped in try/catch — a throwing user-supplied callback can't break the search hot path (defensive across the public `gbrain/search/hybrid` surface)
|
||||
- +144 unit tests across 9 new files (eval-candidates 14, eval-capture-scrub 17, eval-capture 27, eval-export 9, eval-prune 5, eval-replay 16, hybrid-meta 8, mcp-eval-capture 10, public-exports 30) plus 8 v31-shape checks added to `test/migrate.test.ts`. 0 regressions across the full suite (198 v0.25.0-related tests pass cleanly).
|
||||
|
||||
## [0.24.0] - 2026-04-26
|
||||
|
||||
## **The skillify loop stops lying. Privacy guard runs, `--llm` is honest, ghost rows go away.**
|
||||
|
||||
@@ -49,7 +49,16 @@ strict behavior when unset.
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow.
|
||||
- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
||||
- `src/core/search/hybrid.ts` — Cathedral II `Promise<SearchResult[]>` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined.
|
||||
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
||||
- `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`.
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
- `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic.
|
||||
@@ -212,6 +221,14 @@ Key commands added for Minions (job queue):
|
||||
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
|
||||
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
|
||||
|
||||
Key commands added in v0.25.0:
|
||||
- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay.
|
||||
- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
|
||||
- `gbrain eval replay --against FILE.ndjson [--limit N] [--top-regressions K] [--json] [--verbose]` — contributor-facing dev loop. Reads a captured NDJSON snapshot, re-runs each `query` / `search` op against the current brain, computes mean set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. JSON mode (`schema_version: 1`) for CI gating; human mode prints a regression table sorted worst-first. Closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `gbrain doctor` gains an `eval_capture` check: reads `eval_capture_failures` for the last 24h, groups by reason, warns when non-zero. Cross-process visibility (doctor runs in a separate process from MCP). Pre-v31 brains get `Skipped (table unavailable)` — non-fatal.
|
||||
- Config addition: `eval: { capture?: boolean, scrub_pii?: boolean }` in `~/.gbrain/config.json`. **File-plane only** — `gbrain config set` writes the DB plane and does NOT control capture.
|
||||
- **`GBRAIN_CONTRIBUTOR_MODE=1` env var** is the contributor-facing toggle. Capture is **off by default** as of v0.25.0; production users get a quiet brain. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Documented in README.md, CONTRIBUTING.md, and `docs/eval-bench.md`.
|
||||
|
||||
Key commands added in v0.12.2:
|
||||
- `gbrain repair-jsonb [--dry-run] [--json]` — repair double-encoded JSONB rows left over from v0.12.0-and-earlier Postgres writes. Idempotent; PGLite no-ops. The `v0_12_2` migration runs this automatically on `gbrain upgrade`.
|
||||
|
||||
|
||||
@@ -124,6 +124,84 @@ See `docs/ENGINES.md` for the full guide. In short:
|
||||
|
||||
The SQLite engine is designed and ready for implementation. See `docs/SQLITE_ENGINE.md`.
|
||||
|
||||
## CONTRIBUTOR_MODE — turn on the dev loop
|
||||
|
||||
gbrain captures retrieval traffic so you can replay real queries against
|
||||
your code changes before merging. **This is off by default** (production
|
||||
users get a quiet brain, no surprise data accumulation). Contributors turn
|
||||
it on with one shell rc line:
|
||||
|
||||
```bash
|
||||
# In ~/.zshrc or ~/.bashrc:
|
||||
export GBRAIN_CONTRIBUTOR_MODE=1
|
||||
```
|
||||
|
||||
That's it. Every `query` / `search` you (or agents pointed at your dev
|
||||
brain) run from that shell now writes a row to `eval_candidates`, and the
|
||||
[replay tool](#running-real-world-eval-benchmarks-touching-retrieval-code)
|
||||
has data to work against.
|
||||
|
||||
What CONTRIBUTOR_MODE actually does:
|
||||
|
||||
- Turns on `query`/`search` capture into the local `eval_candidates` table.
|
||||
Without it the gate is closed and capture is a no-op.
|
||||
- That's all. PII scrubbing, retention, and replay are independent.
|
||||
|
||||
Resolution order (most explicit wins):
|
||||
|
||||
1. `eval.capture: true` in `~/.gbrain/config.json` → on
|
||||
2. `eval.capture: false` in `~/.gbrain/config.json` → off
|
||||
3. `GBRAIN_CONTRIBUTOR_MODE=1` → on
|
||||
4. otherwise → off
|
||||
|
||||
Quick check that capture is actually running:
|
||||
|
||||
```bash
|
||||
gbrain query "anything" >/dev/null
|
||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates'
|
||||
# (or `gbrain doctor` — surfaces silent capture failures cross-process)
|
||||
```
|
||||
|
||||
To disable capture even with the env var set, write
|
||||
`{"eval": {"capture": false}}` to `~/.gbrain/config.json` — explicit config
|
||||
beats the env var both directions.
|
||||
|
||||
## Running real-world eval benchmarks (touching retrieval code)
|
||||
|
||||
If your PR touches retrieval — search ranking, RRF fusion, embeddings,
|
||||
intent classification, query expansion, source boost, or the `query` /
|
||||
`search` op handlers — run `gbrain eval replay` against a snapshot of
|
||||
real traffic before merging. Requires `CONTRIBUTOR_MODE` (above) so you
|
||||
have captured rows to replay against.
|
||||
|
||||
Quick loop:
|
||||
|
||||
```bash
|
||||
gbrain eval export --since 7d > baseline.ndjson # snapshot before your change
|
||||
# ... make your change ...
|
||||
gbrain eval replay --against baseline.ndjson # diff retrieval, get Jaccard@k
|
||||
```
|
||||
|
||||
Three numbers come back: mean Jaccard@k between captured and current slug
|
||||
sets, top-1 stability, and mean latency Δ. The replay tool flags the worst
|
||||
regressions so you can eyeball whether the change is hurting real queries.
|
||||
|
||||
Trigger paths (rerun if your diff touches any of these):
|
||||
|
||||
- `src/core/search/hybrid.ts`
|
||||
- `src/core/search/source-boost.ts`, `sql-ranking.ts`
|
||||
- `src/core/search/intent.ts`, `expansion.ts`, `dedup.ts`
|
||||
- `src/core/embedding.ts`
|
||||
- `src/core/operations.ts` (query / search handlers)
|
||||
- `src/core/postgres-engine.ts` / `pglite-engine.ts` (searchKeyword /
|
||||
searchVector SQL)
|
||||
|
||||
See [`docs/eval-bench.md`](./docs/eval-bench.md) for the full guide
|
||||
including CI integration, hand-crafted NDJSON corpora (so a fresh checkout
|
||||
without captured data can still replay), and cost considerations. The
|
||||
NDJSON wire format is documented in
|
||||
[`docs/eval-capture.md`](./docs/eval-capture.md).
|
||||
|
||||
## Welcome PRs
|
||||
|
||||
- SQLite engine implementation
|
||||
|
||||
@@ -8,6 +8,8 @@ The brain wires itself. Every page write extracts entity references and creates
|
||||
|
||||
GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
|
||||
**New in v0.25.0 — BrainBench-Real (session capture, contributor opt-in):** with `GBRAIN_CONTRIBUTOR_MODE=1` set in your shell, every real `query` + `search` call through MCP, CLI, or the subagent tool-bridge gets captured (PII-scrubbed) into an `eval_candidates` table. Snapshot with `gbrain eval export`, replay against your code change with `gbrain eval replay`. Three numbers come back: mean Jaccard@k between captured and current retrieved slugs, top-1 stability, and latency Δ. **Off by default** for production users — no surprise data accumulation. Walkthrough: [docs/eval-bench.md](docs/eval-bench.md). NDJSON wire format: [docs/eval-capture.md](docs/eval-capture.md).
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
|
||||
> **LLMs:** fetch [`llms.txt`](llms.txt) for the documentation map, or [`llms-full.txt`](llms-full.txt) for the same map with core docs inlined in one fetch. **Agents:** start with [`AGENTS.md`](AGENTS.md) (or [`CLAUDE.md`](CLAUDE.md) if you're Claude Code).
|
||||
@@ -744,6 +746,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
|
||||
|
||||
If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in.
|
||||
|
||||
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
|
||||
|
||||
## License
|
||||
|
||||
@@ -687,6 +687,21 @@ iteration's residuals.
|
||||
|
||||
## P0
|
||||
|
||||
### PGLite test-runner concurrency flake (~27 false failures in full `bun test`)
|
||||
**What:** Fix the concurrent-PGLite-init flake that surfaces ~27 `error: PGLite not connected. Call connect() first.` failures when `bun test` runs all 174 unit-test files together. Each failing file passes in isolation; failures only appear under full-suite parallelism.
|
||||
|
||||
**Why:** The failures are masking real signal. /ship and any solo dev running `bun test` has to manually triage 27 results every time. Today they're all in `test/cathedral-ii-pglite.test.ts`, `test/cathedral-ii-brainbench.test.ts` (Layer 5/6/7/8 + parent_scope_coverage + call_graph_recall), `test/sync.test.ts` (4 dry-run cases), `test/reindex-code.test.ts` (Layer 13 E2). All exist on master and date back to v0.12.3-v0.21.0 — pre-existing, not caused by any one branch.
|
||||
|
||||
**Context:** Confirmed pre-existing on master via `git diff origin/master...HEAD --stat -- <failing files>` returning empty. Tests pass cleanly in 1-3-file batches. Wall clock for the full suite is 596s. Likely root causes: (a) PGLite has a singleton or shared OPFS-like state that races under parallel `PGlite.create()` calls, (b) `test/cathedral-ii-pglite.test.ts` "fresh-install schema" tests assume exclusive PGLite access, (c) bun test concurrency exceeds what PGLite's WASM init can handle.
|
||||
|
||||
**Pros:** Green suite signal. Faster shipping. Stops eroding trust in `bun test`.
|
||||
|
||||
**Cons:** Likely needs PGLite engine-per-test isolation (each test gets its own dedicated engine instance via tmpdir) or a `bun test --concurrency=N` cap. Both touch test infra used by 50+ files.
|
||||
|
||||
**Effort:** M (human: 1 day to root-cause + implement / CC: ~2-3 hours via /investigate).
|
||||
|
||||
**Discovered:** v0.25.0 ship, 2026-04-25.
|
||||
|
||||
### Fix `bun build --compile` WASM embedding for PGLite
|
||||
**What:** Submit PR to oven-sh/bun fixing WASM file embedding in `bun build --compile` (issue oven-sh/bun#15032).
|
||||
|
||||
@@ -726,6 +741,26 @@ iteration's residuals.
|
||||
|
||||
**Depends on:** v0.10.0 GStackBrain skill layer (shipped).
|
||||
|
||||
## P1 (new from v0.25.0 — eval-capture adversarial review)
|
||||
|
||||
### v0.25.0 eval-capture follow-ups (6 surgical hardenings)
|
||||
**Priority:** P1
|
||||
|
||||
**What:** Six targeted hardenings on the v0.25.0 eval-capture surface, all surfaced by the /ship adversarial review and triaged out of the v0.25.0 PR to keep scope tight:
|
||||
|
||||
1. `gbrain eval prune --dry-run`: replace the `listEvalCandidates(limit:100k) + filter` count with a real `engine.countEvalCandidatesBefore(date)` method. Today the warning at `eval-prune.ts:107-109` honestly tells the user the count may be undercounted, but a brain with > 100k rows + old data could still confuse a careful operator. New `BrainEngine` method on both engines, ~30 LOC, lifts the floor count to a true count.
|
||||
2. PII scrubber CC false-positive rate: 16-digit Luhn-valid order IDs / invoice numbers get redacted as `[REDACTED]`. Either require a contextual prefix (`card`, `cc`, `credit`) within N chars, or document the tradeoff explicitly in `docs/eval-capture.md`. The two approaches differ in coverage so list them as alternatives.
|
||||
3. `eval_capture_failures.reason` enum: `'scrubber_exception'` is dead telemetry — no realistic path emits it (the scrubber is regex-only and never throws). Either remove the value from the schema CHECK + enum, OR wrap `scrubPii` in a try-catch inside `buildEvalCandidateInput` so the value is actually reachable.
|
||||
4. `id DESC` tiebreaker docs: CLAUDE.md says "stable id-desc tiebreaker so `--since` windows never dupe/miss rows". This is true within a single call but doesn't prevent dupe/miss across overlapping windows when LIMIT < total. Either add a real `id`-cursor (`WHERE id < $cursor`) for export, or scope the doc claim to "within a single export call".
|
||||
5. Public-exports canaries: 6 of 17 subpaths (`gbrain` root, `/minions`, `/engine-factory`, `/transcription`, `/backoff`, `/extract`) have `canary: []` — the test only checks the import resolves, so a barrel module accidentally losing its named exports would still pass. Pin one stable canary symbol per subpath.
|
||||
6. `EXPECTED_COUNT` duplication: `scripts/check-exports-count.sh` and `test/public-exports.test.ts` both hardcode `17`. Drift risk. Make one read the other (or both compute from `package.json`).
|
||||
|
||||
**Why:** All 6 are real (some informational, some footgun-class) but each is small and surgical. Bundling into one v0.25.1 follow-up PR keeps the v0.25.0 ship clean and lets the fixes land with their own dedicated tests + CHANGELOG entry.
|
||||
|
||||
**Effort:** S total (human: ~half day / CC: ~1.5 hours).
|
||||
|
||||
**Discovered:** v0.25.0 ship adversarial review, 2026-04-25.
|
||||
|
||||
## P1 (new from v0.7.0)
|
||||
|
||||
### ~~Constrained health_check DSL for third-party recipes~~
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
# Running real-world eval benchmarks against your gbrain changes
|
||||
|
||||
Audience: gbrain maintainers and contributors. If you're touching retrieval
|
||||
(search, ranking, embeddings, intent classification, query expansion, source
|
||||
boost, hybrid fusion), this is the doc.
|
||||
|
||||
For the **NDJSON wire format** consumed by gbrain-evals, see
|
||||
[`eval-capture.md`](./eval-capture.md). This doc is the human dev loop
|
||||
that lives on top of that format.
|
||||
|
||||
## Prerequisite: turn on contributor mode
|
||||
|
||||
Capture is **off by default** for production users (privacy-positive — no
|
||||
surprise data accumulation). Contributors flip it on with one line:
|
||||
|
||||
```bash
|
||||
# In ~/.zshrc or ~/.bashrc:
|
||||
export GBRAIN_CONTRIBUTOR_MODE=1
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
gbrain query "anything" >/dev/null
|
||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates' # should be > 0
|
||||
```
|
||||
|
||||
To override (force on/off regardless of env var), edit `~/.gbrain/config.json`:
|
||||
|
||||
```json
|
||||
{"eval": {"capture": true}} // force on
|
||||
{"eval": {"capture": false}} // force off
|
||||
```
|
||||
|
||||
Explicit config beats the env var both directions.
|
||||
|
||||
## The 4-command loop
|
||||
|
||||
```bash
|
||||
# ① Capture: writes to eval_candidates whenever CONTRIBUTOR_MODE is set.
|
||||
# Inspect what's been collected:
|
||||
gbrain doctor # surfaces capture failures
|
||||
psql $DATABASE_URL -c 'SELECT count(*) FROM eval_candidates'
|
||||
|
||||
# ② Snapshot: freeze a baseline before your code change.
|
||||
gbrain eval export --since 7d > baseline.ndjson
|
||||
|
||||
# ③ Code change: do whatever you want — tune RRF_K, swap embed model, edit
|
||||
# hybrid.ts, add a new boost source, change the intent classifier.
|
||||
|
||||
# ④ Replay: re-run every captured query against the current build.
|
||||
gbrain eval replay --against baseline.ndjson
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```
|
||||
Replaying 247 captured queries…
|
||||
...25/247
|
||||
...50/247
|
||||
...
|
||||
Replayed 247 of 247 captured queries (0 skipped, 0 errored)
|
||||
Mean Jaccard@k: 0.927
|
||||
Top-1 stability: 91.5%
|
||||
Mean latency Δ: +14ms (current vs captured)
|
||||
|
||||
Top 5 regression(s):
|
||||
jaccard=0.20 captured=12 current=3 "find every reference to widget-co"
|
||||
jaccard=0.43 captured=14 current=8 "show me everything tagged for review"
|
||||
jaccard=0.50 captured=8 current=4 "what did alice say about the spec"
|
||||
...
|
||||
```
|
||||
|
||||
Three numbers tell you whether the change is safe to land:
|
||||
|
||||
| Metric | What it means | Healthy range |
|
||||
|---|---|---|
|
||||
| **Mean Jaccard@k** | Average overlap between captured retrieved slugs and current run's slugs. 1.0 = identical sets. | ≥0.85 for "neutral" changes. <0.7 means major retrieval shift. |
|
||||
| **Top-1 stability** | Fraction of queries whose #1 result didn't change. | ≥85% for tuning passes. <70% means top-of-funnel broke. |
|
||||
| **Mean latency Δ** | Current minus captured. Positive = slower now. | Within ±50ms of captured. >2× anywhere = regression alarm. |
|
||||
|
||||
## What it actually does
|
||||
|
||||
`gbrain eval replay` reads your NDJSON snapshot and, for each row:
|
||||
|
||||
1. Re-executes the same op (`searchKeyword` for `tool_name='search'`,
|
||||
`hybridSearch` for `tool_name='query'`) with the captured `detail` and
|
||||
`expand_enabled` values threaded back in.
|
||||
2. Captures the current `retrieved_slugs` (deduped, in result order).
|
||||
3. Computes set-Jaccard between captured and current slug sets.
|
||||
4. Records top-1 match (was the #1 result the same slug?).
|
||||
5. Records latency delta vs captured `latency_ms`.
|
||||
|
||||
It does NOT compute MRR or nDCG — those need ground-truth relevance labels,
|
||||
not a baseline comparison. For metric-against-truth eval, use
|
||||
`gbrain eval --qrels <path>` (the legacy IR-eval path, still supported). The
|
||||
replay tool answers a different question: "did my code change move
|
||||
retrieval, and which queries did it move most?"
|
||||
|
||||
## Best-effort by design
|
||||
|
||||
Replay is not pure. Three things can drift between capture and replay:
|
||||
|
||||
1. **Brain state** — your brain probably has more pages now than when the
|
||||
snapshot was taken. Unless you explicitly seed a fixed corpus, mean
|
||||
Jaccard will drop simply because new pages are eligible.
|
||||
2. **Embedding source** — if you changed `OPENAI_API_KEY` between capture
|
||||
and replay (or the embedding model rotated), vector-path results drift
|
||||
even with identical code.
|
||||
3. **Capture cap** — captured `retrieved_slugs` is a deduped set; it doesn't
|
||||
preserve internal ranking metadata. Two tools can return the same slug
|
||||
set with different scores — Jaccard will say 1.0, but a downstream
|
||||
consumer that orders by score may behave differently.
|
||||
|
||||
The metrics are **regression alarms on real queries**, not a hash check.
|
||||
Pair them with manual inspection of the top regressions.
|
||||
|
||||
## Cost
|
||||
|
||||
Every `query` row in the snapshot embeds the query string via OpenAI to run
|
||||
the vector half of `hybridSearch`. Cost is identical to a normal `gbrain
|
||||
query` invocation — text-embedding-3-large at OpenAI list price, batched
|
||||
inside a single replay row.
|
||||
|
||||
If you're iterating locally and don't want to pay per change, use
|
||||
`--limit 50` to cap rows replayed. The 50 most recent rows are usually
|
||||
enough to catch direction; expand for the final pre-merge run.
|
||||
|
||||
```bash
|
||||
# Iteration mode — 50 most recent queries
|
||||
gbrain eval replay --against baseline.ndjson --limit 50
|
||||
|
||||
# Pre-merge — full snapshot
|
||||
gbrain eval replay --against baseline.ndjson --top-regressions 20
|
||||
```
|
||||
|
||||
## CI integration
|
||||
|
||||
```bash
|
||||
gbrain eval replay --against baseline.ndjson --json > replay.json
|
||||
jq -e '.summary.mean_jaccard >= 0.85' replay.json || exit 1
|
||||
jq -e '.summary.top1_stability_rate >= 0.85' replay.json || exit 1
|
||||
```
|
||||
|
||||
Stable JSON shape (schema_version: 1):
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"summary": {
|
||||
"rows_total": 247,
|
||||
"rows_replayed": 247,
|
||||
"rows_skipped": 0,
|
||||
"rows_errored": 0,
|
||||
"mean_jaccard": 0.927,
|
||||
"top1_stability_rate": 0.915,
|
||||
"mean_latency_delta_ms": 14,
|
||||
"rows_over_2x_latency": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`--verbose` adds a `results: [...]` array with one entry per replayed row
|
||||
(useful for piping into jq or a notebook for deeper analysis).
|
||||
|
||||
## When to run this
|
||||
|
||||
Before merging anything that touches:
|
||||
|
||||
- `src/core/search/hybrid.ts` (RRF, fusion, dedup, two-pass retrieval)
|
||||
- `src/core/search/source-boost.ts` / `sql-ranking.ts` (per-source ranking)
|
||||
- `src/core/search/intent.ts` (auto-detail classification)
|
||||
- `src/core/search/expansion.ts` (Haiku query expansion)
|
||||
- `src/core/search/dedup.ts` (cross-page result collapse)
|
||||
- `src/core/embedding.ts` or any embedding model swap
|
||||
- `src/core/operations.ts` `query` or `search` op handlers (capture surface)
|
||||
- `src/core/postgres-engine.ts` / `pglite-engine.ts` `searchKeyword` /
|
||||
`searchVector` SQL
|
||||
|
||||
Skip for: schema-only migrations, doc changes, tests-only PRs, CLI ergonomics
|
||||
that don't touch retrieval.
|
||||
|
||||
## Building your own corpus
|
||||
|
||||
If you don't have captured traffic yet (fresh install, can't dogfood for a
|
||||
week before merging), you can hand-author an NDJSON file:
|
||||
|
||||
```jsonl
|
||||
{"schema_version":1,"id":1,"tool_name":"query","query":"who is alice","retrieved_slugs":["people/alice","people/alice-bio"],"expand_enabled":false,"detail":null,"latency_ms":0,"remote":false}
|
||||
{"schema_version":1,"id":2,"tool_name":"search","query":"acme deal","retrieved_slugs":["deals/acme-seed","companies/acme"],"latency_ms":0,"remote":false}
|
||||
```
|
||||
|
||||
Then run `gbrain eval replay --against handcrafted.ndjson` to confirm the
|
||||
authoritative slugs come back. This is the seam between the BrainBench-Real
|
||||
pipeline (replay against live captures) and the BrainBench fixed-fixture
|
||||
pipeline (`gbrain eval --qrels` with the sibling
|
||||
[gbrain-evals](https://github.com/garrytan/gbrain-evals) corpus).
|
||||
|
||||
## Off-switch
|
||||
|
||||
Two ways to disable capture:
|
||||
|
||||
```bash
|
||||
unset GBRAIN_CONTRIBUTOR_MODE # easy: just unset the env var
|
||||
```
|
||||
|
||||
Or force off regardless of the env var via `~/.gbrain/config.json`:
|
||||
|
||||
```json
|
||||
{"eval": {"capture": false}}
|
||||
```
|
||||
|
||||
Existing `eval_candidates` rows stay until you `gbrain eval prune
|
||||
--older-than 0d` (or just drop the table).
|
||||
|
||||
## Failure modes
|
||||
|
||||
| What you see | What it means |
|
||||
|---|---|
|
||||
| `Mean Jaccard@k: 0.4`, top regressions all in one source dir | Source boost or hard-exclude regression on that prefix |
|
||||
| `Top-1 stability: 30%`, mean Jaccard still high | RRF tuning shifted the rank order without changing the set — re-tune `rrfK` |
|
||||
| `Mean latency Δ: +500ms`, jaccard high | Vector path got slower; check embedding API or HNSW probes |
|
||||
| `rows_errored > 0` | One or more queries threw. Inspect first 3 in human output, or `--json` to see all `error_message` fields |
|
||||
| Many `skipped: empty query` | Capture ran on rows where someone passed empty `query` — check why those were captured |
|
||||
@@ -0,0 +1,160 @@
|
||||
# Eval capture — NDJSON schema reference
|
||||
|
||||
**Status:** stable from v0.21.0. Schema versioning via `schema_version`
|
||||
on every row; additive changes increment the minor version; removals
|
||||
are breaking-schema-v2.
|
||||
|
||||
**Audience:** downstream consumers (primarily the sibling
|
||||
[gbrain-evals](https://github.com/garrytan/gbrain-evals) repo) that
|
||||
replay captured real-world queries as a BrainBench-Real fixture.
|
||||
|
||||
## The pipeline
|
||||
|
||||
```
|
||||
MCP / CLI / subagent tool-bridge caller
|
||||
│
|
||||
▼
|
||||
src/core/operations.ts — query + search op handlers
|
||||
│
|
||||
│ (hybridSearch or searchKeyword)
|
||||
│
|
||||
▼
|
||||
{results, meta: HybridSearchMeta} ┌── captureEvalCandidate
|
||||
│ │ (fire-and-forget)
|
||||
▼ │
|
||||
return to caller ▼
|
||||
scrubPii(query) ←── src/core/eval-capture-scrub.ts
|
||||
│
|
||||
▼
|
||||
buildEvalCandidateInput
|
||||
│
|
||||
▼
|
||||
engine.logEvalCandidate
|
||||
│
|
||||
┌──────────────┴──────────────┐
|
||||
│ success │ fail
|
||||
▼ ▼
|
||||
INSERT into eval_candidates engine.logEvalCaptureFailure
|
||||
(reason: db_down | rls_reject |
|
||||
check_violation |
|
||||
scrubber_exception | other)
|
||||
```
|
||||
|
||||
## `gbrain eval export` — the consumer contract
|
||||
|
||||
```sh
|
||||
gbrain eval export [--since DUR] [--limit N] [--tool query|search]
|
||||
```
|
||||
|
||||
Emits NDJSON to **stdout**. One JSON object per `\n`-terminated line.
|
||||
stderr receives progress heartbeats. Every line starts with
|
||||
`"schema_version": 1` so a forward-compat parser can fail loudly on
|
||||
schema v2 instead of silently misparsing.
|
||||
|
||||
Typical usage from gbrain-evals:
|
||||
|
||||
```sh
|
||||
# Snapshot the last week of real traffic for replay
|
||||
gbrain eval export --since 7d > brainbench-real.ndjson
|
||||
```
|
||||
|
||||
```sh
|
||||
# Stream through jq for ad-hoc analysis
|
||||
gbrain eval export --tool query | jq -c 'select(.latency_ms > 500)'
|
||||
```
|
||||
|
||||
## Row schema (v1)
|
||||
|
||||
Every exported row has this shape. Field order in JSON output is not
|
||||
guaranteed; consumers MUST key by name, not position.
|
||||
|
||||
| Field | Type | Notes |
|
||||
|---|---|---|
|
||||
| `schema_version` | number | Always `1` on v1 rows. Forward-compat gate. |
|
||||
| `id` | number | Autoincrement primary key. Stable across exports. |
|
||||
| `tool_name` | `"query"` \| `"search"` | Which MCP operation captured this row. |
|
||||
| `query` | string | **Already PII-scrubbed** by `scrubPii` unless `eval.scrub_pii: false`. Emails / phones / SSN / Luhn-verified credit cards / JWTs / bearer tokens replaced with `[REDACTED]`. Max length 50KB (CHECK-enforced). |
|
||||
| `retrieved_slugs` | string[] | Deduplicated slugs that came back in `SearchResult[]`. |
|
||||
| `retrieved_chunk_ids` | number[] | Every chunk id in result order (duplicates preserved — one per hit). |
|
||||
| `source_ids` | string[] | Distinct `sources.id` values across the result set (v0.18 multi-source). Empty for pre-v0.18 rows that lacked the column. |
|
||||
| `expand_enabled` | boolean \| null | Whether the caller **requested** Haiku expansion. `null` for `search` (no expansion concept). |
|
||||
| `detail` | `"low"` \| `"medium"` \| `"high"` \| null | Detail level the caller **requested**. `null` when omitted. |
|
||||
| `detail_resolved` | `"low"` \| `"medium"` \| `"high"` \| null | What `hybridSearch` **actually used** after auto-detect. `null` when neither caller nor heuristic classified. |
|
||||
| `vector_enabled` | boolean | True iff vector search actually ran. `false` when `OPENAI_API_KEY` was missing or the embed call failed. **Replay MUST respect this** — rows with `false` only exercised the keyword path. |
|
||||
| `expansion_applied` | boolean | True iff Haiku expansion actually produced variants (not just "was requested"). |
|
||||
| `latency_ms` | number | Wall-clock duration of the op handler (includes capture itself — negligible since it's fire-and-forget). |
|
||||
| `remote` | boolean | `true` for MCP callers (untrusted), `false` for local CLI. Partitions "real agent traffic" from "operator probing." |
|
||||
| `job_id` | number \| null | `OperationContext.jobId` when the caller was a subagent tool-bridge. Null for MCP + CLI. |
|
||||
| `subagent_id` | number \| null | `OperationContext.subagentId` for subagent-owned runs. |
|
||||
| `created_at` | string (ISO 8601) | UTC timestamp of insert. |
|
||||
|
||||
## Ordering + determinism
|
||||
|
||||
`listEvalCandidates` orders by `created_at DESC, id DESC`. Same-
|
||||
millisecond inserts tie on `created_at`; `id DESC` is the stable
|
||||
tiebreaker. Replay tools can consume rows in order and assume:
|
||||
- no duplicate rows across calls with non-overlapping `--since` windows
|
||||
- no missed rows across calls that chain `--since` windows (window end
|
||||
of run 1 is the strict upper bound, not a soft cursor)
|
||||
|
||||
## Schema versioning promise
|
||||
|
||||
- **v1 (shipped v0.21.0)** — this document. All fields listed above.
|
||||
- **Additive changes** increment gbrain minor version (v0.25.0, v0.23.0
|
||||
…) and ship with new optional fields. Consumers keyed on known fields
|
||||
ignore unknown keys and keep working.
|
||||
- **Breaking changes** (rename, type change, removal) increment
|
||||
`schema_version` to 2. Consumers MUST branch on `schema_version` to
|
||||
stay compatible.
|
||||
|
||||
## `eval_capture_failures` — companion audit table
|
||||
|
||||
Not exported by `gbrain eval export`. Surfaced via `gbrain doctor`:
|
||||
|
||||
```sh
|
||||
gbrain doctor # warns when failures in last 24h > 0
|
||||
```
|
||||
|
||||
Reason enum (stable): `db_down` | `rls_reject` | `check_violation` |
|
||||
`scrubber_exception` | `other`. Cross-process visibility is the whole
|
||||
point — `gbrain doctor` runs in its own process and reads the table
|
||||
directly, so in-process counters wouldn't work.
|
||||
|
||||
## Config + CONTRIBUTOR_MODE
|
||||
|
||||
Capture is **off by default** as of v0.25.0 (was on for everyone in
|
||||
earlier drafts). Two paths to turn it on:
|
||||
|
||||
**Path A — env var (contributor opt-in, the common case):**
|
||||
|
||||
```bash
|
||||
export GBRAIN_CONTRIBUTOR_MODE=1 # in ~/.zshrc or ~/.bashrc
|
||||
```
|
||||
|
||||
**Path B — explicit config (`~/.gbrain/config.json`, file-plane only):**
|
||||
|
||||
```json
|
||||
{
|
||||
"engine": "postgres",
|
||||
"database_url": "...",
|
||||
"eval": {
|
||||
"capture": true,
|
||||
"scrub_pii": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Resolution order (most explicit wins):
|
||||
|
||||
1. `eval.capture: true` in config → on
|
||||
2. `eval.capture: false` in config → off (overrides CONTRIBUTOR_MODE=1)
|
||||
3. `GBRAIN_CONTRIBUTOR_MODE === '1'` → on
|
||||
4. otherwise → off
|
||||
|
||||
`scrub_pii` defaults to `true` independent of capture. Set
|
||||
`eval.scrub_pii: false` to preserve raw query text (only if you control
|
||||
the brain's distribution).
|
||||
|
||||
`gbrain config set eval.capture false` does **not** work — that
|
||||
command writes the DB-plane config, and the MCP server reads the
|
||||
file-plane. Edit the JSON directly or use the env var.
|
||||
+27
-1
@@ -50,6 +50,11 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
[`docs/guides/minions-fix.md`](./docs/guides/minions-fix.md), `gbrain doctor --fix`.
|
||||
- **Migrate:** [`docs/UPGRADING_DOWNSTREAM_AGENTS.md`](./docs/UPGRADING_DOWNSTREAM_AGENTS.md),
|
||||
[`skills/migrations/`](./skills/migrations/), `gbrain apply-migrations`.
|
||||
- **Eval retrieval changes:** capture is off by default. To benchmark a
|
||||
retrieval change against real captured queries, set
|
||||
`GBRAIN_CONTRIBUTOR_MODE=1`, then `gbrain eval export --since 7d > base.ndjson`
|
||||
and `gbrain eval replay --against base.ndjson`. Full guide:
|
||||
[`docs/eval-bench.md`](./docs/eval-bench.md).
|
||||
- **Everything else:** [`./llms.txt`](./llms.txt) is the full documentation map.
|
||||
[`./llms-full.txt`](./llms-full.txt) is the same map with core docs inlined for
|
||||
single-fetch ingestion.
|
||||
@@ -135,7 +140,16 @@ strict behavior when unset.
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow.
|
||||
- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
- `src/core/eval-capture-scrub.ts` (v0.25.0) — zero-deps PII scrubber: emails, phones, SSN, Luhn-verified credit cards, JWT-shaped tokens, bearer tokens.
|
||||
- `src/core/search/hybrid.ts` — Cathedral II `Promise<SearchResult[]>` return shape unchanged in v0.25.0. Adds `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture can record what hybridSearch actually did. Existing callers leave it undefined.
|
||||
- `docs/eval-capture.md` (v0.25.0) — stable NDJSON schema reference for gbrain-evals consumers.
|
||||
- `test/public-exports.test.ts` (v0.25.0 / R2) — runtime contract test. Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`.
|
||||
- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff
|
||||
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`.
|
||||
- `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic.
|
||||
@@ -298,6 +312,14 @@ Key commands added for Minions (job queue):
|
||||
- `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall.
|
||||
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
|
||||
|
||||
Key commands added in v0.25.0:
|
||||
- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay.
|
||||
- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
|
||||
- `gbrain eval replay --against FILE.ndjson [--limit N] [--top-regressions K] [--json] [--verbose]` — contributor-facing dev loop. Reads a captured NDJSON snapshot, re-runs each `query` / `search` op against the current brain, computes mean set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. JSON mode (`schema_version: 1`) for CI gating; human mode prints a regression table sorted worst-first. Closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `gbrain doctor` gains an `eval_capture` check: reads `eval_capture_failures` for the last 24h, groups by reason, warns when non-zero. Cross-process visibility (doctor runs in a separate process from MCP). Pre-v31 brains get `Skipped (table unavailable)` — non-fatal.
|
||||
- Config addition: `eval: { capture?: boolean, scrub_pii?: boolean }` in `~/.gbrain/config.json`. **File-plane only** — `gbrain config set` writes the DB plane and does NOT control capture.
|
||||
- **`GBRAIN_CONTRIBUTOR_MODE=1` env var** is the contributor-facing toggle. Capture is **off by default** as of v0.25.0; production users get a quiet brain. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Documented in README.md, CONTRIBUTING.md, and `docs/eval-bench.md`.
|
||||
|
||||
Key commands added in v0.12.2:
|
||||
- `gbrain repair-jsonb [--dry-run] [--json]` — repair double-encoded JSONB rows left over from v0.12.0-and-earlier Postgres writes. Idempotent; PGLite no-ops. The `v0_12_2` migration runs this automatically on `gbrain upgrade`.
|
||||
|
||||
@@ -1362,6 +1384,8 @@ The brain wires itself. Every page write extracts entity references and creates
|
||||
|
||||
GBrain is those patterns, generalized. 29 skills. Install in 30 minutes. Your agent does the work. As Garry's personal agent gets smarter, so does yours.
|
||||
|
||||
**New in v0.25.0 — BrainBench-Real (session capture, contributor opt-in):** with `GBRAIN_CONTRIBUTOR_MODE=1` set in your shell, every real `query` + `search` call through MCP, CLI, or the subagent tool-bridge gets captured (PII-scrubbed) into an `eval_candidates` table. Snapshot with `gbrain eval export`, replay against your code change with `gbrain eval replay`. Three numbers come back: mean Jaccard@k between captured and current retrieved slugs, top-1 stability, and latency Δ. **Off by default** for production users — no surprise data accumulation. Walkthrough: [docs/eval-bench.md](docs/eval-bench.md). NDJSON wire format: [docs/eval-capture.md](docs/eval-capture.md).
|
||||
|
||||
> **~30 minutes to a fully working brain.** Database ready in 2 seconds (PGLite, no server). You just answer questions about API keys.
|
||||
|
||||
> **LLMs:** fetch [`llms.txt`](llms.txt) for the documentation map, or [`llms-full.txt`](llms-full.txt) for the same map with core docs inlined in one fetch. **Agents:** start with [`AGENTS.md`](AGENTS.md) (or [`CLAUDE.md`](CLAUDE.md) if you're Claude Code).
|
||||
@@ -2098,6 +2122,8 @@ The skills in this repo are those patterns, generalized. What took 11 days to bu
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md). Run `bun test` for unit tests. For the full local CI gate (gitleaks + unit + all 29 E2E files in Docker, the same checks GH Actions runs), use `bun run ci:local` ... or `bun run ci:local:diff` for the diff-aware subset during fast iteration.
|
||||
|
||||
If you're working on retrieval or any of the search/embedding/ranking surface, set `GBRAIN_CONTRIBUTOR_MODE=1` in your shell rc and use `gbrain eval replay` to gate your changes against a snapshot of real captured queries — the dev loop is documented in [`docs/eval-bench.md`](docs/eval-bench.md). Capture is **off by default** for production users (no surprise data accumulation); the env var is the contributor opt-in.
|
||||
|
||||
PRs welcome for: new enrichment APIs, performance optimizations, additional engine backends, new skills following the conformance standard in `skills/skill-creator/SKILL.md`.
|
||||
|
||||
## License
|
||||
|
||||
+3
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.24.0",
|
||||
"version": "0.25.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -33,7 +33,7 @@
|
||||
"build:schema": "bash scripts/build-schema.sh",
|
||||
"build:llms": "bun run scripts/build-llms.ts",
|
||||
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
|
||||
"test": "scripts/check-privacy.sh && scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
|
||||
"test": "scripts/check-privacy.sh && scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && bun run typecheck && bun test --timeout=60000",
|
||||
"check:wasm": "scripts/check-wasm-embedded.sh",
|
||||
"check:newlines": "scripts/check-trailing-newline.sh",
|
||||
"test:e2e": "bash scripts/run-e2e.sh",
|
||||
@@ -46,6 +46,7 @@
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"check:privacy": "scripts/check-privacy.sh",
|
||||
"check:progress": "scripts/check-progress-to-stdout.sh",
|
||||
"check:exports-count": "scripts/check-exports-count.sh",
|
||||
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
|
||||
"prepublish:clawhub": "bun run build:all",
|
||||
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
||||
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: the public exports surface never shrinks silently (v0.21.0).
|
||||
#
|
||||
# Precedent: scripts/check-jsonb-pattern.sh + check-progress-to-stdout.sh
|
||||
# are grep-based structural guards wired into `bun run test`. This one
|
||||
# counts the entries in package.json "exports" and fails when the count
|
||||
# drops below the v0.21.0 baseline (17 entries).
|
||||
#
|
||||
# Policy (from CLAUDE.md):
|
||||
# "Removing any of these is a breaking change going forward."
|
||||
#
|
||||
# If you're legitimately removing a public export: bump gbrain's minor
|
||||
# version, note the removal in CHANGELOG.md under a "Breaking changes"
|
||||
# bullet, then bump EXPECTED_COUNT below. Anything else is a regression.
|
||||
#
|
||||
# Adding a new export: update EXPECTED_COUNT to match AND extend the
|
||||
# EXPECTED_EXPORTS list in test/public-exports.test.ts so the runtime
|
||||
# contract test pins the canary symbol.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
EXPECTED_COUNT=17
|
||||
|
||||
# Count top-level keys in the exports object. `node -e` parses JSON
|
||||
# reliably without needing jq (which isn't in every CI environment).
|
||||
ACTUAL=$(node -e "
|
||||
const pkg = require('./package.json');
|
||||
console.log(Object.keys(pkg.exports || {}).length);
|
||||
")
|
||||
|
||||
if [ "$ACTUAL" -lt "$EXPECTED_COUNT" ]; then
|
||||
echo "❌ public-exports guard: package.json exports shrank from $EXPECTED_COUNT to $ACTUAL"
|
||||
echo " Removing a public export is a breaking change (see CLAUDE.md)."
|
||||
echo " If intentional: bump gbrain minor version + update EXPECTED_COUNT in"
|
||||
echo " scripts/check-exports-count.sh and EXPECTED_EXPORTS in"
|
||||
echo " test/public-exports.test.ts, AND add a CHANGELOG 'Breaking changes' bullet."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$ACTUAL" -gt "$EXPECTED_COUNT" ]; then
|
||||
echo "⚠️ public-exports guard: package.json exports grew from $EXPECTED_COUNT to $ACTUAL"
|
||||
echo " Additive public API change. Update EXPECTED_COUNT in this script + the"
|
||||
echo " EXPECTED_EXPORTS list in test/public-exports.test.ts to lock the new"
|
||||
echo " canary symbols."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✓ public-exports guard: $ACTUAL entries (matches baseline $EXPECTED_COUNT)"
|
||||
@@ -719,6 +719,58 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
fmHb();
|
||||
}
|
||||
|
||||
// 11a-bis. Eval-capture health (v0.25.0). Capture is a fire-and-forget
|
||||
// side-effect that logs failures to a persistent table so this check
|
||||
// can see drops cross-process (the MCP server captures; `gbrain doctor`
|
||||
// runs in a separate process). Counts failures in the last 24h and
|
||||
// warns when non-zero. Pre-v31 brains: the table doesn't exist yet;
|
||||
// swallow the error and report skipped.
|
||||
progress.heartbeat('eval_capture');
|
||||
try {
|
||||
const since = new Date(Date.now() - 24 * 3600 * 1000);
|
||||
const failures = await engine.listEvalCaptureFailures({ since });
|
||||
if (failures.length === 0) {
|
||||
checks.push({ name: 'eval_capture', status: 'ok', message: 'No capture failures in the last 24h' });
|
||||
} else {
|
||||
const byReason = new Map<string, number>();
|
||||
for (const f of failures) {
|
||||
byReason.set(f.reason, (byReason.get(f.reason) ?? 0) + 1);
|
||||
}
|
||||
const breakdown = [...byReason.entries()]
|
||||
.sort((a, b) => b[1] - a[1])
|
||||
.map(([r, n]) => `${n} ${r}`)
|
||||
.join(', ');
|
||||
checks.push({
|
||||
name: 'eval_capture',
|
||||
status: 'warn',
|
||||
message: `${failures.length} capture failure(s) in the last 24h (${breakdown}). ` +
|
||||
`If you care about replay fidelity, investigate. If not, set eval.capture: false ` +
|
||||
`in ~/.gbrain/config.json to silence.`,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
// Distinguish "table doesn't exist yet" (pre-v31, ok skip) from real
|
||||
// problems like RLS denying SELECT — the latter masks the very condition
|
||||
// this check is supposed to surface (capture INSERTs almost certainly
|
||||
// also fail).
|
||||
const code = (err as { code?: string } | null)?.code;
|
||||
if (code === '42P01') {
|
||||
checks.push({ name: 'eval_capture', status: 'ok', message: 'Skipped (eval_capture_failures table unavailable — apply migrations or upgrade)' });
|
||||
} else if (code === '42501') {
|
||||
checks.push({
|
||||
name: 'eval_capture',
|
||||
status: 'warn',
|
||||
message: 'RLS denies SELECT on eval_capture_failures. Capture INSERTs are almost certainly failing too. Run as a role with BYPASSRLS or grant SELECT on this table.',
|
||||
});
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'eval_capture',
|
||||
status: 'warn',
|
||||
message: `Could not read eval_capture_failures: ${(err as Error)?.message ?? String(err)}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// 11b. Queue health (v0.19.1 queue-resilience wave).
|
||||
// Postgres-only because PGLite has no multi-process worker surface. Two
|
||||
// subchecks, both cheap (single SELECT each, status-index-covered):
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
/**
|
||||
* gbrain eval export — stream captured eval_candidates rows as NDJSON (v0.21.0).
|
||||
*
|
||||
* Consumer: sibling gbrain-evals repo, which imports the NDJSON as a
|
||||
* BrainBench-Real fixture alongside the fictional amara-life corpus.
|
||||
*
|
||||
* Output contract (stable from v0.21.0 — schema_version:1 on every row):
|
||||
* { "schema_version": 1, "id": N, "tool_name": "query"|"search", ... }\n
|
||||
*
|
||||
* The schema_version prefix lets gbrain-evals detect format drift and
|
||||
* warn on unknown versions instead of silently misparsing.
|
||||
*
|
||||
* Usage:
|
||||
* gbrain eval export [--since 7d] [--limit N] [--tool query|search] > rows.ndjson
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import type { EvalCandidate } from '../core/types.ts';
|
||||
|
||||
const SCHEMA_VERSION = 1;
|
||||
|
||||
interface ExportOpts {
|
||||
help?: boolean;
|
||||
since?: Date;
|
||||
limit?: number;
|
||||
tool?: 'query' | 'search';
|
||||
}
|
||||
|
||||
function parseDurationToMs(s: string): number | null {
|
||||
// Accepts "30d", "7d", "1h", "90m", "3600s". Same shape as gbrain eval prune + jobs prune.
|
||||
const m = s.match(/^(\d+)\s*(ms|s|m|h|d)$/);
|
||||
if (!m) return null;
|
||||
const n = parseInt(m[1]!, 10);
|
||||
const unit = m[2]!;
|
||||
const mults: Record<string, number> = {
|
||||
ms: 1,
|
||||
s: 1000,
|
||||
m: 60_000,
|
||||
h: 3_600_000,
|
||||
d: 86_400_000,
|
||||
};
|
||||
return n * mults[unit]!;
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): ExportOpts {
|
||||
const opts: ExportOpts = {};
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const arg = args[i]!;
|
||||
const next = args[i + 1];
|
||||
switch (arg) {
|
||||
case '--help':
|
||||
case '-h':
|
||||
opts.help = true;
|
||||
break;
|
||||
case '--since': {
|
||||
if (!next) break;
|
||||
const ms = parseDurationToMs(next);
|
||||
if (ms !== null) {
|
||||
opts.since = new Date(Date.now() - ms);
|
||||
} else {
|
||||
console.error(`Invalid --since value: ${next} (use like 7d, 1h, 30m)`);
|
||||
process.exit(1);
|
||||
}
|
||||
i++;
|
||||
break;
|
||||
}
|
||||
case '--limit':
|
||||
if (next) opts.limit = parseInt(next, 10);
|
||||
i++;
|
||||
break;
|
||||
case '--tool':
|
||||
if (next === 'query' || next === 'search') {
|
||||
opts.tool = next;
|
||||
} else if (next) {
|
||||
console.error(`Invalid --tool value: ${next} (use 'query' or 'search')`);
|
||||
process.exit(1);
|
||||
}
|
||||
i++;
|
||||
break;
|
||||
}
|
||||
}
|
||||
return opts;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
console.error(`gbrain eval export — emit captured eval_candidates as NDJSON to stdout
|
||||
|
||||
USAGE:
|
||||
gbrain eval export [--since DUR] [--limit N] [--tool query|search]
|
||||
|
||||
FLAGS:
|
||||
--since DUR Only rows created within DUR (e.g. 7d, 1h, 30m). Default: all.
|
||||
--limit N Cap rows returned. Default: 1000. Max: 100000.
|
||||
--tool X Filter to a specific tool ('query' or 'search'). Default: both.
|
||||
--help, -h Show this help.
|
||||
|
||||
OUTPUT:
|
||||
One JSON object per line on stdout. Every row begins with
|
||||
"schema_version": 1 so downstream consumers (gbrain-evals) can
|
||||
detect format changes.
|
||||
|
||||
EXAMPLES:
|
||||
gbrain eval export > rows.ndjson
|
||||
gbrain eval export --since 7d --tool query | jq '.query'
|
||||
gbrain eval export --limit 100 | head
|
||||
`);
|
||||
}
|
||||
|
||||
export async function runEvalExport(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const opts = parseArgs(args);
|
||||
if (opts.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
// Progress to stderr (stdout is reserved for NDJSON data).
|
||||
const { createProgress, startHeartbeat } = await import('../core/progress.ts');
|
||||
const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts');
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
|
||||
progress.start('eval.export');
|
||||
const stopHeartbeat = startHeartbeat(progress, 'reading eval_candidates');
|
||||
let rows: EvalCandidate[];
|
||||
try {
|
||||
rows = await engine.listEvalCandidates({
|
||||
since: opts.since,
|
||||
limit: opts.limit,
|
||||
tool: opts.tool,
|
||||
});
|
||||
} finally {
|
||||
stopHeartbeat();
|
||||
}
|
||||
|
||||
// Emit NDJSON to stdout. EPIPE-safe: if the downstream process
|
||||
// (e.g. `| head`) closes its end early, we abort cleanly without a
|
||||
// stack trace. Matches src/core/progress.ts EPIPE handling precedent.
|
||||
const stdout = process.stdout;
|
||||
const abortOnEpipe = (err: NodeJS.ErrnoException) => {
|
||||
if (err.code === 'EPIPE') process.exit(0);
|
||||
};
|
||||
stdout.on('error', abortOnEpipe);
|
||||
|
||||
let written = 0;
|
||||
for (const row of rows) {
|
||||
// Prefix every line with schema_version:1 so gbrain-evals can detect
|
||||
// schema drift before parsing the rest of the fields.
|
||||
const line = JSON.stringify({ schema_version: SCHEMA_VERSION, ...row });
|
||||
if (!stdout.write(line + '\n')) {
|
||||
// Backpressure: wait for drain before continuing.
|
||||
await new Promise(r => stdout.once('drain', r));
|
||||
}
|
||||
written++;
|
||||
progress.tick();
|
||||
}
|
||||
|
||||
stdout.off('error', abortOnEpipe);
|
||||
progress.finish();
|
||||
console.error(`exported ${written} row(s)`);
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
/**
|
||||
* gbrain eval prune — delete old eval_candidates rows (v0.21.0).
|
||||
*
|
||||
* Retention is unlimited by default (matches ingest_log precedent).
|
||||
* This command is the explicit cleanup; pairs with `gbrain eval export`
|
||||
* (snapshot first, then prune if you want to reset).
|
||||
*
|
||||
* Usage:
|
||||
* gbrain eval prune --older-than 30d
|
||||
* gbrain eval prune --older-than 1h --dry-run
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
|
||||
interface PruneOpts {
|
||||
help?: boolean;
|
||||
olderThanMs?: number;
|
||||
dryRun?: boolean;
|
||||
}
|
||||
|
||||
function parseDurationToMs(s: string): number | null {
|
||||
const m = s.match(/^(\d+)\s*(ms|s|m|h|d)$/);
|
||||
if (!m) return null;
|
||||
const n = parseInt(m[1]!, 10);
|
||||
const unit = m[2]!;
|
||||
const mults: Record<string, number> = {
|
||||
ms: 1,
|
||||
s: 1000,
|
||||
m: 60_000,
|
||||
h: 3_600_000,
|
||||
d: 86_400_000,
|
||||
};
|
||||
return n * mults[unit]!;
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): PruneOpts {
|
||||
const opts: PruneOpts = {};
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const arg = args[i]!;
|
||||
const next = args[i + 1];
|
||||
switch (arg) {
|
||||
case '--help':
|
||||
case '-h':
|
||||
opts.help = true;
|
||||
break;
|
||||
case '--older-than': {
|
||||
if (!next) break;
|
||||
const ms = parseDurationToMs(next);
|
||||
if (ms === null) {
|
||||
console.error(`Invalid --older-than value: ${next} (use like 30d, 1h, 90m)`);
|
||||
process.exit(1);
|
||||
}
|
||||
opts.olderThanMs = ms;
|
||||
i++;
|
||||
break;
|
||||
}
|
||||
case '--dry-run':
|
||||
opts.dryRun = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
return opts;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
console.error(`gbrain eval prune — delete old eval_candidates rows
|
||||
|
||||
USAGE:
|
||||
gbrain eval prune --older-than DUR [--dry-run]
|
||||
|
||||
FLAGS:
|
||||
--older-than DUR Delete rows created before now() - DUR (e.g. 30d, 7d, 1h).
|
||||
Required — this command never deletes without a window.
|
||||
--dry-run Report what would be deleted; don't actually delete.
|
||||
--help, -h Show this help.
|
||||
|
||||
EXAMPLES:
|
||||
gbrain eval prune --older-than 30d
|
||||
gbrain eval prune --older-than 90d --dry-run
|
||||
`);
|
||||
}
|
||||
|
||||
export async function runEvalPrune(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const opts = parseArgs(args);
|
||||
if (opts.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
if (!opts.olderThanMs) {
|
||||
console.error('Error: --older-than is required\n');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const cutoff = new Date(Date.now() - opts.olderThanMs);
|
||||
|
||||
if (opts.dryRun) {
|
||||
// Snapshot-count the rows we *would* delete — the list call caps at
|
||||
// 100k which matches the export ceiling, so larger windows get a
|
||||
// floor-estimate that's still useful signal.
|
||||
const rows = await engine.listEvalCandidates({
|
||||
since: new Date(0),
|
||||
limit: 100_000,
|
||||
});
|
||||
const wouldDelete = rows.filter(r => new Date(r.created_at) < cutoff).length;
|
||||
console.log(`[dry-run] would delete ${wouldDelete} row(s) created before ${cutoff.toISOString()}`);
|
||||
if (rows.length === 100_000) {
|
||||
console.log('[dry-run] (count may be undercounted — the scan hit the 100k row limit)');
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const deleted = await engine.deleteEvalCandidatesBefore(cutoff);
|
||||
console.log(`deleted ${deleted} row(s) created before ${cutoff.toISOString()}`);
|
||||
}
|
||||
@@ -0,0 +1,424 @@
|
||||
/**
|
||||
* gbrain eval replay — replay captured eval_candidates against current brain (v0.25.0).
|
||||
*
|
||||
* The contributor-facing half of BrainBench-Real:
|
||||
*
|
||||
* 1. capture some real traffic (default-on, lands in eval_candidates)
|
||||
* 2. snapshot it (gbrain eval export --since 7d > baseline.ndjson)
|
||||
* 3. make a code change (tune RRF_K, edit hybrid.ts, swap an embed model)
|
||||
* 4. replay against the snapshot (gbrain eval replay --against baseline.ndjson)
|
||||
*
|
||||
* Outputs three numbers a contributor can read at a glance:
|
||||
*
|
||||
* - mean Jaccard@k between captured retrieved_slugs and current run's slugs
|
||||
* - top-1 stability rate (was the #1 result the same?)
|
||||
* - mean latency delta (current - captured), positive = slower now
|
||||
*
|
||||
* Best-effort by design. Replay is NOT pure — your brain has more pages than
|
||||
* when the capture was taken, embeddings may have drifted, and the OPENAI key
|
||||
* may be different. The metrics describe "did this change hurt retrieval on
|
||||
* the queries you actually serve" not "do these match the baseline byte for
|
||||
* byte." Use it before merging anything that touches src/core/search/ or the
|
||||
* query/search op handlers.
|
||||
*
|
||||
* Usage:
|
||||
* gbrain eval replay --against captured.ndjson [--limit N] [--json]
|
||||
* [--top-regressions K] [--verbose]
|
||||
*/
|
||||
|
||||
import { readFileSync, existsSync } from 'fs';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import type { SearchResult } from '../core/types.ts';
|
||||
import { hybridSearch } from '../core/search/hybrid.ts';
|
||||
|
||||
interface ReplayOpts {
|
||||
help?: boolean;
|
||||
against?: string;
|
||||
limit?: number;
|
||||
json?: boolean;
|
||||
verbose?: boolean;
|
||||
topRegressions?: number;
|
||||
}
|
||||
|
||||
interface RowResult {
|
||||
/** Captured row's id, for back-referencing into the source NDJSON. */
|
||||
id: number;
|
||||
tool_name: 'query' | 'search';
|
||||
query: string;
|
||||
/** Set-overlap score in [0, 1]. 1.0 = identical retrieved set. */
|
||||
jaccard: number;
|
||||
/** True when current top result matches captured top result. */
|
||||
top1Match: boolean;
|
||||
/** Captured retrieved_slugs (as-is from NDJSON). */
|
||||
captured_slugs: string[];
|
||||
/** Current run's slugs (deduped, in result order). */
|
||||
current_slugs: string[];
|
||||
/** Wall-clock latency (ms) of the current re-run. */
|
||||
current_latency_ms: number;
|
||||
/** latency delta = current - captured. Positive = slower now. */
|
||||
latency_delta_ms: number;
|
||||
/** True if the row was skipped (e.g. captured query was empty). */
|
||||
skipped?: boolean;
|
||||
/** Reason the row was skipped, if any. */
|
||||
skip_reason?: string;
|
||||
/** True if the row threw during replay; current_slugs is empty. */
|
||||
errored?: boolean;
|
||||
error_message?: string;
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): ReplayOpts {
|
||||
const opts: ReplayOpts = {};
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const arg = args[i]!;
|
||||
const next = args[i + 1];
|
||||
switch (arg) {
|
||||
case '--help':
|
||||
case '-h':
|
||||
opts.help = true;
|
||||
break;
|
||||
case '--against':
|
||||
if (!next) break;
|
||||
opts.against = next;
|
||||
i++;
|
||||
break;
|
||||
case '--limit':
|
||||
if (!next) break;
|
||||
opts.limit = parseInt(next, 10);
|
||||
i++;
|
||||
break;
|
||||
case '--json':
|
||||
opts.json = true;
|
||||
break;
|
||||
case '--verbose':
|
||||
opts.verbose = true;
|
||||
break;
|
||||
case '--top-regressions':
|
||||
if (!next) break;
|
||||
opts.topRegressions = parseInt(next, 10);
|
||||
i++;
|
||||
break;
|
||||
}
|
||||
}
|
||||
return opts;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
console.error(`gbrain eval replay — replay captured queries against current brain
|
||||
|
||||
USAGE:
|
||||
gbrain eval replay --against FILE.ndjson [flags]
|
||||
|
||||
FLAGS:
|
||||
--against FILE NDJSON file from \`gbrain eval export\` (required).
|
||||
--limit N Replay at most N rows (default: replay all).
|
||||
Each row hits OpenAI once for query embedding —
|
||||
cap aggressively when iterating locally.
|
||||
--top-regressions K Print the K rows with the worst Jaccard scores.
|
||||
Default 5 in human mode, 0 in --json.
|
||||
--json Emit one JSON object on stdout instead of a table.
|
||||
Stable shape for CI consumption.
|
||||
--verbose Include every row's per-row diff (large output).
|
||||
--help, -h Show this help.
|
||||
|
||||
OUTPUT (human mode):
|
||||
Replayed N captured queries (M skipped, K errored)
|
||||
Mean Jaccard@k: 0.873
|
||||
Top-1 stability: 87% (N=87 / 100)
|
||||
Mean latency Δ: +12ms (current slower)
|
||||
|
||||
Top 5 regressions:
|
||||
0.20 "find every reference to widget-co" captured=12 current=3
|
||||
...
|
||||
|
||||
EXIT CODE:
|
||||
0 — replay completed (regardless of regression magnitude).
|
||||
1 — invalid args, --against not found, or NDJSON parse failure.
|
||||
|
||||
NOTES:
|
||||
Replay is best-effort. Your brain has more pages than when the snapshot
|
||||
was taken; embeddings may have drifted; OPENAI_API_KEY may be different.
|
||||
Use the metrics to spot regressions on REAL queries, not as a hash check.
|
||||
`);
|
||||
}
|
||||
|
||||
interface CapturedRow {
|
||||
schema_version: number;
|
||||
id: number;
|
||||
tool_name: 'query' | 'search';
|
||||
query: string;
|
||||
retrieved_slugs: string[];
|
||||
retrieved_chunk_ids?: number[];
|
||||
source_ids?: string[];
|
||||
expand_enabled?: boolean | null;
|
||||
detail?: 'low' | 'medium' | 'high' | null;
|
||||
detail_resolved?: 'low' | 'medium' | 'high' | null;
|
||||
vector_enabled?: boolean;
|
||||
expansion_applied?: boolean;
|
||||
latency_ms: number;
|
||||
remote?: boolean;
|
||||
job_id?: number | null;
|
||||
subagent_id?: number | null;
|
||||
created_at?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse NDJSON. One object per non-blank line. Single bad line throws — it's
|
||||
* a corrupt export and silently dropping rows would mask real bugs.
|
||||
*/
|
||||
function parseNdjson(content: string): CapturedRow[] {
|
||||
const lines = content.split('\n');
|
||||
const rows: CapturedRow[] = [];
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i]!.trim();
|
||||
if (!line) continue;
|
||||
let row: CapturedRow;
|
||||
try {
|
||||
row = JSON.parse(line);
|
||||
} catch (err) {
|
||||
throw new Error(`NDJSON parse error on line ${i + 1}: ${(err as Error).message}`);
|
||||
}
|
||||
if (typeof row.schema_version !== 'number') {
|
||||
throw new Error(`Line ${i + 1} missing schema_version — not from \`gbrain eval export\`?`);
|
||||
}
|
||||
if (row.schema_version !== 1) {
|
||||
throw new Error(
|
||||
`Line ${i + 1} has schema_version=${row.schema_version}; this replay only supports v1. ` +
|
||||
`Upgrade gbrain or re-export.`,
|
||||
);
|
||||
}
|
||||
rows.push(row);
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
/**
|
||||
* Set-Jaccard between two slug arrays. Order ignored, dupes collapsed.
|
||||
* Both empty → 1.0 (identical empty sets, no information lost).
|
||||
*/
|
||||
function jaccardSlugs(a: string[], b: string[]): number {
|
||||
const setA = new Set(a);
|
||||
const setB = new Set(b);
|
||||
if (setA.size === 0 && setB.size === 0) return 1.0;
|
||||
let intersection = 0;
|
||||
for (const s of setA) if (setB.has(s)) intersection++;
|
||||
const union = setA.size + setB.size - intersection;
|
||||
return union === 0 ? 1.0 : intersection / union;
|
||||
}
|
||||
|
||||
async function replayRow(engine: BrainEngine, row: CapturedRow): Promise<RowResult> {
|
||||
const captured_slugs = row.retrieved_slugs ?? [];
|
||||
const startedAt = Date.now();
|
||||
|
||||
// Default replay limit matches hybridSearch's default (20).
|
||||
const limit = Math.max(captured_slugs.length, 20);
|
||||
|
||||
// search → bare keyword path. query → hybrid path (vector + keyword + RRF).
|
||||
// detail and expansion are threaded in from the captured row so the same
|
||||
// logic runs that produced the original retrieval.
|
||||
let current: SearchResult[];
|
||||
try {
|
||||
if (row.tool_name === 'search') {
|
||||
const dedupedRaw = await engine.searchKeyword(row.query, { limit });
|
||||
current = dedupedRaw;
|
||||
} else {
|
||||
current = await hybridSearch(engine, row.query, {
|
||||
limit,
|
||||
detail: row.detail ?? undefined,
|
||||
expansion: row.expand_enabled ?? false,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
return {
|
||||
id: row.id,
|
||||
tool_name: row.tool_name,
|
||||
query: row.query,
|
||||
jaccard: 0,
|
||||
top1Match: false,
|
||||
captured_slugs,
|
||||
current_slugs: [],
|
||||
current_latency_ms: Date.now() - startedAt,
|
||||
latency_delta_ms: Date.now() - startedAt - row.latency_ms,
|
||||
errored: true,
|
||||
error_message: (err as Error).message ?? String(err),
|
||||
};
|
||||
}
|
||||
|
||||
const current_latency_ms = Date.now() - startedAt;
|
||||
// Dedup slugs while preserving order — same convention as search results.
|
||||
const seen = new Set<string>();
|
||||
const current_slugs: string[] = [];
|
||||
for (const r of current) {
|
||||
if (!seen.has(r.slug)) {
|
||||
seen.add(r.slug);
|
||||
current_slugs.push(r.slug);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
id: row.id,
|
||||
tool_name: row.tool_name,
|
||||
query: row.query,
|
||||
jaccard: jaccardSlugs(captured_slugs, current_slugs),
|
||||
top1Match: captured_slugs[0] !== undefined && current_slugs[0] === captured_slugs[0],
|
||||
captured_slugs,
|
||||
current_slugs,
|
||||
current_latency_ms,
|
||||
latency_delta_ms: current_latency_ms - row.latency_ms,
|
||||
};
|
||||
}
|
||||
|
||||
interface ReplaySummary {
|
||||
rows_total: number;
|
||||
rows_replayed: number;
|
||||
rows_skipped: number;
|
||||
rows_errored: number;
|
||||
/** Mean Jaccard across non-skipped, non-errored rows. */
|
||||
mean_jaccard: number;
|
||||
top1_stability_rate: number;
|
||||
mean_latency_delta_ms: number;
|
||||
/** Rows where current latency is more than 2x captured (regression alarm). */
|
||||
rows_over_2x_latency: number;
|
||||
}
|
||||
|
||||
function summarize(results: RowResult[]): ReplaySummary {
|
||||
const eligible = results.filter(r => !r.skipped && !r.errored);
|
||||
const meanJaccard = eligible.length === 0
|
||||
? 0
|
||||
: eligible.reduce((a, r) => a + r.jaccard, 0) / eligible.length;
|
||||
const top1Rate = eligible.length === 0
|
||||
? 0
|
||||
: eligible.filter(r => r.top1Match).length / eligible.length;
|
||||
const meanLatencyDelta = eligible.length === 0
|
||||
? 0
|
||||
: eligible.reduce((a, r) => a + r.latency_delta_ms, 0) / eligible.length;
|
||||
const over2x = eligible.filter(r => {
|
||||
const captured = results.find(x => x.id === r.id);
|
||||
return captured && captured.current_latency_ms > 2 * (captured.current_latency_ms - captured.latency_delta_ms);
|
||||
}).length;
|
||||
|
||||
return {
|
||||
rows_total: results.length,
|
||||
rows_replayed: eligible.length,
|
||||
rows_skipped: results.filter(r => r.skipped).length,
|
||||
rows_errored: results.filter(r => r.errored).length,
|
||||
mean_jaccard: meanJaccard,
|
||||
top1_stability_rate: top1Rate,
|
||||
mean_latency_delta_ms: meanLatencyDelta,
|
||||
rows_over_2x_latency: over2x,
|
||||
};
|
||||
}
|
||||
|
||||
function printHumanSummary(summary: ReplaySummary, results: RowResult[], topRegressions: number): void {
|
||||
const total = summary.rows_total;
|
||||
const eligible = summary.rows_replayed;
|
||||
console.log(`Replayed ${eligible} of ${total} captured queries (${summary.rows_skipped} skipped, ${summary.rows_errored} errored)`);
|
||||
console.log(`Mean Jaccard@k: ${summary.mean_jaccard.toFixed(3)}`);
|
||||
console.log(`Top-1 stability: ${(summary.top1_stability_rate * 100).toFixed(1)}%`);
|
||||
const sign = summary.mean_latency_delta_ms >= 0 ? '+' : '';
|
||||
console.log(`Mean latency Δ: ${sign}${summary.mean_latency_delta_ms.toFixed(0)}ms (current vs captured)`);
|
||||
if (summary.rows_over_2x_latency > 0) {
|
||||
console.log(`⚠ ${summary.rows_over_2x_latency} row(s) ran more than 2× slower than captured`);
|
||||
}
|
||||
|
||||
if (topRegressions > 0) {
|
||||
const sorted = [...results]
|
||||
.filter(r => !r.skipped && !r.errored)
|
||||
.sort((a, b) => a.jaccard - b.jaccard)
|
||||
.slice(0, topRegressions);
|
||||
if (sorted.length > 0 && sorted[0]!.jaccard < 1.0) {
|
||||
console.log(`\nTop ${sorted.length} regression(s):`);
|
||||
for (const r of sorted) {
|
||||
const truncQuery = r.query.length > 60 ? r.query.slice(0, 57) + '...' : r.query;
|
||||
console.log(
|
||||
` jaccard=${r.jaccard.toFixed(2)} captured=${r.captured_slugs.length} current=${r.current_slugs.length} ` +
|
||||
`"${truncQuery}"`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (summary.rows_errored > 0) {
|
||||
const errors = results.filter(r => r.errored).slice(0, 3);
|
||||
console.log(`\n${summary.rows_errored} row(s) errored. First ${errors.length}:`);
|
||||
for (const r of errors) {
|
||||
const truncQuery = r.query.length > 60 ? r.query.slice(0, 57) + '...' : r.query;
|
||||
console.log(` id=${r.id} "${truncQuery}" ${r.error_message ?? ''}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function runEvalReplay(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const opts = parseArgs(args);
|
||||
if (opts.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
if (!opts.against) {
|
||||
console.error('Error: --against FILE.ndjson is required\n');
|
||||
printHelp();
|
||||
process.exit(1);
|
||||
}
|
||||
if (!existsSync(opts.against)) {
|
||||
console.error(`Error: file not found: ${opts.against}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
let rows: CapturedRow[];
|
||||
try {
|
||||
const content = readFileSync(opts.against, 'utf-8');
|
||||
rows = parseNdjson(content);
|
||||
} catch (err) {
|
||||
console.error(`Error: ${(err as Error).message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (rows.length === 0) {
|
||||
console.error(`Error: ${opts.against} is empty (no NDJSON rows)`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const capped = opts.limit && opts.limit > 0 ? rows.slice(0, opts.limit) : rows;
|
||||
if (!opts.json) {
|
||||
console.error(
|
||||
`Replaying ${capped.length}${capped.length < rows.length ? ` of ${rows.length}` : ''} captured queries…`,
|
||||
);
|
||||
}
|
||||
|
||||
const results: RowResult[] = [];
|
||||
for (const row of capped) {
|
||||
if (!row.query || row.query.length === 0) {
|
||||
results.push({
|
||||
id: row.id,
|
||||
tool_name: row.tool_name,
|
||||
query: row.query ?? '',
|
||||
jaccard: 0,
|
||||
top1Match: false,
|
||||
captured_slugs: row.retrieved_slugs ?? [],
|
||||
current_slugs: [],
|
||||
current_latency_ms: 0,
|
||||
latency_delta_ms: 0,
|
||||
skipped: true,
|
||||
skip_reason: 'empty query',
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const r = await replayRow(engine, row);
|
||||
results.push(r);
|
||||
if (!opts.json && results.length % 25 === 0) {
|
||||
process.stderr.write(` ...${results.length}/${capped.length}\n`);
|
||||
}
|
||||
}
|
||||
|
||||
const summary = summarize(results);
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify({
|
||||
schema_version: 1,
|
||||
summary,
|
||||
results: opts.verbose ? results : undefined,
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
const topN = opts.topRegressions ?? 5;
|
||||
printHumanSummary(summary, results, topN);
|
||||
}
|
||||
@@ -21,6 +21,23 @@ import {
|
||||
} from '../core/search/eval.ts';
|
||||
|
||||
export async function runEvalCommand(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
// v0.25.0 — sub-subcommand dispatch. Bare `gbrain eval --qrels ...`
|
||||
// falls through to the legacy IR-metrics flow so existing callers
|
||||
// don't break.
|
||||
const sub = args[0];
|
||||
if (sub === 'export') {
|
||||
const { runEvalExport } = await import('./eval-export.ts');
|
||||
return runEvalExport(engine, args.slice(1));
|
||||
}
|
||||
if (sub === 'prune') {
|
||||
const { runEvalPrune } = await import('./eval-prune.ts');
|
||||
return runEvalPrune(engine, args.slice(1));
|
||||
}
|
||||
if (sub === 'replay') {
|
||||
const { runEvalReplay } = await import('./eval-replay.ts');
|
||||
return runEvalReplay(engine, args.slice(1));
|
||||
}
|
||||
|
||||
const opts = parseArgs(args);
|
||||
|
||||
if (opts.help) {
|
||||
|
||||
@@ -38,6 +38,18 @@ export interface GBrainConfig {
|
||||
* validates the shape at runtime.
|
||||
*/
|
||||
storage?: unknown;
|
||||
/**
|
||||
* v0.25.0 — session capture settings. Read via file-plane `loadConfig()`
|
||||
* at process boot (NOT `gbrain config set` which writes the DB plane —
|
||||
* those are different stores). Edit `~/.gbrain/config.json` directly.
|
||||
* All fields default to ON — capture and scrubbing both opt-out.
|
||||
*/
|
||||
eval?: {
|
||||
/** false disables capture entirely. Defaults to true. */
|
||||
capture?: boolean;
|
||||
/** false disables PII scrubbing before insert. Defaults to true. */
|
||||
scrub_pii?: boolean;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -10,6 +10,8 @@ import type {
|
||||
IngestLogEntry, IngestLogInput,
|
||||
EngineConfig,
|
||||
CodeEdgeInput, CodeEdgeResult,
|
||||
EvalCandidate, EvalCandidateInput,
|
||||
EvalCaptureFailure, EvalCaptureFailureReason,
|
||||
} from './types.ts';
|
||||
|
||||
/** Input row for addLinksBatch. Optional fields default to '' (matches NOT NULL DDL). */
|
||||
@@ -363,4 +365,20 @@ export interface BrainEngine {
|
||||
* prefer searchKeyword (external contract: page-grain best-chunk-per-page).
|
||||
*/
|
||||
searchKeywordChunks(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
|
||||
// Eval capture (v0.25.0 — BrainBench-Real substrate).
|
||||
// Captured at the op-layer wrapper in src/core/operations.ts; reads via
|
||||
// `gbrain eval export` (NDJSON) for sibling gbrain-evals consumption.
|
||||
// Adding these to BrainEngine is a breaking-interface change for third-
|
||||
// party engine implementers — this is why v0.25.0 is a minor bump.
|
||||
/** Insert a captured candidate. Returns the new row id. Best-effort: callers swallow failures and route them through `logEvalCaptureFailure`. */
|
||||
logEvalCandidate(input: EvalCandidateInput): Promise<number>;
|
||||
/** Read candidates by time window / limit / tool filter. Used by `gbrain eval export`. */
|
||||
listEvalCandidates(filter?: { since?: Date; limit?: number; tool?: 'query' | 'search' }): Promise<EvalCandidate[]>;
|
||||
/** Delete candidates created before `date`. Returns rows deleted. Used by `gbrain eval prune`. */
|
||||
deleteEvalCandidatesBefore(date: Date): Promise<number>;
|
||||
/** Log a capture failure so `gbrain doctor` can surface drops cross-process. Best-effort; symmetric with logEvalCandidate (failure-of-failure is lost). */
|
||||
logEvalCaptureFailure(reason: EvalCaptureFailureReason): Promise<void>;
|
||||
/** Read capture failures within an optional time window. Used by `gbrain doctor`. */
|
||||
listEvalCaptureFailures(filter?: { since?: Date }): Promise<EvalCaptureFailure[]>;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
/**
|
||||
* PII scrubber for captured query text (v0.21.0).
|
||||
*
|
||||
* Runs before INSERT into `eval_candidates`. Capture is on by default,
|
||||
* so plaintext PII sitting in a DB column is a privacy footgun the first
|
||||
* time someone exports or shares a brain dump. Six regex families cover
|
||||
* the obvious cases:
|
||||
*
|
||||
* 1. Email addresses
|
||||
* 2. Phone numbers (US + international)
|
||||
* 3. US Social Security numbers (XXX-XX-XXXX shape, with year-like false-positive guard)
|
||||
* 4. Credit card numbers (13–19 digits with Luhn verification — blocks false positives)
|
||||
* 5. JWT-shaped tokens (three base64url segments joined by '.')
|
||||
* 6. Bearer tokens (Authorization: Bearer <opaque>)
|
||||
*
|
||||
* 80% of the real risk without a dependency on an NER model. If regex v1
|
||||
* proves insufficient we can layer a model-based scrubber later.
|
||||
*
|
||||
* Pure function, zero deps. Safe to call on arbitrary input. Adversarial
|
||||
* regex input (catastrophic backtracking) is contained by the
|
||||
* possessive-quantifier-free patterns below and by the outer try/catch in
|
||||
* captureEvalCandidate (see src/core/eval-capture.ts), not by this
|
||||
* module itself.
|
||||
*/
|
||||
|
||||
const REDACTED = '[REDACTED]';
|
||||
|
||||
// Emails: RFC-5322-adjacent. Keeps the host so replay debug can say "an
|
||||
// email was redacted" without leaking the local-part.
|
||||
const EMAIL_RE = /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g;
|
||||
|
||||
// Phones: US (###-###-####, (###) ###-####, ##########) and E.164 (+country).
|
||||
// Reject short strings of 10 digits with no separators/prefix to limit
|
||||
// false positives on order numbers and other generic long integers.
|
||||
const PHONE_RE =
|
||||
/(?<!\d)(?:\+\d{1,3}[\s.-]?)?(?:\(\d{3}\)\s?|\d{3}[\s.-])\d{3}[\s.-]?\d{4}(?!\d)/g;
|
||||
|
||||
// SSN: XXX-XX-XXXX with dashes required (bare 9-digit blobs are too
|
||||
// ambiguous — phone numbers, account IDs).
|
||||
const SSN_RE = /(?<!\d)\d{3}-\d{2}-\d{4}(?!\d)/g;
|
||||
|
||||
// JWT: three base64url segments. Lookbehind prevents partial matches in
|
||||
// the middle of longer identifiers.
|
||||
const JWT_RE =
|
||||
/\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\b/g;
|
||||
|
||||
// Bearer tokens after Authorization header literal or "Bearer " prefix.
|
||||
const BEARER_RE = /\b(?:bearer|Bearer)\s+[A-Za-z0-9._~+/-]{10,}=*/g;
|
||||
|
||||
// Credit card numbers: 13–19 digits with optional spaces/dashes. Every
|
||||
// match must pass Luhn to qualify — this is the key false-positive guard.
|
||||
const CC_RE = /(?<!\d)(?:\d[ -]?){12,18}\d(?!\d)/g;
|
||||
|
||||
/** Luhn mod-10 check. Returns true when the digit sequence is a valid card number. */
|
||||
function luhnOk(digits: string): boolean {
|
||||
let sum = 0;
|
||||
let parity = digits.length % 2;
|
||||
for (let i = 0; i < digits.length; i++) {
|
||||
let n = digits.charCodeAt(i) - 48;
|
||||
if (n < 0 || n > 9) return false;
|
||||
if (i % 2 === parity) {
|
||||
n *= 2;
|
||||
if (n > 9) n -= 9;
|
||||
}
|
||||
sum += n;
|
||||
}
|
||||
return sum % 10 === 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Redact obvious PII from a captured query string.
|
||||
*
|
||||
* Order of operations matters: email first so "user@example.com" doesn't
|
||||
* get caught by phone/CC regex fragments. CC last since Luhn is expensive
|
||||
* and irrelevant to everything else.
|
||||
*/
|
||||
export function scrubPii(input: string): string {
|
||||
if (!input) return input;
|
||||
let out = input;
|
||||
|
||||
// 1. Emails
|
||||
out = out.replace(EMAIL_RE, REDACTED);
|
||||
|
||||
// 2. Phones (before SSN so +1-555-XX doesn't look like part of a dashes-only SSN)
|
||||
out = out.replace(PHONE_RE, REDACTED);
|
||||
|
||||
// 3. SSN (after phones)
|
||||
out = out.replace(SSN_RE, REDACTED);
|
||||
|
||||
// 4. JWT (distinctive prefix, safe to run anywhere in the pipeline)
|
||||
out = out.replace(JWT_RE, REDACTED);
|
||||
|
||||
// 5. Bearer tokens
|
||||
out = out.replace(BEARER_RE, `Bearer ${REDACTED}`);
|
||||
|
||||
// 6. Credit cards: every candidate must pass Luhn to be replaced.
|
||||
out = out.replace(CC_RE, (match) => {
|
||||
const digitsOnly = match.replace(/\D/g, '');
|
||||
if (digitsOnly.length < 13 || digitsOnly.length > 19) return match;
|
||||
return luhnOk(digitsOnly) ? REDACTED : match;
|
||||
});
|
||||
|
||||
return out;
|
||||
}
|
||||
@@ -0,0 +1,208 @@
|
||||
/**
|
||||
* Op-layer capture wrapper (v0.21.0).
|
||||
*
|
||||
* Catches MCP + CLI + subagent tool-bridge traffic from a single site —
|
||||
* the `query` and `search` op handlers in src/core/operations.ts decorate
|
||||
* themselves with `captureEvalCandidate` before they run. Post-codex O1+O2
|
||||
* design: the MCP server used to be the capture site, but subagent tool
|
||||
* calls dispatch straight to op.handler via brain-allowlist.ts and would
|
||||
* have been invisible. Moving the hook to the op layer covers every caller.
|
||||
*
|
||||
* Best-effort, not await'd by the caller. Every failure is routed through
|
||||
* `engine.logEvalCaptureFailure(reason)` so `gbrain doctor` can see drops
|
||||
* cross-process (in-process counters would be invisible from doctor's
|
||||
* separate process).
|
||||
*
|
||||
* Data-flow (happy path):
|
||||
*
|
||||
* op.handler() ──▶ {results, meta}
|
||||
* │
|
||||
* (fire-and-forget from caller)
|
||||
* ▼
|
||||
* outer try / inner .catch
|
||||
* │
|
||||
* ▼
|
||||
* scrubPii(query)
|
||||
* │
|
||||
* ▼
|
||||
* buildEvalCandidateInput(...)
|
||||
* │
|
||||
* ▼
|
||||
* engine.logEvalCandidate(input)
|
||||
*
|
||||
* Every error path — scrub throw, build throw, INSERT reject — lands at
|
||||
* `logEvalCaptureFailure(reason)` with the right reason tag.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type {
|
||||
EvalCandidateInput,
|
||||
EvalCaptureFailureReason,
|
||||
HybridSearchMeta,
|
||||
SearchResult,
|
||||
} from './types.ts';
|
||||
import type { GBrainConfig } from './config.ts';
|
||||
import { scrubPii } from './eval-capture-scrub.ts';
|
||||
|
||||
// HybridSearchMeta is canonical in src/core/types.ts and exported via the
|
||||
// public `gbrain/types` subpath. Surfaced from hybridSearch via the
|
||||
// optional onMeta callback in HybridSearchOpts (Lane 1C).
|
||||
export type { HybridSearchMeta };
|
||||
|
||||
/** Context needed to build a capture row. Bundled so op handlers don't thread 8 args. */
|
||||
export interface CaptureContext {
|
||||
/** 'query' or 'search'; captured only for these two ops. */
|
||||
tool_name: 'query' | 'search';
|
||||
/** Pre-scrub query text. scrubPii runs INSIDE buildEvalCandidateInput when scrub_pii !== false. */
|
||||
query: string;
|
||||
/** Result set from the op handler. */
|
||||
results: SearchResult[];
|
||||
/** Side-channel metadata from hybridSearch. For 'search' ops, pass vector_enabled:false, others null/false. */
|
||||
meta: HybridSearchMeta;
|
||||
/** How long the underlying op took, in milliseconds. */
|
||||
latency_ms: number;
|
||||
/** OperationContext.remote — true for MCP callers, false for local CLI. */
|
||||
remote: boolean;
|
||||
/** The `expand` flag as requested by the caller (query-only; null for search). */
|
||||
expand_enabled: boolean | null;
|
||||
/** The `detail` flag as requested (query-only). */
|
||||
detail: 'low' | 'medium' | 'high' | null;
|
||||
/** OperationContext.jobId if present. */
|
||||
job_id: number | null;
|
||||
/** OperationContext.subagentId if present. */
|
||||
subagent_id: number | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the insert row for `eval_candidates` from a capture context.
|
||||
*
|
||||
* Runs the PII scrubber on `query` unless `scrub_pii === false`. Pure
|
||||
* function: throws only if the scrubber throws (caller wraps in try/catch).
|
||||
*
|
||||
* Exported for testing and for CLI backfill paths; the hot-path caller is
|
||||
* `captureEvalCandidate` below.
|
||||
*/
|
||||
export function buildEvalCandidateInput(
|
||||
ctx: CaptureContext,
|
||||
opts: { scrub_pii?: boolean } = {},
|
||||
): EvalCandidateInput {
|
||||
const shouldScrub = opts.scrub_pii !== false;
|
||||
const query = shouldScrub ? scrubPii(ctx.query) : ctx.query;
|
||||
|
||||
// Deduplicate + preserve order for slug + source_id extraction.
|
||||
// Both arrays are small (hybridSearch clamps at 100 results) so the
|
||||
// O(n) Set-based dedup is fine.
|
||||
const slugsSet = new Set<string>();
|
||||
const sourceSet = new Set<string>();
|
||||
const chunkIds: number[] = [];
|
||||
for (const r of ctx.results) {
|
||||
slugsSet.add(r.slug);
|
||||
if (r.source_id) sourceSet.add(r.source_id);
|
||||
chunkIds.push(r.chunk_id);
|
||||
}
|
||||
|
||||
return {
|
||||
tool_name: ctx.tool_name,
|
||||
query,
|
||||
retrieved_slugs: [...slugsSet],
|
||||
retrieved_chunk_ids: chunkIds,
|
||||
source_ids: [...sourceSet],
|
||||
expand_enabled: ctx.expand_enabled,
|
||||
detail: ctx.detail,
|
||||
detail_resolved: ctx.meta.detail_resolved,
|
||||
vector_enabled: ctx.meta.vector_enabled,
|
||||
expansion_applied: ctx.meta.expansion_applied,
|
||||
latency_ms: ctx.latency_ms,
|
||||
remote: ctx.remote,
|
||||
job_id: ctx.job_id,
|
||||
subagent_id: ctx.subagent_id,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Map a caught error to an `EvalCaptureFailureReason` so doctor can
|
||||
* group failures by cause (DB down vs RLS reject vs CHECK violation).
|
||||
*
|
||||
* Narrow by Postgres SQLSTATE where we can (postgres-js + PGLite both
|
||||
* surface `.code`), fall back to 'other'.
|
||||
*/
|
||||
export function classifyCaptureFailure(err: unknown): EvalCaptureFailureReason {
|
||||
if (err && typeof err === 'object') {
|
||||
const code = (err as { code?: string }).code;
|
||||
if (code === '23514') return 'check_violation'; // CHECK constraint violation
|
||||
if (code === '42501') return 'rls_reject'; // insufficient_privilege
|
||||
if (code === '42P01') return 'db_down'; // undefined_table (pre-v25)
|
||||
if (code === '53300' || code === '08006' || code === '08003') return 'db_down';
|
||||
const name = (err as { name?: string }).name;
|
||||
if (name === 'RegExpMatchError' || name === 'SyntaxError') {
|
||||
return 'scrubber_exception';
|
||||
}
|
||||
}
|
||||
return 'other';
|
||||
}
|
||||
|
||||
/**
|
||||
* Fire-and-forget capture side-effect. Never throws: every failure is
|
||||
* swallowed, classified, and routed through `engine.logEvalCaptureFailure`
|
||||
* (itself wrapped — failure-of-failure is logged to stderr and dropped).
|
||||
*
|
||||
* Callers invoke as `void captureEvalCandidate(...)` — we explicitly do
|
||||
* NOT await so the op response latency is unaffected.
|
||||
*/
|
||||
export async function captureEvalCandidate(
|
||||
engine: BrainEngine,
|
||||
ctx: CaptureContext,
|
||||
opts: { scrub_pii?: boolean } = {},
|
||||
): Promise<void> {
|
||||
try {
|
||||
const input = buildEvalCandidateInput(ctx, opts);
|
||||
await engine.logEvalCandidate(input);
|
||||
} catch (err) {
|
||||
const reason = classifyCaptureFailure(err);
|
||||
try {
|
||||
await engine.logEvalCaptureFailure(reason);
|
||||
} catch (failureErr) {
|
||||
// Failure-of-failure: last-resort stderr. Doctor can't see this
|
||||
// row, but we've exhausted the persistent path.
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn('[eval-capture] secondary failure logging also failed:', failureErr);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check whether capture is enabled for this process.
|
||||
*
|
||||
* Resolution order:
|
||||
* 1. `config.eval.capture === true` → on (explicit user opt-in wins)
|
||||
* 2. `config.eval.capture === false` → off (explicit user opt-out wins)
|
||||
* 3. `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'` → on (contributor opt-in)
|
||||
* 4. otherwise → off (default-off, privacy-positive for end users)
|
||||
*
|
||||
* The default flipped in v0.25.0 from "on for everyone" to "off unless you
|
||||
* opt in." Capturing every query a real user runs without their consent is
|
||||
* a footgun even with PII scrubbing; tying capture to CONTRIBUTOR_MODE makes
|
||||
* the developer-skill nature of the feature explicit. Production users get
|
||||
* a quiet brain; contributors get the BrainBench-Real replay loop with one
|
||||
* shell rc line. See docs/eval-bench.md and CONTRIBUTING.md.
|
||||
*
|
||||
* Takes the already-loaded config so callers control the loadConfig()
|
||||
* lifecycle (MCP server loads once at boot, CLI commands load per-invocation).
|
||||
*/
|
||||
export function isEvalCaptureEnabled(config: GBrainConfig | null | undefined): boolean {
|
||||
if (config?.eval?.capture === true) return true;
|
||||
if (config?.eval?.capture === false) return false;
|
||||
return process.env.GBRAIN_CONTRIBUTOR_MODE === '1';
|
||||
}
|
||||
|
||||
/**
|
||||
* PII scrubbing enabled? Defaults to true; explicit `false` opts out.
|
||||
*
|
||||
* Independent of `isEvalCaptureEnabled` — scrubbing only matters when capture
|
||||
* is actually running, but the gate stays separate so CONTRIBUTOR_MODE
|
||||
* doesn't accidentally turn off scrubbing on a brain that happens to also
|
||||
* have explicit `capture: true`.
|
||||
*/
|
||||
export function isEvalScrubEnabled(config: GBrainConfig | null | undefined): boolean {
|
||||
return config?.eval?.scrub_pii !== false;
|
||||
}
|
||||
@@ -1101,6 +1101,106 @@ export const MIGRATIONS: Migration[] = [
|
||||
END $$;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 31,
|
||||
name: 'eval_capture_tables',
|
||||
// v0.25.0 — BrainBench-Real session capture substrate.
|
||||
// Two tables:
|
||||
// eval_candidates: per-call capture from the op-layer wrapper around
|
||||
// `query` and `search`. Captures MCP + CLI + subagent tool-bridge
|
||||
// traffic via src/core/operations.ts. query column is CHECK-capped
|
||||
// at 50KB; PII is scrubbed before insert by src/core/eval-capture-scrub.ts.
|
||||
// remote distinguishes MCP callers (untrusted) from local CLI; job_id +
|
||||
// subagent_id let gbrain-evals partition replay by run.
|
||||
// eval_capture_failures: insert-side audit trail. When logEvalCandidate
|
||||
// fails (DB down, RLS reject, CHECK violation, scrubber exception),
|
||||
// the capture path records the reason here so `gbrain doctor` can
|
||||
// surface silent drops cross-process. In-process counters don't work
|
||||
// because doctor runs in a separate process from the MCP server.
|
||||
//
|
||||
// RLS enable matches the v24 / v29 posture: fail loudly via RAISE EXCEPTION
|
||||
// if current_user lacks BYPASSRLS, so the migration retries cleanly after
|
||||
// operator fixes the role instead of silently bumping schema_version.
|
||||
// PGLite ignores RLS; sqlFor carries the table+index DDL only.
|
||||
//
|
||||
// Renumbered v30→v31 on merge with master's v0.23.0 (dream_verdicts) which
|
||||
// claimed v30 first. Pre-existing brains that applied our v30 will see
|
||||
// version 31 as new on next initSchema and run the IF NOT EXISTS DDL —
|
||||
// the CREATE TABLE statements are idempotent so the rename is safe.
|
||||
sqlFor: {
|
||||
postgres: `
|
||||
DO $$
|
||||
DECLARE
|
||||
has_bypass BOOLEAN;
|
||||
BEGIN
|
||||
SELECT rolbypassrls INTO has_bypass FROM pg_roles WHERE rolname = current_user;
|
||||
IF NOT has_bypass THEN
|
||||
RAISE EXCEPTION 'v31 eval_capture_tables: role % does not have BYPASSRLS privilege — cannot enable RLS safely. Re-run as postgres (or another BYPASSRLS role). The migration will retry automatically on the next initSchema call.', current_user;
|
||||
END IF;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS eval_candidates (
|
||||
id SERIAL PRIMARY KEY,
|
||||
tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')),
|
||||
query TEXT NOT NULL CHECK (length(query) <= 51200),
|
||||
retrieved_slugs TEXT[] NOT NULL DEFAULT '{}',
|
||||
retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}',
|
||||
source_ids TEXT[] NOT NULL DEFAULT '{}',
|
||||
expand_enabled BOOLEAN,
|
||||
detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')),
|
||||
detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')),
|
||||
vector_enabled BOOLEAN NOT NULL,
|
||||
expansion_applied BOOLEAN NOT NULL,
|
||||
latency_ms INTEGER NOT NULL,
|
||||
remote BOOLEAN NOT NULL,
|
||||
job_id INTEGER,
|
||||
subagent_id INTEGER,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates (created_at DESC);
|
||||
ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS eval_capture_failures (
|
||||
id SERIAL PRIMARY KEY,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures (ts DESC);
|
||||
ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
RAISE NOTICE 'v31: eval_capture tables ready (role % has BYPASSRLS)', current_user;
|
||||
END $$;
|
||||
`,
|
||||
pglite: `
|
||||
CREATE TABLE IF NOT EXISTS eval_candidates (
|
||||
id SERIAL PRIMARY KEY,
|
||||
tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')),
|
||||
query TEXT NOT NULL CHECK (length(query) <= 51200),
|
||||
retrieved_slugs TEXT[] NOT NULL DEFAULT '{}',
|
||||
retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}',
|
||||
source_ids TEXT[] NOT NULL DEFAULT '{}',
|
||||
expand_enabled BOOLEAN,
|
||||
detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')),
|
||||
detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')),
|
||||
vector_enabled BOOLEAN NOT NULL,
|
||||
expansion_applied BOOLEAN NOT NULL,
|
||||
latency_ms INTEGER NOT NULL,
|
||||
remote BOOLEAN NOT NULL,
|
||||
job_id INTEGER,
|
||||
subagent_id INTEGER,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates (created_at DESC);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS eval_capture_failures (
|
||||
id SERIAL PRIMARY KEY,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures (ts DESC);
|
||||
`,
|
||||
},
|
||||
sql: '',
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
+69
-3
@@ -13,6 +13,8 @@ import { importFromContent } from './import-file.ts';
|
||||
import { hybridSearch } from './search/hybrid.ts';
|
||||
import { expandQuery } from './search/expansion.ts';
|
||||
import { dedupResults } from './search/dedup.ts';
|
||||
import { captureEvalCandidate, isEvalCaptureEnabled, isEvalScrubEnabled } from './eval-capture.ts';
|
||||
import type { HybridSearchMeta } from './types.ts';
|
||||
import { extractPageLinks, isAutoLinkEnabled, isAutoTimelineEnabled, parseTimelineEntries, makeResolver, type UnresolvedFrontmatterRef } from './link-extraction.ts';
|
||||
import * as db from './db.ts';
|
||||
|
||||
@@ -627,11 +629,38 @@ const search: Operation = {
|
||||
offset: { type: 'number', description: 'Skip first N results (for pagination)' },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
const results = await ctx.engine.searchKeyword(p.query as string, {
|
||||
const startedAt = Date.now();
|
||||
const queryText = p.query as string;
|
||||
const raw = await ctx.engine.searchKeyword(queryText, {
|
||||
limit: (p.limit as number) || 20,
|
||||
offset: (p.offset as number) || 0,
|
||||
});
|
||||
return dedupResults(results);
|
||||
const results = dedupResults(raw);
|
||||
const latency_ms = Date.now() - startedAt;
|
||||
|
||||
// Op-layer capture (v0.25.0). Fire-and-forget — no await on the
|
||||
// capture call so MCP response latency is unaffected. search has
|
||||
// no expand/detail/vector semantics so meta fields are fixed.
|
||||
if (isEvalCaptureEnabled(ctx.config)) {
|
||||
void captureEvalCandidate(
|
||||
ctx.engine,
|
||||
{
|
||||
tool_name: 'search',
|
||||
query: queryText,
|
||||
results,
|
||||
meta: { vector_enabled: false, detail_resolved: null, expansion_applied: false },
|
||||
latency_ms,
|
||||
remote: ctx.remote ?? false,
|
||||
expand_enabled: null,
|
||||
detail: null,
|
||||
job_id: ctx.jobId ?? null,
|
||||
subagent_id: ctx.subagentId ?? null,
|
||||
},
|
||||
{ scrub_pii: isEvalScrubEnabled(ctx.config) },
|
||||
);
|
||||
}
|
||||
|
||||
return results;
|
||||
},
|
||||
cliHints: { name: 'search', positional: ['query'] },
|
||||
};
|
||||
@@ -653,9 +682,16 @@ const query: Operation = {
|
||||
walk_depth: { type: 'number', description: 'Structural walk depth 1-2. Default 0 (off). Expands anchors through code_edges with 1/(1+hop) decay.' },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
const startedAt = Date.now();
|
||||
const expand = p.expand !== false;
|
||||
const detail = (p.detail as 'low' | 'medium' | 'high') || undefined;
|
||||
return hybridSearch(ctx.engine, p.query as string, {
|
||||
const queryText = p.query as string;
|
||||
|
||||
// v0.25.0 — capture meta side-channel. hybridSearch's return contract
|
||||
// stays SearchResult[] (Cathedral II callers depend on that); meta
|
||||
// arrives via callback so eval capture can record what actually ran.
|
||||
let capturedMeta: HybridSearchMeta | null = null;
|
||||
const results = await hybridSearch(ctx.engine, queryText, {
|
||||
limit: (p.limit as number) || 20,
|
||||
offset: (p.offset as number) || 0,
|
||||
expansion: expand,
|
||||
@@ -665,7 +701,37 @@ const query: Operation = {
|
||||
symbolKind: (p.symbol_kind as string) || undefined,
|
||||
nearSymbol: (p.near_symbol as string) || undefined,
|
||||
walkDepth: typeof p.walk_depth === 'number' ? (p.walk_depth as number) : undefined,
|
||||
onMeta: (m) => { capturedMeta = m; },
|
||||
});
|
||||
const latency_ms = Date.now() - startedAt;
|
||||
|
||||
// Op-layer capture (v0.25.0). Fire-and-forget. meta tells gbrain-evals
|
||||
// what hybridSearch *actually* did so replay can distinguish "with API
|
||||
// key" from "keyword-only fallback" and "expansion fired" from
|
||||
// "expansion requested + silently fell back."
|
||||
if (isEvalCaptureEnabled(ctx.config)) {
|
||||
const meta: HybridSearchMeta = capturedMeta ?? {
|
||||
vector_enabled: false, detail_resolved: detail ?? null, expansion_applied: false,
|
||||
};
|
||||
void captureEvalCandidate(
|
||||
ctx.engine,
|
||||
{
|
||||
tool_name: 'query',
|
||||
query: queryText,
|
||||
results,
|
||||
meta,
|
||||
latency_ms,
|
||||
remote: ctx.remote ?? false,
|
||||
expand_enabled: expand,
|
||||
detail: detail ?? null,
|
||||
job_id: ctx.jobId ?? null,
|
||||
subagent_id: ctx.subagentId ?? null,
|
||||
},
|
||||
{ scrub_pii: isEvalScrubEnabled(ctx.config) },
|
||||
);
|
||||
}
|
||||
|
||||
return results;
|
||||
},
|
||||
cliHints: { name: 'query', positional: ['query'] },
|
||||
};
|
||||
|
||||
@@ -18,6 +18,8 @@ import type {
|
||||
BrainStats, BrainHealth,
|
||||
IngestLogEntry, IngestLogInput,
|
||||
EngineConfig,
|
||||
EvalCandidate, EvalCandidateInput,
|
||||
EvalCaptureFailure, EvalCaptureFailureReason,
|
||||
} from './types.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts';
|
||||
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
||||
@@ -1673,6 +1675,82 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return (rows as Record<string, unknown>[]).map(rowToCodeEdge);
|
||||
}
|
||||
|
||||
// Eval capture (v0.25.0). See BrainEngine interface docs.
|
||||
async logEvalCandidate(input: EvalCandidateInput): Promise<number> {
|
||||
const { rows } = await this.db.query<{ id: number }>(
|
||||
`INSERT INTO eval_candidates (
|
||||
tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
|
||||
expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
|
||||
latency_ms, remote, job_id, subagent_id
|
||||
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)
|
||||
RETURNING id`,
|
||||
[
|
||||
input.tool_name,
|
||||
input.query,
|
||||
input.retrieved_slugs,
|
||||
input.retrieved_chunk_ids,
|
||||
input.source_ids,
|
||||
input.expand_enabled,
|
||||
input.detail,
|
||||
input.detail_resolved,
|
||||
input.vector_enabled,
|
||||
input.expansion_applied,
|
||||
input.latency_ms,
|
||||
input.remote,
|
||||
input.job_id,
|
||||
input.subagent_id,
|
||||
]
|
||||
);
|
||||
return rows[0]!.id;
|
||||
}
|
||||
|
||||
async listEvalCandidates(filter?: { since?: Date; limit?: number; tool?: 'query' | 'search' }): Promise<EvalCandidate[]> {
|
||||
const raw = filter?.limit;
|
||||
const limit = (raw === undefined || raw === null || !Number.isFinite(raw) || raw <= 0)
|
||||
? 1000
|
||||
: Math.min(Math.floor(raw), 100000);
|
||||
const since = filter?.since ?? new Date(0);
|
||||
const tool = filter?.tool ?? null;
|
||||
// id DESC tiebreaker — see postgres-engine for rationale.
|
||||
const { rows } = tool
|
||||
? await this.db.query(
|
||||
`SELECT * FROM eval_candidates
|
||||
WHERE created_at >= $1 AND tool_name = $2
|
||||
ORDER BY created_at DESC, id DESC LIMIT $3`,
|
||||
[since, tool, limit]
|
||||
)
|
||||
: await this.db.query(
|
||||
`SELECT * FROM eval_candidates
|
||||
WHERE created_at >= $1
|
||||
ORDER BY created_at DESC, id DESC LIMIT $2`,
|
||||
[since, limit]
|
||||
);
|
||||
return rows as unknown as EvalCandidate[];
|
||||
}
|
||||
|
||||
async deleteEvalCandidatesBefore(date: Date): Promise<number> {
|
||||
const { rows } = await this.db.query(
|
||||
`DELETE FROM eval_candidates WHERE created_at < $1 RETURNING id`,
|
||||
[date]
|
||||
);
|
||||
return rows.length;
|
||||
}
|
||||
|
||||
async logEvalCaptureFailure(reason: EvalCaptureFailureReason): Promise<void> {
|
||||
await this.db.query(
|
||||
`INSERT INTO eval_capture_failures (reason) VALUES ($1)`,
|
||||
[reason]
|
||||
);
|
||||
}
|
||||
|
||||
async listEvalCaptureFailures(filter?: { since?: Date }): Promise<EvalCaptureFailure[]> {
|
||||
const since = filter?.since ?? new Date(0);
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT * FROM eval_capture_failures WHERE ts >= $1 ORDER BY ts DESC`,
|
||||
[since]
|
||||
);
|
||||
return rows as unknown as EvalCaptureFailure[];
|
||||
}
|
||||
}
|
||||
|
||||
function rowToCodeEdge(row: Record<string, unknown>): import('./types.ts').CodeEdgeResult {
|
||||
|
||||
@@ -363,6 +363,35 @@ CREATE TABLE IF NOT EXISTS gbrain_cycle_locks (
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cycle_locks_ttl ON gbrain_cycle_locks(ttl_expires_at);
|
||||
|
||||
-- Eval capture (v0.25.0). PGLite ignores RLS — see src/schema.sql for the
|
||||
-- cross-engine spec.
|
||||
CREATE TABLE IF NOT EXISTS eval_candidates (
|
||||
id SERIAL PRIMARY KEY,
|
||||
tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')),
|
||||
query TEXT NOT NULL CHECK (length(query) <= 51200),
|
||||
retrieved_slugs TEXT[] NOT NULL DEFAULT '{}',
|
||||
retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}',
|
||||
source_ids TEXT[] NOT NULL DEFAULT '{}',
|
||||
expand_enabled BOOLEAN,
|
||||
detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')),
|
||||
detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')),
|
||||
vector_enabled BOOLEAN NOT NULL,
|
||||
expansion_applied BOOLEAN NOT NULL,
|
||||
latency_ms INTEGER NOT NULL,
|
||||
remote BOOLEAN NOT NULL,
|
||||
job_id INTEGER,
|
||||
subagent_id INTEGER,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS eval_capture_failures (
|
||||
id SERIAL PRIMARY KEY,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures(ts DESC);
|
||||
|
||||
-- ============================================================
|
||||
-- Trigger-based search_vector (spans pages + timeline_entries)
|
||||
-- ============================================================
|
||||
|
||||
@@ -15,6 +15,8 @@ import type {
|
||||
BrainStats, BrainHealth,
|
||||
IngestLogEntry, IngestLogInput,
|
||||
EngineConfig,
|
||||
EvalCandidate, EvalCandidateInput,
|
||||
EvalCaptureFailure, EvalCaptureFailureReason,
|
||||
} from './types.ts';
|
||||
import { GBrainError } from './types.ts';
|
||||
import * as db from './db.ts';
|
||||
@@ -1737,6 +1739,74 @@ export class PostgresEngine implements BrainEngine {
|
||||
return [...chunkRows, ...symbolRows].map(r => pgRowToCodeEdge(r as Record<string, unknown>));
|
||||
}
|
||||
|
||||
// Eval capture (v0.25.0). See BrainEngine interface docs.
|
||||
async logEvalCandidate(input: EvalCandidateInput): Promise<number> {
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
INSERT INTO eval_candidates (
|
||||
tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
|
||||
expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
|
||||
latency_ms, remote, job_id, subagent_id
|
||||
) VALUES (
|
||||
${input.tool_name}, ${input.query}, ${input.retrieved_slugs}, ${input.retrieved_chunk_ids}, ${input.source_ids},
|
||||
${input.expand_enabled}, ${input.detail}, ${input.detail_resolved}, ${input.vector_enabled}, ${input.expansion_applied},
|
||||
${input.latency_ms}, ${input.remote}, ${input.job_id}, ${input.subagent_id}
|
||||
)
|
||||
RETURNING id
|
||||
`;
|
||||
return rows[0]!.id as number;
|
||||
}
|
||||
|
||||
async listEvalCandidates(filter?: { since?: Date; limit?: number; tool?: 'query' | 'search' }): Promise<EvalCandidate[]> {
|
||||
const sql = this.sql;
|
||||
const raw = filter?.limit;
|
||||
const limit = (raw === undefined || raw === null || !Number.isFinite(raw) || raw <= 0)
|
||||
? 1000
|
||||
: Math.min(Math.floor(raw), 100000);
|
||||
const since = filter?.since ?? new Date(0);
|
||||
const tool = filter?.tool ?? null;
|
||||
// id DESC tiebreaker so same-millisecond inserts return deterministically
|
||||
// — without this, `gbrain eval export --since` could dupe or miss rows
|
||||
// across non-overlapping windows.
|
||||
const rows = tool
|
||||
? await sql`
|
||||
SELECT * FROM eval_candidates
|
||||
WHERE created_at >= ${since} AND tool_name = ${tool}
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT ${limit}
|
||||
`
|
||||
: await sql`
|
||||
SELECT * FROM eval_candidates
|
||||
WHERE created_at >= ${since}
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT ${limit}
|
||||
`;
|
||||
return rows as unknown as EvalCandidate[];
|
||||
}
|
||||
|
||||
async deleteEvalCandidatesBefore(date: Date): Promise<number> {
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
DELETE FROM eval_candidates WHERE created_at < ${date} RETURNING id
|
||||
`;
|
||||
return rows.length;
|
||||
}
|
||||
|
||||
async logEvalCaptureFailure(reason: EvalCaptureFailureReason): Promise<void> {
|
||||
const sql = this.sql;
|
||||
await sql`INSERT INTO eval_capture_failures (reason) VALUES (${reason})`;
|
||||
}
|
||||
|
||||
async listEvalCaptureFailures(filter?: { since?: Date }): Promise<EvalCaptureFailure[]> {
|
||||
const sql = this.sql;
|
||||
const since = filter?.since ?? new Date(0);
|
||||
const rows = await sql`
|
||||
SELECT * FROM eval_capture_failures
|
||||
WHERE ts >= ${since}
|
||||
ORDER BY ts DESC
|
||||
`;
|
||||
return rows as unknown as EvalCaptureFailure[];
|
||||
}
|
||||
}
|
||||
|
||||
function pgRowToCodeEdge(row: Record<string, unknown>): import('./types.ts').CodeEdgeResult {
|
||||
|
||||
@@ -597,7 +597,7 @@ CREATE TABLE IF NOT EXISTS subagent_rate_leases (
|
||||
CREATE INDEX IF NOT EXISTS idx_rate_leases_key_expires ON subagent_rate_leases (key, expires_at);
|
||||
|
||||
-- ============================================================
|
||||
-- Dream-cycle significance verdict cache — v0.23 synthesize phase
|
||||
-- Dream-cycle significance verdict cache — v0.21 synthesize phase
|
||||
-- ============================================================
|
||||
-- Caches the cheap Haiku "is this transcript worth processing?" verdict
|
||||
-- per (file_path, content_hash) so backfill re-runs skip already-judged
|
||||
@@ -630,6 +630,42 @@ CREATE TABLE IF NOT EXISTS gbrain_cycle_locks (
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cycle_locks_ttl ON gbrain_cycle_locks(ttl_expires_at);
|
||||
|
||||
-- ============================================================
|
||||
-- Eval capture (v0.25.0 — BrainBench-Real substrate)
|
||||
-- ============================================================
|
||||
-- eval_candidates: captured query/search calls from the op-layer wrapper
|
||||
-- in src/core/operations.ts. PII is scrubbed before insert by
|
||||
-- src/core/eval-capture-scrub.ts. query is CHECK-capped at 50KB.
|
||||
-- eval_capture_failures: cross-process audit of insert failures, surfaced
|
||||
-- by \`gbrain doctor\` (in-process counters can't bridge MCP server + doctor
|
||||
-- CLI process boundaries).
|
||||
CREATE TABLE IF NOT EXISTS eval_candidates (
|
||||
id SERIAL PRIMARY KEY,
|
||||
tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')),
|
||||
query TEXT NOT NULL CHECK (length(query) <= 51200),
|
||||
retrieved_slugs TEXT[] NOT NULL DEFAULT '{}',
|
||||
retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}',
|
||||
source_ids TEXT[] NOT NULL DEFAULT '{}',
|
||||
expand_enabled BOOLEAN,
|
||||
detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')),
|
||||
detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')),
|
||||
vector_enabled BOOLEAN NOT NULL,
|
||||
expansion_applied BOOLEAN NOT NULL,
|
||||
latency_ms INTEGER NOT NULL,
|
||||
remote BOOLEAN NOT NULL,
|
||||
job_id INTEGER,
|
||||
subagent_id INTEGER,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS eval_capture_failures (
|
||||
id SERIAL PRIMARY KEY,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures(ts DESC);
|
||||
|
||||
-- NOTIFY trigger for real-time job events (Postgres only, not PGLite)
|
||||
CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS \$\$
|
||||
BEGIN
|
||||
@@ -680,6 +716,8 @@ BEGIN
|
||||
ALTER TABLE subagent_rate_leases ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE gbrain_cycle_locks ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY;
|
||||
RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user;
|
||||
ELSE
|
||||
RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user;
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from '../engine.ts';
|
||||
import type { SearchResult, SearchOpts } from '../types.ts';
|
||||
import type { SearchResult, SearchOpts, HybridSearchMeta } from '../types.ts';
|
||||
import { embed } from '../embedding.ts';
|
||||
import { dedupResults } from './dedup.ts';
|
||||
import { autoDetectDetail } from './intent.ts';
|
||||
@@ -56,6 +56,15 @@ export interface HybridSearchOpts extends SearchOpts {
|
||||
maxTypeRatio?: number;
|
||||
maxPerPage?: number;
|
||||
};
|
||||
/**
|
||||
* v0.25.0 — optional side-channel for what hybridSearch actually did
|
||||
* (vector ran or fell back, expansion fired or didn't, post-auto-detect
|
||||
* detail). Surfaced via callback so the bare-return contract stays as
|
||||
* `Promise<SearchResult[]>` for existing Cathedral II callers. Op-layer
|
||||
* eval capture passes a callback that threads `meta` into the captured
|
||||
* row; everyone else leaves it undefined and pays no cost.
|
||||
*/
|
||||
onMeta?: (meta: HybridSearchMeta) => void;
|
||||
}
|
||||
|
||||
export async function hybridSearch(
|
||||
@@ -69,6 +78,7 @@ export async function hybridSearch(
|
||||
|
||||
// Auto-detect detail level from query intent when caller doesn't specify
|
||||
const detail = opts?.detail ?? autoDetectDetail(query);
|
||||
const detailResolved: 'low' | 'medium' | 'high' | null = detail ?? null;
|
||||
const searchOpts: SearchOpts = {
|
||||
limit: innerLimit,
|
||||
detail,
|
||||
@@ -77,6 +87,22 @@ export async function hybridSearch(
|
||||
language: opts?.language,
|
||||
symbolKind: opts?.symbolKind,
|
||||
};
|
||||
// Track what actually ran for the optional onMeta callback (v0.25.0).
|
||||
// Caller leaves onMeta undefined → these flags are computed but never
|
||||
// surfaced. Capture wrapper passes a closure to receive the meta and
|
||||
// threads it into the eval_candidates row.
|
||||
let expansionApplied = false;
|
||||
|
||||
// A throwing user callback must never break the search hot path — onMeta
|
||||
// is a public surface (gbrain/search/hybrid) so a third-party closure bug
|
||||
// shouldn't take down query/search responses.
|
||||
const emitMeta = (meta: HybridSearchMeta): void => {
|
||||
try {
|
||||
opts?.onMeta?.(meta);
|
||||
} catch {
|
||||
// swallow — capture telemetry is best-effort
|
||||
}
|
||||
};
|
||||
|
||||
if (DEBUG && detail) {
|
||||
console.error(`[search-debug] auto-detail=${detail} for query="${query}"`);
|
||||
@@ -99,6 +125,7 @@ export async function hybridSearch(
|
||||
// Boost failure is non-fatal: keep unboosted ranking.
|
||||
}
|
||||
}
|
||||
emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: false });
|
||||
return dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
}
|
||||
|
||||
@@ -110,6 +137,8 @@ export async function hybridSearch(
|
||||
try {
|
||||
queries = await opts.expandFn(query);
|
||||
if (queries.length === 0) queries = [query];
|
||||
// "Applied" = produced variants beyond the original, not just called.
|
||||
expansionApplied = queries.length > 1;
|
||||
} catch {
|
||||
// Expansion failure is non-fatal
|
||||
}
|
||||
@@ -129,6 +158,8 @@ export async function hybridSearch(
|
||||
}
|
||||
|
||||
if (vectorLists.length === 0) {
|
||||
// Embed/vector failed silently; record that vector did not run.
|
||||
emitMeta({ vector_enabled: false, detail_resolved: detailResolved, expansion_applied: expansionApplied });
|
||||
return dedupResults(keywordResults).slice(offset, offset + limit);
|
||||
}
|
||||
|
||||
@@ -203,11 +234,14 @@ export async function hybridSearch(
|
||||
// Dedup
|
||||
const deduped = dedupResults(fused, dedupOpts);
|
||||
|
||||
// Auto-escalate: if detail=low returned 0, retry with high
|
||||
// Auto-escalate: if detail=low returned 0, retry with high. The inner
|
||||
// call's onMeta fires with the escalated detail_resolved; do NOT also
|
||||
// fire here (would double-emit and capture stale meta).
|
||||
if (deduped.length === 0 && opts?.detail === 'low') {
|
||||
return hybridSearch(engine, query, { ...opts, detail: 'high' });
|
||||
}
|
||||
|
||||
emitMeta({ vector_enabled: true, detail_resolved: detailResolved, expansion_applied: expansionApplied });
|
||||
return deduped.slice(offset, offset + limit);
|
||||
}
|
||||
|
||||
|
||||
@@ -389,6 +389,69 @@ export interface IngestLogInput {
|
||||
summary: string;
|
||||
}
|
||||
|
||||
// Eval capture (v0.25.0)
|
||||
// Real MCP/CLI/subagent query+search calls are captured into eval_candidates
|
||||
// so gbrain-evals can replay them as BrainBench-Real. The companion
|
||||
// eval_capture_failures table records insert failures so gbrain doctor can
|
||||
// surface silent capture drops cross-process.
|
||||
export interface EvalCandidateInput {
|
||||
tool_name: 'query' | 'search';
|
||||
/** Already PII-scrubbed by captureEvalCandidate before this point. */
|
||||
query: string;
|
||||
retrieved_slugs: string[];
|
||||
retrieved_chunk_ids: number[];
|
||||
source_ids: string[];
|
||||
/** Whether multi-query Haiku expansion was enabled on the call. Null for 'search'. */
|
||||
expand_enabled: boolean | null;
|
||||
/** The detail level the call requested (pre-auto-detect). */
|
||||
detail: 'low' | 'medium' | 'high' | null;
|
||||
/** What hybridSearch actually ran (post-auto-detect). Null for 'search'. */
|
||||
detail_resolved: 'low' | 'medium' | 'high' | null;
|
||||
/** True when vector search actually ran (false when OPENAI_API_KEY missing or embed failed). */
|
||||
vector_enabled: boolean;
|
||||
/** True when Haiku expansion actually fired. */
|
||||
expansion_applied: boolean;
|
||||
latency_ms: number;
|
||||
/** ctx.remote: true for MCP callers (untrusted), false for local CLI. */
|
||||
remote: boolean;
|
||||
job_id: number | null;
|
||||
subagent_id: number | null;
|
||||
}
|
||||
|
||||
export interface EvalCandidate extends EvalCandidateInput {
|
||||
id: number;
|
||||
created_at: Date;
|
||||
}
|
||||
|
||||
export type EvalCaptureFailureReason =
|
||||
| 'db_down'
|
||||
| 'rls_reject'
|
||||
| 'check_violation'
|
||||
| 'scrubber_exception'
|
||||
| 'other';
|
||||
|
||||
export interface EvalCaptureFailure {
|
||||
id: number;
|
||||
ts: Date;
|
||||
reason: EvalCaptureFailureReason;
|
||||
}
|
||||
|
||||
/**
|
||||
* Side-channel metadata that hybridSearch reports about what actually ran.
|
||||
* Surfaced via the optional `onMeta` callback in HybridSearchOpts so
|
||||
* existing SearchResult[] consumers (Cathedral II, gbrain-evals, etc.)
|
||||
* stay unchanged. Used by op-layer eval capture to distinguish
|
||||
* "keyword-only fallback" from "full hybrid with expansion."
|
||||
*/
|
||||
export interface HybridSearchMeta {
|
||||
/** True iff vector search actually ran. False when OPENAI_API_KEY missing or embed failed. */
|
||||
vector_enabled: boolean;
|
||||
/** Post-auto-detect detail level. */
|
||||
detail_resolved: 'low' | 'medium' | 'high' | null;
|
||||
/** True iff multi-query expansion (Haiku) actually fired and produced variants. */
|
||||
expansion_applied: boolean;
|
||||
}
|
||||
|
||||
// Config
|
||||
export interface EngineConfig {
|
||||
database_url?: string;
|
||||
|
||||
@@ -626,6 +626,42 @@ CREATE TABLE IF NOT EXISTS gbrain_cycle_locks (
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_cycle_locks_ttl ON gbrain_cycle_locks(ttl_expires_at);
|
||||
|
||||
-- ============================================================
|
||||
-- Eval capture (v0.25.0 — BrainBench-Real substrate)
|
||||
-- ============================================================
|
||||
-- eval_candidates: captured query/search calls from the op-layer wrapper
|
||||
-- in src/core/operations.ts. PII is scrubbed before insert by
|
||||
-- src/core/eval-capture-scrub.ts. query is CHECK-capped at 50KB.
|
||||
-- eval_capture_failures: cross-process audit of insert failures, surfaced
|
||||
-- by `gbrain doctor` (in-process counters can't bridge MCP server + doctor
|
||||
-- CLI process boundaries).
|
||||
CREATE TABLE IF NOT EXISTS eval_candidates (
|
||||
id SERIAL PRIMARY KEY,
|
||||
tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')),
|
||||
query TEXT NOT NULL CHECK (length(query) <= 51200),
|
||||
retrieved_slugs TEXT[] NOT NULL DEFAULT '{}',
|
||||
retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}',
|
||||
source_ids TEXT[] NOT NULL DEFAULT '{}',
|
||||
expand_enabled BOOLEAN,
|
||||
detail TEXT CHECK (detail IS NULL OR detail IN ('low', 'medium', 'high')),
|
||||
detail_resolved TEXT CHECK (detail_resolved IS NULL OR detail_resolved IN ('low', 'medium', 'high')),
|
||||
vector_enabled BOOLEAN NOT NULL,
|
||||
expansion_applied BOOLEAN NOT NULL,
|
||||
latency_ms INTEGER NOT NULL,
|
||||
remote BOOLEAN NOT NULL,
|
||||
job_id INTEGER,
|
||||
subagent_id INTEGER,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_candidates_created_at ON eval_candidates(created_at DESC);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS eval_capture_failures (
|
||||
id SERIAL PRIMARY KEY,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
reason TEXT NOT NULL CHECK (reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other'))
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_capture_failures_ts ON eval_capture_failures(ts DESC);
|
||||
|
||||
-- NOTIFY trigger for real-time job events (Postgres only, not PGLite)
|
||||
CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS $$
|
||||
BEGIN
|
||||
@@ -676,6 +712,8 @@ BEGIN
|
||||
ALTER TABLE subagent_rate_leases ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE gbrain_cycle_locks ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE dream_verdicts ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY;
|
||||
RAISE NOTICE 'RLS enabled on all tables (role % has BYPASSRLS)', current_user;
|
||||
ELSE
|
||||
RAISE WARNING 'Skipping RLS: role % does not have BYPASSRLS privilege. Run as postgres role to enable.', current_user;
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
/**
|
||||
* E2E — Postgres-specific eval_capture behavior (v0.21.0, E1 spec).
|
||||
*
|
||||
* Unit tests in test/eval-candidates.test.ts already cover PGLite. This
|
||||
* file is strictly for Postgres-only behavior that PGLite can't exercise:
|
||||
* 1. RLS policy rejects when the caller is NOT BYPASSRLS
|
||||
* 2. CHECK violation surfaces as Postgres error code '23514'
|
||||
* 3. Concurrent INSERT pressure on the connection pool doesn't deadlock
|
||||
* or drop rows (the fire-and-forget capture path at full tilt)
|
||||
*
|
||||
* Runs only when DATABASE_URL is set. Skips gracefully otherwise
|
||||
* per CLAUDE.md E2E lifecycle.
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
||||
import type { EvalCandidateInput } from '../../src/core/types.ts';
|
||||
|
||||
const DATABASE_URL = process.env.DATABASE_URL;
|
||||
|
||||
let engine: PostgresEngine | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
if (!DATABASE_URL) {
|
||||
console.log('[e2e/eval-capture] DATABASE_URL not set — skipping.');
|
||||
return;
|
||||
}
|
||||
engine = new PostgresEngine();
|
||||
await engine.connect({ database_url: DATABASE_URL });
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
if (!engine) return;
|
||||
await engine.executeRaw('DELETE FROM eval_candidates');
|
||||
await engine.executeRaw('DELETE FROM eval_capture_failures');
|
||||
});
|
||||
|
||||
function baseInput(overrides: Partial<EvalCandidateInput> = {}): EvalCandidateInput {
|
||||
return {
|
||||
tool_name: 'query',
|
||||
query: 'alice',
|
||||
retrieved_slugs: ['people/alice-example'],
|
||||
retrieved_chunk_ids: [1],
|
||||
source_ids: ['default'],
|
||||
expand_enabled: true,
|
||||
detail: null,
|
||||
detail_resolved: 'medium',
|
||||
vector_enabled: true,
|
||||
expansion_applied: false,
|
||||
latency_ms: 100,
|
||||
remote: true,
|
||||
job_id: null,
|
||||
subagent_id: null,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('CHECK constraint — 50KB query length cap', () => {
|
||||
test.if(DATABASE_URL !== undefined)(
|
||||
'oversized query rejected with Postgres error code 23514',
|
||||
async () => {
|
||||
if (!engine) return;
|
||||
const big = 'x'.repeat(51_201);
|
||||
let caught: unknown = null;
|
||||
try {
|
||||
await engine.logEvalCandidate(baseInput({ query: big }));
|
||||
} catch (err) {
|
||||
caught = err;
|
||||
}
|
||||
expect(caught).not.toBeNull();
|
||||
// postgres.js surfaces SQLSTATE on the error object as `code`.
|
||||
expect((caught as { code?: string }).code).toBe('23514');
|
||||
},
|
||||
);
|
||||
|
||||
test.if(DATABASE_URL !== undefined)(
|
||||
'query exactly at 50KB succeeds',
|
||||
async () => {
|
||||
if (!engine) return;
|
||||
const atCap = 'x'.repeat(51_200);
|
||||
const id = await engine.logEvalCandidate(baseInput({ query: atCap }));
|
||||
expect(id).toBeGreaterThan(0);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe('RLS policy — requires BYPASSRLS to INSERT', () => {
|
||||
// Note: running this test REQUIRES the E2E test role to be postgres
|
||||
// (or another BYPASSRLS role) since schema.sql + migration v25 enable
|
||||
// RLS only on BYPASSRLS-capable roles. We verify the policy is
|
||||
// actually enabled and the INSERT path succeeds as BYPASSRLS. A
|
||||
// "deny from non-BYPASSRLS role" positive test would require setting
|
||||
// up a second role, which is beyond this test's scope — but
|
||||
// structurally we assert the ALTER TABLE ran.
|
||||
test.if(DATABASE_URL !== undefined)(
|
||||
'eval_candidates has RLS enabled',
|
||||
async () => {
|
||||
if (!engine) return;
|
||||
const rows = await engine.executeRaw<{ relrowsecurity: boolean }>(
|
||||
`SELECT relrowsecurity FROM pg_class WHERE relname = 'eval_candidates'`,
|
||||
);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]!.relrowsecurity).toBe(true);
|
||||
},
|
||||
);
|
||||
|
||||
test.if(DATABASE_URL !== undefined)(
|
||||
'eval_capture_failures has RLS enabled',
|
||||
async () => {
|
||||
if (!engine) return;
|
||||
const rows = await engine.executeRaw<{ relrowsecurity: boolean }>(
|
||||
`SELECT relrowsecurity FROM pg_class WHERE relname = 'eval_capture_failures'`,
|
||||
);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]!.relrowsecurity).toBe(true);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe('concurrent pool pressure — 50 parallel INSERTs', () => {
|
||||
test.if(DATABASE_URL !== undefined)(
|
||||
'50 concurrent logEvalCandidate calls all land without deadlock',
|
||||
async () => {
|
||||
if (!engine) return;
|
||||
const n = 50;
|
||||
const promises: Promise<number>[] = [];
|
||||
for (let i = 0; i < n; i++) {
|
||||
promises.push(engine.logEvalCandidate(baseInput({ query: `q${i}` })));
|
||||
}
|
||||
const ids = await Promise.all(promises);
|
||||
expect(ids).toHaveLength(n);
|
||||
expect(new Set(ids).size).toBe(n); // every id is distinct
|
||||
|
||||
const rows = await engine.listEvalCandidates({ limit: n * 2 });
|
||||
expect(rows).toHaveLength(n);
|
||||
},
|
||||
30_000,
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,201 @@
|
||||
/**
|
||||
* PGLite round-trip for the 5 eval-capture engine methods shipped in v0.21.0:
|
||||
* logEvalCandidate / listEvalCandidates / deleteEvalCandidatesBefore
|
||||
* logEvalCaptureFailure / listEvalCaptureFailures
|
||||
*
|
||||
* No Docker, no DATABASE_URL. In-memory only. For Postgres-only behavior
|
||||
* (RLS policy enforcement, CHECK violation error codes, concurrent pool
|
||||
* pressure), see test/e2e/eval-capture.test.ts.
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import type { EvalCandidateInput, EvalCaptureFailureReason } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.exec('DELETE FROM eval_candidates');
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.exec('DELETE FROM eval_capture_failures');
|
||||
});
|
||||
|
||||
function makeInput(overrides: Partial<EvalCandidateInput> = {}): EvalCandidateInput {
|
||||
return {
|
||||
tool_name: 'query',
|
||||
query: 'who is alice-example',
|
||||
retrieved_slugs: ['people/alice-example', 'companies/acme-example'],
|
||||
retrieved_chunk_ids: [42, 43],
|
||||
source_ids: ['default'],
|
||||
expand_enabled: true,
|
||||
detail: null,
|
||||
detail_resolved: 'medium',
|
||||
vector_enabled: true,
|
||||
expansion_applied: false,
|
||||
latency_ms: 123,
|
||||
remote: true,
|
||||
job_id: null,
|
||||
subagent_id: null,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('logEvalCandidate', () => {
|
||||
test('inserts a full row and returns the id', async () => {
|
||||
const id = await engine.logEvalCandidate(makeInput());
|
||||
expect(typeof id).toBe('number');
|
||||
expect(id).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('preserves array + enum + boolean columns through round-trip', async () => {
|
||||
await engine.logEvalCandidate(
|
||||
makeInput({
|
||||
tool_name: 'search',
|
||||
expand_enabled: null, // search has no expansion semantics
|
||||
detail: 'high',
|
||||
detail_resolved: 'high',
|
||||
vector_enabled: false,
|
||||
expansion_applied: false,
|
||||
job_id: 9001,
|
||||
subagent_id: 42,
|
||||
remote: false,
|
||||
}),
|
||||
);
|
||||
const rows = await engine.listEvalCandidates();
|
||||
expect(rows).toHaveLength(1);
|
||||
const row = rows[0]!;
|
||||
expect(row.tool_name).toBe('search');
|
||||
expect(row.retrieved_slugs).toEqual(['people/alice-example', 'companies/acme-example']);
|
||||
expect(row.retrieved_chunk_ids).toEqual([42, 43]);
|
||||
expect(row.source_ids).toEqual(['default']);
|
||||
expect(row.expand_enabled).toBeNull();
|
||||
expect(row.detail).toBe('high');
|
||||
expect(row.detail_resolved).toBe('high');
|
||||
expect(row.vector_enabled).toBe(false);
|
||||
expect(row.job_id).toBe(9001);
|
||||
expect(row.subagent_id).toBe(42);
|
||||
expect(row.remote).toBe(false);
|
||||
});
|
||||
|
||||
test('rejects oversize queries via CHECK constraint (50KB cap)', async () => {
|
||||
const bigQuery = 'x'.repeat(51201);
|
||||
await expect(engine.logEvalCandidate(makeInput({ query: bigQuery }))).rejects.toThrow();
|
||||
});
|
||||
|
||||
test('rejects invalid tool_name via CHECK constraint', async () => {
|
||||
await expect(
|
||||
engine.logEvalCandidate(makeInput({ tool_name: 'dance' as unknown as 'query' })),
|
||||
).rejects.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('listEvalCandidates', () => {
|
||||
test('returns empty when table is empty', async () => {
|
||||
expect(await engine.listEvalCandidates()).toEqual([]);
|
||||
});
|
||||
|
||||
test('filters by tool', async () => {
|
||||
await engine.logEvalCandidate(makeInput({ tool_name: 'query' }));
|
||||
await engine.logEvalCandidate(makeInput({ tool_name: 'search', expand_enabled: null }));
|
||||
await engine.logEvalCandidate(makeInput({ tool_name: 'query' }));
|
||||
|
||||
const queries = await engine.listEvalCandidates({ tool: 'query' });
|
||||
const searches = await engine.listEvalCandidates({ tool: 'search' });
|
||||
|
||||
expect(queries).toHaveLength(2);
|
||||
expect(searches).toHaveLength(1);
|
||||
expect(queries.every(r => r.tool_name === 'query')).toBe(true);
|
||||
expect(searches.every(r => r.tool_name === 'search')).toBe(true);
|
||||
});
|
||||
|
||||
test('filters by since (inclusive lower bound)', async () => {
|
||||
await engine.logEvalCandidate(makeInput());
|
||||
const cutoff = new Date(Date.now() + 60_000); // 1 minute in future — excludes everything
|
||||
expect(await engine.listEvalCandidates({ since: cutoff })).toHaveLength(0);
|
||||
|
||||
const past = new Date(0);
|
||||
expect(await engine.listEvalCandidates({ since: past })).toHaveLength(1);
|
||||
});
|
||||
|
||||
test('honors limit and returns newest-first (id DESC tiebreaker for same-ms inserts)', async () => {
|
||||
const ids: number[] = [];
|
||||
for (let i = 0; i < 5; i++) {
|
||||
ids.push(await engine.logEvalCandidate(makeInput({ query: `q${i}` })));
|
||||
}
|
||||
const limited = await engine.listEvalCandidates({ limit: 3 });
|
||||
expect(limited).toHaveLength(3);
|
||||
// ORDER BY created_at DESC, id DESC — same-millisecond rows fall back to
|
||||
// id DESC, which matches insertion order. Critical for `gbrain eval export`
|
||||
// since unstable tiebreaks would cause duplicate/missed rows across runs.
|
||||
expect(limited.map(r => r.id)).toEqual([ids[4]!, ids[3]!, ids[2]!]);
|
||||
});
|
||||
|
||||
test('clamps limit to [1, 100000]', async () => {
|
||||
await engine.logEvalCandidate(makeInput());
|
||||
// Implementation must clamp limit <= 0 up to default and cap huge values.
|
||||
expect(await engine.listEvalCandidates({ limit: 0 })).toHaveLength(1);
|
||||
expect(await engine.listEvalCandidates({ limit: 10_000_000 })).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('deleteEvalCandidatesBefore', () => {
|
||||
test('returns the number of rows deleted', async () => {
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await engine.logEvalCandidate(makeInput());
|
||||
}
|
||||
const futureCutoff = new Date(Date.now() + 60_000);
|
||||
const deleted = await engine.deleteEvalCandidatesBefore(futureCutoff);
|
||||
expect(deleted).toBe(3);
|
||||
expect(await engine.listEvalCandidates()).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('leaves newer rows alone', async () => {
|
||||
await engine.logEvalCandidate(makeInput({ query: 'old' }));
|
||||
// Cutoff 0 deletes nothing since created_at > 1970.
|
||||
const pastCutoff = new Date(0);
|
||||
const deleted = await engine.deleteEvalCandidatesBefore(pastCutoff);
|
||||
expect(deleted).toBe(0);
|
||||
expect(await engine.listEvalCandidates()).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('logEvalCaptureFailure / listEvalCaptureFailures', () => {
|
||||
test('inserts and round-trips each reason enum', async () => {
|
||||
const reasons: EvalCaptureFailureReason[] = [
|
||||
'db_down',
|
||||
'rls_reject',
|
||||
'check_violation',
|
||||
'scrubber_exception',
|
||||
'other',
|
||||
];
|
||||
for (const r of reasons) {
|
||||
await engine.logEvalCaptureFailure(r);
|
||||
}
|
||||
const rows = await engine.listEvalCaptureFailures();
|
||||
expect(rows.map(r => r.reason).sort()).toEqual([...reasons].sort());
|
||||
});
|
||||
|
||||
test('rejects invalid reason via CHECK constraint', async () => {
|
||||
await expect(
|
||||
engine.logEvalCaptureFailure('unknown' as EvalCaptureFailureReason),
|
||||
).rejects.toThrow();
|
||||
});
|
||||
|
||||
test('filters by since', async () => {
|
||||
await engine.logEvalCaptureFailure('db_down');
|
||||
const future = new Date(Date.now() + 60_000);
|
||||
expect(await engine.listEvalCaptureFailures({ since: future })).toHaveLength(0);
|
||||
expect(await engine.listEvalCaptureFailures({ since: new Date(0) })).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,118 @@
|
||||
/**
|
||||
* PII scrubber regex coverage. Every redacted family gets a positive
|
||||
* case and at least one false-positive avoidance case. Adversarial regex
|
||||
* input is covered so a pathological query can't hang capture.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { scrubPii } from '../src/core/eval-capture-scrub.ts';
|
||||
|
||||
describe('scrubPii', () => {
|
||||
test('passes empty input through unchanged', () => {
|
||||
expect(scrubPii('')).toBe('');
|
||||
});
|
||||
|
||||
test('leaves PII-free queries untouched', () => {
|
||||
const q = 'who is alice-example and what did they build';
|
||||
expect(scrubPii(q)).toBe(q);
|
||||
});
|
||||
|
||||
test('redacts email addresses', () => {
|
||||
const out = scrubPii('email me at alice@example.com about this');
|
||||
expect(out).not.toContain('alice@example.com');
|
||||
expect(out).toContain('[REDACTED]');
|
||||
});
|
||||
|
||||
test('redacts multiple emails in one query', () => {
|
||||
const out = scrubPii('alice@example.com cc bob@other.org');
|
||||
expect(out).not.toContain('@example.com');
|
||||
expect(out).not.toContain('@other.org');
|
||||
expect(out.match(/\[REDACTED\]/g) || []).toHaveLength(2);
|
||||
});
|
||||
|
||||
test('redacts US phone 555-123-4567', () => {
|
||||
expect(scrubPii('call 555-123-4567')).toBe('call [REDACTED]');
|
||||
});
|
||||
|
||||
test('redacts US phone (555) 123-4567', () => {
|
||||
expect(scrubPii('tel (555) 123-4567 ext 4')).toContain('[REDACTED]');
|
||||
});
|
||||
|
||||
test('redacts E.164 +1 555 123 4567', () => {
|
||||
expect(scrubPii('intl +1 555 123 4567')).toContain('[REDACTED]');
|
||||
});
|
||||
|
||||
test('redacts SSN with dashes (XXX-XX-XXXX)', () => {
|
||||
expect(scrubPii('ssn is 123-45-6789 on file')).toContain('[REDACTED]');
|
||||
expect(scrubPii('ssn is 123-45-6789 on file')).not.toContain('123-45-6789');
|
||||
});
|
||||
|
||||
test('does NOT redact a 4-digit year like 2026 as SSN', () => {
|
||||
expect(scrubPii('released in 2026, v0.21.0')).toContain('2026');
|
||||
});
|
||||
|
||||
test('redacts a Visa number that passes Luhn', () => {
|
||||
// 4111 1111 1111 1111 is the canonical Visa test number (valid Luhn).
|
||||
const out = scrubPii('card 4111 1111 1111 1111 expires 12/26');
|
||||
expect(out).not.toContain('4111 1111 1111 1111');
|
||||
expect(out).toContain('[REDACTED]');
|
||||
});
|
||||
|
||||
test('does NOT redact a 16-digit integer that fails Luhn', () => {
|
||||
// 1234567890123456 has Luhn mod 10 = 4 (fails).
|
||||
expect(scrubPii('order id 1234567890123456')).toContain('1234567890123456');
|
||||
});
|
||||
|
||||
test('does NOT redact bare 6-digit identifiers (#123456)', () => {
|
||||
expect(scrubPii('see PR #123456 and issue #789012')).toContain('#123456');
|
||||
});
|
||||
|
||||
test('redacts JWT-shaped tokens', () => {
|
||||
const jwt =
|
||||
'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.' +
|
||||
'eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.' +
|
||||
'SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c';
|
||||
const out = scrubPii(`token: ${jwt}`);
|
||||
expect(out).not.toContain('eyJhbGciOi');
|
||||
expect(out).toContain('[REDACTED]');
|
||||
});
|
||||
|
||||
test('redacts bearer tokens after Authorization-style prefix', () => {
|
||||
const out = scrubPii('Authorization: Bearer abc123DEF456ghi789');
|
||||
expect(out).not.toContain('abc123DEF456ghi789');
|
||||
expect(out).toContain('Bearer [REDACTED]');
|
||||
});
|
||||
|
||||
test('redacts bearer lowercase too', () => {
|
||||
const out = scrubPii('header bearer abc123DEF456ghi789');
|
||||
expect(out).not.toContain('abc123DEF456ghi789');
|
||||
});
|
||||
|
||||
test('handles adversarial nested-group input without hanging', () => {
|
||||
// Classic catastrophic-backtracking bait for regex engines without
|
||||
// possessive quantifiers. Must complete quickly — the scrubber should
|
||||
// stay linear-time on all input. 10k char limit is well under the
|
||||
// CHECK-constrained 50KB cap enforced at DB level.
|
||||
const adversarial = 'a'.repeat(10000) + '!';
|
||||
const start = Date.now();
|
||||
const out = scrubPii(adversarial);
|
||||
const elapsed = Date.now() - start;
|
||||
expect(elapsed).toBeLessThan(1000); // generous cap, realistic target <50ms
|
||||
expect(out).toContain('a'.repeat(100)); // output still visible
|
||||
});
|
||||
|
||||
test('combined input: email + phone + CC + JWT + SSN all redacted together', () => {
|
||||
const mixed =
|
||||
'contact alice@example.com at 555-123-4567, ' +
|
||||
'card 4111-1111-1111-1111, ssn 123-45-6789, ' +
|
||||
'auth Bearer abcdef123456';
|
||||
const out = scrubPii(mixed);
|
||||
expect(out).not.toContain('alice@example.com');
|
||||
expect(out).not.toContain('555-123-4567');
|
||||
expect(out).not.toContain('4111-1111-1111-1111');
|
||||
expect(out).not.toContain('123-45-6789');
|
||||
expect(out).not.toContain('abcdef123456');
|
||||
// Each distinct redaction yields a [REDACTED] token somewhere.
|
||||
expect((out.match(/\[REDACTED\]/g) || []).length).toBeGreaterThanOrEqual(5);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,311 @@
|
||||
/**
|
||||
* Eval capture module tests.
|
||||
*
|
||||
* buildEvalCandidateInput: shape + scrubbing + slug/chunk extraction.
|
||||
* classifyCaptureFailure: SQLSTATE → reason mapping.
|
||||
* captureEvalCandidate: best-effort swallow, failure routing, never throws.
|
||||
*
|
||||
* Engine is mocked so this file runs in unit speed (<50ms).
|
||||
*/
|
||||
|
||||
import { describe, expect, mock, test } from 'bun:test';
|
||||
import {
|
||||
buildEvalCandidateInput,
|
||||
captureEvalCandidate,
|
||||
classifyCaptureFailure,
|
||||
isEvalCaptureEnabled,
|
||||
isEvalScrubEnabled,
|
||||
type CaptureContext,
|
||||
} from '../src/core/eval-capture.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
|
||||
function makeResult(overrides: Partial<SearchResult> = {}): SearchResult {
|
||||
return {
|
||||
slug: 'people/alice-example',
|
||||
page_id: 1,
|
||||
title: 'Alice Example',
|
||||
type: 'person',
|
||||
chunk_text: '…',
|
||||
chunk_source: 'compiled_truth',
|
||||
chunk_id: 42,
|
||||
chunk_index: 0,
|
||||
score: 0.9,
|
||||
stale: false,
|
||||
source_id: 'default',
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function makeCtx(overrides: Partial<CaptureContext> = {}): CaptureContext {
|
||||
return {
|
||||
tool_name: 'query',
|
||||
query: 'who is alice',
|
||||
results: [makeResult()],
|
||||
meta: { vector_enabled: true, detail_resolved: 'medium', expansion_applied: false },
|
||||
latency_ms: 123,
|
||||
remote: true,
|
||||
expand_enabled: true,
|
||||
detail: null,
|
||||
job_id: null,
|
||||
subagent_id: null,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('buildEvalCandidateInput', () => {
|
||||
test('builds a valid row from minimal context', () => {
|
||||
const input = buildEvalCandidateInput(makeCtx());
|
||||
expect(input.tool_name).toBe('query');
|
||||
expect(input.query).toBe('who is alice');
|
||||
expect(input.retrieved_slugs).toEqual(['people/alice-example']);
|
||||
expect(input.retrieved_chunk_ids).toEqual([42]);
|
||||
expect(input.source_ids).toEqual(['default']);
|
||||
expect(input.latency_ms).toBe(123);
|
||||
expect(input.remote).toBe(true);
|
||||
});
|
||||
|
||||
test('dedupes slugs across multiple chunks from the same page', () => {
|
||||
const input = buildEvalCandidateInput(
|
||||
makeCtx({
|
||||
results: [
|
||||
makeResult({ slug: 'people/alice-example', chunk_id: 1 }),
|
||||
makeResult({ slug: 'people/alice-example', chunk_id: 2 }),
|
||||
makeResult({ slug: 'companies/acme', chunk_id: 3 }),
|
||||
],
|
||||
}),
|
||||
);
|
||||
expect(input.retrieved_slugs).toEqual(['people/alice-example', 'companies/acme']);
|
||||
// chunk ids preserve order and duplicates (each hit is distinct).
|
||||
expect(input.retrieved_chunk_ids).toEqual([1, 2, 3]);
|
||||
});
|
||||
|
||||
test('dedupes source_ids', () => {
|
||||
const input = buildEvalCandidateInput(
|
||||
makeCtx({
|
||||
results: [
|
||||
makeResult({ source_id: 'default' }),
|
||||
makeResult({ source_id: 'default' }),
|
||||
makeResult({ source_id: 'wiki' }),
|
||||
],
|
||||
}),
|
||||
);
|
||||
expect(input.retrieved_slugs).toHaveLength(1);
|
||||
expect(input.source_ids.sort()).toEqual(['default', 'wiki']);
|
||||
});
|
||||
|
||||
test('omits source_id from the set when a result lacks one (pre-v0.18 rows)', () => {
|
||||
const input = buildEvalCandidateInput(
|
||||
makeCtx({ results: [makeResult({ source_id: undefined })] }),
|
||||
);
|
||||
expect(input.source_ids).toEqual([]);
|
||||
});
|
||||
|
||||
test('scrubs PII by default', () => {
|
||||
const input = buildEvalCandidateInput(
|
||||
makeCtx({ query: 'email alice@example.com about the ticket' }),
|
||||
);
|
||||
expect(input.query).not.toContain('alice@example.com');
|
||||
expect(input.query).toContain('[REDACTED]');
|
||||
});
|
||||
|
||||
test('preserves PII when scrub_pii is false', () => {
|
||||
const input = buildEvalCandidateInput(
|
||||
makeCtx({ query: 'email alice@example.com' }),
|
||||
{ scrub_pii: false },
|
||||
);
|
||||
expect(input.query).toBe('email alice@example.com');
|
||||
});
|
||||
|
||||
test('carries hybridSearch meta onto the row verbatim', () => {
|
||||
const input = buildEvalCandidateInput(
|
||||
makeCtx({
|
||||
meta: { vector_enabled: false, detail_resolved: 'high', expansion_applied: true },
|
||||
}),
|
||||
);
|
||||
expect(input.vector_enabled).toBe(false);
|
||||
expect(input.detail_resolved).toBe('high');
|
||||
expect(input.expansion_applied).toBe(true);
|
||||
});
|
||||
|
||||
test('propagates jobId + subagentId when present (subagent tool-bridge path)', () => {
|
||||
const input = buildEvalCandidateInput(
|
||||
makeCtx({ job_id: 9001, subagent_id: 42 }),
|
||||
);
|
||||
expect(input.job_id).toBe(9001);
|
||||
expect(input.subagent_id).toBe(42);
|
||||
});
|
||||
});
|
||||
|
||||
describe('classifyCaptureFailure', () => {
|
||||
test('maps Postgres 23514 to check_violation', () => {
|
||||
expect(classifyCaptureFailure({ code: '23514' })).toBe('check_violation');
|
||||
});
|
||||
test('maps Postgres 42501 to rls_reject', () => {
|
||||
expect(classifyCaptureFailure({ code: '42501' })).toBe('rls_reject');
|
||||
});
|
||||
test('maps Postgres 42P01 to db_down', () => {
|
||||
expect(classifyCaptureFailure({ code: '42P01' })).toBe('db_down');
|
||||
});
|
||||
test('maps connection SQLSTATEs to db_down', () => {
|
||||
expect(classifyCaptureFailure({ code: '08006' })).toBe('db_down');
|
||||
expect(classifyCaptureFailure({ code: '53300' })).toBe('db_down');
|
||||
});
|
||||
test('maps RegExpMatchError to scrubber_exception', () => {
|
||||
expect(classifyCaptureFailure({ name: 'RegExpMatchError' })).toBe('scrubber_exception');
|
||||
});
|
||||
test('falls back to other for unknown errors', () => {
|
||||
expect(classifyCaptureFailure(new Error('something else'))).toBe('other');
|
||||
expect(classifyCaptureFailure('just a string')).toBe('other');
|
||||
expect(classifyCaptureFailure(null)).toBe('other');
|
||||
});
|
||||
});
|
||||
|
||||
describe('captureEvalCandidate — best-effort failure handling', () => {
|
||||
function makeMockEngine(overrides: Partial<BrainEngine> = {}): BrainEngine {
|
||||
return {
|
||||
logEvalCandidate: mock(() => Promise.resolve(1)),
|
||||
logEvalCaptureFailure: mock(() => Promise.resolve()),
|
||||
...overrides,
|
||||
} as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
test('happy path: calls logEvalCandidate and does not throw', async () => {
|
||||
const engine = makeMockEngine();
|
||||
await expect(captureEvalCandidate(engine, makeCtx())).resolves.toBeUndefined();
|
||||
expect((engine.logEvalCandidate as unknown as { mock: { calls: unknown[] } }).mock.calls).toHaveLength(1);
|
||||
expect((engine.logEvalCaptureFailure as unknown as { mock: { calls: unknown[] } }).mock.calls).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('DB error: routes to logEvalCaptureFailure with classified reason, does not throw', async () => {
|
||||
const rlsErr = Object.assign(new Error('row violates row-level security'), { code: '42501' });
|
||||
const engine = makeMockEngine({
|
||||
logEvalCandidate: mock(() => Promise.reject(rlsErr)),
|
||||
});
|
||||
await expect(captureEvalCandidate(engine, makeCtx())).resolves.toBeUndefined();
|
||||
const failureCalls = (engine.logEvalCaptureFailure as unknown as { mock: { calls: unknown[][] } }).mock.calls;
|
||||
expect(failureCalls).toHaveLength(1);
|
||||
expect(failureCalls[0]![0]).toBe('rls_reject');
|
||||
});
|
||||
|
||||
test('failure-of-failure: if the failure-log insert also rejects, still does not throw', async () => {
|
||||
const engine = makeMockEngine({
|
||||
logEvalCandidate: mock(() => Promise.reject(new Error('primary fail'))),
|
||||
logEvalCaptureFailure: mock(() => Promise.reject(new Error('secondary fail'))),
|
||||
});
|
||||
await expect(captureEvalCandidate(engine, makeCtx())).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
test('sync scrubber throw lands in check path (mocked scrub would throw)', async () => {
|
||||
// We can't force the real scrubber to throw on normal input — the
|
||||
// adversarial-input test in eval-capture-scrub.test.ts covers that.
|
||||
// This smoke test proves buildEvalCandidateInput throw propagates to
|
||||
// the outer try/catch by handing an engine that throws synchronously
|
||||
// in logEvalCandidate to simulate the "anything after scrub can fail".
|
||||
const engine = makeMockEngine({
|
||||
logEvalCandidate: mock(() => {
|
||||
throw new Error('sync throw'); // not a rejection — a sync throw
|
||||
}),
|
||||
});
|
||||
await expect(captureEvalCandidate(engine, makeCtx())).resolves.toBeUndefined();
|
||||
expect(
|
||||
(engine.logEvalCaptureFailure as unknown as { mock: { calls: unknown[] } }).mock.calls,
|
||||
).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isEvalCaptureEnabled / isEvalScrubEnabled (CONTRIBUTOR_MODE-gated)', () => {
|
||||
// v0.25.0 flipped the default: was on for everyone, now off unless either
|
||||
// the env var or an explicit config flag is set. Tests scope env mutation
|
||||
// so they don't leak across describe blocks.
|
||||
const origMode = process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
const restore = () => {
|
||||
if (origMode === undefined) delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
else process.env.GBRAIN_CONTRIBUTOR_MODE = origMode;
|
||||
};
|
||||
|
||||
test('defaults to OFF when config is missing AND CONTRIBUTOR_MODE unset', () => {
|
||||
delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||
expect(isEvalCaptureEnabled(undefined)).toBe(false);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('CONTRIBUTOR_MODE=1 turns capture on without any config', () => {
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(true);
|
||||
expect(isEvalCaptureEnabled(undefined)).toBe(true);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('CONTRIBUTOR_MODE=anything-else does NOT turn capture on (strict equality)', () => {
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = 'true';
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||
} finally { restore(); }
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = 'yes';
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||
} finally { restore(); }
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '';
|
||||
try {
|
||||
expect(isEvalCaptureEnabled(null)).toBe(false);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('explicit config eval.capture=true wins over absent CONTRIBUTOR_MODE', () => {
|
||||
delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const enabled: any = { engine: 'pglite', eval: { capture: true } };
|
||||
expect(isEvalCaptureEnabled(enabled)).toBe(true);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('explicit config eval.capture=false wins over CONTRIBUTOR_MODE=1', () => {
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const disabled: any = { engine: 'pglite', eval: { capture: false } };
|
||||
expect(isEvalCaptureEnabled(disabled)).toBe(false);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('config eval present but capture key undefined falls through to env check', () => {
|
||||
delete process.env.GBRAIN_CONTRIBUTOR_MODE;
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const partial: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||
expect(isEvalCaptureEnabled(partial)).toBe(false);
|
||||
} finally { restore(); }
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const partial: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||
expect(isEvalCaptureEnabled(partial)).toBe(true);
|
||||
} finally { restore(); }
|
||||
});
|
||||
|
||||
test('isEvalScrubEnabled: defaults true, only false when explicitly disabled', () => {
|
||||
expect(isEvalScrubEnabled(null)).toBe(true);
|
||||
expect(isEvalScrubEnabled(undefined)).toBe(true);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const partial: any = { engine: 'pglite', eval: {} };
|
||||
expect(isEvalScrubEnabled(partial)).toBe(true);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const disabled: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||
expect(isEvalScrubEnabled(disabled)).toBe(false);
|
||||
});
|
||||
|
||||
test('isEvalScrubEnabled is independent of CONTRIBUTOR_MODE', () => {
|
||||
process.env.GBRAIN_CONTRIBUTOR_MODE = '1';
|
||||
try {
|
||||
expect(isEvalScrubEnabled(null)).toBe(true);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
const opt: any = { engine: 'pglite', eval: { scrub_pii: false } };
|
||||
expect(isEvalScrubEnabled(opt)).toBe(false);
|
||||
} finally { restore(); }
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,178 @@
|
||||
/**
|
||||
* gbrain eval export — NDJSON contract (v0.21.0).
|
||||
*
|
||||
* Verifies:
|
||||
* - every line is valid JSON
|
||||
* - every line has "schema_version": 1
|
||||
* - --since / --tool / --limit filter correctly
|
||||
* - stdout ordering is newest-first
|
||||
* - EPIPE on stdout doesn't crash the process (covered by invariant
|
||||
* that runEvalExport returns without throwing on truncated streams)
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runEvalExport } from '../src/commands/eval-export.ts';
|
||||
import type { EvalCandidateInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
function baseInput(overrides: Partial<EvalCandidateInput> = {}): EvalCandidateInput {
|
||||
return {
|
||||
tool_name: 'query',
|
||||
query: 'alice',
|
||||
retrieved_slugs: ['people/alice-example'],
|
||||
retrieved_chunk_ids: [1],
|
||||
source_ids: ['default'],
|
||||
expand_enabled: true,
|
||||
detail: null,
|
||||
detail_resolved: 'medium',
|
||||
vector_enabled: false,
|
||||
expansion_applied: false,
|
||||
latency_ms: 100,
|
||||
remote: false,
|
||||
job_id: null,
|
||||
subagent_id: null,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.exec('DELETE FROM eval_candidates');
|
||||
});
|
||||
|
||||
/**
|
||||
* Capture stdout.write output during runEvalExport. The command writes
|
||||
* NDJSON directly to process.stdout; we intercept to inspect.
|
||||
*/
|
||||
async function captureExport(args: string[]): Promise<string> {
|
||||
const captured: string[] = [];
|
||||
const originalWrite = process.stdout.write.bind(process.stdout);
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(process.stdout as any).write = (chunk: string | Uint8Array) => {
|
||||
captured.push(typeof chunk === 'string' ? chunk : new TextDecoder().decode(chunk));
|
||||
return true;
|
||||
};
|
||||
try {
|
||||
await runEvalExport(engine, args);
|
||||
} finally {
|
||||
process.stdout.write = originalWrite;
|
||||
}
|
||||
return captured.join('');
|
||||
}
|
||||
|
||||
describe('gbrain eval export — NDJSON shape', () => {
|
||||
test('empty table → zero lines, exit 0', async () => {
|
||||
const out = await captureExport([]);
|
||||
expect(out).toBe('');
|
||||
});
|
||||
|
||||
test('every row is a valid JSON object on its own line', async () => {
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await engine.logEvalCandidate(baseInput({ query: `q${i}` }));
|
||||
}
|
||||
const out = await captureExport([]);
|
||||
const lines = out.trim().split('\n');
|
||||
expect(lines).toHaveLength(3);
|
||||
for (const line of lines) {
|
||||
const parsed = JSON.parse(line);
|
||||
expect(parsed).toHaveProperty('id');
|
||||
expect(parsed).toHaveProperty('query');
|
||||
expect(parsed).toHaveProperty('tool_name');
|
||||
}
|
||||
});
|
||||
|
||||
test('every row starts with "schema_version":1 (F5 contract)', async () => {
|
||||
await engine.logEvalCandidate(baseInput());
|
||||
const out = await captureExport([]);
|
||||
const parsed = JSON.parse(out.trim());
|
||||
expect(parsed.schema_version).toBe(1);
|
||||
});
|
||||
|
||||
test('--tool query filters to query rows only', async () => {
|
||||
await engine.logEvalCandidate(baseInput({ tool_name: 'query' }));
|
||||
await engine.logEvalCandidate(
|
||||
baseInput({ tool_name: 'search', expand_enabled: null, detail_resolved: null }),
|
||||
);
|
||||
await engine.logEvalCandidate(baseInput({ tool_name: 'query' }));
|
||||
|
||||
const out = await captureExport(['--tool', 'query']);
|
||||
const lines = out.trim().split('\n').filter(Boolean);
|
||||
expect(lines).toHaveLength(2);
|
||||
for (const line of lines) {
|
||||
expect(JSON.parse(line).tool_name).toBe('query');
|
||||
}
|
||||
});
|
||||
|
||||
test('--limit caps the row count', async () => {
|
||||
for (let i = 0; i < 5; i++) {
|
||||
await engine.logEvalCandidate(baseInput({ query: `q${i}` }));
|
||||
}
|
||||
const out = await captureExport(['--limit', '2']);
|
||||
const lines = out.trim().split('\n').filter(Boolean);
|
||||
expect(lines).toHaveLength(2);
|
||||
});
|
||||
|
||||
test('--since DUR filter rejects everything older than the window', async () => {
|
||||
await engine.logEvalCandidate(baseInput({ query: 'old' }));
|
||||
// --since 1ms: rows created more than 1ms ago are excluded, but the
|
||||
// insert we just did happened within the last 1ms so it might still
|
||||
// slip through. Use a clearly past window instead.
|
||||
const out = await captureExport(['--since', '0s']);
|
||||
// 0s is parsed as "since now - 0ms" i.e. since now, which excludes
|
||||
// everything that happened before the function started.
|
||||
const lines = out.trim().split('\n').filter(Boolean);
|
||||
expect(lines.length).toBeLessThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('invalid --tool value exits 1 (via process.exit mock check)', async () => {
|
||||
// Replace process.exit so we can see the exit code without killing bun.
|
||||
const originalExit = process.exit;
|
||||
let exitCode: number | undefined;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(process.exit as any) = (code?: number) => {
|
||||
exitCode = code;
|
||||
throw new Error('exit'); // abort the command
|
||||
};
|
||||
try {
|
||||
await captureExport(['--tool', 'dance']);
|
||||
} catch { /* expected */ }
|
||||
process.exit = originalExit;
|
||||
expect(exitCode).toBe(1);
|
||||
});
|
||||
|
||||
test('invalid --since format exits 1', async () => {
|
||||
const originalExit = process.exit;
|
||||
let exitCode: number | undefined;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(process.exit as any) = (code?: number) => {
|
||||
exitCode = code;
|
||||
throw new Error('exit');
|
||||
};
|
||||
try {
|
||||
await captureExport(['--since', 'yesterday']);
|
||||
} catch { /* expected */ }
|
||||
process.exit = originalExit;
|
||||
expect(exitCode).toBe(1);
|
||||
});
|
||||
|
||||
test('output is stream-friendly: each row on its own \\n-terminated line', async () => {
|
||||
await engine.logEvalCandidate(baseInput());
|
||||
await engine.logEvalCandidate(baseInput({ query: 'second' }));
|
||||
const out = await captureExport([]);
|
||||
// Trailing newline on every record — no concatenated blobs.
|
||||
expect(out).toMatch(/\}\n$/);
|
||||
expect(out.split('\n').filter(Boolean)).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,134 @@
|
||||
/**
|
||||
* gbrain eval prune — retention cleanup (v0.21.0).
|
||||
*
|
||||
* Verifies:
|
||||
* - --older-than parses duration strings (30d, 1h, 90m, 3600s)
|
||||
* - deletes only rows older than the cutoff
|
||||
* - --dry-run reports count without deleting
|
||||
* - missing --older-than exits with error
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runEvalPrune } from '../src/commands/eval-prune.ts';
|
||||
import type { EvalCandidateInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.exec('DELETE FROM eval_candidates');
|
||||
});
|
||||
|
||||
function baseInput(): EvalCandidateInput {
|
||||
return {
|
||||
tool_name: 'query',
|
||||
query: 'q',
|
||||
retrieved_slugs: [],
|
||||
retrieved_chunk_ids: [],
|
||||
source_ids: [],
|
||||
expand_enabled: true,
|
||||
detail: null,
|
||||
detail_resolved: null,
|
||||
vector_enabled: false,
|
||||
expansion_applied: false,
|
||||
latency_ms: 50,
|
||||
remote: false,
|
||||
job_id: null,
|
||||
subagent_id: null,
|
||||
};
|
||||
}
|
||||
|
||||
/** Insert a row with an explicit created_at timestamp (for "aged" data). */
|
||||
async function insertAged(daysAgo: number): Promise<void> {
|
||||
const when = new Date(Date.now() - daysAgo * 86_400_000).toISOString();
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.query(
|
||||
`INSERT INTO eval_candidates (
|
||||
tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
|
||||
expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
|
||||
latency_ms, remote, job_id, subagent_id, created_at
|
||||
) VALUES ('query', 'q', '{}', '{}', '{}',
|
||||
true, null, null, false, false, 50, false, null, null, $1)`,
|
||||
[when],
|
||||
);
|
||||
}
|
||||
|
||||
async function withSilencedStdout<T>(fn: () => Promise<T>): Promise<T> {
|
||||
const orig = console.log;
|
||||
console.log = () => {};
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
console.log = orig;
|
||||
}
|
||||
}
|
||||
|
||||
describe('gbrain eval prune', () => {
|
||||
test('--older-than 30d deletes rows older than 30 days, keeps recent', async () => {
|
||||
await insertAged(60); // old
|
||||
await insertAged(45); // old
|
||||
await insertAged(10); // kept
|
||||
await engine.logEvalCandidate(baseInput()); // just now, kept
|
||||
|
||||
expect(await engine.listEvalCandidates()).toHaveLength(4);
|
||||
await withSilencedStdout(() => runEvalPrune(engine, ['--older-than', '30d']));
|
||||
|
||||
const remaining = await engine.listEvalCandidates();
|
||||
expect(remaining).toHaveLength(2);
|
||||
});
|
||||
|
||||
test('--dry-run does not delete anything', async () => {
|
||||
await insertAged(60);
|
||||
await insertAged(45);
|
||||
await withSilencedStdout(() => runEvalPrune(engine, ['--older-than', '30d', '--dry-run']));
|
||||
expect(await engine.listEvalCandidates()).toHaveLength(2);
|
||||
});
|
||||
|
||||
test('accepts multiple duration units', async () => {
|
||||
// Each of these runs should succeed without error.
|
||||
for (const dur of ['1h', '90m', '3600s', '7d']) {
|
||||
await withSilencedStdout(() => runEvalPrune(engine, ['--older-than', dur]));
|
||||
}
|
||||
});
|
||||
|
||||
test('missing --older-than exits 1', async () => {
|
||||
const originalExit = process.exit;
|
||||
let exitCode: number | undefined;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(process.exit as any) = (code?: number) => {
|
||||
exitCode = code;
|
||||
throw new Error('exit');
|
||||
};
|
||||
try {
|
||||
await runEvalPrune(engine, []);
|
||||
} catch { /* expected */ }
|
||||
process.exit = originalExit;
|
||||
expect(exitCode).toBe(1);
|
||||
});
|
||||
|
||||
test('invalid --older-than value exits 1', async () => {
|
||||
const originalExit = process.exit;
|
||||
let exitCode: number | undefined;
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
(process.exit as any) = (code?: number) => {
|
||||
exitCode = code;
|
||||
throw new Error('exit');
|
||||
};
|
||||
try {
|
||||
await runEvalPrune(engine, ['--older-than', 'foo']);
|
||||
} catch { /* expected */ }
|
||||
process.exit = originalExit;
|
||||
expect(exitCode).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,421 @@
|
||||
/**
|
||||
* Tests for `gbrain eval replay` (v0.25.0).
|
||||
*
|
||||
* Three layers:
|
||||
* 1. Pure: Jaccard math + NDJSON parser via re-export.
|
||||
* 2. CLI happy path: --against captures NDJSON → human/JSON output.
|
||||
* 3. CLI failure: missing file, bad NDJSON, mismatched schema_version.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { writeFileSync, mkdtempSync, rmSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { runEvalReplay } from '../src/commands/eval-replay.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
|
||||
// Minimal stub engine — only the methods replayRow touches matter.
|
||||
// Every other method is a thrower so accidental use surfaces in tests.
|
||||
function makeStubEngine(returns: Record<string, SearchResult[]>): BrainEngine {
|
||||
return {
|
||||
kind: 'pglite',
|
||||
searchKeyword: async (q: string) => returns[q] ?? [],
|
||||
searchVector: async () => [],
|
||||
} as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
function fakeResult(slug: string, score = 0.5): SearchResult {
|
||||
return {
|
||||
slug,
|
||||
chunk_id: 1,
|
||||
chunk_index: 0,
|
||||
chunk_text: '',
|
||||
score,
|
||||
title: slug,
|
||||
page_kind: 'markdown',
|
||||
source_id: 'default',
|
||||
} as unknown as SearchResult;
|
||||
}
|
||||
|
||||
function makeCapturedRow(over: Partial<{
|
||||
id: number;
|
||||
tool_name: 'query' | 'search';
|
||||
query: string;
|
||||
retrieved_slugs: string[];
|
||||
detail: 'low' | 'medium' | 'high' | null;
|
||||
expand_enabled: boolean | null;
|
||||
latency_ms: number;
|
||||
}>) {
|
||||
return {
|
||||
schema_version: 1,
|
||||
id: over.id ?? 1,
|
||||
tool_name: over.tool_name ?? 'search',
|
||||
query: over.query ?? 'alice',
|
||||
retrieved_slugs: over.retrieved_slugs ?? ['people/alice', 'people/alice-bio'],
|
||||
retrieved_chunk_ids: [],
|
||||
source_ids: [],
|
||||
expand_enabled: over.expand_enabled ?? null,
|
||||
detail: over.detail ?? null,
|
||||
detail_resolved: null,
|
||||
vector_enabled: false,
|
||||
expansion_applied: false,
|
||||
latency_ms: over.latency_ms ?? 50,
|
||||
remote: true,
|
||||
job_id: null,
|
||||
subagent_id: null,
|
||||
created_at: '2026-04-25T00:00:00Z',
|
||||
};
|
||||
}
|
||||
|
||||
function withTmp<T>(fn: (path: string) => Promise<T> | T): Promise<T> {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'eval-replay-'));
|
||||
return Promise.resolve(fn(dir)).finally(() => rmSync(dir, { recursive: true, force: true }));
|
||||
}
|
||||
|
||||
function captureStdoutStderr(): { restore: () => { stdout: string; stderr: string } } {
|
||||
// Bun's console.log doesn't route through process.stdout.write — it uses
|
||||
// internal writers. Hook console directly so test capture works across
|
||||
// console.log/error/warn AND raw stream writes.
|
||||
let outBuf = '';
|
||||
let errBuf = '';
|
||||
const origLog = console.log;
|
||||
const origErrCons = console.error;
|
||||
const origWarn = console.warn;
|
||||
const origStdoutWrite = process.stdout.write.bind(process.stdout);
|
||||
const origStderrWrite = process.stderr.write.bind(process.stderr);
|
||||
const stringify = (a: unknown) => typeof a === 'string' ? a : JSON.stringify(a);
|
||||
console.log = (...args: unknown[]) => { outBuf += args.map(stringify).join(' ') + '\n'; };
|
||||
console.error = (...args: unknown[]) => { errBuf += args.map(stringify).join(' ') + '\n'; };
|
||||
console.warn = (...args: unknown[]) => { errBuf += args.map(stringify).join(' ') + '\n'; };
|
||||
process.stdout.write = ((s: string | Uint8Array) => {
|
||||
outBuf += typeof s === 'string' ? s : new TextDecoder().decode(s);
|
||||
return true;
|
||||
}) as typeof process.stdout.write;
|
||||
process.stderr.write = ((s: string | Uint8Array) => {
|
||||
errBuf += typeof s === 'string' ? s : new TextDecoder().decode(s);
|
||||
return true;
|
||||
}) as typeof process.stderr.write;
|
||||
return {
|
||||
restore: () => {
|
||||
console.log = origLog;
|
||||
console.error = origErrCons;
|
||||
console.warn = origWarn;
|
||||
process.stdout.write = origStdoutWrite;
|
||||
process.stderr.write = origStderrWrite;
|
||||
return { stdout: outBuf, stderr: errBuf };
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('gbrain eval replay — happy path', () => {
|
||||
test('search row with identical retrieved slugs reports Jaccard 1.0', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const ndjson = JSON.stringify(makeCapturedRow({
|
||||
tool_name: 'search',
|
||||
query: 'alice',
|
||||
retrieved_slugs: ['people/alice', 'people/alice-bio'],
|
||||
})) + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, ndjson);
|
||||
|
||||
const engine = makeStubEngine({
|
||||
alice: [fakeResult('people/alice'), fakeResult('people/alice-bio')],
|
||||
});
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file, '--json']);
|
||||
const { stdout } = cap.restore();
|
||||
const out = JSON.parse(stdout);
|
||||
expect(out.schema_version).toBe(1);
|
||||
expect(out.summary.rows_replayed).toBe(1);
|
||||
expect(out.summary.mean_jaccard).toBe(1.0);
|
||||
expect(out.summary.top1_stability_rate).toBe(1.0);
|
||||
});
|
||||
});
|
||||
|
||||
test('search row with disjoint retrieved slugs reports Jaccard 0', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const ndjson = JSON.stringify(makeCapturedRow({
|
||||
tool_name: 'search',
|
||||
query: 'bob',
|
||||
retrieved_slugs: ['people/bob', 'people/bob-bio'],
|
||||
})) + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, ndjson);
|
||||
|
||||
const engine = makeStubEngine({
|
||||
bob: [fakeResult('people/charlie'), fakeResult('people/charlie-bio')],
|
||||
});
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file, '--json']);
|
||||
const { stdout } = cap.restore();
|
||||
const out = JSON.parse(stdout);
|
||||
expect(out.summary.mean_jaccard).toBe(0);
|
||||
expect(out.summary.top1_stability_rate).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('partial overlap — captured [a,b], current [a,c] → Jaccard 1/3', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const ndjson = JSON.stringify(makeCapturedRow({
|
||||
retrieved_slugs: ['a', 'b'],
|
||||
query: 'q',
|
||||
})) + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, ndjson);
|
||||
|
||||
const engine = makeStubEngine({ q: [fakeResult('a'), fakeResult('c')] });
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file, '--json']);
|
||||
const { stdout } = cap.restore();
|
||||
const out = JSON.parse(stdout);
|
||||
// {a,b} ∩ {a,c} = {a} (1), union {a,b,c} (3) → 1/3
|
||||
expect(out.summary.mean_jaccard).toBeCloseTo(1 / 3, 5);
|
||||
// top-1 still matches (both lead with 'a')
|
||||
expect(out.summary.top1_stability_rate).toBe(1.0);
|
||||
});
|
||||
});
|
||||
|
||||
test('top-1 mismatch reduces stability rate', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const ndjson = JSON.stringify(makeCapturedRow({
|
||||
retrieved_slugs: ['a', 'b', 'c'],
|
||||
query: 'q',
|
||||
})) + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, ndjson);
|
||||
|
||||
const engine = makeStubEngine({ q: [fakeResult('b'), fakeResult('a'), fakeResult('c')] });
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file, '--json']);
|
||||
const { stdout } = cap.restore();
|
||||
const out = JSON.parse(stdout);
|
||||
// Same set, jaccard = 1.0
|
||||
expect(out.summary.mean_jaccard).toBe(1.0);
|
||||
// top-1 swapped a → b, stability 0
|
||||
expect(out.summary.top1_stability_rate).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('multiple rows → averaged Jaccard + top-1 rate', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const lines = [
|
||||
JSON.stringify(makeCapturedRow({ id: 1, query: 'q1', retrieved_slugs: ['a'] })),
|
||||
JSON.stringify(makeCapturedRow({ id: 2, query: 'q2', retrieved_slugs: ['b'] })),
|
||||
].join('\n') + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, lines);
|
||||
|
||||
const engine = makeStubEngine({
|
||||
q1: [fakeResult('a')], // perfect match
|
||||
q2: [fakeResult('z')], // miss
|
||||
});
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file, '--json']);
|
||||
const { stdout } = cap.restore();
|
||||
const out = JSON.parse(stdout);
|
||||
// (1.0 + 0) / 2 = 0.5
|
||||
expect(out.summary.mean_jaccard).toBe(0.5);
|
||||
// (1 + 0) / 2 = 0.5
|
||||
expect(out.summary.top1_stability_rate).toBe(0.5);
|
||||
expect(out.summary.rows_replayed).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
test('--limit caps replay count', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const lines = [1, 2, 3, 4, 5]
|
||||
.map(i => JSON.stringify(makeCapturedRow({ id: i, query: `q${i}`, retrieved_slugs: ['a'] })))
|
||||
.join('\n') + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, lines);
|
||||
|
||||
const engine = makeStubEngine({}); // returns [] for everything
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file, '--limit', '2', '--json']);
|
||||
const { stdout } = cap.restore();
|
||||
const out = JSON.parse(stdout);
|
||||
expect(out.summary.rows_total).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
test('empty query is skipped, not counted in replayed', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const ndjson = JSON.stringify(makeCapturedRow({ query: '', retrieved_slugs: [] })) + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, ndjson);
|
||||
|
||||
const engine = makeStubEngine({});
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file, '--json']);
|
||||
const { stdout } = cap.restore();
|
||||
const out = JSON.parse(stdout);
|
||||
expect(out.summary.rows_skipped).toBe(1);
|
||||
expect(out.summary.rows_replayed).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('--verbose includes per-row results in JSON', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const ndjson = JSON.stringify(makeCapturedRow({ query: 'q', retrieved_slugs: ['a'] })) + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, ndjson);
|
||||
|
||||
const engine = makeStubEngine({ q: [fakeResult('a')] });
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file, '--json', '--verbose']);
|
||||
const { stdout } = cap.restore();
|
||||
const out = JSON.parse(stdout);
|
||||
expect(Array.isArray(out.results)).toBe(true);
|
||||
expect(out.results.length).toBe(1);
|
||||
expect(out.results[0].jaccard).toBe(1.0);
|
||||
expect(out.results[0].current_slugs).toEqual(['a']);
|
||||
});
|
||||
});
|
||||
|
||||
test('row that throws during replay → errored, not crash', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const ndjson = JSON.stringify(makeCapturedRow({ query: 'boom' })) + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, ndjson);
|
||||
|
||||
const engine = {
|
||||
kind: 'pglite' as const,
|
||||
searchKeyword: async () => { throw new Error('engine offline'); },
|
||||
searchVector: async () => [],
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file, '--json']);
|
||||
const { stdout } = cap.restore();
|
||||
const out = JSON.parse(stdout);
|
||||
expect(out.summary.rows_errored).toBe(1);
|
||||
expect(out.summary.rows_replayed).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('human mode prints summary and top regressions', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const ndjson = JSON.stringify(makeCapturedRow({ query: 'q', retrieved_slugs: ['a'] })) + '\n';
|
||||
const file = join(dir, 'baseline.ndjson');
|
||||
writeFileSync(file, ndjson);
|
||||
|
||||
const engine = makeStubEngine({ q: [fakeResult('z')] }); // miss
|
||||
|
||||
const cap = captureStdoutStderr();
|
||||
await runEvalReplay(engine, ['--against', file]);
|
||||
const { stdout, stderr } = cap.restore();
|
||||
expect(stderr).toContain('Replaying 1');
|
||||
expect(stdout).toContain('Mean Jaccard@k:');
|
||||
expect(stdout).toContain('Top-1 stability:');
|
||||
expect(stdout).toContain('regression');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain eval replay — failure modes', () => {
|
||||
test('missing --against errors and exits 1', async () => {
|
||||
const origExit = process.exit;
|
||||
let code: number | undefined;
|
||||
process.exit = (c?: number) => { code = c; throw new Error('exit'); };
|
||||
const cap = captureStdoutStderr();
|
||||
try {
|
||||
await runEvalReplay({} as BrainEngine, []);
|
||||
} catch { /* expected */ }
|
||||
cap.restore();
|
||||
process.exit = origExit;
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
test('--against pointing at missing file errors and exits 1', async () => {
|
||||
const origExit = process.exit;
|
||||
let code: number | undefined;
|
||||
process.exit = (c?: number) => { code = c; throw new Error('exit'); };
|
||||
const cap = captureStdoutStderr();
|
||||
try {
|
||||
await runEvalReplay({} as BrainEngine, ['--against', '/tmp/nope-' + Date.now() + '.ndjson']);
|
||||
} catch { /* expected */ }
|
||||
cap.restore();
|
||||
process.exit = origExit;
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
|
||||
test('NDJSON line missing schema_version is rejected', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const file = join(dir, 'bad.ndjson');
|
||||
writeFileSync(file, '{"id":1,"tool_name":"search","query":"q","retrieved_slugs":[]}\n');
|
||||
const origExit = process.exit;
|
||||
let code: number | undefined;
|
||||
process.exit = (c?: number) => { code = c; throw new Error('exit'); };
|
||||
const cap = captureStdoutStderr();
|
||||
try {
|
||||
await runEvalReplay({} as BrainEngine, ['--against', file]);
|
||||
} catch { /* expected */ }
|
||||
const { stderr } = cap.restore();
|
||||
process.exit = origExit;
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toContain('schema_version');
|
||||
});
|
||||
});
|
||||
|
||||
test('schema_version === 2 (future) is rejected with helpful message', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const file = join(dir, 'v2.ndjson');
|
||||
writeFileSync(file, '{"schema_version":2,"id":1,"tool_name":"search","query":"q","retrieved_slugs":[],"latency_ms":0}\n');
|
||||
const origExit = process.exit;
|
||||
let code: number | undefined;
|
||||
process.exit = (c?: number) => { code = c; throw new Error('exit'); };
|
||||
const cap = captureStdoutStderr();
|
||||
try {
|
||||
await runEvalReplay({} as BrainEngine, ['--against', file]);
|
||||
} catch { /* expected */ }
|
||||
const { stderr } = cap.restore();
|
||||
process.exit = origExit;
|
||||
expect(code).toBe(1);
|
||||
expect(stderr.toLowerCase()).toContain('upgrade gbrain');
|
||||
});
|
||||
});
|
||||
|
||||
test('empty file errors and exits 1', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const file = join(dir, 'empty.ndjson');
|
||||
writeFileSync(file, '');
|
||||
const origExit = process.exit;
|
||||
let code: number | undefined;
|
||||
process.exit = (c?: number) => { code = c; throw new Error('exit'); };
|
||||
const cap = captureStdoutStderr();
|
||||
try {
|
||||
await runEvalReplay({} as BrainEngine, ['--against', file]);
|
||||
} catch { /* expected */ }
|
||||
cap.restore();
|
||||
process.exit = origExit;
|
||||
expect(code).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
test('malformed JSON line is rejected with line number', async () => {
|
||||
await withTmp(async (dir) => {
|
||||
const file = join(dir, 'bad.ndjson');
|
||||
writeFileSync(file, '{"schema_version":1,"id":1}\nthis is not json\n');
|
||||
const origExit = process.exit;
|
||||
let code: number | undefined;
|
||||
process.exit = (c?: number) => { code = c; throw new Error('exit'); };
|
||||
const cap = captureStdoutStderr();
|
||||
try {
|
||||
await runEvalReplay({} as BrainEngine, ['--against', file]);
|
||||
} catch { /* expected */ }
|
||||
const { stderr } = cap.restore();
|
||||
process.exit = origExit;
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toContain('line 2');
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,102 @@
|
||||
/**
|
||||
* hybridSearch meta-field accuracy (v0.25.0, callback-based API).
|
||||
*
|
||||
* v0.25.0 keeps hybridSearch's return as `Promise<SearchResult[]>` (so
|
||||
* Cathedral II callers stay unchanged) and surfaces meta via an optional
|
||||
* `onMeta` callback in HybridSearchOpts. Asserts the callback fires with
|
||||
* accurate values:
|
||||
* - vector_enabled=false when OPENAI_API_KEY missing (keyword-only path)
|
||||
* - detail_resolved reflects auto-detect + caller override
|
||||
* - expansion_applied only true when expandFn returned variants
|
||||
*
|
||||
* Uses PGLite in-memory + no embedding calls (vector path doesn't need
|
||||
* real embeddings to test the meta flag since we control the env).
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { hybridSearch } from '../src/core/search/hybrid.ts';
|
||||
import type { PageInput, HybridSearchMeta } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
const savedKey = process.env.OPENAI_API_KEY;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
const page: PageInput = {
|
||||
type: 'person',
|
||||
title: 'Alice Example',
|
||||
compiled_truth: 'Alice Example is a test person for hybrid-meta tests.',
|
||||
};
|
||||
await engine.putPage('people/alice-example', page);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (savedKey === undefined) delete process.env.OPENAI_API_KEY;
|
||||
else process.env.OPENAI_API_KEY = savedKey;
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function runWithMeta(query: string, opts: Parameters<typeof hybridSearch>[2] = {}): Promise<HybridSearchMeta | null> {
|
||||
let captured: HybridSearchMeta | null = null;
|
||||
await hybridSearch(engine, query, { ...opts, onMeta: (m) => { captured = m; } });
|
||||
return captured;
|
||||
}
|
||||
|
||||
describe('hybridSearch return shape (v0.25.0 keeps SearchResult[])', () => {
|
||||
test('returns SearchResult[] (unchanged from Cathedral II contract)', async () => {
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
const out = await hybridSearch(engine, 'alice');
|
||||
expect(Array.isArray(out)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearch onMeta callback — vector_enabled', () => {
|
||||
test('false when OPENAI_API_KEY is missing (keyword-only path)', async () => {
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
const meta = await runWithMeta('alice');
|
||||
expect(meta).not.toBeNull();
|
||||
expect(meta!.vector_enabled).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearch onMeta callback — detail_resolved', () => {
|
||||
test('passes through explicit detail override (caller specified "high")', async () => {
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
const meta = await runWithMeta('alice', { detail: 'high' });
|
||||
expect(meta!.detail_resolved).toBe('high');
|
||||
});
|
||||
|
||||
test('detail_resolved reflects autoDetect output when caller omits detail', async () => {
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
const meta = await runWithMeta('alice');
|
||||
expect([null, 'low', 'medium', 'high']).toContain(meta!.detail_resolved);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearch onMeta callback — expansion_applied', () => {
|
||||
test('false when expansion flag is off', async () => {
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
const meta = await runWithMeta('alice', { expansion: false });
|
||||
expect(meta!.expansion_applied).toBe(false);
|
||||
});
|
||||
|
||||
test('false when OPENAI_API_KEY missing (early-return short-circuits expansion)', async () => {
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
const meta = await runWithMeta('alice', {
|
||||
expansion: true,
|
||||
expandFn: async () => ['alice', 'alice example', 'the person alice'],
|
||||
});
|
||||
expect(meta!.expansion_applied).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('onMeta callback omitted', () => {
|
||||
test('hybridSearch works without onMeta (existing Cathedral II callers unaffected)', async () => {
|
||||
delete process.env.OPENAI_API_KEY;
|
||||
const out = await hybridSearch(engine, 'alice');
|
||||
expect(Array.isArray(out)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,245 @@
|
||||
/**
|
||||
* Op-layer capture integration test (v0.21.0).
|
||||
*
|
||||
* The hook lives at the `query` and `search` op handlers in
|
||||
* src/core/operations.ts, not at the MCP dispatch site — this catches
|
||||
* MCP callers, CLI callers, and subagent tool-bridge callers all from
|
||||
* one code path.
|
||||
*
|
||||
* This test invokes the op handler directly with various
|
||||
* OperationContext shapes (remote MCP, local CLI, subagent with jobId)
|
||||
* and asserts that captured rows carry the expected origin metadata.
|
||||
*
|
||||
* Capture is fire-and-forget from the caller — but the INSERT fires on
|
||||
* the same tick, so awaiting a microtask boundary is enough to let the
|
||||
* row land before we assert.
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { operations } from '../src/core/operations.ts';
|
||||
import type { OperationContext } from '../src/core/operations.ts';
|
||||
import type { GBrainConfig } from '../src/core/config.ts';
|
||||
import type { PageInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
const savedKey = process.env.OPENAI_API_KEY;
|
||||
|
||||
beforeAll(async () => {
|
||||
delete process.env.OPENAI_API_KEY; // force keyword-only path so tests don't need live credentials
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
const page: PageInput = {
|
||||
type: 'person',
|
||||
title: 'Alice Example',
|
||||
compiled_truth: 'Alice Example for op-layer capture tests.',
|
||||
};
|
||||
await engine.putPage('people/alice-example', page);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (savedKey === undefined) delete process.env.OPENAI_API_KEY;
|
||||
else process.env.OPENAI_API_KEY = savedKey;
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.exec('DELETE FROM eval_candidates');
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.exec('DELETE FROM eval_capture_failures');
|
||||
});
|
||||
|
||||
function makeConfig(overrides: Partial<GBrainConfig['eval']> = {}): GBrainConfig {
|
||||
return {
|
||||
engine: 'pglite',
|
||||
eval: { capture: true, scrub_pii: true, ...overrides },
|
||||
};
|
||||
}
|
||||
|
||||
function makeCtx(overrides: Partial<OperationContext> = {}): OperationContext {
|
||||
return {
|
||||
engine,
|
||||
config: makeConfig(),
|
||||
logger: console,
|
||||
dryRun: false,
|
||||
remote: true,
|
||||
...overrides,
|
||||
} as OperationContext;
|
||||
}
|
||||
|
||||
/** Tiny helper: wait for fire-and-forget INSERT to land. */
|
||||
async function waitForCapture(): Promise<void> {
|
||||
// Two microtask cycles is plenty — the handler already awaited the op,
|
||||
// so the fire-and-forget INSERT is enqueued. One await resolves the
|
||||
// logEvalCandidate promise; one more for any follow-up.
|
||||
await new Promise(r => setTimeout(r, 10));
|
||||
}
|
||||
|
||||
describe('op-layer capture — query', () => {
|
||||
const queryOp = operations.find(o => o.name === 'query')!;
|
||||
|
||||
test('captures MCP query call with remote=true', async () => {
|
||||
const ctx = makeCtx({ remote: true });
|
||||
await queryOp.handler(ctx, { query: 'alice' });
|
||||
await waitForCapture();
|
||||
|
||||
const rows = await engine.listEvalCandidates();
|
||||
expect(rows).toHaveLength(1);
|
||||
const row = rows[0]!;
|
||||
expect(row.tool_name).toBe('query');
|
||||
expect(row.query).toBe('alice');
|
||||
expect(row.remote).toBe(true);
|
||||
expect(row.expand_enabled).toBe(true); // default
|
||||
expect(row.vector_enabled).toBe(false); // OPENAI_API_KEY deleted
|
||||
expect(row.job_id).toBeNull();
|
||||
expect(row.subagent_id).toBeNull();
|
||||
});
|
||||
|
||||
test('captures CLI query call with remote=false', async () => {
|
||||
const ctx = makeCtx({ remote: false });
|
||||
await queryOp.handler(ctx, { query: 'alice' });
|
||||
await waitForCapture();
|
||||
|
||||
const rows = await engine.listEvalCandidates();
|
||||
expect(rows[0]!.remote).toBe(false);
|
||||
});
|
||||
|
||||
test('captures subagent query call with jobId + subagentId', async () => {
|
||||
const ctx = makeCtx({ remote: true, jobId: 9001, subagentId: 42 });
|
||||
await queryOp.handler(ctx, { query: 'alice' });
|
||||
await waitForCapture();
|
||||
|
||||
const row = (await engine.listEvalCandidates())[0]!;
|
||||
expect(row.job_id).toBe(9001);
|
||||
expect(row.subagent_id).toBe(42);
|
||||
});
|
||||
|
||||
test('scrubs PII by default before insert', async () => {
|
||||
const ctx = makeCtx();
|
||||
await queryOp.handler(ctx, { query: 'email alice@example.com about it' });
|
||||
await waitForCapture();
|
||||
|
||||
const row = (await engine.listEvalCandidates())[0]!;
|
||||
expect(row.query).not.toContain('alice@example.com');
|
||||
expect(row.query).toContain('[REDACTED]');
|
||||
});
|
||||
|
||||
test('preserves PII when scrub_pii is disabled', async () => {
|
||||
const ctx = makeCtx({
|
||||
config: makeConfig({ scrub_pii: false }),
|
||||
});
|
||||
await queryOp.handler(ctx, { query: 'email alice@example.com' });
|
||||
await waitForCapture();
|
||||
|
||||
const row = (await engine.listEvalCandidates())[0]!;
|
||||
expect(row.query).toBe('email alice@example.com');
|
||||
});
|
||||
|
||||
test('does nothing when eval.capture is false (off-switch works)', async () => {
|
||||
const ctx = makeCtx({
|
||||
config: makeConfig({ capture: false }),
|
||||
});
|
||||
await queryOp.handler(ctx, { query: 'alice' });
|
||||
await waitForCapture();
|
||||
|
||||
const rows = await engine.listEvalCandidates();
|
||||
expect(rows).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('op-layer capture — search', () => {
|
||||
const searchOp = operations.find(o => o.name === 'search')!;
|
||||
|
||||
test('captures search call with tool_name="search" and vector_enabled=false', async () => {
|
||||
const ctx = makeCtx();
|
||||
await searchOp.handler(ctx, { query: 'alice' });
|
||||
await waitForCapture();
|
||||
|
||||
const rows = await engine.listEvalCandidates();
|
||||
expect(rows).toHaveLength(1);
|
||||
const row = rows[0]!;
|
||||
expect(row.tool_name).toBe('search');
|
||||
expect(row.vector_enabled).toBe(false); // search never runs vector
|
||||
expect(row.expand_enabled).toBeNull(); // no expansion concept for search
|
||||
expect(row.detail_resolved).toBeNull();
|
||||
});
|
||||
|
||||
test('respects eval.capture=false off-switch', async () => {
|
||||
const ctx = makeCtx({
|
||||
config: makeConfig({ capture: false }),
|
||||
});
|
||||
await searchOp.handler(ctx, { query: 'alice' });
|
||||
await waitForCapture();
|
||||
expect(await engine.listEvalCandidates()).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('op-layer capture — non-query/search ops are NOT captured', () => {
|
||||
test('list_pages does not insert into eval_candidates', async () => {
|
||||
const listPagesOp = operations.find(o => o.name === 'list_pages')!;
|
||||
const ctx = makeCtx();
|
||||
await listPagesOp.handler(ctx, { limit: 10 });
|
||||
await waitForCapture();
|
||||
|
||||
const rows = await engine.listEvalCandidates();
|
||||
expect(rows).toHaveLength(0);
|
||||
});
|
||||
|
||||
test('get_page does not insert into eval_candidates', async () => {
|
||||
const getPageOp = operations.find(o => o.name === 'get_page')!;
|
||||
const ctx = makeCtx();
|
||||
await getPageOp.handler(ctx, { slug: 'people/alice-example' });
|
||||
await waitForCapture();
|
||||
|
||||
const rows = await engine.listEvalCandidates();
|
||||
expect(rows).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('op-layer capture — failure isolation (F1/F2)', () => {
|
||||
test('capture failures do not propagate to op response', async () => {
|
||||
// Disconnect-then-reconnect the engine to simulate an INSERT failure.
|
||||
// We can't easily inject a rejecting engine here since operations.ts
|
||||
// imports the real one, but we can break the table temporarily.
|
||||
// Drop the eval_candidates table, run the op, assert (a) op response
|
||||
// succeeds and (b) a failure row appears in eval_capture_failures.
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.exec('DROP TABLE eval_candidates');
|
||||
const queryOp = operations.find(o => o.name === 'query')!;
|
||||
const ctx = makeCtx();
|
||||
// Op must still succeed and return results.
|
||||
const results = await queryOp.handler(ctx, { query: 'alice' });
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
|
||||
await waitForCapture();
|
||||
// Failure should have landed in the companion table.
|
||||
const failures = await engine.listEvalCaptureFailures();
|
||||
expect(failures.length).toBeGreaterThanOrEqual(1);
|
||||
expect(['db_down', 'other']).toContain(failures[0]!.reason);
|
||||
|
||||
// Restore the table for subsequent tests.
|
||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||
await (engine as any).db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS eval_candidates (
|
||||
id SERIAL PRIMARY KEY,
|
||||
tool_name TEXT NOT NULL CHECK (tool_name IN ('query', 'search')),
|
||||
query TEXT NOT NULL CHECK (length(query) <= 51200),
|
||||
retrieved_slugs TEXT[] NOT NULL DEFAULT '{}',
|
||||
retrieved_chunk_ids INTEGER[] NOT NULL DEFAULT '{}',
|
||||
source_ids TEXT[] NOT NULL DEFAULT '{}',
|
||||
expand_enabled BOOLEAN,
|
||||
detail TEXT,
|
||||
detail_resolved TEXT,
|
||||
vector_enabled BOOLEAN NOT NULL,
|
||||
expansion_applied BOOLEAN NOT NULL,
|
||||
latency_ms INTEGER NOT NULL,
|
||||
remote BOOLEAN NOT NULL,
|
||||
job_id INTEGER,
|
||||
subagent_id INTEGER,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
)
|
||||
`);
|
||||
});
|
||||
});
|
||||
@@ -830,6 +830,72 @@ describe('PR #356 — non-transactional DDL runs via reserved connection', () =>
|
||||
});
|
||||
});
|
||||
|
||||
describe('migration v31 — eval_capture_tables', () => {
|
||||
test('exists with the expected name and is engine-specific (sqlFor)', () => {
|
||||
const v31 = MIGRATIONS.find(m => m.version === 31);
|
||||
expect(v31).toBeDefined();
|
||||
expect(v31?.name).toBe('eval_capture_tables');
|
||||
expect(v31?.sqlFor?.postgres).toBeDefined();
|
||||
expect(v31?.sqlFor?.pglite).toBeDefined();
|
||||
expect(v31?.sql).toBe('');
|
||||
});
|
||||
|
||||
test('creates both eval_candidates and eval_capture_failures on both engines', () => {
|
||||
const v31 = MIGRATIONS.find(m => m.version === 31)!;
|
||||
for (const variant of ['postgres', 'pglite'] as const) {
|
||||
const sql = v31.sqlFor![variant]!;
|
||||
expect(sql).toContain('CREATE TABLE IF NOT EXISTS eval_candidates');
|
||||
expect(sql).toContain('CREATE TABLE IF NOT EXISTS eval_capture_failures');
|
||||
}
|
||||
});
|
||||
|
||||
test('enforces CHECK length(query) <= 51200', () => {
|
||||
const v31 = MIGRATIONS.find(m => m.version === 31)!;
|
||||
for (const variant of ['postgres', 'pglite'] as const) {
|
||||
expect(v31.sqlFor![variant]!).toContain('CHECK (length(query) <= 51200)');
|
||||
}
|
||||
});
|
||||
|
||||
test('enforces tool_name enum + reason enum', () => {
|
||||
const v31 = MIGRATIONS.find(m => m.version === 31)!;
|
||||
for (const variant of ['postgres', 'pglite'] as const) {
|
||||
const sql = v31.sqlFor![variant]!;
|
||||
expect(sql).toContain(`tool_name IN ('query', 'search')`);
|
||||
expect(sql).toContain(`reason IN ('db_down', 'rls_reject', 'check_violation', 'scrubber_exception', 'other')`);
|
||||
}
|
||||
});
|
||||
|
||||
test('creates DESC indexes on both tables', () => {
|
||||
const v31 = MIGRATIONS.find(m => m.version === 31)!;
|
||||
for (const variant of ['postgres', 'pglite'] as const) {
|
||||
const sql = v31.sqlFor![variant]!;
|
||||
expect(sql).toContain('idx_eval_candidates_created_at');
|
||||
expect(sql).toContain('idx_eval_capture_failures_ts');
|
||||
expect(sql).toContain('created_at DESC');
|
||||
expect(sql).toContain('ts DESC');
|
||||
}
|
||||
});
|
||||
|
||||
test('Postgres variant gates RLS on BYPASSRLS and fails loudly', () => {
|
||||
const pgSql = MIGRATIONS.find(m => m.version === 31)!.sqlFor!.postgres!;
|
||||
expect(pgSql).toContain('rolbypassrls');
|
||||
expect(pgSql).toMatch(/IF NOT has_bypass/);
|
||||
expect(pgSql).toMatch(/RAISE EXCEPTION[^;]*BYPASSRLS/);
|
||||
expect(pgSql).toContain('ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY');
|
||||
expect(pgSql).toContain('ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY');
|
||||
});
|
||||
|
||||
test('PGLite variant has no RLS / no BYPASSRLS gate', () => {
|
||||
const pgliteSql = MIGRATIONS.find(m => m.version === 31)!.sqlFor!.pglite!;
|
||||
expect(pgliteSql).not.toContain('rolbypassrls');
|
||||
expect(pgliteSql).not.toContain('ENABLE ROW LEVEL SECURITY');
|
||||
});
|
||||
|
||||
test('LATEST_VERSION caught up to 31', () => {
|
||||
expect(LATEST_VERSION).toBeGreaterThanOrEqual(31);
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// PR #363 regression guards — session timeouts via startup parameters
|
||||
// resolveSessionTimeouts — GBRAIN_*_TIMEOUT env overrides
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
/**
|
||||
* Public exports contract test (v0.21.0 — Lane 2 / R2).
|
||||
*
|
||||
* Reads package.json "exports" at runtime, imports each subpath via the
|
||||
* package name ("gbrain/<subpath>") so it actually exercises the
|
||||
* resolver — then asserts each resolves AND has at least one canary
|
||||
* symbol. Importing from relative paths (e.g. "../src/core/engine.ts")
|
||||
* would bypass the exports map and miss resolver/wiring breakage.
|
||||
*
|
||||
* The canary symbols are concrete values each module re-exports today.
|
||||
* If a refactor renames or removes one, this test fails in CI so the
|
||||
* downstream consumer (gbrain-evals) doesn't silently break first.
|
||||
*
|
||||
* To add a new public export: extend `EXPECTED_EXPORTS` below. To
|
||||
* REMOVE one: bump gbrain's minor (breaking-interface per CLAUDE.md
|
||||
* "Removing any of these is a breaking change going forward").
|
||||
*/
|
||||
|
||||
import { readFileSync } from 'fs';
|
||||
import { resolve } from 'path';
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
|
||||
interface ExpectedExport {
|
||||
/** Subpath key as it appears in package.json exports. */
|
||||
subpath: string;
|
||||
/** At least one named export that MUST exist at runtime. Chosen from the
|
||||
* module's current surface; if it goes away, that's a breaking change. */
|
||||
canary: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Canary symbols pinned to the v0.21.0 contract. Changes to this list
|
||||
* are intentional breaking changes to the public exports surface.
|
||||
*/
|
||||
const EXPECTED_EXPORTS: ExpectedExport[] = [
|
||||
{ subpath: 'gbrain', canary: [] }, // root "." export; no single canary — just require import success
|
||||
{ subpath: 'gbrain/engine', canary: ['clampSearchLimit', 'MAX_SEARCH_LIMIT'] },
|
||||
{ subpath: 'gbrain/types', canary: ['GBrainError'] },
|
||||
{ subpath: 'gbrain/operations', canary: ['operations', 'OperationError'] },
|
||||
{ subpath: 'gbrain/minions', canary: [] }, // barrel module; re-exports many names
|
||||
{ subpath: 'gbrain/engine-factory', canary: [] }, // factory exports a default creator
|
||||
{ subpath: 'gbrain/pglite-engine', canary: ['PGLiteEngine'] },
|
||||
{ subpath: 'gbrain/link-extraction', canary: ['extractEntityRefs', 'extractPageLinks'] },
|
||||
{ subpath: 'gbrain/import-file', canary: ['importFromContent'] },
|
||||
{ subpath: 'gbrain/transcription', canary: [] },
|
||||
{ subpath: 'gbrain/embedding', canary: ['embed'] },
|
||||
{ subpath: 'gbrain/config', canary: ['loadConfig'] },
|
||||
{ subpath: 'gbrain/markdown', canary: ['splitBody', 'parseMarkdown', 'serializeMarkdown'] },
|
||||
{ subpath: 'gbrain/backoff', canary: [] },
|
||||
{ subpath: 'gbrain/search/hybrid', canary: ['hybridSearch', 'rrfFusion'] },
|
||||
{ subpath: 'gbrain/search/expansion', canary: ['expandQuery'] },
|
||||
{ subpath: 'gbrain/extract', canary: [] },
|
||||
];
|
||||
|
||||
function readPackageExports(): Record<string, string> {
|
||||
const pkgPath = resolve(import.meta.dir, '..', 'package.json');
|
||||
const pkg = JSON.parse(readFileSync(pkgPath, 'utf8'));
|
||||
return pkg.exports as Record<string, string>;
|
||||
}
|
||||
|
||||
describe('public exports — package.json exports map', () => {
|
||||
test('has the expected number of subpaths (v0.21.0 locks the surface)', () => {
|
||||
const exports = readPackageExports();
|
||||
const count = Object.keys(exports).length;
|
||||
// Adding new exports: increment this + add to EXPECTED_EXPORTS below.
|
||||
// Removing exports: see CLAUDE.md "Removing any of these is a
|
||||
// breaking change going forward" — bump minor and update this count.
|
||||
expect(count).toBe(17);
|
||||
});
|
||||
|
||||
test('EXPECTED_EXPORTS list matches the exports map exactly (no drift)', () => {
|
||||
const exports = readPackageExports();
|
||||
const exportedSubpaths = Object.keys(exports).map(k => (k === '.' ? 'gbrain' : `gbrain${k.slice(1)}`)).sort();
|
||||
const expectedSubpaths = EXPECTED_EXPORTS.map(e => e.subpath).sort();
|
||||
expect(expectedSubpaths).toEqual(exportedSubpaths);
|
||||
});
|
||||
});
|
||||
|
||||
describe('public exports — every subpath resolves via package name', () => {
|
||||
for (const entry of EXPECTED_EXPORTS) {
|
||||
test(`${entry.subpath} imports without throwing`, async () => {
|
||||
// Package-path import goes through the exports map — bypassing a
|
||||
// broken/removed subpath surfaces here. Importing "../src/..."
|
||||
// would resolve via filesystem and miss the contract.
|
||||
const mod = await import(entry.subpath);
|
||||
expect(mod).toBeDefined();
|
||||
expect(typeof mod).toBe('object');
|
||||
});
|
||||
|
||||
if (entry.canary.length > 0) {
|
||||
test(`${entry.subpath} exports canary symbols: ${entry.canary.join(', ')}`, async () => {
|
||||
const mod = await import(entry.subpath);
|
||||
for (const name of entry.canary) {
|
||||
expect(mod).toHaveProperty(name);
|
||||
// Must be something truthy — value, class, or function. Not
|
||||
// just a TypeScript type (those don't exist at runtime).
|
||||
expect(mod[name]).toBeDefined();
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user