mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 19:56:25 +00:00
* feat(v0.22.0): eval_candidates + eval_capture_failures schema (Lane 1A)
R1 substrate for BrainBench-Real, replayed onto master after Cathedral II
landed. Migration v30 (slotted after master's v25-v29 Cathedral II wave)
creates two tables:
eval_candidates: per-call capture of MCP/CLI/subagent query+search
traffic. Column set lets gbrain-evals replay with full fidelity —
source_ids from v0.18 multi-source, vector_enabled/detail_resolved/
expansion_applied so replay knows what hybridSearch actually did,
remote + job_id + subagent_id so rows are traceable to their origin.
query is CHECK-capped at 50KB; PII scrubber (Lane 1B) runs before insert.
eval_capture_failures: cross-process audit trail. In-process counters
don't work because `gbrain doctor` runs in a separate process from
the MCP server. Persistent rows let doctor query capture health via
COUNT(*) GROUP BY reason over the last 24h.
Both tables get RLS on Postgres gated on BYPASSRLS (matches v24/v29
posture). PGLite ignores RLS; sqlFor split carries only DDL.
5 new BrainEngine methods (breaking-interface addition, drives v0.22.0
minor bump): logEvalCandidate, listEvalCandidates,
deleteEvalCandidatesBefore, logEvalCaptureFailure, listEvalCaptureFailures.
listEvalCandidates uses ORDER BY created_at DESC, id DESC so
`gbrain eval export` is deterministic across same-millisecond inserts.
Also adds HybridSearchMeta type for the side-channel callback used by
Lane 1C's op-layer capture (no change to hybridSearch return shape —
that respects Cathedral II's existing SearchResult[] contract).
Tests: 14 PGLite round-trip cases + 8 v30 structural assertions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.22.0): PII scrubber + op-layer capture module (Lane 1B)
Replayed onto master post-Cathedral II. Same semantics as the original
v0.21.0 work — only adjusted to import HybridSearchMeta from types.ts
(canonical home) instead of redeclaring it locally.
src/core/eval-capture-scrub.ts — pure-function regex scrubber with 6
pattern families: emails, phones (US + E.164), SSN (year-aware),
Luhn-verified credit cards, JWT-shaped tokens, bearer tokens. Zero
deps. Adversarial-input safe.
src/core/eval-capture.ts — op-layer hook helper:
- buildEvalCandidateInput(ctx, {scrub_pii}) — pure row builder
- classifyCaptureFailure(err) — Postgres SQLSTATE → reason tag
- captureEvalCandidate(engine, ctx, opts) — best-effort, never throws
- isEvalCaptureEnabled / isEvalScrubEnabled — file-plane config checks
GBrainConfig gains `eval?: {capture?, scrub_pii?}`. Both default ON.
File-plane only — `gbrain config set` writes the DB plane, doesn't
control capture.
Tests: 17 scrubber + 21 capture-module cases. Zero regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.22.0): hybridSearch onMeta callback + op-layer capture (Lane 1C)
Replayed onto master. Adapted from the original v0.21.0 work to keep
Cathedral II's contract intact: hybridSearch's return stays
`Promise<SearchResult[]>` (unchanged), and meta surfaces via an optional
`onMeta?: (meta: HybridSearchMeta) => void` callback in HybridSearchOpts.
Cathedral II callers leave onMeta undefined and pay no cost. The
op-layer capture wrapper passes a closure that threads meta into the
captured row so gbrain-evals can distinguish:
- "with OPENAI_API_KEY" vs "keyword-only fallback" (vector_enabled)
- "expansion fired" vs "expansion requested + silently fell back" (expansion_applied)
- what hybridSearch actually used after auto-detect (detail_resolved)
Op-layer capture wired into both `query` and `search` op handlers in
src/core/operations.ts. Single hook site catches MCP dispatch + CLI +
subagent tool-bridge from the same place. Fire-and-forget, never throws,
respects ctx.config.eval.capture off-switch.
Tests:
- test/hybrid-meta.test.ts (8 cases) — onMeta accuracy across the 4
return paths in hybridSearch + verification that omitting onMeta
leaves Cathedral II callers unchanged.
- test/mcp-eval-capture.test.ts (10 cases) — query/search ops capture
correctly with MCP/CLI/subagent contexts, scrub on/off, capture=false
off-switch, non-captured ops (list_pages, get_page), F1 failure
isolation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.22.0): gbrain eval export/prune + doctor eval_capture check (Lane 1D)
Replayed onto master. Same semantics as the original v0.21.0 work.
CLI:
gbrain eval export [--since DUR] [--limit N] [--tool query|search]
NDJSON to stdout, every row prefixed with "schema_version":1 per
docs/eval-capture.md contract. EPIPE-safe streaming, stderr
heartbeats, deterministic ordering (created_at DESC, id DESC).
gbrain eval prune --older-than DUR [--dry-run]
Explicit retention cleanup. Requires --older-than (never deletes
without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
Legacy bare `gbrain eval --qrels …` still works via sub-subcommand
fall-through.
gbrain doctor gains an eval_capture check between markdown_body_completeness
and queue_health: reads eval_capture_failures for the last 24h, groups by
reason, warns when non-zero. Pre-v30 brains get "Skipped (table
unavailable)" — non-fatal.
docs/eval-capture.md ships the stable NDJSON schema reference for
gbrain-evals consumers.
Tests: 9 export cases + 5 prune cases. Doctor check covered by
existing doctor tests on master.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.22.0): public-exports contract test + CI count guard (Lane 2 / R2)
Master locks 17 public subpath exports as gbrain's stable third-party
contract. Zero enforcement existed. This PR locks the surface in two
layers:
1. test/public-exports.test.ts — runtime contract test.
Reads package.json "exports" at startup. For each subpath, imports
via the package name ("gbrain/engine"), NOT the relative filesystem
path — that's the difference between exercising the actual resolver
and bypassing it. Every subpath gets a canary symbol pinned (e.g.
gbrain/search/hybrid must export hybridSearch + rrfFusion) so a
refactor that renames or removes one fails CI before downstream
consumers (gbrain-evals) silently break.
2. scripts/check-exports-count.sh — CI structural guard.
Wired into `bun test` after check-jsonb-pattern.sh +
check-progress-to-stdout.sh + check-wasm-embedded.sh per master's
precedent. EXPECTED_COUNT=17 baseline — shrinks fail loudly,
growth also fails so the new canary must be pinned in the runtime
test deliberately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs+e2e(v0.22.0): VERSION/CHANGELOG/CLAUDE/README + Postgres E2E (Lane 3)
Bump VERSION + package.json to 0.22.0 (next free slot after master's
v0.21.0 Code Cathedral II minor).
CHANGELOG.md v0.22.0 entry follows the Garry voice template:
- Bold 2-line headline
- Lead paragraph contextualizing v0.20 + v0.21 + v0.22 progression
- Numbers-that-matter table comparing v0.21.0 → v0.22.0
- "What this means for you" sectioned by audience
- "## To take advantage of v0.22.0" operator runbook
- Itemized changes
CLAUDE.md updates:
- Key files: 8 new module entries (eval-capture*, eval-export,
eval-prune, docs/eval-capture.md, public-exports test).
hybrid.ts entry rewritten to reflect the additive `onMeta` callback
(return shape unchanged).
- Key commands: new v0.22.0 section for `gbrain eval export`,
`gbrain eval prune`, and the doctor `eval_capture` check, with the
file-plane vs DB-plane config gotcha called out.
README.md: one-paragraph pointer after the BrainBench blurb so anyone
reading the landing page sees the new session-capture feature.
llms.txt + llms-full.txt regenerated to pick up the doc additions.
test/e2e/eval-capture.test.ts (Postgres-only E1 spec):
- CHECK violation surfaces as Postgres SQLSTATE 23514 on oversize input
- RLS is actually enabled on both eval_candidates + eval_capture_failures
- 50 concurrent logEvalCandidate calls — no deadlock, all distinct IDs
Skips gracefully when DATABASE_URL is unset.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(todos): P0 — PGLite test-runner concurrency flake
Pre-existing on master, surfaces ~27 false failures when bun test runs all
174 files together. Each failing file passes in isolation. Tracked for a
dedicated investigation branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v0.22.0): adversarial review post-fixes (doctor RLS, onMeta safety)
Two surgical fixes from /ship adversarial review, plus 6 follow-ups TODO'd
into v0.22.1:
- doctor.ts: distinguish pre-v30 missing-table (42P01, ok skip) from
RLS-denied SELECT (42501, warn) and other DB errors (warn). The check
exists specifically to surface capture-failure misconfigs cross-process,
so silently reporting "ok / skipped" on the most diagnostic class
defeated the purpose.
- hybrid.ts: wrap onMeta invocation in try/catch via small emitMeta
helper. The callback is part of the public gbrain/search/hybrid
contract; a throwing user-supplied closure must never break the search
hot path.
- TODOS.md: 6 P1 follow-ups (eval prune real COUNT, scrubber CC false
positives, dead 'scrubber_exception' enum value, id-cursor for
cross-window dedup, public-export canary pinning, EXPECTED_COUNT dedup).
- TODOS.md: P0 entry for the pre-existing PGLite test-runner concurrency
flake (~27 false failures in full bun test on master).
- CHANGELOG.md: 2 bullets noting the doctor + onMeta hardening.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(version): bump v0.22.0 → v0.25.0 (queue-aware version pick)
Master is at v0.21.0. Open PRs claim v0.21.1 (#432) and v0.24.0 (#387).
v0.25 is the first uncontested slot, so this branch claims it. Pure
rename across VERSION, package.json, CHANGELOG header, and every "v0.22.0"
reference in CLAUDE.md / README.md / TODOS.md / docs/eval-capture.md /
src/ / test/ files. CHANGELOG date bumped to 2026-04-26.
llms.txt + llms-full.txt regenerated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.25.0): gbrain eval replay + contributor doc + CONTRIBUTING link
Closes the gap between "session capture works" (this PR's core) and
"contributors actually use it before merging." Three artifacts:
- src/commands/eval-replay.ts (~340 LOC) — reads NDJSON from `gbrain eval
export`, re-runs each captured query/search against the current brain,
computes set-Jaccard@k, top-1 stability, and latency delta. Stable JSON
shape (schema_version:1) for CI gating; human mode prints a regression
table sorted worst-first. Pure Bun, zero new deps. Stub-engine tests
cover Jaccard math, NDJSON parser (including v2 forward-compat
rejection + line-numbered errors), --limit, --verbose, --json, and
graceful per-row error handling. 16/16 passing.
- docs/eval-bench.md (~80 lines) — contributor guide. The 4-command loop
(export → change → replay → diff), metric definitions with healthy
ranges (Jaccard ≥0.85, top-1 ≥85%, latency Δ within ±50ms), trigger
paths, CI integration snippet, hand-crafted NDJSON corpus path for
fresh installs, and the off-switch. Pairs with the existing
docs/eval-capture.md which is the consumer-facing wire format.
- CONTRIBUTING.md gains a "Running real-world eval benchmarks (touching
retrieval code)" section with the trigger paths and a link to
docs/eval-bench.md. Reviewers now have a one-line ask: "did you run
replay?"
CLAUDE.md key files updated. CHANGELOG bullets added. llms.txt
regenerated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.25.0): CONTRIBUTOR_MODE flag — capture off by default for users
Eval capture was on for everyone in the v0.25.0 draft. Privacy footgun:
end users had retrieval traffic accumulate in their brain DB without
asking, even with PII scrubbing. Flips to off by default + explicit
opt-in for contributors who actually use the replay loop.
Resolution order in isEvalCaptureEnabled():
1. config.eval.capture === true → on
2. config.eval.capture === false → off
3. process.env.GBRAIN_CONTRIBUTOR_MODE === '1' → on
4. otherwise → off
The env var is the contributor-facing toggle (one line in .zshrc, no
JSON edit). Explicit config wins both directions for users who want to
override per-brain.
PII scrubbing gate stays independent — default true regardless of
CONTRIBUTOR_MODE — so any brain that does capture still scrubs.
Tests rewritten: env var hygiene per-test (origMode preserved + restored
in finally). 9/9 pass; total v0.25.0 suite is 198/198.
Docs:
- README.md gains a Contributing-section pointer to the env var.
- CONTRIBUTING.md gains a "CONTRIBUTOR_MODE — turn on the dev loop"
section with verification commands and resolution-order table.
- docs/eval-bench.md leads with the prerequisite (must set the env var
for the rest of the doc to be useful).
- docs/eval-capture.md "Config" section split into Path A (env var) +
Path B (config) with explicit resolution-order rules.
- CHANGELOG v0.25.0 entry corrected ("on by default" was wrong) plus a
new top itemized bullet calling out the gate change.
- CLAUDE.md eval-capture entry annotated with the new gate logic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: post-ship documentation pass for v0.25.0
Cross-references every doc against the final state of the branch
(CONTRIBUTOR_MODE flag, eval replay tool, off-by-default capture):
- README.md: top callout rewritten — was implying capture-on-by-default
contradicting the gate landed in 7a80ce25. Now leads with
"contributor opt-in" and links docs/eval-bench.md alongside
docs/eval-capture.md.
- AGENTS.md: new "Eval retrieval changes" task entry with the
CONTRIBUTOR_MODE+replay one-liner so non-Claude agents (Codex, Cursor,
Aider) have the same path.
- CLAUDE.md: "Key commands added in v0.25.0" gains the replay command and
a CONTRIBUTOR_MODE bullet covering the resolution order.
- CHANGELOG.md: headline rewritten to match the actual feature ("benchmark
retrieval changes against real captured queries before merging" — was
"every real query is captured"). Stale "v0.22 ships the substrate"
→ v0.25. Test count corrected 82 → 144 (added 16 replay + 9
CONTRIBUTOR_MODE + 8 v31-shape tests since the original count). Two
metric rows added to the numbers table: default-off posture, in-tree
replay tooling. "To take advantage" block split into user vs
contributor branches with shell-rc instructions.
- TODOS.md: v0.22.1 follow-up reference corrected to v0.25.1.
llms.txt + llms-full.txt regenerated. Typecheck clean. 198/198 v0.25.0
tests still green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
135 lines
4.1 KiB
TypeScript
135 lines
4.1 KiB
TypeScript
/**
|
|
* gbrain eval prune — retention cleanup (v0.21.0).
|
|
*
|
|
* Verifies:
|
|
* - --older-than parses duration strings (30d, 1h, 90m, 3600s)
|
|
* - deletes only rows older than the cutoff
|
|
* - --dry-run reports count without deleting
|
|
* - missing --older-than exits with error
|
|
*/
|
|
|
|
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
|
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
|
import { runEvalPrune } from '../src/commands/eval-prune.ts';
|
|
import type { EvalCandidateInput } from '../src/core/types.ts';
|
|
|
|
let engine: PGLiteEngine;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({});
|
|
await engine.initSchema();
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
await (engine as any).db.exec('DELETE FROM eval_candidates');
|
|
});
|
|
|
|
function baseInput(): EvalCandidateInput {
|
|
return {
|
|
tool_name: 'query',
|
|
query: 'q',
|
|
retrieved_slugs: [],
|
|
retrieved_chunk_ids: [],
|
|
source_ids: [],
|
|
expand_enabled: true,
|
|
detail: null,
|
|
detail_resolved: null,
|
|
vector_enabled: false,
|
|
expansion_applied: false,
|
|
latency_ms: 50,
|
|
remote: false,
|
|
job_id: null,
|
|
subagent_id: null,
|
|
};
|
|
}
|
|
|
|
/** Insert a row with an explicit created_at timestamp (for "aged" data). */
|
|
async function insertAged(daysAgo: number): Promise<void> {
|
|
const when = new Date(Date.now() - daysAgo * 86_400_000).toISOString();
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
await (engine as any).db.query(
|
|
`INSERT INTO eval_candidates (
|
|
tool_name, query, retrieved_slugs, retrieved_chunk_ids, source_ids,
|
|
expand_enabled, detail, detail_resolved, vector_enabled, expansion_applied,
|
|
latency_ms, remote, job_id, subagent_id, created_at
|
|
) VALUES ('query', 'q', '{}', '{}', '{}',
|
|
true, null, null, false, false, 50, false, null, null, $1)`,
|
|
[when],
|
|
);
|
|
}
|
|
|
|
async function withSilencedStdout<T>(fn: () => Promise<T>): Promise<T> {
|
|
const orig = console.log;
|
|
console.log = () => {};
|
|
try {
|
|
return await fn();
|
|
} finally {
|
|
console.log = orig;
|
|
}
|
|
}
|
|
|
|
describe('gbrain eval prune', () => {
|
|
test('--older-than 30d deletes rows older than 30 days, keeps recent', async () => {
|
|
await insertAged(60); // old
|
|
await insertAged(45); // old
|
|
await insertAged(10); // kept
|
|
await engine.logEvalCandidate(baseInput()); // just now, kept
|
|
|
|
expect(await engine.listEvalCandidates()).toHaveLength(4);
|
|
await withSilencedStdout(() => runEvalPrune(engine, ['--older-than', '30d']));
|
|
|
|
const remaining = await engine.listEvalCandidates();
|
|
expect(remaining).toHaveLength(2);
|
|
});
|
|
|
|
test('--dry-run does not delete anything', async () => {
|
|
await insertAged(60);
|
|
await insertAged(45);
|
|
await withSilencedStdout(() => runEvalPrune(engine, ['--older-than', '30d', '--dry-run']));
|
|
expect(await engine.listEvalCandidates()).toHaveLength(2);
|
|
});
|
|
|
|
test('accepts multiple duration units', async () => {
|
|
// Each of these runs should succeed without error.
|
|
for (const dur of ['1h', '90m', '3600s', '7d']) {
|
|
await withSilencedStdout(() => runEvalPrune(engine, ['--older-than', dur]));
|
|
}
|
|
});
|
|
|
|
test('missing --older-than exits 1', async () => {
|
|
const originalExit = process.exit;
|
|
let exitCode: number | undefined;
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
(process.exit as any) = (code?: number) => {
|
|
exitCode = code;
|
|
throw new Error('exit');
|
|
};
|
|
try {
|
|
await runEvalPrune(engine, []);
|
|
} catch { /* expected */ }
|
|
process.exit = originalExit;
|
|
expect(exitCode).toBe(1);
|
|
});
|
|
|
|
test('invalid --older-than value exits 1', async () => {
|
|
const originalExit = process.exit;
|
|
let exitCode: number | undefined;
|
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
(process.exit as any) = (code?: number) => {
|
|
exitCode = code;
|
|
throw new Error('exit');
|
|
};
|
|
try {
|
|
await runEvalPrune(engine, ['--older-than', 'foo']);
|
|
} catch { /* expected */ }
|
|
process.exit = originalExit;
|
|
expect(exitCode).toBe(1);
|
|
});
|
|
});
|