Claude adversarial pass (post-red-team): (1) the same-hash drift defense had
a foreign-fixtures_hash carve-out a doctored committed baseline could slip
through — removed; ANY committed-vs-main divergence must now be
receipts-backed by the actual run (foreign-corpus callers opt out via
--committed-baseline); (2) same corpus must yield identical gold_total —
free invariant now breached on any scorer/loader drift; (3) hybrid
continuity writers can no longer double-count into the write-back cell;
(4) retrieval gold on assistant turns (never replayed) is now a validation
error. Hygiene: gate script cleans its defaulted temp artifact, CI uses
runner.temp instead of a fixed /tmp path, codex preamble failures are loud,
accepted residuals (justification is human-judged, count-preserving
dilution, no auto-ratchet) documented in BRAINBENCH.md.
The catch that mattered (testing + maintainability, independently): --llm
extraction metrics were computed per fixture but never reached any scoreboard
cell — dead matched_any_gold/stored_rows aggregation fields proved the
unfinished wiring. Now aggregated Σ-style into the write-back cell when llm
is on, pinned by a harness-level stubbed-transport test.
Also: RUN-scoped --llm BudgetTracker (a per-invocation cap multiplied by
fixture count — ~$550 worst case — now one tracker, exhaustion aborts the
run loudly); continuity loop collapsed to one prep per pair + read-only
reader per harness (the writer replay was provably observable-effect-free
and orderings rebuilt byte-identical brains — 90 preps → 15, runtime ~12s →
~7s, identical scores; baseline counts updated 24 → 12 honestly);
factKeywordProbe escapes ILIKE metacharacters; validator rejects unpassable
slug-less retrieve-turns; ci-gate fails HARD on a broken ref instead of
silently running ungated; privacy year-scan covers the gold dir; e2e tests
self-sufficient (shared artifacts in beforeAll) with a minimal-env --llm
gate; round4/cellKey single-sourced; explicit return in eval.ts dispatch;
stale comment + type anchor fixes.
run-all once-per-sweep semantics (ONE n/a record across two modes, 12 cells),
the write-back --llm lane via stubbed transport (extraction recall/precision),
file-vs-file --compare, the --llm availability gate (bare-HOME subprocess),
--json stdout completeness, render-brainbench-delta output + usage exit,
factKeywordProbe AND/expired semantics + the continuity miss path,
parseBaseline error paths, and the privacy guard's violation branches
(BRAINBENCH_PRIVACY_DIR env seam added for the negative test).
Coverage audit: 76% → ~94% of traced paths.
No-DB cli.ts route (brings its own PGLite), --out as the canonical CI
artifact, explicit grace-tick process.exit(verdict) (PGLite exitCode-hijack
+ Bun stdout-discard guards), anti-vacuous-pass + seed-failure exit-2 paths,
_meta.metric_glossary block, --update-baseline canonical writer,
--compare with main-baseline governance. eval run-all's brainbench stub is
now wired in-process: one record per sweep, schema_version 3, mode 'n/a'.
Subprocess e2e pins literal exit codes 0/1/2 end to end.
Garry Tan