(1) test-status never read needs.brainbench.result: a failing gate job left
the branch-protection aggregate green — merge-bypassable. Now in the loop.
(2) The first-landing path verifies the run against the COMMITTED baseline
(an unverified initial merge could seed a doctored baseline for every future
PR). (3) One stored fact row may satisfy at most one gold probe (merged/
broad extractions inflated fidelity via unconstrained find()). (4) The
slug-granularity limit of source-isolation detection is disclosed in the
methodology doc with a pointer to the engine-layer fuzz that covers the
content-leak case. Plus 30s timeouts on engine-touching tests (observed 5s
default flaking at load avg 20).
The catch that mattered (testing + maintainability, independently): --llm
extraction metrics were computed per fixture but never reached any scoreboard
cell — dead matched_any_gold/stored_rows aggregation fields proved the
unfinished wiring. Now aggregated Σ-style into the write-back cell when llm
is on, pinned by a harness-level stubbed-transport test.
Also: RUN-scoped --llm BudgetTracker (a per-invocation cap multiplied by
fixture count — ~$550 worst case — now one tracker, exhaustion aborts the
run loudly); continuity loop collapsed to one prep per pair + read-only
reader per harness (the writer replay was provably observable-effect-free
and orderings rebuilt byte-identical brains — 90 preps → 15, runtime ~12s →
~7s, identical scores; baseline counts updated 24 → 12 honestly);
factKeywordProbe escapes ILIKE metacharacters; validator rejects unpassable
slug-less retrieve-turns; ci-gate fails HARD on a broken ref instead of
silently running ungated; privacy year-scan covers the gold dir; e2e tests
self-sufficient (shared artifacts in beforeAll) with a minimal-env --llm
gate; round4/cellKey single-sourced; explicit return in eval.ts dispatch;
stale comment + type anchor fixes.
run-all once-per-sweep semantics (ONE n/a record across two modes, 12 cells),
the write-back --llm lane via stubbed transport (extraction recall/precision),
file-vs-file --compare, the --llm availability gate (bare-HOME subprocess),
--json stdout completeness, render-brainbench-delta output + usage exit,
factKeywordProbe AND/expired semantics + the continuity miss path,
parseBaseline error paths, and the privacy guard's violation branches
(BRAINBENCH_PRIVACY_DIR env seam added for the negative test).
Coverage audit: 76% → ~94% of traced paths.
Garry Tan