(1) test-status never read needs.brainbench.result: a failing gate job left
the branch-protection aggregate green — merge-bypassable. Now in the loop.
(2) The first-landing path verifies the run against the COMMITTED baseline
(an unverified initial merge could seed a doctored baseline for every future
PR). (3) One stored fact row may satisfy at most one gold probe (merged/
broad extractions inflated fidelity via unconstrained find()). (4) The
slug-granularity limit of source-isolation detection is disclosed in the
methodology doc with a pointer to the engine-layer fuzz that covers the
content-leak case. Plus 30s timeouts on engine-touching tests (observed 5s
default flaking at load avg 20).
Claude adversarial pass (post-red-team): (1) the same-hash drift defense had
a foreign-fixtures_hash carve-out a doctored committed baseline could slip
through — removed; ANY committed-vs-main divergence must now be
receipts-backed by the actual run (foreign-corpus callers opt out via
--committed-baseline); (2) same corpus must yield identical gold_total —
free invariant now breached on any scorer/loader drift; (3) hybrid
continuity writers can no longer double-count into the write-back cell;
(4) retrieval gold on assistant turns (never replayed) is now a validation
error. Hygiene: gate script cleans its defaulted temp artifact, CI uses
runner.temp instead of a fixed /tmp path, codex preamble failures are loud,
accepted residuals (justification is human-judged, count-preserving
dilution, no auto-ratchet) documented in BRAINBENCH.md.
Five red-team criticals, all closed and test-pinned:
(1) two-PR gate poisoning: a same-hash committed-baseline edit must now be
receipts-backed (byte-match THIS run) or the gate is inconclusive — and a
receipts-backed same-hash regression still needs a justification;
(2) corpus hollowing: bless-mode gold_total shrink (deleting failing
fixtures / flipping holdout) now requires a justification like any adverse
move; (3) continuity cross-source detection was structurally blind to
writer-seeded slugs (the zero-gate was vacuous) — writer+reader slug→source
maps now merge; (4) --update-baseline writes BEFORE compare (the one-shot
bless flow no longer trains users to ignore exit 2) and REFUSES to write
when seed failures invalidate the run; (5) 15 continuity writers' write-back
gold was computed and dropped — now accumulated, so the write-back cell is
flag-independent (gold 46 → 61). Plus: baselines bind their run config
(holdout/llm/harness/suite — fixtures_hash covers files only), the CI gate
script refuses a working-tree baseline deletion, and renderInlineTime no
longer straddles UTC midnight for non-Z offsets.
The catch that mattered (testing + maintainability, independently): --llm
extraction metrics were computed per fixture but never reached any scoreboard
cell — dead matched_any_gold/stored_rows aggregation fields proved the
unfinished wiring. Now aggregated Σ-style into the write-back cell when llm
is on, pinned by a harness-level stubbed-transport test.
Also: RUN-scoped --llm BudgetTracker (a per-invocation cap multiplied by
fixture count — ~$550 worst case — now one tracker, exhaustion aborts the
run loudly); continuity loop collapsed to one prep per pair + read-only
reader per harness (the writer replay was provably observable-effect-free
and orderings rebuilt byte-identical brains — 90 preps → 15, runtime ~12s →
~7s, identical scores; baseline counts updated 24 → 12 honestly);
factKeywordProbe escapes ILIKE metacharacters; validator rejects unpassable
slug-less retrieve-turns; ci-gate fails HARD on a broken ref instead of
silently running ungated; privacy year-scan covers the gold dir; e2e tests
self-sufficient (shared artifacts in beforeAll) with a minimal-env --llm
gate; round4/cellKey single-sourced; explicit return in eval.ts dispatch;
stale comment + type anchor fixes.
git show origin/master:evals/brainbench/baselines/main.json is the compare
target, so a PR cannot rewrite the thing it's graded by (plan decision 4).
Hermetic, zero keys, ~15s. ci-brainbench-gate.sh is local parity (with an
ungated first-landing path); render-brainbench-delta.ts emits the step-summary
/ PR-body delta block from the --out artifact.
Garry Tan