admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-29 19:01:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: codex adversarial round — CI aggregate gap + first-landing verification + scoring honesty

Browse Source 
(1) test-status never read needs.brainbench.result: a failing gate job left
the branch-protection aggregate green — merge-bypassable. Now in the loop.
(2) The first-landing path verifies the run against the COMMITTED baseline
(an unverified initial merge could seed a doctored baseline for every future
PR). (3) One stored fact row may satisfy at most one gold probe (merged/
broad extractions inflated fidelity via unconstrained find()). (4) The
slug-granularity limit of source-isolation detection is disclosed in the
methodology doc with a pointer to the engine-layer fuzz that covers the
content-leak case. Plus 30s timeouts on engine-touching tests (observed 5s
default flaking at load avg 20).
This commit is contained in:
Garry Tan
2026-06-12 13:24:34 -07:00
parent 1077868e36
commit b3fd66bfc6
7 changed files with 46 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/ci-brainbench-gate.sh
+12 -4
View File
@@ -52,8 +52,16 @@ if git show "${MAIN_REF}:${BASELINE_PATH}" > "$MAIN_BASELINE" 2>/dev/null; then
echo "[brainbench-gate] comparing against ${MAIN_REF}:${BASELINE_PATH}"
bun src/cli.ts eval brainbench --compare "$MAIN_BASELINE" --out "$OUT"
else
# First landing: the ref exists but carries no baseline yet. Run without a
# gate so the PR that introduces BrainBench can commit the initial baseline.
echo "[brainbench-gate] no baseline on ${MAIN_REF} yet — running ungated (initial-landing path)"
bun src/cli.ts eval brainbench --out "$OUT"
# First landing: the ref exists but carries no baseline yet. The COMMITTED
# baseline still gets verified against the actual run (codex adversarial
# finding: an unverified first landing could seed a doctored baseline for
# every future PR to compare against). Same-hash + committed==run logic
# inside --compare does the verification.
if [ -f "$BASELINE_PATH" ]; then
echo "[brainbench-gate] no baseline on ${MAIN_REF} yet — verifying the run against the COMMITTED baseline (first-landing path)"
bun src/cli.ts eval brainbench --compare "$BASELINE_PATH" --out "$OUT"
else
echo "[brainbench-gate] no baseline on ${MAIN_REF} and none committed — running ungated (pre-baseline tree)"
bun src/cli.ts eval brainbench --out "$OUT"
fi
fi