fix: codex adversarial round — CI aggregate gap + first-landing verification + scoring honesty

(1) test-status never read needs.brainbench.result: a failing gate job left
the branch-protection aggregate green — merge-bypassable. Now in the loop.
(2) The first-landing path verifies the run against the COMMITTED baseline
(an unverified initial merge could seed a doctored baseline for every future
PR). (3) One stored fact row may satisfy at most one gold probe (merged/
broad extractions inflated fidelity via unconstrained find()). (4) The
slug-granularity limit of source-isolation detection is disclosed in the
methodology doc with a pointer to the engine-layer fuzz that covers the
content-leak case. Plus 30s timeouts on engine-touching tests (observed 5s
default flaking at load avg 20).
This commit is contained in:
Garry Tan
2026-06-12 13:24:34 -07:00
parent 1077868e36
commit b3fd66bfc6
7 changed files with 46 additions and 30 deletions
+1 -1
View File
@@ -49,7 +49,7 @@ All micro-averaged per (harness × suite) cell; registered in
- `write_back_fidelity` = |gold facts that survive the PRODUCTION conversation→memory pipeline and are keyword-findable with correct entity attribution| / |gold facts|. The deterministic mode injects a gold extractor at the pipeline's extractor seam so segmentation, batching, dedup, and provenance stamping execute shipped code with zero LLM calls.
- `provenance_accuracy` = |surviving facts with correct {source, source_session, source_markdown_slug}| / |surviving facts|.
- `continuity_rate` = |decision probes recalled by the reader| / |probes|, per READER harness. The writer fixture's decisions persist through the production write-back pipeline — which is harness-INDEPENDENT in v1 — so each pair preps once and every harness replays the read-only reader against the same persisted state (an ordered writer×reader sweep would rebuild byte-identical brains for identical scores). A probe succeeds via pointer injection or stored-fact keyword lookup. The per-writer axis activates when harness-specific write paths land.
- `source_isolation_violations` = count of injected slugs from a non-active source. **Gates at zero**, every run, regardless of baseline — cross-source leakage is the data-leak invariant.
- `source_isolation_violations` = count of injected slugs from a non-active source. **Gates at zero**, every run, regardless of baseline — cross-source leakage is the data-leak invariant. Granularity disclosure: detection is slug-keyed, so it catches injection of slugs seeded ONLY in a foreign source; a same-slug cross-source CONTENT leak would require the engine's source-scoped SQL itself to fail, which the engine-layer source-isolation fuzz (gbrain-evals Cat 22) covers directly.
- `avg_injected_tokens` = mean estimated tokens (chars/4) of injected context per replayed turn. Intrusion-budget diagnostic; reported, NOT gated (gating awaits calibration data — filed TODO).
- `extraction_recall` / `extraction_precision``--llm` runs only: the real extractor's output vs gold keyword probes.