mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 11:22:34 +00:00
fix: codex adversarial round — CI aggregate gap + first-landing verification + scoring honesty
(1) test-status never read needs.brainbench.result: a failing gate job left the branch-protection aggregate green — merge-bypassable. Now in the loop. (2) The first-landing path verifies the run against the COMMITTED baseline (an unverified initial merge could seed a doctored baseline for every future PR). (3) One stored fact row may satisfy at most one gold probe (merged/ broad extractions inflated fidelity via unconstrained find()). (4) The slug-granularity limit of source-isolation detection is disclosed in the methodology doc with a pointer to the engine-layer fuzz that covers the content-leak case. Plus 30s timeouts on engine-touching tests (observed 5s default flaking at load avg 20).
This commit is contained in:
@@ -49,7 +49,7 @@ All micro-averaged per (harness × suite) cell; registered in
|
||||
- `write_back_fidelity` = |gold facts that survive the PRODUCTION conversation→memory pipeline and are keyword-findable with correct entity attribution| / |gold facts|. The deterministic mode injects a gold extractor at the pipeline's extractor seam so segmentation, batching, dedup, and provenance stamping execute shipped code with zero LLM calls.
|
||||
- `provenance_accuracy` = |surviving facts with correct {source, source_session, source_markdown_slug}| / |surviving facts|.
|
||||
- `continuity_rate` = |decision probes recalled by the reader| / |probes|, per READER harness. The writer fixture's decisions persist through the production write-back pipeline — which is harness-INDEPENDENT in v1 — so each pair preps once and every harness replays the read-only reader against the same persisted state (an ordered writer×reader sweep would rebuild byte-identical brains for identical scores). A probe succeeds via pointer injection or stored-fact keyword lookup. The per-writer axis activates when harness-specific write paths land.
|
||||
- `source_isolation_violations` = count of injected slugs from a non-active source. **Gates at zero**, every run, regardless of baseline — cross-source leakage is the data-leak invariant.
|
||||
- `source_isolation_violations` = count of injected slugs from a non-active source. **Gates at zero**, every run, regardless of baseline — cross-source leakage is the data-leak invariant. Granularity disclosure: detection is slug-keyed, so it catches injection of slugs seeded ONLY in a foreign source; a same-slug cross-source CONTENT leak would require the engine's source-scoped SQL itself to fail, which the engine-layer source-isolation fuzz (gbrain-evals Cat 22) covers directly.
|
||||
- `avg_injected_tokens` = mean estimated tokens (chars/4) of injected context per replayed turn. Intrusion-budget diagnostic; reported, NOT gated (gating awaits calibration data — filed TODO).
|
||||
- `extraction_recall` / `extraction_precision` — `--llm` runs only: the real extractor's output vs gold keyword probes.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user