mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
Garry TanandClaude Opus 4.6
33e2e73940
**eval/runner/cat5-provenance.ts** — BrainBench Cat 5 scoring. Samples claims from gbrain brain-export and classifies each against its source material via a dedicated Haiku judge (classify_claim tool with a three-label enum: supported | unsupported | over-generalized). Separate from judge.ts by design: Cat 5 is a single three-way classification per claim, not a weighted rubric. Rather than overload judge.ts with a mode switch, Cat 5 has its own tool definition (CLASSIFY_CLAIM_TOOL) and prompt. The retry-once pattern, $20 cost gate semantics, and structured parsing are mirrored from judge.ts so failures look the same across Cats. Metric: `citation_accuracy` = fraction where predicted label equals gold expected_label. Threshold (informational): >0.90 per design-doc METRICS.md. v1 ships with `enableThreshold: false` so the verdict is always baseline_only — we don't have hand-authored gold claims yet, and codex flagged that threshold gating should wait until the amara-life-v1 corpus + gold file authoring lands in Day 3b. runCat5 uses a bounded-concurrency worker pool (default 4) to respect Haiku rate limits across 100+ claim batches. Evidence pages are looked up by slug from a caller-provided pagesBySlug map — missing pages don't crash, they just pass an empty source list to the judge (correct behavior for genuinely unsupported claims). **Tests (23):** classifyClaim happy/retry/fallback paths with stubbed Haiku, aggregate accuracy math, threshold gating (pass/fail vs baseline_only), runCat5 concurrency + missing-page handling, renderClaimPrompt embeds claim + sources correctly, parseClassification rejects invalid enum values + plain-text responses. Total eval suite now: 250 pass, 0 fail. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>