mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 03:12:32 +00:00
Garry TanandClaude Opus 4.8
073d311327
Surfaced by the SkillOpt real-LLM eval (Track B). Two coupled bugs that made
a budget-capped Haiku run report a vacuous "0/N" measurement in ~2ms with
zero LLM calls — indistinguishable from a real deficient-skill score:
1. Claude Haiku 4.5's canonical dateless id (`claude-haiku-4-5`) was missing
from anthropic-pricing.ts (only the dated `-20251001` was present). With
`--max-cost` set, BudgetTracker.reserve() threw no_pricing on the FIRST
chat() of every rollout. Added the dateless entry (sonnet already had its
dateless form).
2. runValidationGate swallowed that BUDGET_EXHAUSTED error — runWithLimit
settled it as {ok:false}, which the gate turned into median:0. A pricing/cap
crash became a fake score. The gate now scans settled results for
isMustAbortError() and re-throws so the caller aborts loudly; ordinary
(non-abort) rollout errors still fail-open to 0 (judge-hiccup posture kept).
Pinned by test/skillopt/validate-gate-abort.test.ts (3 cases). Folds into the
open v0.42.9.0 PR (#1759).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>