Files
gbrain/test
Paolo BelcastroandClaude Fable 5 e6d392ad08 fix(eval,probe): nightly probe reaches real verdicts — SDK model strip, live default slots, gold-answer QA judging
Three more layers of the same never-exercised enable path, each found by
running the probe end-to-end after the previous fix:

1. eval-longmemeval passed resolveModel's RECIPE ids (anthropic:claude-…)
   straight to the raw Anthropic SDK — the API 404s on the prefixed string,
   so every answer/extractor call failed and batches were all-upstream_error.
   (think avoids this by routing through the gateway; this eval's raw-SDK
   client is deliberate — hermetic — so strip via splitProviderModelId.)

2. DEFAULT_SLOTS pinned openai:gpt-4o after the OpenAI recipe dropped it, so
   judge slot A errored 'not listed for OpenAI chat' on every install and
   the panel could not reach its 2-model quorum without a Google key —
   verdicts were permanently inconclusive. Bumped to a listed model and
   added a slots↔recipes consistency test so defaults can't drift again.

3. The judge panel scored terse LongMemEval facts ('in widget-co') on a
   rich-response rubric (DEPTH/SOURCING ≥7) with no gold answer and no
   haystack — guaranteed FAIL even when healthy. eval-longmemeval now emits
   the dataset's gold answer in each row; batch mode folds it into the
   judge task when present; the probe adapter passes QA-shaped dimensions
   (CORRECTNESS + DIRECTNESS; deliberately no grounding dimension — judges
   who never see the haystack read correct extra detail as invented).

Verified end-to-end on a real brain: 9/10 fixture questions pass; the one
failure is a genuine multi-session aggregation miss (2 of 3 evidence
sessions retrieved), i.e. the probe now surfaces real signal instead of
plumbing artifacts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FQgByq4aqQq2PP8UHCdnfk
2026-07-06 14:11:14 +02:00
..