Files
gbrain/test
Garry TanandClaude Opus 4.7 b3e4fa5a07 cycle: grade_takes phase + take_grade_cache verdict pipeline (T4)
Walks unresolved takes that are old enough to have outcome data, retrieves
evidence from the brain, asks a judge model to verdict each one. Writes
verdicts to take_grade_cache. Optionally — only when operator has flipped
the opt-in config flag — auto-applies high-confidence verdicts to the
canonical takes table via engine.resolveTake.

Auto-resolve posture (D17 — DISABLED by default):
  On a fresh install, grade_takes runs and writes verdicts to the cache,
  but applied=false on every row. Operator reviews the queue, then flips
  `cycle.grade_takes.auto_resolve.enabled: true` once trust is earned.
  Mirrors the propose_takes review-queue posture: queue exists, mutation
  requires explicit opt-in.

Conservative threshold (D12):
  When auto_resolve.enabled is true, a verdict auto-applies only when
  confidence >= 0.95 (single-judge path). T5 ensemble path lands next,
  tightening this further with 3/3 unanimous requirement.

  'unresolvable' verdict NEVER auto-applies even at confidence=1.0 —
  there's no canonical column for "we tried and there's no evidence yet."

Evidence retrieval status (v0.36.0.0 ship state):
  The default evidence retriever returns an "evidence-retrieval not yet
  wired" placeholder. Most verdicts produced by the stub-judge against
  the stub-evidence will be 'unresolvable'. Real retrieval (hybrid search
  over pages newer than the take's since_date, optionally augmented by a
  gateway web-search recipe in v0.37+) lands as a follow-up. Documented
  limitation per CDX-8 + D17 — the phase ships now so the wiring is real
  and the cache table accumulates verdicts even if early ones are
  conservative.

Cache key:
  Composite primary key on take_grade_cache is
  (take_id, prompt_version, judge_model_id, evidence_signature). Prompt
  edits OR evidence changes OR judge swap cleanly invalidate prior
  verdicts. Mirrors the v0.32.6 eval_contradictions_cache pattern.

  evidence_signature = SHA-256 of (judge_model_id + '|' + evidence_text)
  so identical evidence under a different judge does NOT collide.

Architecture:
  GradeTakesPhase extends BaseCyclePhase. Inherits source-scope threading,
  budget metering (cycle.grade_takes.budget_usd, default $3/cycle), error
  envelope. Test seam: opts.judge + opts.evidenceRetriever injection so
  the phase runs hermetically.

  parseJudgeOutput defends against fence-wrapping, leading prose,
  out-of-range confidence (clamps to [0,1]), invalid verdict labels,
  oversized reasoning (truncated at 400 chars). Returns null on
  unrecoverable parse — caller treats null as "judge_output_parse_failed
  / unresolvable at confidence 0.0" so the row still lands in cache with
  the parse failure surfaced via warnings.

  takeIsOldEnough gates on since_date (default 6 months). Tolerates
  YYYY-MM-DD and YYYY-MM formats. Returns false on null/unparseable
  since_date so takes without dates never get graded (we'd be
  hallucinating temporal context).

Tests: 23 cases covering parseJudgeOutput (7 cases), evidenceSignature
(3), takeIsOldEnough (5), and 8 phase integration scenarios — happy path,
D17 auto-resolve-off default, D12 above-threshold auto-apply, below-
threshold cache-only, unresolvable-NEVER-applies, cache hit, too-recent
gate, judge-throw warning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 16:08:20 -07:00
..