Files
gbrain/test
Garry TanandClaude Opus 4.7 fd9a4ae1ce cycle: grade_takes ensemble tiebreaker for borderline verdicts (T5 / E2)
Multi-judge ensemble tiebreaker, additive on top of T4's single-judge
foundation. Reuses gateway.chat as the per-model judge interface; runs
three judges in parallel via Promise.allSettled. Pure aggregation logic
in aggregateEnsemble() — no SQL, no LLM, hermetically testable.

When ensemble fires (T5 trigger band):
  Only when ALL of:
    - opts.useEnsemble === true (default false)
    - opts.ensembleJudges array is non-empty
    - single-model confidence in [0.6, 0.95) (configurable via
      opts.ensembleTriggerBand)
    - single-model verdict !== 'unresolvable'

  Above 0.95 the single judge is already sufficient (T4 path). Below 0.6
  the verdict is clearly review-only — ensemble wouldn't change the
  posture. 'unresolvable' from single-judge means no evidence yet; calling
  three more judges on the same evidence won't manufacture some.

Conservative auto-apply (D12):
  Ensemble verdict auto-applies via engine.resolveTake only when ALL of:
    - autoResolve === true (operator opt-in per D17)
    - ensemble.agreement === 3 (3/3 unanimous)
    - ensemble.minConfidence >= ensembleThreshold (default 0.85)
    - winning verdict !== 'unresolvable'

  Schema-level monotonic-tightening guard for ensembleThreshold lives in
  the takes resolution layer.

Cache identity:
  When ensemble fires, the cache row's judge_model_id becomes
  'ensemble:<modelA>+<modelB>+<modelC>' — a future re-run with different
  ensemble membership doesn't collide with prior verdicts. evidence_signature
  is recomputed because it includes the judge_model_id.

aggregateEnsemble (pure):
  - 3/3 unanimous → agreement=3, minConfidence=min across the three
  - 2/3 majority → agreement=2, minConfidence across the agreeing two
  - 1/1/1 disagreement → tie-break: prefer non-'unresolvable', then
    alphabetical for determinism
  - 'unresolvable' from one model NEVER tips a 2-vote majority toward
    'unresolvable' — by-label tally only counts a model toward its own
    label
  - All three judges failing (allSettled rejected) → verdict='unresolvable'
    with agreement=0; auto-apply path blocked
  - Single judge survives + two fail → agreement=1; the lone verdict wins
    but auto-apply gated by the 3/3 requirement

Tests: 16 cases.
  aggregateEnsemble (6): 3/3, 2/3, 1/1/1, unresolvable-tipping-resistance,
  all-failed, partial-failed-but-survives.
  Phase trigger conditions (5): useEnsemble=false default, useEnsemble=true
  in borderline band, single >= 0.95 skip, single < 0.6 skip, single =
  'unresolvable' skip.
  Phase auto-apply rules (5): 3/3+threshold+autoResolve, 2/3 majority no
  apply, 3/3 below threshold no apply, one ensemble judge throws still
  aggregates from allSettled, empty ensembleJudges falls through to
  single.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 16:12:34 -07:00
..