Files
gbrain/test
Garry TanandClaude Opus 4.7 c45b4329f7 calibration: A/B harness for think + ab-report (T18 / D19 CDX-18)
Structural answer to CDX-18 (anti-bias rewrite may make advice worse).
We don't have to guess whether calibration helps — we measure.

Architecture:
  runAbTrial(input) — calls thinkRunner TWICE on the same question
  (baseline + --with-calibration), surfaces both answers to a
  preferenceResolver, persists the trial to think_ab_results.

  buildAbReport(engine, { days }) — aggregates the table over the last
  N days (default 30). Computes win counts, ties, neither, and a
  with_calibration_win_rate over DECISIVE trials only (excludes
  neither/tie). Flags calibration_net_negative when n >= 20 AND win
  rate < 45%.

  formatAbReport(report, days) — pretty-prints for stdout; emits the
  calibration_net_negative warning block when triggered.

CLI:
  gbrain calibration ab-report [--days N] [--json]
    Reads the table, prints the breakdown. Replaces the v0.36.0.0
    ship-state placeholder in src/commands/calibration.ts.

  gbrain think --ab "<question>"
    Wires into runAbTrial via the dispatch in src/commands/think.ts —
    follow-up commit. This commit lands the harness layer + schema +
    report surface; the --ab flag itself flips on in a one-line wiring
    commit when the runRecall path is ready.

Schema (migration v72 / think_ab_results):
  source_id, wave_version, ran_at, question, baseline_answer,
  with_calibration_answer, preferred (CHECK in {baseline,
  with_calibration, neither, tie}), model_id, notes.

  CHECK constraint enforces preferred enum. Default wave_version
  'v0.36.0.0' stamped so --undo-wave can scrub these too.

  Index on (source_id, ran_at DESC) supports the report's
  "last N days" query.

  schema.sql + pglite-schema.ts both updated for fresh-install parity.
  schema-embedded.ts regenerated via build:schema.

calibration_net_negative threshold (D19):
  Triggers when:
    - decisive_trials (baseline + with_calibration) >= 20
    - with_calibration_win_rate < 0.45 (NOT <= — exact 45% is OK)

  Small-sample guard (n < 20) prevents the warning from firing on
  early data with sampling noise. Confidence-flat threshold (no Wilson
  CI yet) keeps the math simple; v0.37+ adds CI bounds.

Tests: 12 cases in test/think-ab.test.ts.
  runAbTrial (4): both runner calls fire, preferenceResolver receives
    both answers, INSERT row params shape, throws when thinkRunner
    missing.
  buildAbReport (5): zero trials, aggregation, net_negative trigger at
    n>=20 + win<45%, no trigger at n<20 (small-sample guard), no
    trigger at exact 45% boundary.
  formatAbReport (3): zero-state message, decisive-trials breakdown,
    net_negative warning block.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 16:58:56 -07:00
..