Garry Tan and Claude Opus 4.7
c45b4329f7
calibration: A/B harness for think + ab-report (T18 / D19 CDX-18)
...
Structural answer to CDX-18 (anti-bias rewrite may make advice worse).
We don't have to guess whether calibration helps — we measure.
Architecture:
runAbTrial(input) — calls thinkRunner TWICE on the same question
(baseline + --with-calibration), surfaces both answers to a
preferenceResolver, persists the trial to think_ab_results.
buildAbReport(engine, { days }) — aggregates the table over the last
N days (default 30). Computes win counts, ties, neither, and a
with_calibration_win_rate over DECISIVE trials only (excludes
neither/tie). Flags calibration_net_negative when n >= 20 AND win
rate < 45%.
formatAbReport(report, days) — pretty-prints for stdout; emits the
calibration_net_negative warning block when triggered.
CLI:
gbrain calibration ab-report [--days N] [--json]
Reads the table, prints the breakdown. Replaces the v0.36.0.0
ship-state placeholder in src/commands/calibration.ts.
gbrain think --ab "<question>"
Wires into runAbTrial via the dispatch in src/commands/think.ts —
follow-up commit. This commit lands the harness layer + schema +
report surface; the --ab flag itself flips on in a one-line wiring
commit when the runRecall path is ready.
Schema (migration v72 / think_ab_results):
source_id, wave_version, ran_at, question, baseline_answer,
with_calibration_answer, preferred (CHECK in {baseline,
with_calibration, neither, tie}), model_id, notes.
CHECK constraint enforces preferred enum. Default wave_version
'v0.36.0.0' stamped so --undo-wave can scrub these too.
Index on (source_id, ran_at DESC) supports the report's
"last N days" query.
schema.sql + pglite-schema.ts both updated for fresh-install parity.
schema-embedded.ts regenerated via build:schema.
calibration_net_negative threshold (D19):
Triggers when:
- decisive_trials (baseline + with_calibration) >= 20
- with_calibration_win_rate < 0.45 (NOT <= — exact 45% is OK)
Small-sample guard (n < 20) prevents the warning from firing on
early data with sampling noise. Confidence-flat threshold (no Wilson
CI yet) keeps the math simple; v0.37+ adds CI bounds.
Tests: 12 cases in test/think-ab.test.ts.
runAbTrial (4): both runner calls fire, preferenceResolver receives
both answers, INSERT row params shape, throws when thinkRunner
missing.
buildAbReport (5): zero trials, aggregation, net_negative trigger at
n>=20 + win<45%, no trigger at n<20 (small-sample guard), no
trigger at exact 45% boundary.
formatAbReport (3): zero-state message, decisive-trials breakdown,
net_negative warning block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-17 16:58:56 -07:00
..
2026-05-15 06:48:36 -07:00
2026-05-11 22:54:35 -07:00
2026-05-14 19:13:14 -07:00
2026-05-17 15:57:03 -07:00
2026-05-17 08:32:03 -07:00
2026-05-12 14:33:29 -07:00
2026-05-17 14:02:45 -07:00
2026-05-17 08:32:03 -07:00
2026-05-15 06:48:36 -07:00
2026-05-10 20:06:31 -07:00
2026-04-29 23:46:36 -07:00
2026-05-07 21:52:58 -07:00
2026-05-10 20:06:31 -07:00
2026-05-11 19:25:48 -07:00
2026-05-02 08:41:11 -07:00
2026-05-11 22:54:35 -07:00
2026-05-06 21:14:34 -07:00
2026-04-18 16:57:38 +08:00
2026-04-21 18:35:45 -07:00
2026-05-14 21:06:42 -07:00
2026-05-08 13:25:48 -07:00
2026-05-08 13:25:48 -07:00
2026-04-11 21:25:44 -10:00
2026-04-14 19:41:34 -10:00
2026-04-22 01:34:22 -07:00
2026-05-14 19:13:14 -07:00
2026-05-14 19:13:14 -07:00
2026-05-07 21:52:58 -07:00
2026-05-14 20:15:29 -07:00
2026-04-30 20:47:32 -07:00
2026-05-07 21:52:58 -07:00
2026-05-10 20:50:40 -07:00
2026-05-02 22:01:05 -07:00
2026-04-20 23:14:38 +08:00
2026-04-29 10:12:10 -07:00
2026-05-10 20:06:31 -07:00
2026-04-21 11:51:32 -07:00
2026-05-17 16:25:13 -07:00
2026-05-17 16:19:27 -07:00
2026-04-25 12:25:34 -07:00
2026-05-10 13:20:07 -07:00
2026-05-02 08:41:11 -07:00
2026-05-11 19:25:48 -07:00
2026-04-09 12:25:04 -10:00
2026-05-14 21:06:42 -07:00
2026-04-25 12:25:34 -07:00
2026-05-14 19:13:14 -07:00
2026-04-25 12:25:34 -07:00
2026-05-11 22:54:35 -07:00
2026-04-29 23:46:36 -07:00
2026-05-09 19:47:56 -07:00
2026-05-07 13:41:46 -07:00
2026-05-09 19:47:56 -07:00
2026-05-02 08:41:11 -07:00
2026-05-07 10:03:38 -07:00
2026-05-09 20:46:34 -07:00
2026-04-25 12:25:34 -07:00
2026-04-25 12:25:34 -07:00
2026-04-25 12:25:34 -07:00
2026-05-14 19:50:50 -04:00
2026-05-13 13:14:58 -04:00
2026-05-09 22:58:19 -07:00
2026-05-13 13:14:58 -04:00
2026-04-06 16:50:15 -07:00
2026-05-08 13:25:48 -07:00
2026-04-26 15:49:48 -07:00
2026-05-11 21:49:57 -07:00
2026-05-17 16:45:07 -07:00
2026-05-06 20:39:56 -07:00
2026-05-06 20:39:56 -07:00
2026-05-06 20:39:56 -07:00
2026-04-29 22:13:05 -07:00
2026-05-09 16:57:47 -07:00
2026-04-30 01:23:29 -07:00
2026-05-08 17:07:51 -07:00
2026-05-09 20:46:34 -07:00
2026-05-11 23:02:03 -07:00
2026-05-09 20:46:34 -07:00
2026-04-14 19:41:34 -10:00
2026-05-08 13:25:48 -07:00
2026-04-22 16:24:23 -07:00
2026-05-03 20:41:39 -07:00
2026-04-29 22:21:07 -07:00
2026-05-17 16:40:11 -07:00
2026-04-20 21:54:36 +08:00
2026-04-26 15:49:48 -07:00
2026-05-10 18:59:11 -07:00
2026-05-07 22:56:22 -07:00
2026-05-13 13:14:58 -04:00
2026-05-17 14:27:31 -07:00
2026-04-30 01:23:29 -07:00
2026-05-02 22:01:05 -07:00
2026-05-13 13:14:58 -04:00
2026-04-20 21:54:36 +08:00
2026-05-14 19:13:14 -07:00
2026-05-07 21:52:58 -07:00
2026-05-14 21:22:10 -07:00
2026-05-06 20:58:19 -07:00
2026-05-15 18:54:36 -07:00
2026-05-07 21:52:58 -07:00
2026-04-11 00:01:09 -10:00
2026-05-07 10:03:38 -07:00
2026-05-10 06:34:40 -07:00
2026-04-14 19:41:34 -10:00
2026-05-11 23:02:03 -07:00
2026-05-17 08:51:33 -07:00
2026-05-17 08:51:33 -07:00
2026-04-25 12:25:34 -07:00
2026-05-01 09:02:43 -07:00
2026-05-01 09:02:43 -07:00
2026-05-01 09:02:43 -07:00
2026-05-13 13:14:58 -04:00
2026-05-17 08:32:03 -07:00
2026-05-17 08:32:03 -07:00
2026-05-17 16:31:34 -07:00
2026-05-11 22:42:54 -07:00
2026-05-17 08:32:03 -07:00
2026-05-11 22:42:54 -07:00
2026-05-17 08:32:03 -07:00
2026-05-17 08:32:03 -07:00
2026-05-17 08:32:03 -07:00
2026-05-11 22:42:54 -07:00
2026-05-17 08:32:03 -07:00
2026-05-11 22:42:54 -07:00
2026-05-17 08:32:03 -07:00
2026-05-17 08:32:03 -07:00
2026-05-17 08:32:03 -07:00
2026-05-17 08:32:03 -07:00
2026-05-01 09:02:43 -07:00
2026-05-16 13:19:04 -07:00
2026-05-01 09:02:43 -07:00
2026-05-01 09:02:43 -07:00
2026-05-13 13:14:58 -04:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-12 14:33:29 -07:00
2026-04-13 21:03:40 -10:00
2026-05-17 14:27:31 -07:00
2026-05-02 22:01:05 -07:00
2026-05-11 19:25:48 -07:00
2026-05-11 19:25:48 -07:00
2026-05-09 20:46:34 -07:00
2026-04-29 10:12:10 -07:00
2026-05-10 06:34:40 -07:00
2026-05-06 21:14:34 -07:00
2026-05-09 20:46:34 -07:00
2026-05-10 12:49:14 -07:00
2026-05-09 16:57:47 -07:00
2026-05-09 16:57:47 -07:00
2026-05-10 12:49:14 -07:00
2026-05-17 08:51:33 -07:00
2026-05-09 16:57:47 -07:00
2026-05-10 12:49:14 -07:00
2026-05-09 16:57:47 -07:00
2026-05-10 12:49:14 -07:00
2026-05-09 16:57:47 -07:00
2026-05-10 12:49:14 -07:00
2026-05-10 12:49:14 -07:00
2026-05-10 20:06:31 -07:00
2026-05-10 12:49:14 -07:00
2026-05-10 12:49:14 -07:00
2026-05-11 19:25:48 -07:00
2026-05-11 19:25:48 -07:00
2026-05-09 16:57:47 -07:00
2026-05-09 16:57:47 -07:00
2026-05-09 16:57:47 -07:00
2026-05-09 16:57:47 -07:00
2026-05-09 16:57:47 -07:00
2026-05-09 16:57:47 -07:00
2026-05-09 16:57:47 -07:00
2026-04-20 07:30:00 +08:00
2026-04-14 21:40:48 -10:00
2026-04-25 12:25:34 -07:00
2026-05-17 08:51:33 -07:00
2026-04-09 10:17:13 -07:00
2026-04-13 07:49:13 -10:00
2026-04-16 23:03:15 -07:00
2026-04-13 07:49:13 -10:00
2026-04-23 19:34:27 -07:00
2026-05-12 14:33:29 -07:00
2026-04-29 23:46:36 -07:00
2026-04-29 23:46:36 -07:00
2026-04-26 20:45:05 -07:00
2026-04-29 23:26:25 -07:00
2026-04-26 20:45:05 -07:00
2026-04-29 23:46:36 -07:00
2026-05-14 19:13:14 -07:00
2026-05-17 08:00:29 -07:00
2026-05-17 16:12:34 -07:00
2026-05-17 16:08:20 -07:00
2026-04-18 18:16:18 +08:00
2026-05-17 16:36:37 -07:00
2026-04-29 22:13:05 -07:00
2026-05-09 22:58:19 -07:00
2026-05-01 09:02:43 -07:00
2026-05-13 13:14:58 -04:00
2026-05-14 20:51:11 -07:00
2026-05-11 22:54:35 -07:00
2026-05-07 10:03:38 -07:00
2026-05-14 20:51:11 -07:00
2026-04-12 07:48:47 -10:00
2026-04-25 12:25:34 -07:00
2026-05-07 22:56:22 -07:00
2026-04-18 16:57:38 +08:00
2026-05-13 13:14:58 -04:00
2026-05-11 19:25:48 -07:00
2026-04-16 23:03:15 -07:00
2026-04-22 01:34:22 -07:00
2026-05-13 13:14:58 -04:00
2026-04-25 12:25:34 -07:00
2026-04-25 12:25:34 -07:00
2026-05-11 21:49:57 -07:00
2026-04-26 20:45:05 -07:00
2026-04-11 21:25:44 -10:00
2026-05-07 13:41:46 -07:00
2026-05-07 19:49:46 -07:00
2026-04-26 20:45:05 -07:00
2026-04-24 00:08:54 -07:00
2026-05-09 19:47:56 -07:00
2026-05-07 22:56:22 -07:00
2026-05-04 21:11:15 -07:00
2026-05-01 09:02:43 -07:00
2026-05-17 08:00:29 -07:00
2026-05-13 13:14:58 -04:00
2026-05-08 13:25:48 -07:00
2026-05-14 21:22:10 -07:00
2026-04-26 20:45:05 -07:00
2026-05-10 12:49:14 -07:00
2026-05-09 20:46:34 -07:00
2026-05-09 20:46:34 -07:00
2026-05-11 22:54:35 -07:00
2026-04-18 16:57:38 +08:00
2026-04-18 16:57:38 +08:00
2026-04-18 18:16:18 +08:00
2026-04-19 07:14:24 +08:00
2026-04-22 01:34:22 -07:00
2026-04-20 07:30:00 +08:00
2026-04-29 23:46:36 -07:00
2026-04-22 01:34:22 -07:00
2026-04-25 12:25:34 -07:00
2026-04-25 12:25:34 -07:00
2026-04-26 20:45:05 -07:00
2026-05-07 10:03:38 -07:00
2026-05-11 19:25:48 -07:00
2026-05-10 06:34:40 -07:00
2026-04-22 01:34:22 -07:00
2026-04-24 01:09:28 -07:00
2026-04-29 23:06:16 -07:00
2026-05-10 20:06:31 -07:00
2026-05-02 22:01:05 -07:00
2026-05-02 22:01:05 -07:00
2026-05-10 13:33:05 -07:00
2026-05-03 20:41:39 -07:00
2026-05-10 12:49:14 -07:00
2026-05-17 16:42:47 -07:00
2026-05-09 19:47:56 -07:00
2026-05-14 20:15:29 -07:00
2026-05-14 20:15:29 -07:00
2026-05-14 19:13:14 -07:00
2026-04-30 01:23:29 -07:00
2026-05-07 21:52:58 -07:00
2026-05-17 14:02:45 -07:00
2026-05-06 21:14:34 -07:00
2026-05-07 10:03:38 -07:00
2026-05-03 20:41:39 -07:00
2026-04-25 12:25:34 -07:00
2026-04-08 23:26:11 -10:00
2026-05-09 20:46:34 -07:00
2026-05-14 21:22:10 -07:00
2026-04-12 17:38:48 -07:00
2026-04-23 19:34:27 -07:00
2026-05-02 08:41:11 -07:00
2026-04-20 07:30:00 +08:00
2026-04-21 18:35:45 -07:00
2026-05-09 20:46:34 -07:00
2026-05-03 20:16:15 -07:00
2026-05-11 19:25:48 -07:00
2026-04-29 23:46:36 -07:00
2026-04-30 20:47:32 -07:00
2026-05-17 16:03:42 -07:00
2026-05-15 18:54:36 -07:00
2026-04-12 13:23:55 -10:00
2026-05-14 19:13:14 -07:00
2026-04-25 12:25:34 -07:00
2026-05-13 13:14:58 -04:00
2026-05-13 13:14:58 -04:00
2026-05-07 10:03:38 -07:00
2026-05-07 21:52:58 -07:00
2026-05-07 21:52:58 -07:00
2026-04-16 23:03:15 -07:00
2026-04-22 01:34:22 -07:00
2026-04-22 01:34:22 -07:00
2026-05-11 23:20:17 -07:00
2026-05-17 16:53:15 -07:00
2026-05-11 23:20:17 -07:00
2026-05-07 21:52:58 -07:00
2026-05-07 21:52:58 -07:00
2026-05-03 20:16:15 -07:00
2026-04-23 19:34:27 -07:00
2026-04-25 12:25:34 -07:00
2026-05-11 22:54:35 -07:00
2026-04-19 07:14:24 +08:00
2026-05-10 13:20:07 -07:00
2026-04-11 21:25:44 -10:00
2026-04-25 12:25:34 -07:00
2026-05-15 06:48:36 -07:00
2026-04-21 18:35:45 -07:00
2026-05-10 13:20:07 -07:00
2026-04-24 01:39:58 -07:00
2026-05-17 08:00:29 -07:00
2026-05-06 16:32:11 -07:00
2026-05-08 13:25:48 -07:00
2026-04-30 23:17:54 -07:00
2026-04-23 19:34:27 -07:00
2026-05-07 21:52:58 -07:00
2026-04-29 23:46:36 -07:00
2026-05-17 14:02:45 -07:00
2026-04-27 22:59:08 -07:00
2026-05-06 21:14:34 -07:00
2026-05-07 10:03:38 -07:00
2026-04-25 12:25:34 -07:00
2026-04-16 23:03:15 -07:00
2026-05-15 06:48:36 -07:00
2026-05-13 13:14:58 -04:00
2026-05-12 14:33:29 -07:00
2026-04-18 18:16:18 +08:00
2026-04-29 23:46:36 -07:00
2026-04-30 20:47:32 -07:00
2026-05-07 13:17:27 -07:00
2026-05-14 20:15:29 -07:00
2026-04-09 10:17:13 -07:00
2026-04-23 19:34:27 -07:00
2026-04-22 02:07:00 -07:00
2026-05-06 20:39:56 -07:00
2026-04-18 16:57:38 +08:00
2026-04-30 23:17:54 -07:00
2026-04-23 19:34:27 -07:00
2026-05-02 08:41:11 -07:00
2026-04-14 19:41:34 -10:00
2026-05-11 22:54:35 -07:00
2026-05-14 20:51:11 -07:00
2026-05-14 19:13:14 -07:00
2026-04-29 22:21:07 -07:00
2026-05-14 19:13:14 -07:00
2026-05-06 21:14:34 -07:00
2026-05-06 21:14:34 -07:00
2026-04-22 16:24:23 -07:00
2026-05-06 11:54:06 -07:00
2026-05-09 22:58:19 -07:00
2026-05-03 20:41:39 -07:00
2026-04-22 16:24:23 -07:00
2026-04-29 22:21:07 -07:00
2026-04-29 22:21:07 -07:00
2026-04-29 22:21:07 -07:00
2026-04-29 22:21:07 -07:00
2026-05-17 14:02:45 -07:00
2026-04-13 07:49:13 -10:00
2026-05-17 08:51:33 -07:00
2026-04-21 21:14:17 -07:00
2026-04-21 21:14:17 -07:00
2026-04-22 01:34:22 -07:00
2026-05-08 17:07:51 -07:00
2026-04-22 01:34:22 -07:00
2026-04-09 10:17:13 -07:00
2026-05-17 14:27:31 -07:00
2026-05-06 11:54:06 -07:00
2026-05-14 21:06:42 -07:00
2026-05-17 16:50:56 -07:00
2026-04-25 12:25:34 -07:00
2026-04-29 22:53:41 -07:00
2026-04-25 12:25:34 -07:00
2026-05-09 20:46:34 -07:00
2026-04-29 22:53:41 -07:00
2026-04-25 12:25:34 -07:00
2026-05-09 21:18:53 -07:00
2026-05-17 14:02:45 -07:00
2026-05-10 06:34:40 -07:00
2026-05-17 16:33:43 -07:00
2026-05-07 23:07:00 -07:00
2026-05-09 20:46:34 -07:00
2026-05-07 23:07:00 -07:00
2026-05-10 06:34:40 -07:00
2026-05-10 06:34:40 -07:00
2026-05-09 20:46:34 -07:00
2026-05-07 23:07:00 -07:00
2026-05-10 06:34:40 -07:00
2026-05-11 23:20:17 -07:00
2026-05-10 18:59:11 -07:00
2026-05-17 16:58:56 -07:00
2026-05-17 14:02:45 -07:00
2026-05-06 21:14:34 -07:00
2026-05-17 16:29:31 -07:00
2026-05-13 13:14:58 -04:00
2026-04-29 23:46:36 -07:00
2026-04-14 19:41:34 -10:00
2026-05-07 21:52:58 -07:00
2026-04-20 23:14:38 +08:00
2026-05-04 21:11:15 -07:00
2026-04-25 12:25:34 -07:00
2026-05-17 16:55:44 -07:00
2026-05-08 13:25:48 -07:00
2026-05-11 22:54:35 -07:00
2026-05-09 20:46:34 -07:00
2026-05-08 13:25:48 -07:00
2026-04-19 18:23:02 +08:00
2026-05-07 21:52:58 -07:00
2026-05-08 13:25:48 -07:00
2026-05-17 16:19:27 -07:00
2026-05-07 13:41:46 -07:00
2026-05-13 12:21:21 -04:00
2026-04-22 01:34:22 -07:00
2026-05-06 21:14:34 -07:00
2026-05-12 14:33:29 -07:00
2026-05-12 14:33:29 -07:00
2026-05-14 21:06:42 -07:00
2026-05-06 11:54:06 -07:00
2026-04-20 07:30:00 +08:00
2026-04-09 10:17:13 -07:00
2026-05-06 11:54:06 -07:00