Files
gbrain/test
Garry TanandClaude Opus 4.7 69a71c9de4 calibration: expand synthetic corpus + add hand-labeled ground-truth (T19)
Adds 8 new synthetic pages modeled on the genre mix observed in the
real brain (concepts-with-timeline, meeting-notes, daily-journal,
people-pages, essays). Companion .gradeable-claims.json files carry
hand-labeled answer keys — what a tuned propose_takes prompt SHOULD
extract per page. Closes the F1 gate gap from the plan's T19/D19:

  Training corpus (test/fixtures/calibration/extract-takes-corpus/):
    + concept-startup-market-dynamics.md     (10 claims)
    + meeting-2026-04-10-fundraise-fund-a.md (6 claims)
    + daily-2026-04-15.md                    (5 claims)

  Blind holdout (test/fixtures/calibration/holdout/):
    + concept-founder-execution.md           (6 claims, F1 >= 0.80)
    + daily-2026-04-18.md                    (4 claims, F1 >= 0.80)
    + meeting-2026-04-17-hiring-charlie.md   (5 claims, F1 >= 0.80)
    + essay-on-conviction.md                 (7 claims, F1 >= 0.80)
    + people-bob-example.md                  (5 claims, F1 >= 0.80)

Privacy:
  - No real-brain content read into any committed artifact. Pages
    written from scratch using the canonical placeholder set
    (alice-example, charlie-example, bob-example, acme-example,
    widget-co, fund-a/b/c). Real-name grep confirms zero leakage:
    wintermute, garrytan, paul-graham, sam-altman, etc. → 0 hits.
  - scripts/check-synthetic-corpus-privacy.sh passes: 0 violations
    across 14 pages (was 6).

Genre fidelity:
  - concept-with-timeline pages mirror the dated-assertion structure
    real brain uses (verb framing varies: "argues / predicts / I
    think / I bet / strong conviction / moderate conviction").
  - meeting-notes pages carry both prose claims (extracted via
    hedging language) and explicit ## Takes sections.
  - daily-journal pages test probabilistic framing ("75/25 in favor",
    "call it ~0.5") and self-tagged conviction values.
  - essay-on-conviction is the meta-page that names the author's
    own bias patterns — primary signal for calibration_profile.
  - people pages test claim-about-third-party extraction.

Each JSON ground-truth lists per-claim:
  - claim_text + kind (prediction|judgment|bet) + domain
  - conviction (0..1)
  - since_date
  - rationale (why this claim is gradeable + how a tuned prompt
    should infer conviction from the prose)

This is the corpus that gates the T19 prompt-tune iteration:
  - F1 >= 0.85 on training (10+6+5 = 21 claims across 3 pages
    plus the existing 5 fixtures already shipped)
  - F1 >= 0.80 on holdout (27 claims across 5 pages)

Plan reference: ~/.claude/plans/system-instruction-you-are-working-rippling-knuth.md
Privacy gate: scripts/check-synthetic-corpus-privacy.sh (wired into bun run verify).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-18 13:56:03 -07:00
..