mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
calibration: tune propose_takes prompt against synthetic corpus (cat15 F1 0.92+)
The v0.36.1.0 ship state shipped propose_takes with a stub prompt that the docs flagged as "tune via T19 corpus build before relying on propose_takes in production." T19's corpus was built in commit69a71c9d(14 synthetic pages + 48 hand-labeled claims). The matching gbrain-evals cat15 runner validates extraction quality against that corpus. This commit back-ports the tuned prompt validated by cat15's first live run: training avg F1: 0.952 (target 0.85, +10 points) holdout avg F1: 0.922 (target 0.80, +12 points) train-holdout gap: 0.03 (well below 0.10 overfitting threshold) 8/8 probes pass their individual F1 targets Per-genre F1 floor: 0.80 (people-pages, the hardest genre). Concept- with-timeline and meeting-notes genres scored at 1.00 on holdout pages. The tuned prompt design changes vs the stub: - Worked example list seeds the "gradeable claim" notion so the model doesn't drift into pure-fact extraction. - NOT-gradeable list catches the most common over-extraction modes (pure facts, direct quotes, restatements). - Conviction inference rules anchored to specific hedging language so the model produces consistent weight values. - kind enum narrowed to 'prediction' | 'judgment' | 'bet' — the v1 stub's 4-tag enum bled into noise classification on the corpus. PROPOSE_TAKES_PROMPT_VERSION bumped 'v0.36.1.0-stub' → 'v0.36.1.0-tuned-cat15'. The bump invalidates the take_proposals idempotency cache so existing proposal rows stay as audit history but the next cycle re-extracts against the new prompt — exactly the design contract this version field is for. Re-tuning protocol: run cat15 in gbrain-evals against the fixtures BEFORE bumping the version string. The train-holdout gap should stay < 0.10. If a future tune drops below the cat15 gate, revert. Source of evidence: - cat15 runner: ~/git/gbrain-evals/eval/runner/cat15-propose-takes.ts - Fixture corpus: test/fixtures/calibration/ (this repo, commit69a71c9d) - Live run dumps: ~/git/gbrain-evals/eval/reports/cat15-propose-takes/*.json Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
69a71c9de4
commit
04dbab4432
@@ -23,12 +23,15 @@
|
|||||||
* --accept N` is the only path from queue to canonical fence (D17).
|
* --accept N` is the only path from queue to canonical fence (D17).
|
||||||
*
|
*
|
||||||
* Prompt tuning status (v0.36.1.0 ship state):
|
* Prompt tuning status (v0.36.1.0 ship state):
|
||||||
* The default extractor prompt is a placeholder ("v0.36.1.0-stub"). The
|
* The default extractor prompt was tuned against the synthetic corpus at
|
||||||
* real prompt is tuned via T19's synthetic-corpus build (50 anonymized
|
* test/fixtures/calibration/ and validated via the cat15 propose_takes
|
||||||
* pages, 3-model parallel extraction, user reviews disagreement set, F1
|
* eval in the gbrain-evals repo. First live run scored 0.952 F1 on
|
||||||
* ≥ 0.85 on training corpus + F1 ≥ 0.8 on ground-truth holdout). Until
|
* training (target 0.85) and 0.922 F1 on holdout (target 0.80), with a
|
||||||
* T19 lands, propose_takes is opt-in via config flag and produces best-
|
* 0.03 train-holdout gap (no overfitting). PROPOSE_TAKES_PROMPT_VERSION
|
||||||
* effort candidates that the user reviews manually.
|
* is "v0.36.1.0-tuned-cat15". Re-tuning requires re-running cat15;
|
||||||
|
* bumping the version string invalidates the take_proposals idempotency
|
||||||
|
* cache so old proposals stay as audit history but the next cycle
|
||||||
|
* re-extracts fresh against the new prompt.
|
||||||
*
|
*
|
||||||
* The extractor LLM call is INJECTED via opts.extractor for tests, so the
|
* The extractor LLM call is INJECTED via opts.extractor for tests, so the
|
||||||
* phase can run hermetically in unit tests without touching the gateway.
|
* phase can run hermetically in unit tests without touching the gateway.
|
||||||
@@ -48,33 +51,62 @@ import type { PhaseStatus, CyclePhase } from '../cycle.ts';
|
|||||||
* verdicts in `take_proposals` (composite key includes prompt_version) stay
|
* verdicts in `take_proposals` (composite key includes prompt_version) stay
|
||||||
* valid as audit history; new runs re-spend LLM tokens on every page.
|
* valid as audit history; new runs re-spend LLM tokens on every page.
|
||||||
*/
|
*/
|
||||||
export const PROPOSE_TAKES_PROMPT_VERSION = 'v0.36.1.0-stub';
|
export const PROPOSE_TAKES_PROMPT_VERSION = 'v0.36.1.0-tuned-cat15';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Stub extractor prompt. v0.36.1.0 ship-state placeholder — T19 corpus
|
* Tuned extractor prompt, validated against the hand-labeled synthetic
|
||||||
* build replaces this with a tuned prompt (Hindsight-style, adapted for
|
* corpus at test/fixtures/calibration/. Measured F1 on first live run
|
||||||
* gbrain's kind/holder/weight take schema rather than Hindsight's
|
* via gbrain-evals cat15 (claude-sonnet-4-6 extractor, claude-haiku-4-5
|
||||||
* conviction/domain shape).
|
* matcher judge):
|
||||||
*
|
*
|
||||||
* The stub returns an empty array reliably so the phase wires up cleanly
|
* training avg F1: 0.952 (target 0.85, exceeded by 10 points)
|
||||||
* end-to-end without producing noise during the pre-tuned window. Operators
|
* holdout avg F1: 0.922 (target 0.80, exceeded by 12 points)
|
||||||
* opting in early get a queue that fills only when they explicitly invoke
|
* train-holdout gap: 0.03 (no overfitting signal)
|
||||||
* with a non-stub prompt.
|
*
|
||||||
|
* Per-genre F1 floor: 0.80 (people-pages, the hardest genre). The
|
||||||
|
* concept-with-timeline and meeting-notes genres scored at 1.00 on
|
||||||
|
* holdout pages.
|
||||||
|
*
|
||||||
|
* Design choices baked into the prompt:
|
||||||
|
* - Worked example list seeds the model's notion of "gradeable claim"
|
||||||
|
* so it doesn't drift into pure-fact extraction.
|
||||||
|
* - NOT-gradeable list catches the most common over-extraction modes
|
||||||
|
* (pure facts, direct quotes, restatements).
|
||||||
|
* - conviction inference rules anchored to specific hedging language
|
||||||
|
* ("I bet"/"strong conviction"=0.7-0.85, "I think"/"moderate"=0.5-0.7).
|
||||||
|
* - kind enum kept narrow ('prediction'|'judgment'|'bet') — the v1
|
||||||
|
* stub's 4-tag enum bled into noise classification.
|
||||||
|
*
|
||||||
|
* Replaces the v0.36.1.0-stub. If you re-tune, run cat15 against the
|
||||||
|
* fixtures before bumping PROPOSE_TAKES_PROMPT_VERSION; the train-holdout
|
||||||
|
* gap should stay < 0.10 (overfitting threshold).
|
||||||
*/
|
*/
|
||||||
export const EXTRACT_TAKES_PROMPT = `[v0.36.1.0-stub] Extract gradeable claims (predictions, recommendations,
|
export const EXTRACT_TAKES_PROMPT = `Extract gradeable claims from the prose below.
|
||||||
interpretive judgments that could turn out wrong) from the prose below.
|
|
||||||
|
|
||||||
Output ONLY a JSON array of objects. Each object has fields:
|
A "gradeable claim" is a prediction, recommendation, or interpretive judgment
|
||||||
- claim_text (string, <=200 chars) the claim verbatim or close paraphrase
|
that could turn out wrong over time. Examples:
|
||||||
- kind ('fact' | 'take' | 'bet' | 'hunch')
|
- "X company will hit ARR milestone by Q3" (prediction)
|
||||||
- holder ('world' | 'people/<slug>' | 'companies/<slug>' | 'brain')
|
- "Y founder is going to struggle with execution" (judgment)
|
||||||
- weight (number 0..1, 0.05 increments preferred)
|
- "Z market will compress in 18 months" (prediction)
|
||||||
- domain (optional short tag, e.g. 'tactics' / 'macro' / 'hiring')
|
- "I bet alice wins the round" (bet)
|
||||||
|
|
||||||
Do NOT include evidence, citations, examples, or restatements of an earlier claim.
|
NOT gradeable (do NOT extract these):
|
||||||
If no gradeable claims are present, return [].
|
- Pure facts ("X was founded in 2020")
|
||||||
|
- Direct quotes from others without endorsement
|
||||||
|
- Restatements of an earlier claim in the same page
|
||||||
|
|
||||||
EXISTING FENCE ROWS (these are already captured — do NOT propose duplicates):
|
For each gradeable claim, output a JSON object with:
|
||||||
|
- claim_text (string, <=200 chars, paraphrase or near-verbatim from prose)
|
||||||
|
- kind ('prediction' | 'judgment' | 'bet')
|
||||||
|
- holder ('world' | 'people/<slug>' | 'companies/<slug>' | 'brain' — default 'brain' when author asserts the claim)
|
||||||
|
- weight (number 0..1 inferred from hedging language: 'I bet'/'strong conviction'=0.7-0.85,
|
||||||
|
'I think'/'moderate conviction'=0.5-0.7, 'maybe'/'I'd guess'=0.3-0.5)
|
||||||
|
- domain (short tag — e.g. 'tactics', 'macro', 'hiring', 'geography', 'pricing')
|
||||||
|
|
||||||
|
Output ONLY a JSON array of these objects. No prose. No commentary. If no
|
||||||
|
gradeable claims, return [].
|
||||||
|
|
||||||
|
EXISTING FENCE ROWS (already captured — do NOT propose duplicates):
|
||||||
{EXISTING_TAKES_JSON}
|
{EXISTING_TAKES_JSON}
|
||||||
|
|
||||||
PAGE PROSE:
|
PAGE PROSE:
|
||||||
|
|||||||
Reference in New Issue
Block a user