mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-29 19:01:39 +00:00
Garry Tan
a1476af084
Deterministic generator (Mulberry32 seed 42, whole-cloth fictional universe, template prose with controlled difficulty incl. documented v1 reflex limits) emits 135 fixtures + sealed gold across 7 categories with a 15% holdout split; 6 hand-authored spike fixtures froze the schema. Published JSON Schemas are the foreign-runner contract. Gold validated by blind double-label (96.4% agreement; findings fixed, receipt in _ledger.json). Privacy guard extended over the new corpus dirs.
27 lines
834 B
JSON
27 lines
834 B
JSON
{
|
|
"name": "brainbench-corpus",
|
|
"version": 1,
|
|
"seed": 42,
|
|
"generated_fixtures": 135,
|
|
"holdout_fixtures": 23,
|
|
"gold_turns": 241,
|
|
"categories": {
|
|
"kta-pos": 25,
|
|
"kta-neg": 15,
|
|
"push": 20,
|
|
"write-back": 20,
|
|
"continuity_pairs": 15,
|
|
"multi-source": 10,
|
|
"adversarial": 15
|
|
},
|
|
"prose": "template-synthesized, PRNG-varied; deliberately no LLM pass (controlled difficulty; see README)",
|
|
"rebuild": "bun evals/brainbench/generator/gen.ts",
|
|
"generation_cost_usd": 0,
|
|
"gold_validation": {
|
|
"method": "blind double-label, stratified 10% sample (14 fixtures, 28 labeled items)",
|
|
"date": "2026-06-12",
|
|
"agreement": 0.964,
|
|
"findings": "continuity-writer rationale-clause drift (5 gold files) fixed in this corpus version; wb-001 MRR fact added; conventions documented in README"
|
|
}
|
|
}
|