mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
Garry Tan
18baca6629
Adds the v1→v2 contract boundary for BrainBench. 6 JSON schemas at
eval/schemas/ pin the shape of every artifact a stack must emit to be
scorable: corpus-manifest, public-probe (PublicQuery with gold stripped),
tool-schema (12 read + 3 dry_run tools, 32K tool-output cap), transcript,
scorecard (N ∈ {1, 5, 10}), evidence-contract (structured judge input).
8 gold file templates at eval/data/gold/ scaffold the sealed qrels,
contradictions, poison items, and citation labels. Empty-but-valid
skeletons; Day 3b fills them with real content once the amara-life-v1
corpus generates.
48 tests validate schema syntax, $schema/$id/title/type headers,
round-trip stability, and cross-schema coherence (new Page types in
manifest enum, tool counts, token cap, N enum).
When v2 ports to Python + Inspect AI + Docker, these schemas are the
boundary. Same fixtures, same tool contracts, zero rework.
18 lines
557 B
JSON
18 lines
557 B
JSON
{
|
|
"version": 1,
|
|
"_comment": "3 implicit preferences planted in amara-life-v1. Never stated directly; only inferable from patterns (what Amara says, does, responds to). Cat 9 measures implicit_preference_recall.",
|
|
"preferences": [
|
|
{
|
|
"_example": "true",
|
|
"id": "pref-001",
|
|
"label": "hates-morning-meetings",
|
|
"evidence_pages": [
|
|
"emails/em-0007",
|
|
"slack/sl-0023",
|
|
"notes/2026-03-14-calendar-notes"
|
|
],
|
|
"surface_in_tasks": ["q-0103-schedule-next-week", "q-0201-should-i-take-7am"]
|
|
}
|
|
]
|
|
}
|