mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
Garry Tan
18baca6629
Adds the v1→v2 contract boundary for BrainBench. 6 JSON schemas at
eval/schemas/ pin the shape of every artifact a stack must emit to be
scorable: corpus-manifest, public-probe (PublicQuery with gold stripped),
tool-schema (12 read + 3 dry_run tools, 32K tool-output cap), transcript,
scorecard (N ∈ {1, 5, 10}), evidence-contract (structured judge input).
8 gold file templates at eval/data/gold/ scaffold the sealed qrels,
contradictions, poison items, and citation labels. Empty-but-valid
skeletons; Day 3b fills them with real content once the amara-life-v1
corpus generates.
48 tests validate schema syntax, $schema/$id/title/type headers,
round-trip stability, and cross-schema coherence (new Page types in
manifest enum, tool counts, token cap, N enum).
When v2 ports to Python + Inspect AI + Docker, these schemas are the
boundary. Same fixtures, same tool contracts, zero rework.
21 lines
1.1 KiB
JSON
21 lines
1.1 KiB
JSON
{
|
|
"version": 1,
|
|
"_comment": "Judge rubrics for Cat 9 (50 scenarios) + Cat 8 (100 probes). Each probe has 3-5 criteria with weights 1-2. Judge prompt versions are pinned here; scorecard config_card records the version.",
|
|
"judge_prompt_version": "2026-04-19-v1",
|
|
"judge_system_prompt_template": "You grade an agent's answer against a rubric using ONLY the ground_truth_pages as the world-of-facts. Anything in final_answer_text not grounded in ground_truth_pages is a hallucination. Score each criterion 0-5 where 5=fully satisfied, 0=absent or contradicted. Return via the score_answer tool.",
|
|
"probes": [
|
|
{
|
|
"_example": "true",
|
|
"id": "q-0100",
|
|
"category": 9,
|
|
"query": "Prep me for my 2pm with Mina",
|
|
"ground_truth_keys": ["cal/evt-0042", "meeting/mtg-0003", "people/mina"],
|
|
"rubric": [
|
|
{"id": "names_attendee", "criterion": "Names Mina as the meeting counterparty", "weight": 1},
|
|
{"id": "cites_last_context", "criterion": "References the most recent prior meeting or thread", "weight": 2},
|
|
{"id": "no_hallucination", "criterion": "No facts outside ground_truth_pages", "weight": 2}
|
|
]
|
|
}
|
|
]
|
|
}
|