CLAUDE.md: adds a full BrainBench section to the Key Files list — 14 new
entries covering eval/README.md, multi-adapter.ts, types.ts (with new
PublicPage/PublicQuery), adapters/, queries/, type-accuracy.ts,
adversarial.ts, all.ts, world.ts/gen.ts, world-html.ts, amara-life.ts,
amara-life-gen.ts, schemas/, data/world-v1/, data/gold/,
data/amara-life-v1/, docs/benchmarks/, and test/eval/. Adds 3 new
test/eval/ lines to the unit-tests catalog.
eval/README.md: file tree updated to reflect v0.15 additions —
data/amara-life-v1/, data/gold/, schemas/, generators/amara-life.ts +
amara-life-gen.ts, runner/all.ts + adversarial.ts.
README.md: updates hero benchmark numbers (L7 intro + L353 mid-page)
from v0.10.5 PR #188 numbers (R@5 83→95, P@5 39→45) to current v0.12.1
4-adapter numbers (P@5 49.1% · R@5 97.9% · +31.4 pts vs hybrid-nograph).
Adds the v0.11→v0.12 regression comparison as the secondary reference.
Deeper-section tables (L422+) labeled "BrainBench v1 (PR #188)" are
preserved as historical data.
CHANGELOG is untouched — /ship already wrote the v0.15.0 entry.
TODOS.md is untouched — Cat 5/6/8/9/11 remain open (only foundations
shipped in v0.15.0; Cat runners ship in v1 Complete follow-ups).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ships the contributor-onboarding surface promised in the plan. With this
commit, external researchers have a self-serve path from clone to PR in
under 5 minutes.
Added:
eval/README.md — 5-minute quickstart,
directory map, methodology
one-pager, adapter scorecard
eval/CONTRIBUTING.md — three contributor paths:
1. Write Tier 5.5 queries
2. Submit an external adapter
3. Reproduce a scorecard
eval/RUNBOOK.md — operational troubleshooting:
generation failures, runner
failures, query validation,
world.html rendering, CI
eval/CREDITS.md — contributor attribution
(synthetic-outsider-v1 labeled
as placeholder; real submissions
land here)
.github/PULL_REQUEST_TEMPLATE/tier5-queries.md — structured PR template
for Tier 5.5 submissions
.github/workflows/eval-tests.yml — CI: validates queries,
runs all eval unit tests,
renders world.html on every PR
touching eval/** or
src/core/link-extraction.ts
CI scope (intentionally narrow):
- Triggers on paths: eval/**, src/core/link-extraction.ts, src/core/search/**
- Runs: bun run eval:query:validate (80 queries), test:eval (57 tests),
eval:world:render (smoke-test the HTML renderer)
- Pinned actions by commit SHA (matches existing .github/workflows/test.yml)
- Zero API calls — all Opus/OpenAI paths stubbed or skipped in unit tests
- Fast: ~30s total wall clock
Contributor TTHW (clone → first merged PR):
- Path 1 (Tier 5.5 queries): ~5 min
- Path 2 (external adapter): ~30 min for a simple adapter
- Path 3 (reproduce scorecard): ~15 min wall clock (N=5 run)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
