docs(eval): Phase 3 contributor docs + CI workflow for eval/ tests

Ships the contributor-onboarding surface promised in the plan. With this
commit, external researchers have a self-serve path from clone to PR in
under 5 minutes.

Added:
  eval/README.md                                — 5-minute quickstart,
                                                  directory map, methodology
                                                  one-pager, adapter scorecard
  eval/CONTRIBUTING.md                          — three contributor paths:
                                                    1. Write Tier 5.5 queries
                                                    2. Submit an external adapter
                                                    3. Reproduce a scorecard
  eval/RUNBOOK.md                               — operational troubleshooting:
                                                  generation failures, runner
                                                  failures, query validation,
                                                  world.html rendering, CI
  eval/CREDITS.md                               — contributor attribution
                                                  (synthetic-outsider-v1 labeled
                                                  as placeholder; real submissions
                                                  land here)
  .github/PULL_REQUEST_TEMPLATE/tier5-queries.md — structured PR template
                                                  for Tier 5.5 submissions
  .github/workflows/eval-tests.yml              — CI: validates queries,
                                                  runs all eval unit tests,
                                                  renders world.html on every PR
                                                  touching eval/** or
                                                  src/core/link-extraction.ts

CI scope (intentionally narrow):
  - Triggers on paths: eval/**, src/core/link-extraction.ts, src/core/search/**
  - Runs: bun run eval:query:validate (80 queries), test:eval (57 tests),
          eval:world:render (smoke-test the HTML renderer)
  - Pinned actions by commit SHA (matches existing .github/workflows/test.yml)
  - Zero API calls — all Opus/OpenAI paths stubbed or skipped in unit tests
  - Fast: ~30s total wall clock

Contributor TTHW (clone → first merged PR):
  - Path 1 (Tier 5.5 queries): ~5 min
  - Path 2 (external adapter): ~30 min for a simple adapter
  - Path 3 (reproduce scorecard): ~15 min wall clock (N=5 run)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-19 00:18:28 +08:00
co-authored by Claude Opus 4.7
parent f0649e2f32
commit b81373d4c2
6 changed files with 563 additions and 0 deletions
@@ -0,0 +1,39 @@
<!--
Tier 5.5 Externally-Authored Query Submission template
See eval/CONTRIBUTING.md for the full workflow.
-->
## Summary
Submitting **N** Tier 5.5 queries for BrainBench.
- Author handle: `@your-handle`
- File location: `eval/external-authors/your-handle/queries.json`
- Queries authored fresh (not copy-pasted from a model output)
- Slugs verified against `eval/data/world-v1/` (via `bun run eval:world:view`)
## Checklist
- [ ] `bun run eval:query:validate eval/external-authors/your-handle/queries.json` passes
- [ ] At least 20 queries
- [ ] Each query has either `gold.relevant` (with real slugs) or `gold.expected_abstention: true`
- [ ] Temporal queries have `as_of_date` set (`corpus-end` | `per-source` | ISO-8601)
- [ ] Phrasing is varied (not all the same template)
- [ ] `author` field matches my handle
## Phrasing variety (optional self-audit)
Tick the styles represented in your batch:
- [ ] Full sentence questions
- [ ] Fragment-style ("crypto founder Goldman Sachs background")
- [ ] Comparison ("X vs Y")
- [ ] Follow-up ("And who else...")
- [ ] Imperative ("Pull up Alice Davis")
- [ ] Trait-based ("the demanding engineering leader")
- [ ] Abstention bait (answer is "not in corpus")
## Notes to reviewer
Anything worth flagging — ambiguous cases, corpus gaps you found, specific
phrasings you were uncertain about.
+40
View File
@@ -0,0 +1,40 @@
name: Eval tests
on:
push:
branches: [master]
paths:
- 'eval/**'
- 'src/core/link-extraction.ts'
- 'src/core/search/**'
pull_request:
branches: [master]
paths:
- 'eval/**'
- 'src/core/link-extraction.ts'
- 'src/core/search/**'
permissions:
contents: read
jobs:
eval-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: latest
- run: bun install
# Validate the built-in Tier 5 + 5.5 query set.
- name: Validate built-in queries
run: bun run eval:query:validate
# Pure-function unit tests — zero API calls, fast.
- name: Run eval unit tests
run: bun run test:eval
# Smoke-test the world.html renderer against the committed corpus.
- name: Render world.html
run: bun run eval:world:render