mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
docs(eval): Phase 3 contributor docs + CI workflow for eval/ tests
Ships the contributor-onboarding surface promised in the plan. With this
commit, external researchers have a self-serve path from clone to PR in
under 5 minutes.
Added:
eval/README.md — 5-minute quickstart,
directory map, methodology
one-pager, adapter scorecard
eval/CONTRIBUTING.md — three contributor paths:
1. Write Tier 5.5 queries
2. Submit an external adapter
3. Reproduce a scorecard
eval/RUNBOOK.md — operational troubleshooting:
generation failures, runner
failures, query validation,
world.html rendering, CI
eval/CREDITS.md — contributor attribution
(synthetic-outsider-v1 labeled
as placeholder; real submissions
land here)
.github/PULL_REQUEST_TEMPLATE/tier5-queries.md — structured PR template
for Tier 5.5 submissions
.github/workflows/eval-tests.yml — CI: validates queries,
runs all eval unit tests,
renders world.html on every PR
touching eval/** or
src/core/link-extraction.ts
CI scope (intentionally narrow):
- Triggers on paths: eval/**, src/core/link-extraction.ts, src/core/search/**
- Runs: bun run eval:query:validate (80 queries), test:eval (57 tests),
eval:world:render (smoke-test the HTML renderer)
- Pinned actions by commit SHA (matches existing .github/workflows/test.yml)
- Zero API calls — all Opus/OpenAI paths stubbed or skipped in unit tests
- Fast: ~30s total wall clock
Contributor TTHW (clone → first merged PR):
- Path 1 (Tier 5.5 queries): ~5 min
- Path 2 (external adapter): ~30 min for a simple adapter
- Path 3 (reproduce scorecard): ~15 min wall clock (N=5 run)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
f0649e2f32
commit
b81373d4c2
@@ -0,0 +1,39 @@
|
||||
<!--
|
||||
Tier 5.5 Externally-Authored Query Submission template
|
||||
See eval/CONTRIBUTING.md for the full workflow.
|
||||
-->
|
||||
|
||||
## Summary
|
||||
|
||||
Submitting **N** Tier 5.5 queries for BrainBench.
|
||||
|
||||
- Author handle: `@your-handle`
|
||||
- File location: `eval/external-authors/your-handle/queries.json`
|
||||
- Queries authored fresh (not copy-pasted from a model output)
|
||||
- Slugs verified against `eval/data/world-v1/` (via `bun run eval:world:view`)
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] `bun run eval:query:validate eval/external-authors/your-handle/queries.json` passes
|
||||
- [ ] At least 20 queries
|
||||
- [ ] Each query has either `gold.relevant` (with real slugs) or `gold.expected_abstention: true`
|
||||
- [ ] Temporal queries have `as_of_date` set (`corpus-end` | `per-source` | ISO-8601)
|
||||
- [ ] Phrasing is varied (not all the same template)
|
||||
- [ ] `author` field matches my handle
|
||||
|
||||
## Phrasing variety (optional self-audit)
|
||||
|
||||
Tick the styles represented in your batch:
|
||||
|
||||
- [ ] Full sentence questions
|
||||
- [ ] Fragment-style ("crypto founder Goldman Sachs background")
|
||||
- [ ] Comparison ("X vs Y")
|
||||
- [ ] Follow-up ("And who else...")
|
||||
- [ ] Imperative ("Pull up Alice Davis")
|
||||
- [ ] Trait-based ("the demanding engineering leader")
|
||||
- [ ] Abstention bait (answer is "not in corpus")
|
||||
|
||||
## Notes to reviewer
|
||||
|
||||
Anything worth flagging — ambiguous cases, corpus gaps you found, specific
|
||||
phrasings you were uncertain about.
|
||||
@@ -0,0 +1,40 @@
|
||||
name: Eval tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
paths:
|
||||
- 'eval/**'
|
||||
- 'src/core/link-extraction.ts'
|
||||
- 'src/core/search/**'
|
||||
pull_request:
|
||||
branches: [master]
|
||||
paths:
|
||||
- 'eval/**'
|
||||
- 'src/core/link-extraction.ts'
|
||||
- 'src/core/search/**'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
eval-tests:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: latest
|
||||
- run: bun install
|
||||
|
||||
# Validate the built-in Tier 5 + 5.5 query set.
|
||||
- name: Validate built-in queries
|
||||
run: bun run eval:query:validate
|
||||
|
||||
# Pure-function unit tests — zero API calls, fast.
|
||||
- name: Run eval unit tests
|
||||
run: bun run test:eval
|
||||
|
||||
# Smoke-test the world.html renderer against the committed corpus.
|
||||
- name: Render world.html
|
||||
run: bun run eval:world:render
|
||||
Reference in New Issue
Block a user