mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
Garry TanandClaude Opus 4.7
b81373d4c2
Ships the contributor-onboarding surface promised in the plan. With this
commit, external researchers have a self-serve path from clone to PR in
under 5 minutes.
Added:
eval/README.md — 5-minute quickstart,
directory map, methodology
one-pager, adapter scorecard
eval/CONTRIBUTING.md — three contributor paths:
1. Write Tier 5.5 queries
2. Submit an external adapter
3. Reproduce a scorecard
eval/RUNBOOK.md — operational troubleshooting:
generation failures, runner
failures, query validation,
world.html rendering, CI
eval/CREDITS.md — contributor attribution
(synthetic-outsider-v1 labeled
as placeholder; real submissions
land here)
.github/PULL_REQUEST_TEMPLATE/tier5-queries.md — structured PR template
for Tier 5.5 submissions
.github/workflows/eval-tests.yml — CI: validates queries,
runs all eval unit tests,
renders world.html on every PR
touching eval/** or
src/core/link-extraction.ts
CI scope (intentionally narrow):
- Triggers on paths: eval/**, src/core/link-extraction.ts, src/core/search/**
- Runs: bun run eval:query:validate (80 queries), test:eval (57 tests),
eval:world:render (smoke-test the HTML renderer)
- Pinned actions by commit SHA (matches existing .github/workflows/test.yml)
- Zero API calls — all Opus/OpenAI paths stubbed or skipped in unit tests
- Fast: ~30s total wall clock
Contributor TTHW (clone → first merged PR):
- Path 1 (Tier 5.5 queries): ~5 min
- Path 2 (external adapter): ~30 min for a simple adapter
- Path 3 (reproduce scorecard): ~15 min wall clock (N=5 run)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
41 lines
1006 B
YAML
41 lines
1006 B
YAML
name: Eval tests
|
|
|
|
on:
|
|
push:
|
|
branches: [master]
|
|
paths:
|
|
- 'eval/**'
|
|
- 'src/core/link-extraction.ts'
|
|
- 'src/core/search/**'
|
|
pull_request:
|
|
branches: [master]
|
|
paths:
|
|
- 'eval/**'
|
|
- 'src/core/link-extraction.ts'
|
|
- 'src/core/search/**'
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
eval-tests:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
|
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
|
with:
|
|
bun-version: latest
|
|
- run: bun install
|
|
|
|
# Validate the built-in Tier 5 + 5.5 query set.
|
|
- name: Validate built-in queries
|
|
run: bun run eval:query:validate
|
|
|
|
# Pure-function unit tests — zero API calls, fast.
|
|
- name: Run eval unit tests
|
|
run: bun run test:eval
|
|
|
|
# Smoke-test the world.html renderer against the committed corpus.
|
|
- name: Render world.html
|
|
run: bun run eval:world:render
|