mirror of https://github.com/garrytan/gbrain.git synced 2026-07-31 04:07:52 +00:00

Files

T

Garry TanandClaude Opus 4.7 b81373d4c2 docs(eval): Phase 3 contributor docs + CI workflow for eval/ tests

Ships the contributor-onboarding surface promised in the plan. With this
commit, external researchers have a self-serve path from clone to PR in
under 5 minutes.

Added:
  eval/README.md                                — 5-minute quickstart,
                                                  directory map, methodology
                                                  one-pager, adapter scorecard
  eval/CONTRIBUTING.md                          — three contributor paths:
                                                    1. Write Tier 5.5 queries
                                                    2. Submit an external adapter
                                                    3. Reproduce a scorecard
  eval/RUNBOOK.md                               — operational troubleshooting:
                                                  generation failures, runner
                                                  failures, query validation,
                                                  world.html rendering, CI
  eval/CREDITS.md                               — contributor attribution
                                                  (synthetic-outsider-v1 labeled
                                                  as placeholder; real submissions
                                                  land here)
  .github/PULL_REQUEST_TEMPLATE/tier5-queries.md — structured PR template
                                                  for Tier 5.5 submissions
  .github/workflows/eval-tests.yml              — CI: validates queries,
                                                  runs all eval unit tests,
                                                  renders world.html on every PR
                                                  touching eval/** or
                                                  src/core/link-extraction.ts

CI scope (intentionally narrow):
  - Triggers on paths: eval/**, src/core/link-extraction.ts, src/core/search/**
  - Runs: bun run eval:query:validate (80 queries), test:eval (57 tests),
          eval:world:render (smoke-test the HTML renderer)
  - Pinned actions by commit SHA (matches existing .github/workflows/test.yml)
  - Zero API calls — all Opus/OpenAI paths stubbed or skipped in unit tests
  - Fast: ~30s total wall clock

Contributor TTHW (clone → first merged PR):
  - Path 1 (Tier 5.5 queries): ~5 min
  - Path 2 (external adapter): ~30 min for a simple adapter
  - Path 3 (reproduce scorecard): ~15 min wall clock (N=5 run)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-19 00:18:28 +08:00

1.8 KiB

Raw Blame History

BrainBench credits

Core team

garrytan — BrainBench v1 + v1.1 architecture, adapter interface, extraction regex residuals (v0.10.5), multi-axis type-accuracy runner
Claude Opus 4.7 — pair programming, test coverage, documentation

External query authors (Tier 5.5)

No human external authors yet. The Tier 5.5 query set currently comprises 50 synthetic queries labeled author: "synthetic-outsider-v1" as a placeholder. Real submissions via eval/external-authors/<handle>/queries.json PRs supersede synthetic entries.

Want to be credited here? See eval/CONTRIBUTING.md.

External adapters

No third-party adapters yet. The shipping adapter set:

gbrain-after — gbrain v0.10.3+ (internal; the system under test)
hybrid-nograph — gbrain hybrid search with graph layer disabled (internal comparator; closest apples-to-apples to gbrain-after)
ripgrep-bm25 — classic IR baseline built in an afternoon
vector-only — commodity vector RAG, same embedder as gbrain

Third-party submissions (mem0, supermemory, Letta, Cognee, etc.) via eval/runner/adapters/<adapter>.ts PRs. See eval/CONTRIBUTING.md for the adapter interface and submission flow.

Data

Corpus generator: Claude Opus
Canonical world: eval/data/world-v1/ (committed, 240 entities)
Generation cost: ~$3.14 USD (one-time)

Inspiration

SWE-bench — taught us that a benchmark's credibility comes from real baselines, not from the authoring team saying nice things about their own stack
Codex — cold-read critique that "this isn't a standard, it's an internal test" drove the Phase 2 external-baselines work that became the headline of this PR
MTEB — embedding-model reproducibility card pattern; we copy the "pin every version in every scorecard" discipline

1.8 KiB Raw Blame History