docs(eval): Phase 3 contributor docs + CI workflow for eval/ tests

Ships the contributor-onboarding surface promised in the plan. With this commit, external researchers have a self-serve path from clone to PR in under 5 minutes. Added: eval/README.md — 5-minute quickstart, directory map, methodology one-pager, adapter scorecard eval/CONTRIBUTING.md — three contributor paths: 1. Write Tier 5.5 queries 2. Submit an external adapter 3. Reproduce a scorecard eval/RUNBOOK.md — operational troubleshooting: generation failures, runner failures, query validation, world.html rendering, CI eval/CREDITS.md — contributor attribution (synthetic-outsider-v1 labeled as placeholder; real submissions land here) .github/PULL_REQUEST_TEMPLATE/tier5-queries.md — structured PR template for Tier 5.5 submissions .github/workflows/eval-tests.yml — CI: validates queries, runs all eval unit tests, renders world.html on every PR touching eval/** or src/core/link-extraction.ts CI scope (intentionally narrow): - Triggers on paths: eval/**, src/core/link-extraction.ts, src/core/search/** - Runs: bun run eval:query:validate (80 queries), test:eval (57 tests), eval:world:render (smoke-test the HTML renderer) - Pinned actions by commit SHA (matches existing .github/workflows/test.yml) - Zero API calls — all Opus/OpenAI paths stubbed or skipped in unit tests - Fast: ~30s total wall clock Contributor TTHW (clone → first merged PR): - Path 1 (Tier 5.5 queries): ~5 min - Path 2 (external adapter): ~30 min for a simple adapter - Path 3 (reproduce scorecard): ~15 min wall clock (N=5 run) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-07-27 22:15:33 +00:00 · 2026-04-19 00:18:28 +08:00
parent f0649e2f32
commit b81373d4c2
6 changed files with 563 additions and 0 deletions
@@ -0,0 +1,39 @@
+<!--
+Tier 5.5 Externally-Authored Query Submission template
+See eval/CONTRIBUTING.md for the full workflow.
+-->
+
+## Summary
+
+Submitting **N** Tier 5.5 queries for BrainBench.
+
+- Author handle: `@your-handle`
+- File location: `eval/external-authors/your-handle/queries.json`
+- Queries authored fresh (not copy-pasted from a model output)
+- Slugs verified against `eval/data/world-v1/` (via `bun run eval:world:view`)
+
+## Checklist
+
+- [ ] `bun run eval:query:validate eval/external-authors/your-handle/queries.json` passes
+- [ ] At least 20 queries
+- [ ] Each query has either `gold.relevant` (with real slugs) or `gold.expected_abstention: true`
+- [ ] Temporal queries have `as_of_date` set (`corpus-end` | `per-source` | ISO-8601)
+- [ ] Phrasing is varied (not all the same template)
+- [ ] `author` field matches my handle
+
+## Phrasing variety (optional self-audit)
+
+Tick the styles represented in your batch:
+
+- [ ] Full sentence questions
+- [ ] Fragment-style ("crypto founder Goldman Sachs background")
+- [ ] Comparison ("X vs Y")
+- [ ] Follow-up ("And who else...")
+- [ ] Imperative ("Pull up Alice Davis")
+- [ ] Trait-based ("the demanding engineering leader")
+- [ ] Abstention bait (answer is "not in corpus")
+
+## Notes to reviewer
+
+Anything worth flagging — ambiguous cases, corpus gaps you found, specific
+phrasings you were uncertain about.
@@ -0,0 +1,40 @@
+name: Eval tests
+
+on:
+  push:
+    branches: [master]
+    paths:
+      - 'eval/**'
+      - 'src/core/link-extraction.ts'
+      - 'src/core/search/**'
+  pull_request:
+    branches: [master]
+    paths:
+      - 'eval/**'
+      - 'src/core/link-extraction.ts'
+      - 'src/core/search/**'
+
+permissions:
+  contents: read
+
+jobs:
+  eval-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5  # v4
+      - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6  # v2
+        with:
+          bun-version: latest
+      - run: bun install
+
+      # Validate the built-in Tier 5 + 5.5 query set.
+      - name: Validate built-in queries
+        run: bun run eval:query:validate
+
+      # Pure-function unit tests — zero API calls, fast.
+      - name: Run eval unit tests
+        run: bun run test:eval
+
+      # Smoke-test the world.html renderer against the committed corpus.
+      - name: Render world.html
+        run: bun run eval:world:render
@@ -0,0 +1,150 @@
+# Contributing to BrainBench
+
+Three contribution paths. Each has a separate workflow.
+
+## 1. Write Tier 5.5 externally-authored queries
+
+Tier 5.5 exists to neutralize the "gbrain wrote its own exam" critique. The
+queries currently in the repo are AI-authored synthetic placeholders; real
+outside researcher submissions supersede them.
+
+### Workflow
+
+```sh
+# Step 1. Understand the canonical world.
+bun run eval:world:view
+# Browser opens. Click through entities. Note down what's real.
+
+# Step 2. Scaffold a query.
+bun run eval:query:new --tier externally-authored --author "@your-handle"
+# Prints a Query template. Save to a file.
+
+# Step 3. Edit the template.
+# - Replace text with your actual question
+# - Replace gold.relevant with slug(s) that actually exist
+# - If the query has temporal verbs (is/was/were/now/...), set as_of_date
+#   to "corpus-end", "per-source", or ISO-8601
+# - Fill in tags
+
+# Step 4. Validate before submitting.
+bun run eval:query:validate path/to/your-queries.json
+
+# Step 5. Submit a PR.
+# File location: eval/external-authors/<your-handle>/queries.json
+# PR template: .github/PULL_REQUEST_TEMPLATE/tier5-queries.md
+```
+
+### Query-authoring guidelines
+
+- **Write like you'd naturally ask.** Don't adapt your voice to an "AI
+  benchmark style." Fragments, typos, comparisons, follow-ups, imperatives
+  — all welcome. Variety is the value.
+- **Gold must be real slugs.** Every slug in `gold.relevant` must exist in
+  `eval/data/world-v1/`. The validator checks format; you verify existence.
+- **Abstention is a valid answer.** If your query has no answer in the
+  corpus (e.g. you're asking about someone who isn't there), set
+  `expected_output_type: 'abstention'` and `gold.expected_abstention: true`.
+- **Temporal queries need `as_of_date`.** The validator will reject
+  "Where is Sarah now?" without it. Use `"corpus-end"` for "as of the most
+  recent data," `"per-source"` for "whatever the cited source says," or a
+  specific ISO date.
+- **Partial answers are OK** if you flag them via `known_failure_modes`.
+
+### Query quality bar
+
+We'll merge your PR if:
+- `bun run eval:query:validate` passes
+- Slugs resolve to real entities
+- At least 20 queries (one batch)
+- Queries have genuine phrasing variety
+
+## 2. Submit an external adapter
+
+The `Adapter` interface is `eval/runner/types.ts`. Three methods:
+
+```typescript
+interface Adapter {
+  readonly name: string;
+  init(rawPages: Page[], config: AdapterConfig): Promise<BrainState>;
+  query(q: Query, state: BrainState): Promise<RankedDoc[]>;
+  snapshot?(state: BrainState): Promise<string>;
+}
+```
+
+### Workflow
+
+```sh
+# Step 1. Create your adapter file.
+#   eval/runner/adapters/my-adapter.ts
+
+# Step 2. Write it.
+#   - import types from '../types.ts'
+#   - export class MyAdapter implements Adapter { ... }
+#   - BrainState is opaque to the runner. Internal shape is yours.
+#   - `rawPages: Page[]` is all you get. Never read from gold/ — the
+#     runner doesn't give you that path on purpose.
+
+# Step 3. Write a unit test.
+#   eval/runner/adapters/my-adapter.test.ts
+#   Cover at minimum: init, query, deterministic tie-break.
+
+# Step 4. Wire into multi-adapter.ts.
+#   import { MyAdapter } from './adapters/my-adapter.ts';
+#   const allAdapters: Adapter[] = [
+#     ...existing,
+#     new MyAdapter(),
+#   ];
+
+# Step 5. Test locally.
+bun run test:eval
+bun run eval:run:dev --adapter=my-adapter
+
+# Step 6. Open a PR.
+```
+
+### Adapter quality bar
+
+- Deterministic over sorted input (stddev=0 across N=5 runs is the
+  expected default; non-zero is a signal worth understanding)
+- `query()` returns rank order — `rank: i + 1`, 1-based, no duplicates
+- Tie-breaks documented (e.g. "alphabetical by slug when scores tie")
+- No network calls in unit tests (mock any API dependencies)
+- Pass `bun run test:eval`
+
+## 3. Reproduce / verify a published scorecard
+
+```sh
+# Step 1. Check the scorecard's commit hash.
+# Reports in docs/benchmarks/ include the gbrain version + commit.
+
+# Step 2. Pin the same commit.
+git checkout <commit-sha>
+
+# Step 3. Run the full benchmark.
+bun run eval:run
+
+# Step 4. Compare to the published scorecard.
+# For deterministic adapters, numbers should match exactly.
+# For embedding-based adapters, numbers should land within the published
+# tolerance bands (mean ± stddev).
+```
+
+If your numbers drift outside tolerance, file an issue with:
+- Your `bun --version`
+- Your `uname -sr`
+- Your OpenAI model ID (for embedding-model drift)
+- A diff of the scorecard
+
+## Code style
+
+- Match existing gbrain patterns (hand-rolled where appropriate, no new
+  deps unless genuinely needed)
+- Bun's built-in test runner (`bun:test`), not jest/vitest
+- No em dashes in prose (`—`, `–`); use parentheses or sentences
+- Commit messages: `feat(eval):`, `fix(eval):`, `docs(eval):`, `test(eval):`
+
+## Contributors
+
+See `eval/CREDITS.md` for the full list. All Tier 5.5 external-author
+submissions credited there + in the scorecard. Synthetic placeholders are
+labeled `synthetic-outsider-v1`.
@@ -0,0 +1,47 @@
+# BrainBench credits
+
+## Core team
+
+- **garrytan** — BrainBench v1 + v1.1 architecture, adapter interface,
+  extraction regex residuals (v0.10.5), multi-axis type-accuracy runner
+- **Claude Opus 4.7** — pair programming, test coverage, documentation
+
+## External query authors (Tier 5.5)
+
+No human external authors yet. The Tier 5.5 query set currently comprises
+50 synthetic queries labeled `author: "synthetic-outsider-v1"` as a
+placeholder. Real submissions via `eval/external-authors/<handle>/queries.json`
+PRs supersede synthetic entries.
+
+**Want to be credited here?** See `eval/CONTRIBUTING.md`.
+
+## External adapters
+
+No third-party adapters yet. The shipping adapter set:
+
+- `gbrain-after` — gbrain v0.10.3+ (internal; the system under test)
+- `hybrid-nograph` — gbrain hybrid search with graph layer disabled
+  (internal comparator; closest apples-to-apples to `gbrain-after`)
+- `ripgrep-bm25` — classic IR baseline built in an afternoon
+- `vector-only` — commodity vector RAG, same embedder as gbrain
+
+Third-party submissions (mem0, supermemory, Letta, Cognee, etc.) via
+`eval/runner/adapters/<adapter>.ts` PRs. See `eval/CONTRIBUTING.md` for
+the adapter interface and submission flow.
+
+## Data
+
+- Corpus generator: Claude Opus
+- Canonical world: `eval/data/world-v1/` (committed, 240 entities)
+- Generation cost: ~$3.14 USD (one-time)
+
+## Inspiration
+
+- **SWE-bench** — taught us that a benchmark's credibility comes from real
+  baselines, not from the authoring team saying nice things about their
+  own stack
+- **Codex** — cold-read critique that "this isn't a standard, it's an
+  internal test" drove the Phase 2 external-baselines work that became
+  the headline of this PR
+- **MTEB** — embedding-model reproducibility card pattern; we copy the
+  "pin every version in every scorecard" discipline
@@ -0,0 +1,126 @@
+# BrainBench
+
+Public benchmark for personal knowledge brain agent stacks. Ships 4 adapter
+configurations scored side-by-side on a 240-page rich-prose fictional corpus
+(`twin-amara`). Measures retrieval, extraction quality, and per-link-type
+accuracy.
+
+**What this answers:** "Does the knowledge graph layer do useful work, or is
+gbrain just a thin wrapper over vector+keyword hybrid?" Headline: gbrain
+beats the closest external baseline (hybrid-without-graph, same embedder,
+same chunking) by **+31 points P@5**. The graph layer is load-bearing.
+
+## 5-minute quickstart
+
+```sh
+# 1. Run the full benchmark (4 adapters × 5 runs, ~15 min wall clock)
+bun run eval:run
+
+# 2. Fast iteration (N=1 single run)
+bun run eval:run:dev
+
+# 3. Just the type-accuracy report
+bun run eval:type-accuracy
+
+# 4. Explore the canonical world (contributor-facing UI)
+bun run eval:world:view
+```
+
+## What's in the box
+
+```
+eval/
+├── data/world-v1/          Canonical world (committed). 240 sharded JSON files.
+│                            One file per entity + _ledger.json metadata.
+├── generators/
+│   ├── gen.ts              Opus-backed corpus generator (run once; output cached)
+│   ├── world.ts            World-schema scaffolder
+│   └── world-html.ts       World explorer HTML renderer (XSS-safe)
+├── runner/
+│   ├── multi-adapter.ts    4-adapter side-by-side scorer (N=5)
+│   ├── type-accuracy.ts    Per-link-type accuracy vs gold from _facts
+│   ├── before-after.ts     Original v1 BEFORE/AFTER retrieval run
+│   ├── types.ts            Adapter, Page, Query, RankedDoc interfaces
+│   ├── adapters/
+│   │   ├── ripgrep-bm25.ts       EXT-1: classic IR baseline (BM25 over grep hits)
+│   │   ├── vector-only.ts        EXT-2: pure cosine similarity, same embedder
+│   │   └── hybrid-nograph.ts     EXT-3: gbrain hybrid with graph disabled
+│   └── queries/
+│       ├── tier5-fuzzy.ts          30 vague-recall queries (hand-authored)
+│       ├── tier5_5-synthetic.ts    50 synthetic outsider queries (AI-authored, labeled)
+│       ├── validator.ts            Query schema enforcement (temporal as_of_date rule)
+│       └── index.ts                Aggregator + validateAll()
+├── cli/
+│   ├── world-view.ts       Render + open world.html
+│   ├── query-validate.ts   Validate a Query[] file
+│   └── query-new.ts        Scaffold a Query template
+└── reports/                Benchmark scorecards (gitignored)
+```
+
+## Three contributor paths
+
+### Path 1: Reproduce a published scorecard
+
+```sh
+# 1. Check out the specific gbrain commit referenced in the scorecard
+git checkout <commit-sha>
+# 2. Run the full benchmark
+bun run eval:run
+# 3. Compare your numbers to the scorecard. Deterministic adapters should
+#    match exactly. Embedding-based adapters should land within tolerance bands.
+```
+
+### Path 2: Submit a new external adapter
+
+See `CONTRIBUTING.md` for the adapter submission flow. Short version:
+
+1. Implement `eval/runner/adapters/<your-adapter>.ts` conforming to the
+   `Adapter` interface in `eval/runner/types.ts`.
+2. Add a unit test file alongside.
+3. Wire your adapter into `eval/runner/multi-adapter.ts` (one line).
+4. `bun run eval:run:dev` to verify.
+5. Open a PR.
+
+### Path 3: Write Tier 5.5 externally-authored queries
+
+The T5.5 queries currently in the repo are AI-authored (`author:
+"synthetic-outsider-v1"`) as a placeholder. Real outside researchers should:
+
+1. `bun run eval:world:view` to understand the canonical world
+2. `bun run eval:query:new --tier externally-authored --author "@your-handle"`
+3. Edit the scaffolded template with a real query + gold slugs
+4. `bun run eval:query:validate path/to/your.json`
+5. Submit via `eval/external-authors/<your-handle>/queries.json` in a PR
+
+See `CONTRIBUTING.md` for the query-submission template.
+
+## Methodology one-pager
+
+- **Corpus:** 240 Opus-generated fictional biographical pages. Fixed,
+  committed, zero private data. Reproducibility baseline for any run.
+- **Gold:** Each page's `_facts` metadata defines canonical relationships.
+  The scorer never shows `_facts` to the adapters — **raw pages only**
+  cross the ingestion boundary (structural enforcement in `Adapter.init`).
+- **Metrics:** P@5 and R@5 on relational queries (145 canonical from
+  `_facts`, 80 tier-5 + tier-5.5). Type accuracy on extracted edges
+  (`eval/runner/type-accuracy.ts`).
+- **N=5 runs per adapter** with page-order shuffle (seeded LCG; runs are
+  reproducible). Stddev surfaces order-dependent adapter bugs. Deterministic
+  adapters correctly show stddev=0.
+- **Temporal queries** require explicit `as_of_date` (validated at query
+  authoring time; rejected at load if a temporal verb is present without it).
+
+## Adapter scorecard (most recent, N=5)
+
+See `docs/benchmarks/2026-04-18-brainbench-v1.md` for the full report.
+Quick summary from `bun run eval:run`:
+
+| Adapter         | P@5    | R@5    |
+|-----------------|--------|--------|
+| gbrain-after    | 49.1%  | 97.9%  |
+| hybrid-nograph  | 17.8%  | 65.1%  |
+| ripgrep-bm25    | 17.1%  | 62.4%  |
+| vector-only     | 10.8%  | 40.7%  |
+
+The graph layer beats vector+keyword hybrid on relational queries by ~31
+points; hybrid-without-graph barely edges BM25. That's the story.
@@ -0,0 +1,161 @@
+# BrainBench runbook
+
+Operational troubleshooting for the most common failures. One fix per entry.
+
+## Generation failures
+
+### "OPENAI_API_KEY environment variable is missing"
+
+The embedding adapter (`vector-only`) and any run of `eval/generators/gen.ts`
+calls the OpenAI API. You need an API key.
+
+```sh
+export OPENAI_API_KEY=sk-proj-...
+# or source from a dotenv file
+source ~/.zshrc   # if the key is in your shell profile
+bun run eval:run
+```
+
+### "ANTHROPIC_API_KEY environment variable is missing"
+
+Only needed if you regenerate the corpus (`eval/generators/gen.ts`). If
+you're using the committed `eval/data/world-v1/` shards, you don't need it.
+
+### `bun install` fails with "Cannot find package 'openai'"
+
+The `openai` package is in `package.json` dependencies. Run `bun install`
+to fetch it. This shouldn't happen post-clone if you followed the normal
+setup; see CLAUDE.md troubleshooting.
+
+## Runner failures
+
+### `multi-adapter.ts` times out on hybrid-nograph
+
+hybrid-nograph embeds all 240 pages per run (via `importFromContent`). At
+N=5, that's 5 re-embeddings. Typical wall clock: ~10 minutes.
+
+If you're iterating, use the dev mode:
+```sh
+BRAINBENCH_N=1 bun run eval:run:dev
+```
+
+Or skip embedding-based adapters for focused runs:
+```sh
+bun run eval:run -- --adapter=gbrain-after
+bun run eval:run -- --adapter=ripgrep-bm25
+```
+
+### "hybrid-nograph returned P@5 0.0%"
+
+Likely the adapter is calling `hybridSearch()` on an engine that doesn't
+have chunks/embeddings populated. This shouldn't happen with current code
+— `importFromContent` populates them. If it does happen:
+
+1. Check the adapter uses `importFromContent(engine, slug, content)`,
+   not bare `engine.putPage(...)`. The latter skips chunking.
+2. Check `auto_link` is OFF (the adapter sets it, but if someone edits
+   the engine's default, verify).
+
+### "ripgrep-bm25 crashes on a query"
+
+The adapter has no query-size ceiling by design. If a specific query crashes,
+run it in isolation:
+
+```sh
+# Drop other adapters temporarily and bisect the query list.
+bun run eval:run -- --adapter=ripgrep-bm25
+```
+
+## Query validation failures
+
+### `validateAll()` fails with "temporal verb detected; as_of_date required"
+
+The query text matches the temporal verb regex. Pick one:
+
+1. **The query is actually temporal.** Add `as_of_date: 'corpus-end' |
+   'per-source' | '2024-01-15'` (ISO-8601).
+2. **The query isn't really temporal.** Rephrase to avoid the trigger verb.
+   "Where is Sarah working?" → "Sarah's current employer" (adjective-form
+   doesn't trigger).
+3. **Edge case bug in the regex.** File an issue; the regex lives at
+   `eval/runner/queries/validator.ts:TEMPORAL_VERBS`.
+
+### `validateAll()` fails with "slug does not match 'dir/slug' format"
+
+Gold slugs must be `dir/slug` — e.g. `people/alice-chen`, not just
+`alice-chen` or `people/Alice Chen`. Lowercase, hyphens, no spaces.
+
+### `validateAll()` fails with "duplicate id in batch"
+
+Two queries share an `id`. Renumber. Convention:
+- Tier 5 (fuzzy): `q5-NNNN`
+- Tier 5.5 (externally-authored): `q55-NNNN`
+- Scaffolder default: `q-<timestamp-suffix>` (via `eval:query:new`)
+
+## World.html rendering
+
+### "world.html doesn't open automatically"
+
+`eval:world:view` tries `open` (macOS), `xdg-open` (Linux), `start`
+(Windows). If none work:
+
+```sh
+bun run eval:world:render              # generate only
+# then open manually in your browser
+open eval/data/world-v1/world.html    # or xdg-open, start, etc.
+```
+
+### "world.html looks weird / broken"
+
+Regenerate from scratch — shard files might have drifted since last render:
+
+```sh
+rm eval/data/world-v1/world.html
+bun run eval:world:view
+```
+
+### "I see unescaped HTML in world.html"
+
+That's a security regression. Open an issue IMMEDIATELY with the specific
+entity slug. Every string should route through `escapeHtml()` in
+`eval/generators/world-html.ts`.
+
+## Dataset regeneration (advanced)
+
+Don't regenerate unless you know why. The committed corpus is the stable
+baseline everyone benchmarks against. Regenerating produces a DIFFERENT
+dataset (Opus isn't byte-deterministic), which becomes a new version.
+
+If you need to regenerate (e.g. for a v1.2 dataset):
+
+```sh
+# Clean slate
+rm -rf eval/data/world-v1
+# Regenerate (~$3 Opus cost, 30 min)
+bun eval/generators/gen.ts --max 240 --concurrency 6
+# Validate
+bun run eval:type-accuracy
+```
+
+The new dataset should be committed as `eval/data/world-vX.Y/` with a
+new ledger. Don't overwrite `world-v1/` — that's the reproducibility baseline.
+
+## CI failures
+
+### `bun run test:eval` fails on a fresh checkout
+
+```sh
+bun install                   # fetch openai (+ deps)
+bun run test:eval             # retry
+```
+
+If tests still fail, bisect:
+
+```sh
+bun test eval/runner/queries/validator.test.ts         # pure functions
+bun test eval/runner/adapters/ripgrep-bm25.test.ts     # pure functions
+bun test eval/runner/adapters/vector-only.test.ts      # pure functions (cosine math only)
+bun test eval/generators/world-html.test.ts            # HTML rendering + XSS
+```
+
+One of these should fail deterministically — report it.