diff --git a/.github/PULL_REQUEST_TEMPLATE/tier5-queries.md b/.github/PULL_REQUEST_TEMPLATE/tier5-queries.md new file mode 100644 index 000000000..c75043597 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE/tier5-queries.md @@ -0,0 +1,39 @@ + + +## Summary + +Submitting **N** Tier 5.5 queries for BrainBench. + +- Author handle: `@your-handle` +- File location: `eval/external-authors/your-handle/queries.json` +- Queries authored fresh (not copy-pasted from a model output) +- Slugs verified against `eval/data/world-v1/` (via `bun run eval:world:view`) + +## Checklist + +- [ ] `bun run eval:query:validate eval/external-authors/your-handle/queries.json` passes +- [ ] At least 20 queries +- [ ] Each query has either `gold.relevant` (with real slugs) or `gold.expected_abstention: true` +- [ ] Temporal queries have `as_of_date` set (`corpus-end` | `per-source` | ISO-8601) +- [ ] Phrasing is varied (not all the same template) +- [ ] `author` field matches my handle + +## Phrasing variety (optional self-audit) + +Tick the styles represented in your batch: + +- [ ] Full sentence questions +- [ ] Fragment-style ("crypto founder Goldman Sachs background") +- [ ] Comparison ("X vs Y") +- [ ] Follow-up ("And who else...") +- [ ] Imperative ("Pull up Alice Davis") +- [ ] Trait-based ("the demanding engineering leader") +- [ ] Abstention bait (answer is "not in corpus") + +## Notes to reviewer + +Anything worth flagging — ambiguous cases, corpus gaps you found, specific +phrasings you were uncertain about. diff --git a/.github/workflows/eval-tests.yml b/.github/workflows/eval-tests.yml new file mode 100644 index 000000000..0e8f5f960 --- /dev/null +++ b/.github/workflows/eval-tests.yml @@ -0,0 +1,40 @@ +name: Eval tests + +on: + push: + branches: [master] + paths: + - 'eval/**' + - 'src/core/link-extraction.ts' + - 'src/core/search/**' + pull_request: + branches: [master] + paths: + - 'eval/**' + - 'src/core/link-extraction.ts' + - 'src/core/search/**' + +permissions: + contents: read + +jobs: + eval-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 + with: + bun-version: latest + - run: bun install + + # Validate the built-in Tier 5 + 5.5 query set. + - name: Validate built-in queries + run: bun run eval:query:validate + + # Pure-function unit tests — zero API calls, fast. + - name: Run eval unit tests + run: bun run test:eval + + # Smoke-test the world.html renderer against the committed corpus. + - name: Render world.html + run: bun run eval:world:render diff --git a/eval/CONTRIBUTING.md b/eval/CONTRIBUTING.md new file mode 100644 index 000000000..43439c306 --- /dev/null +++ b/eval/CONTRIBUTING.md @@ -0,0 +1,150 @@ +# Contributing to BrainBench + +Three contribution paths. Each has a separate workflow. + +## 1. Write Tier 5.5 externally-authored queries + +Tier 5.5 exists to neutralize the "gbrain wrote its own exam" critique. The +queries currently in the repo are AI-authored synthetic placeholders; real +outside researcher submissions supersede them. + +### Workflow + +```sh +# Step 1. Understand the canonical world. +bun run eval:world:view +# Browser opens. Click through entities. Note down what's real. + +# Step 2. Scaffold a query. +bun run eval:query:new --tier externally-authored --author "@your-handle" +# Prints a Query template. Save to a file. + +# Step 3. Edit the template. +# - Replace text with your actual question +# - Replace gold.relevant with slug(s) that actually exist +# - If the query has temporal verbs (is/was/were/now/...), set as_of_date +# to "corpus-end", "per-source", or ISO-8601 +# - Fill in tags + +# Step 4. Validate before submitting. +bun run eval:query:validate path/to/your-queries.json + +# Step 5. Submit a PR. +# File location: eval/external-authors//queries.json +# PR template: .github/PULL_REQUEST_TEMPLATE/tier5-queries.md +``` + +### Query-authoring guidelines + +- **Write like you'd naturally ask.** Don't adapt your voice to an "AI + benchmark style." Fragments, typos, comparisons, follow-ups, imperatives + — all welcome. Variety is the value. +- **Gold must be real slugs.** Every slug in `gold.relevant` must exist in + `eval/data/world-v1/`. The validator checks format; you verify existence. +- **Abstention is a valid answer.** If your query has no answer in the + corpus (e.g. you're asking about someone who isn't there), set + `expected_output_type: 'abstention'` and `gold.expected_abstention: true`. +- **Temporal queries need `as_of_date`.** The validator will reject + "Where is Sarah now?" without it. Use `"corpus-end"` for "as of the most + recent data," `"per-source"` for "whatever the cited source says," or a + specific ISO date. +- **Partial answers are OK** if you flag them via `known_failure_modes`. + +### Query quality bar + +We'll merge your PR if: +- `bun run eval:query:validate` passes +- Slugs resolve to real entities +- At least 20 queries (one batch) +- Queries have genuine phrasing variety + +## 2. Submit an external adapter + +The `Adapter` interface is `eval/runner/types.ts`. Three methods: + +```typescript +interface Adapter { + readonly name: string; + init(rawPages: Page[], config: AdapterConfig): Promise; + query(q: Query, state: BrainState): Promise; + snapshot?(state: BrainState): Promise; +} +``` + +### Workflow + +```sh +# Step 1. Create your adapter file. +# eval/runner/adapters/my-adapter.ts + +# Step 2. Write it. +# - import types from '../types.ts' +# - export class MyAdapter implements Adapter { ... } +# - BrainState is opaque to the runner. Internal shape is yours. +# - `rawPages: Page[]` is all you get. Never read from gold/ — the +# runner doesn't give you that path on purpose. + +# Step 3. Write a unit test. +# eval/runner/adapters/my-adapter.test.ts +# Cover at minimum: init, query, deterministic tie-break. + +# Step 4. Wire into multi-adapter.ts. +# import { MyAdapter } from './adapters/my-adapter.ts'; +# const allAdapters: Adapter[] = [ +# ...existing, +# new MyAdapter(), +# ]; + +# Step 5. Test locally. +bun run test:eval +bun run eval:run:dev --adapter=my-adapter + +# Step 6. Open a PR. +``` + +### Adapter quality bar + +- Deterministic over sorted input (stddev=0 across N=5 runs is the + expected default; non-zero is a signal worth understanding) +- `query()` returns rank order — `rank: i + 1`, 1-based, no duplicates +- Tie-breaks documented (e.g. "alphabetical by slug when scores tie") +- No network calls in unit tests (mock any API dependencies) +- Pass `bun run test:eval` + +## 3. Reproduce / verify a published scorecard + +```sh +# Step 1. Check the scorecard's commit hash. +# Reports in docs/benchmarks/ include the gbrain version + commit. + +# Step 2. Pin the same commit. +git checkout + +# Step 3. Run the full benchmark. +bun run eval:run + +# Step 4. Compare to the published scorecard. +# For deterministic adapters, numbers should match exactly. +# For embedding-based adapters, numbers should land within the published +# tolerance bands (mean ± stddev). +``` + +If your numbers drift outside tolerance, file an issue with: +- Your `bun --version` +- Your `uname -sr` +- Your OpenAI model ID (for embedding-model drift) +- A diff of the scorecard + +## Code style + +- Match existing gbrain patterns (hand-rolled where appropriate, no new + deps unless genuinely needed) +- Bun's built-in test runner (`bun:test`), not jest/vitest +- No em dashes in prose (`—`, `–`); use parentheses or sentences +- Commit messages: `feat(eval):`, `fix(eval):`, `docs(eval):`, `test(eval):` + +## Contributors + +See `eval/CREDITS.md` for the full list. All Tier 5.5 external-author +submissions credited there + in the scorecard. Synthetic placeholders are +labeled `synthetic-outsider-v1`. diff --git a/eval/CREDITS.md b/eval/CREDITS.md new file mode 100644 index 000000000..b2832435d --- /dev/null +++ b/eval/CREDITS.md @@ -0,0 +1,47 @@ +# BrainBench credits + +## Core team + +- **garrytan** — BrainBench v1 + v1.1 architecture, adapter interface, + extraction regex residuals (v0.10.5), multi-axis type-accuracy runner +- **Claude Opus 4.7** — pair programming, test coverage, documentation + +## External query authors (Tier 5.5) + +No human external authors yet. The Tier 5.5 query set currently comprises +50 synthetic queries labeled `author: "synthetic-outsider-v1"` as a +placeholder. Real submissions via `eval/external-authors//queries.json` +PRs supersede synthetic entries. + +**Want to be credited here?** See `eval/CONTRIBUTING.md`. + +## External adapters + +No third-party adapters yet. The shipping adapter set: + +- `gbrain-after` — gbrain v0.10.3+ (internal; the system under test) +- `hybrid-nograph` — gbrain hybrid search with graph layer disabled + (internal comparator; closest apples-to-apples to `gbrain-after`) +- `ripgrep-bm25` — classic IR baseline built in an afternoon +- `vector-only` — commodity vector RAG, same embedder as gbrain + +Third-party submissions (mem0, supermemory, Letta, Cognee, etc.) via +`eval/runner/adapters/.ts` PRs. See `eval/CONTRIBUTING.md` for +the adapter interface and submission flow. + +## Data + +- Corpus generator: Claude Opus +- Canonical world: `eval/data/world-v1/` (committed, 240 entities) +- Generation cost: ~$3.14 USD (one-time) + +## Inspiration + +- **SWE-bench** — taught us that a benchmark's credibility comes from real + baselines, not from the authoring team saying nice things about their + own stack +- **Codex** — cold-read critique that "this isn't a standard, it's an + internal test" drove the Phase 2 external-baselines work that became + the headline of this PR +- **MTEB** — embedding-model reproducibility card pattern; we copy the + "pin every version in every scorecard" discipline diff --git a/eval/README.md b/eval/README.md new file mode 100644 index 000000000..6113e759a --- /dev/null +++ b/eval/README.md @@ -0,0 +1,126 @@ +# BrainBench + +Public benchmark for personal knowledge brain agent stacks. Ships 4 adapter +configurations scored side-by-side on a 240-page rich-prose fictional corpus +(`twin-amara`). Measures retrieval, extraction quality, and per-link-type +accuracy. + +**What this answers:** "Does the knowledge graph layer do useful work, or is +gbrain just a thin wrapper over vector+keyword hybrid?" Headline: gbrain +beats the closest external baseline (hybrid-without-graph, same embedder, +same chunking) by **+31 points P@5**. The graph layer is load-bearing. + +## 5-minute quickstart + +```sh +# 1. Run the full benchmark (4 adapters × 5 runs, ~15 min wall clock) +bun run eval:run + +# 2. Fast iteration (N=1 single run) +bun run eval:run:dev + +# 3. Just the type-accuracy report +bun run eval:type-accuracy + +# 4. Explore the canonical world (contributor-facing UI) +bun run eval:world:view +``` + +## What's in the box + +``` +eval/ +├── data/world-v1/ Canonical world (committed). 240 sharded JSON files. +│ One file per entity + _ledger.json metadata. +├── generators/ +│ ├── gen.ts Opus-backed corpus generator (run once; output cached) +│ ├── world.ts World-schema scaffolder +│ └── world-html.ts World explorer HTML renderer (XSS-safe) +├── runner/ +│ ├── multi-adapter.ts 4-adapter side-by-side scorer (N=5) +│ ├── type-accuracy.ts Per-link-type accuracy vs gold from _facts +│ ├── before-after.ts Original v1 BEFORE/AFTER retrieval run +│ ├── types.ts Adapter, Page, Query, RankedDoc interfaces +│ ├── adapters/ +│ │ ├── ripgrep-bm25.ts EXT-1: classic IR baseline (BM25 over grep hits) +│ │ ├── vector-only.ts EXT-2: pure cosine similarity, same embedder +│ │ └── hybrid-nograph.ts EXT-3: gbrain hybrid with graph disabled +│ └── queries/ +│ ├── tier5-fuzzy.ts 30 vague-recall queries (hand-authored) +│ ├── tier5_5-synthetic.ts 50 synthetic outsider queries (AI-authored, labeled) +│ ├── validator.ts Query schema enforcement (temporal as_of_date rule) +│ └── index.ts Aggregator + validateAll() +├── cli/ +│ ├── world-view.ts Render + open world.html +│ ├── query-validate.ts Validate a Query[] file +│ └── query-new.ts Scaffold a Query template +└── reports/ Benchmark scorecards (gitignored) +``` + +## Three contributor paths + +### Path 1: Reproduce a published scorecard + +```sh +# 1. Check out the specific gbrain commit referenced in the scorecard +git checkout +# 2. Run the full benchmark +bun run eval:run +# 3. Compare your numbers to the scorecard. Deterministic adapters should +# match exactly. Embedding-based adapters should land within tolerance bands. +``` + +### Path 2: Submit a new external adapter + +See `CONTRIBUTING.md` for the adapter submission flow. Short version: + +1. Implement `eval/runner/adapters/.ts` conforming to the + `Adapter` interface in `eval/runner/types.ts`. +2. Add a unit test file alongside. +3. Wire your adapter into `eval/runner/multi-adapter.ts` (one line). +4. `bun run eval:run:dev` to verify. +5. Open a PR. + +### Path 3: Write Tier 5.5 externally-authored queries + +The T5.5 queries currently in the repo are AI-authored (`author: +"synthetic-outsider-v1"`) as a placeholder. Real outside researchers should: + +1. `bun run eval:world:view` to understand the canonical world +2. `bun run eval:query:new --tier externally-authored --author "@your-handle"` +3. Edit the scaffolded template with a real query + gold slugs +4. `bun run eval:query:validate path/to/your.json` +5. Submit via `eval/external-authors//queries.json` in a PR + +See `CONTRIBUTING.md` for the query-submission template. + +## Methodology one-pager + +- **Corpus:** 240 Opus-generated fictional biographical pages. Fixed, + committed, zero private data. Reproducibility baseline for any run. +- **Gold:** Each page's `_facts` metadata defines canonical relationships. + The scorer never shows `_facts` to the adapters — **raw pages only** + cross the ingestion boundary (structural enforcement in `Adapter.init`). +- **Metrics:** P@5 and R@5 on relational queries (145 canonical from + `_facts`, 80 tier-5 + tier-5.5). Type accuracy on extracted edges + (`eval/runner/type-accuracy.ts`). +- **N=5 runs per adapter** with page-order shuffle (seeded LCG; runs are + reproducible). Stddev surfaces order-dependent adapter bugs. Deterministic + adapters correctly show stddev=0. +- **Temporal queries** require explicit `as_of_date` (validated at query + authoring time; rejected at load if a temporal verb is present without it). + +## Adapter scorecard (most recent, N=5) + +See `docs/benchmarks/2026-04-18-brainbench-v1.md` for the full report. +Quick summary from `bun run eval:run`: + +| Adapter | P@5 | R@5 | +|-----------------|--------|--------| +| gbrain-after | 49.1% | 97.9% | +| hybrid-nograph | 17.8% | 65.1% | +| ripgrep-bm25 | 17.1% | 62.4% | +| vector-only | 10.8% | 40.7% | + +The graph layer beats vector+keyword hybrid on relational queries by ~31 +points; hybrid-without-graph barely edges BM25. That's the story. diff --git a/eval/RUNBOOK.md b/eval/RUNBOOK.md new file mode 100644 index 000000000..6216d2da4 --- /dev/null +++ b/eval/RUNBOOK.md @@ -0,0 +1,161 @@ +# BrainBench runbook + +Operational troubleshooting for the most common failures. One fix per entry. + +## Generation failures + +### "OPENAI_API_KEY environment variable is missing" + +The embedding adapter (`vector-only`) and any run of `eval/generators/gen.ts` +calls the OpenAI API. You need an API key. + +```sh +export OPENAI_API_KEY=sk-proj-... +# or source from a dotenv file +source ~/.zshrc # if the key is in your shell profile +bun run eval:run +``` + +### "ANTHROPIC_API_KEY environment variable is missing" + +Only needed if you regenerate the corpus (`eval/generators/gen.ts`). If +you're using the committed `eval/data/world-v1/` shards, you don't need it. + +### `bun install` fails with "Cannot find package 'openai'" + +The `openai` package is in `package.json` dependencies. Run `bun install` +to fetch it. This shouldn't happen post-clone if you followed the normal +setup; see CLAUDE.md troubleshooting. + +## Runner failures + +### `multi-adapter.ts` times out on hybrid-nograph + +hybrid-nograph embeds all 240 pages per run (via `importFromContent`). At +N=5, that's 5 re-embeddings. Typical wall clock: ~10 minutes. + +If you're iterating, use the dev mode: +```sh +BRAINBENCH_N=1 bun run eval:run:dev +``` + +Or skip embedding-based adapters for focused runs: +```sh +bun run eval:run -- --adapter=gbrain-after +bun run eval:run -- --adapter=ripgrep-bm25 +``` + +### "hybrid-nograph returned P@5 0.0%" + +Likely the adapter is calling `hybridSearch()` on an engine that doesn't +have chunks/embeddings populated. This shouldn't happen with current code +— `importFromContent` populates them. If it does happen: + +1. Check the adapter uses `importFromContent(engine, slug, content)`, + not bare `engine.putPage(...)`. The latter skips chunking. +2. Check `auto_link` is OFF (the adapter sets it, but if someone edits + the engine's default, verify). + +### "ripgrep-bm25 crashes on a query" + +The adapter has no query-size ceiling by design. If a specific query crashes, +run it in isolation: + +```sh +# Drop other adapters temporarily and bisect the query list. +bun run eval:run -- --adapter=ripgrep-bm25 +``` + +## Query validation failures + +### `validateAll()` fails with "temporal verb detected; as_of_date required" + +The query text matches the temporal verb regex. Pick one: + +1. **The query is actually temporal.** Add `as_of_date: 'corpus-end' | + 'per-source' | '2024-01-15'` (ISO-8601). +2. **The query isn't really temporal.** Rephrase to avoid the trigger verb. + "Where is Sarah working?" → "Sarah's current employer" (adjective-form + doesn't trigger). +3. **Edge case bug in the regex.** File an issue; the regex lives at + `eval/runner/queries/validator.ts:TEMPORAL_VERBS`. + +### `validateAll()` fails with "slug does not match 'dir/slug' format" + +Gold slugs must be `dir/slug` — e.g. `people/alice-chen`, not just +`alice-chen` or `people/Alice Chen`. Lowercase, hyphens, no spaces. + +### `validateAll()` fails with "duplicate id in batch" + +Two queries share an `id`. Renumber. Convention: +- Tier 5 (fuzzy): `q5-NNNN` +- Tier 5.5 (externally-authored): `q55-NNNN` +- Scaffolder default: `q-` (via `eval:query:new`) + +## World.html rendering + +### "world.html doesn't open automatically" + +`eval:world:view` tries `open` (macOS), `xdg-open` (Linux), `start` +(Windows). If none work: + +```sh +bun run eval:world:render # generate only +# then open manually in your browser +open eval/data/world-v1/world.html # or xdg-open, start, etc. +``` + +### "world.html looks weird / broken" + +Regenerate from scratch — shard files might have drifted since last render: + +```sh +rm eval/data/world-v1/world.html +bun run eval:world:view +``` + +### "I see unescaped HTML in world.html" + +That's a security regression. Open an issue IMMEDIATELY with the specific +entity slug. Every string should route through `escapeHtml()` in +`eval/generators/world-html.ts`. + +## Dataset regeneration (advanced) + +Don't regenerate unless you know why. The committed corpus is the stable +baseline everyone benchmarks against. Regenerating produces a DIFFERENT +dataset (Opus isn't byte-deterministic), which becomes a new version. + +If you need to regenerate (e.g. for a v1.2 dataset): + +```sh +# Clean slate +rm -rf eval/data/world-v1 +# Regenerate (~$3 Opus cost, 30 min) +bun eval/generators/gen.ts --max 240 --concurrency 6 +# Validate +bun run eval:type-accuracy +``` + +The new dataset should be committed as `eval/data/world-vX.Y/` with a +new ledger. Don't overwrite `world-v1/` — that's the reproducibility baseline. + +## CI failures + +### `bun run test:eval` fails on a fresh checkout + +```sh +bun install # fetch openai (+ deps) +bun run test:eval # retry +``` + +If tests still fail, bisect: + +```sh +bun test eval/runner/queries/validator.test.ts # pure functions +bun test eval/runner/adapters/ripgrep-bm25.test.ts # pure functions +bun test eval/runner/adapters/vector-only.test.ts # pure functions (cosine math only) +bun test eval/generators/world-html.test.ts # HTML rendering + XSS +``` + +One of these should fail deterministically — report it.