mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
docs(eval): Phase 3 contributor docs + CI workflow for eval/ tests
Ships the contributor-onboarding surface promised in the plan. With this
commit, external researchers have a self-serve path from clone to PR in
under 5 minutes.
Added:
eval/README.md — 5-minute quickstart,
directory map, methodology
one-pager, adapter scorecard
eval/CONTRIBUTING.md — three contributor paths:
1. Write Tier 5.5 queries
2. Submit an external adapter
3. Reproduce a scorecard
eval/RUNBOOK.md — operational troubleshooting:
generation failures, runner
failures, query validation,
world.html rendering, CI
eval/CREDITS.md — contributor attribution
(synthetic-outsider-v1 labeled
as placeholder; real submissions
land here)
.github/PULL_REQUEST_TEMPLATE/tier5-queries.md — structured PR template
for Tier 5.5 submissions
.github/workflows/eval-tests.yml — CI: validates queries,
runs all eval unit tests,
renders world.html on every PR
touching eval/** or
src/core/link-extraction.ts
CI scope (intentionally narrow):
- Triggers on paths: eval/**, src/core/link-extraction.ts, src/core/search/**
- Runs: bun run eval:query:validate (80 queries), test:eval (57 tests),
eval:world:render (smoke-test the HTML renderer)
- Pinned actions by commit SHA (matches existing .github/workflows/test.yml)
- Zero API calls — all Opus/OpenAI paths stubbed or skipped in unit tests
- Fast: ~30s total wall clock
Contributor TTHW (clone → first merged PR):
- Path 1 (Tier 5.5 queries): ~5 min
- Path 2 (external adapter): ~30 min for a simple adapter
- Path 3 (reproduce scorecard): ~15 min wall clock (N=5 run)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
f0649e2f32
commit
b81373d4c2
@@ -0,0 +1,39 @@
|
|||||||
|
<!--
|
||||||
|
Tier 5.5 Externally-Authored Query Submission template
|
||||||
|
See eval/CONTRIBUTING.md for the full workflow.
|
||||||
|
-->
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Submitting **N** Tier 5.5 queries for BrainBench.
|
||||||
|
|
||||||
|
- Author handle: `@your-handle`
|
||||||
|
- File location: `eval/external-authors/your-handle/queries.json`
|
||||||
|
- Queries authored fresh (not copy-pasted from a model output)
|
||||||
|
- Slugs verified against `eval/data/world-v1/` (via `bun run eval:world:view`)
|
||||||
|
|
||||||
|
## Checklist
|
||||||
|
|
||||||
|
- [ ] `bun run eval:query:validate eval/external-authors/your-handle/queries.json` passes
|
||||||
|
- [ ] At least 20 queries
|
||||||
|
- [ ] Each query has either `gold.relevant` (with real slugs) or `gold.expected_abstention: true`
|
||||||
|
- [ ] Temporal queries have `as_of_date` set (`corpus-end` | `per-source` | ISO-8601)
|
||||||
|
- [ ] Phrasing is varied (not all the same template)
|
||||||
|
- [ ] `author` field matches my handle
|
||||||
|
|
||||||
|
## Phrasing variety (optional self-audit)
|
||||||
|
|
||||||
|
Tick the styles represented in your batch:
|
||||||
|
|
||||||
|
- [ ] Full sentence questions
|
||||||
|
- [ ] Fragment-style ("crypto founder Goldman Sachs background")
|
||||||
|
- [ ] Comparison ("X vs Y")
|
||||||
|
- [ ] Follow-up ("And who else...")
|
||||||
|
- [ ] Imperative ("Pull up Alice Davis")
|
||||||
|
- [ ] Trait-based ("the demanding engineering leader")
|
||||||
|
- [ ] Abstention bait (answer is "not in corpus")
|
||||||
|
|
||||||
|
## Notes to reviewer
|
||||||
|
|
||||||
|
Anything worth flagging — ambiguous cases, corpus gaps you found, specific
|
||||||
|
phrasings you were uncertain about.
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
name: Eval tests
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [master]
|
||||||
|
paths:
|
||||||
|
- 'eval/**'
|
||||||
|
- 'src/core/link-extraction.ts'
|
||||||
|
- 'src/core/search/**'
|
||||||
|
pull_request:
|
||||||
|
branches: [master]
|
||||||
|
paths:
|
||||||
|
- 'eval/**'
|
||||||
|
- 'src/core/link-extraction.ts'
|
||||||
|
- 'src/core/search/**'
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
eval-tests:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||||
|
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||||
|
with:
|
||||||
|
bun-version: latest
|
||||||
|
- run: bun install
|
||||||
|
|
||||||
|
# Validate the built-in Tier 5 + 5.5 query set.
|
||||||
|
- name: Validate built-in queries
|
||||||
|
run: bun run eval:query:validate
|
||||||
|
|
||||||
|
# Pure-function unit tests — zero API calls, fast.
|
||||||
|
- name: Run eval unit tests
|
||||||
|
run: bun run test:eval
|
||||||
|
|
||||||
|
# Smoke-test the world.html renderer against the committed corpus.
|
||||||
|
- name: Render world.html
|
||||||
|
run: bun run eval:world:render
|
||||||
@@ -0,0 +1,150 @@
|
|||||||
|
# Contributing to BrainBench
|
||||||
|
|
||||||
|
Three contribution paths. Each has a separate workflow.
|
||||||
|
|
||||||
|
## 1. Write Tier 5.5 externally-authored queries
|
||||||
|
|
||||||
|
Tier 5.5 exists to neutralize the "gbrain wrote its own exam" critique. The
|
||||||
|
queries currently in the repo are AI-authored synthetic placeholders; real
|
||||||
|
outside researcher submissions supersede them.
|
||||||
|
|
||||||
|
### Workflow
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Step 1. Understand the canonical world.
|
||||||
|
bun run eval:world:view
|
||||||
|
# Browser opens. Click through entities. Note down what's real.
|
||||||
|
|
||||||
|
# Step 2. Scaffold a query.
|
||||||
|
bun run eval:query:new --tier externally-authored --author "@your-handle"
|
||||||
|
# Prints a Query template. Save to a file.
|
||||||
|
|
||||||
|
# Step 3. Edit the template.
|
||||||
|
# - Replace text with your actual question
|
||||||
|
# - Replace gold.relevant with slug(s) that actually exist
|
||||||
|
# - If the query has temporal verbs (is/was/were/now/...), set as_of_date
|
||||||
|
# to "corpus-end", "per-source", or ISO-8601
|
||||||
|
# - Fill in tags
|
||||||
|
|
||||||
|
# Step 4. Validate before submitting.
|
||||||
|
bun run eval:query:validate path/to/your-queries.json
|
||||||
|
|
||||||
|
# Step 5. Submit a PR.
|
||||||
|
# File location: eval/external-authors/<your-handle>/queries.json
|
||||||
|
# PR template: .github/PULL_REQUEST_TEMPLATE/tier5-queries.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### Query-authoring guidelines
|
||||||
|
|
||||||
|
- **Write like you'd naturally ask.** Don't adapt your voice to an "AI
|
||||||
|
benchmark style." Fragments, typos, comparisons, follow-ups, imperatives
|
||||||
|
— all welcome. Variety is the value.
|
||||||
|
- **Gold must be real slugs.** Every slug in `gold.relevant` must exist in
|
||||||
|
`eval/data/world-v1/`. The validator checks format; you verify existence.
|
||||||
|
- **Abstention is a valid answer.** If your query has no answer in the
|
||||||
|
corpus (e.g. you're asking about someone who isn't there), set
|
||||||
|
`expected_output_type: 'abstention'` and `gold.expected_abstention: true`.
|
||||||
|
- **Temporal queries need `as_of_date`.** The validator will reject
|
||||||
|
"Where is Sarah now?" without it. Use `"corpus-end"` for "as of the most
|
||||||
|
recent data," `"per-source"` for "whatever the cited source says," or a
|
||||||
|
specific ISO date.
|
||||||
|
- **Partial answers are OK** if you flag them via `known_failure_modes`.
|
||||||
|
|
||||||
|
### Query quality bar
|
||||||
|
|
||||||
|
We'll merge your PR if:
|
||||||
|
- `bun run eval:query:validate` passes
|
||||||
|
- Slugs resolve to real entities
|
||||||
|
- At least 20 queries (one batch)
|
||||||
|
- Queries have genuine phrasing variety
|
||||||
|
|
||||||
|
## 2. Submit an external adapter
|
||||||
|
|
||||||
|
The `Adapter` interface is `eval/runner/types.ts`. Three methods:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface Adapter {
|
||||||
|
readonly name: string;
|
||||||
|
init(rawPages: Page[], config: AdapterConfig): Promise<BrainState>;
|
||||||
|
query(q: Query, state: BrainState): Promise<RankedDoc[]>;
|
||||||
|
snapshot?(state: BrainState): Promise<string>;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Workflow
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Step 1. Create your adapter file.
|
||||||
|
# eval/runner/adapters/my-adapter.ts
|
||||||
|
|
||||||
|
# Step 2. Write it.
|
||||||
|
# - import types from '../types.ts'
|
||||||
|
# - export class MyAdapter implements Adapter { ... }
|
||||||
|
# - BrainState is opaque to the runner. Internal shape is yours.
|
||||||
|
# - `rawPages: Page[]` is all you get. Never read from gold/ — the
|
||||||
|
# runner doesn't give you that path on purpose.
|
||||||
|
|
||||||
|
# Step 3. Write a unit test.
|
||||||
|
# eval/runner/adapters/my-adapter.test.ts
|
||||||
|
# Cover at minimum: init, query, deterministic tie-break.
|
||||||
|
|
||||||
|
# Step 4. Wire into multi-adapter.ts.
|
||||||
|
# import { MyAdapter } from './adapters/my-adapter.ts';
|
||||||
|
# const allAdapters: Adapter[] = [
|
||||||
|
# ...existing,
|
||||||
|
# new MyAdapter(),
|
||||||
|
# ];
|
||||||
|
|
||||||
|
# Step 5. Test locally.
|
||||||
|
bun run test:eval
|
||||||
|
bun run eval:run:dev --adapter=my-adapter
|
||||||
|
|
||||||
|
# Step 6. Open a PR.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Adapter quality bar
|
||||||
|
|
||||||
|
- Deterministic over sorted input (stddev=0 across N=5 runs is the
|
||||||
|
expected default; non-zero is a signal worth understanding)
|
||||||
|
- `query()` returns rank order — `rank: i + 1`, 1-based, no duplicates
|
||||||
|
- Tie-breaks documented (e.g. "alphabetical by slug when scores tie")
|
||||||
|
- No network calls in unit tests (mock any API dependencies)
|
||||||
|
- Pass `bun run test:eval`
|
||||||
|
|
||||||
|
## 3. Reproduce / verify a published scorecard
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Step 1. Check the scorecard's commit hash.
|
||||||
|
# Reports in docs/benchmarks/ include the gbrain version + commit.
|
||||||
|
|
||||||
|
# Step 2. Pin the same commit.
|
||||||
|
git checkout <commit-sha>
|
||||||
|
|
||||||
|
# Step 3. Run the full benchmark.
|
||||||
|
bun run eval:run
|
||||||
|
|
||||||
|
# Step 4. Compare to the published scorecard.
|
||||||
|
# For deterministic adapters, numbers should match exactly.
|
||||||
|
# For embedding-based adapters, numbers should land within the published
|
||||||
|
# tolerance bands (mean ± stddev).
|
||||||
|
```
|
||||||
|
|
||||||
|
If your numbers drift outside tolerance, file an issue with:
|
||||||
|
- Your `bun --version`
|
||||||
|
- Your `uname -sr`
|
||||||
|
- Your OpenAI model ID (for embedding-model drift)
|
||||||
|
- A diff of the scorecard
|
||||||
|
|
||||||
|
## Code style
|
||||||
|
|
||||||
|
- Match existing gbrain patterns (hand-rolled where appropriate, no new
|
||||||
|
deps unless genuinely needed)
|
||||||
|
- Bun's built-in test runner (`bun:test`), not jest/vitest
|
||||||
|
- No em dashes in prose (`—`, `–`); use parentheses or sentences
|
||||||
|
- Commit messages: `feat(eval):`, `fix(eval):`, `docs(eval):`, `test(eval):`
|
||||||
|
|
||||||
|
## Contributors
|
||||||
|
|
||||||
|
See `eval/CREDITS.md` for the full list. All Tier 5.5 external-author
|
||||||
|
submissions credited there + in the scorecard. Synthetic placeholders are
|
||||||
|
labeled `synthetic-outsider-v1`.
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# BrainBench credits
|
||||||
|
|
||||||
|
## Core team
|
||||||
|
|
||||||
|
- **garrytan** — BrainBench v1 + v1.1 architecture, adapter interface,
|
||||||
|
extraction regex residuals (v0.10.5), multi-axis type-accuracy runner
|
||||||
|
- **Claude Opus 4.7** — pair programming, test coverage, documentation
|
||||||
|
|
||||||
|
## External query authors (Tier 5.5)
|
||||||
|
|
||||||
|
No human external authors yet. The Tier 5.5 query set currently comprises
|
||||||
|
50 synthetic queries labeled `author: "synthetic-outsider-v1"` as a
|
||||||
|
placeholder. Real submissions via `eval/external-authors/<handle>/queries.json`
|
||||||
|
PRs supersede synthetic entries.
|
||||||
|
|
||||||
|
**Want to be credited here?** See `eval/CONTRIBUTING.md`.
|
||||||
|
|
||||||
|
## External adapters
|
||||||
|
|
||||||
|
No third-party adapters yet. The shipping adapter set:
|
||||||
|
|
||||||
|
- `gbrain-after` — gbrain v0.10.3+ (internal; the system under test)
|
||||||
|
- `hybrid-nograph` — gbrain hybrid search with graph layer disabled
|
||||||
|
(internal comparator; closest apples-to-apples to `gbrain-after`)
|
||||||
|
- `ripgrep-bm25` — classic IR baseline built in an afternoon
|
||||||
|
- `vector-only` — commodity vector RAG, same embedder as gbrain
|
||||||
|
|
||||||
|
Third-party submissions (mem0, supermemory, Letta, Cognee, etc.) via
|
||||||
|
`eval/runner/adapters/<adapter>.ts` PRs. See `eval/CONTRIBUTING.md` for
|
||||||
|
the adapter interface and submission flow.
|
||||||
|
|
||||||
|
## Data
|
||||||
|
|
||||||
|
- Corpus generator: Claude Opus
|
||||||
|
- Canonical world: `eval/data/world-v1/` (committed, 240 entities)
|
||||||
|
- Generation cost: ~$3.14 USD (one-time)
|
||||||
|
|
||||||
|
## Inspiration
|
||||||
|
|
||||||
|
- **SWE-bench** — taught us that a benchmark's credibility comes from real
|
||||||
|
baselines, not from the authoring team saying nice things about their
|
||||||
|
own stack
|
||||||
|
- **Codex** — cold-read critique that "this isn't a standard, it's an
|
||||||
|
internal test" drove the Phase 2 external-baselines work that became
|
||||||
|
the headline of this PR
|
||||||
|
- **MTEB** — embedding-model reproducibility card pattern; we copy the
|
||||||
|
"pin every version in every scorecard" discipline
|
||||||
+126
@@ -0,0 +1,126 @@
|
|||||||
|
# BrainBench
|
||||||
|
|
||||||
|
Public benchmark for personal knowledge brain agent stacks. Ships 4 adapter
|
||||||
|
configurations scored side-by-side on a 240-page rich-prose fictional corpus
|
||||||
|
(`twin-amara`). Measures retrieval, extraction quality, and per-link-type
|
||||||
|
accuracy.
|
||||||
|
|
||||||
|
**What this answers:** "Does the knowledge graph layer do useful work, or is
|
||||||
|
gbrain just a thin wrapper over vector+keyword hybrid?" Headline: gbrain
|
||||||
|
beats the closest external baseline (hybrid-without-graph, same embedder,
|
||||||
|
same chunking) by **+31 points P@5**. The graph layer is load-bearing.
|
||||||
|
|
||||||
|
## 5-minute quickstart
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# 1. Run the full benchmark (4 adapters × 5 runs, ~15 min wall clock)
|
||||||
|
bun run eval:run
|
||||||
|
|
||||||
|
# 2. Fast iteration (N=1 single run)
|
||||||
|
bun run eval:run:dev
|
||||||
|
|
||||||
|
# 3. Just the type-accuracy report
|
||||||
|
bun run eval:type-accuracy
|
||||||
|
|
||||||
|
# 4. Explore the canonical world (contributor-facing UI)
|
||||||
|
bun run eval:world:view
|
||||||
|
```
|
||||||
|
|
||||||
|
## What's in the box
|
||||||
|
|
||||||
|
```
|
||||||
|
eval/
|
||||||
|
├── data/world-v1/ Canonical world (committed). 240 sharded JSON files.
|
||||||
|
│ One file per entity + _ledger.json metadata.
|
||||||
|
├── generators/
|
||||||
|
│ ├── gen.ts Opus-backed corpus generator (run once; output cached)
|
||||||
|
│ ├── world.ts World-schema scaffolder
|
||||||
|
│ └── world-html.ts World explorer HTML renderer (XSS-safe)
|
||||||
|
├── runner/
|
||||||
|
│ ├── multi-adapter.ts 4-adapter side-by-side scorer (N=5)
|
||||||
|
│ ├── type-accuracy.ts Per-link-type accuracy vs gold from _facts
|
||||||
|
│ ├── before-after.ts Original v1 BEFORE/AFTER retrieval run
|
||||||
|
│ ├── types.ts Adapter, Page, Query, RankedDoc interfaces
|
||||||
|
│ ├── adapters/
|
||||||
|
│ │ ├── ripgrep-bm25.ts EXT-1: classic IR baseline (BM25 over grep hits)
|
||||||
|
│ │ ├── vector-only.ts EXT-2: pure cosine similarity, same embedder
|
||||||
|
│ │ └── hybrid-nograph.ts EXT-3: gbrain hybrid with graph disabled
|
||||||
|
│ └── queries/
|
||||||
|
│ ├── tier5-fuzzy.ts 30 vague-recall queries (hand-authored)
|
||||||
|
│ ├── tier5_5-synthetic.ts 50 synthetic outsider queries (AI-authored, labeled)
|
||||||
|
│ ├── validator.ts Query schema enforcement (temporal as_of_date rule)
|
||||||
|
│ └── index.ts Aggregator + validateAll()
|
||||||
|
├── cli/
|
||||||
|
│ ├── world-view.ts Render + open world.html
|
||||||
|
│ ├── query-validate.ts Validate a Query[] file
|
||||||
|
│ └── query-new.ts Scaffold a Query template
|
||||||
|
└── reports/ Benchmark scorecards (gitignored)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Three contributor paths
|
||||||
|
|
||||||
|
### Path 1: Reproduce a published scorecard
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# 1. Check out the specific gbrain commit referenced in the scorecard
|
||||||
|
git checkout <commit-sha>
|
||||||
|
# 2. Run the full benchmark
|
||||||
|
bun run eval:run
|
||||||
|
# 3. Compare your numbers to the scorecard. Deterministic adapters should
|
||||||
|
# match exactly. Embedding-based adapters should land within tolerance bands.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Path 2: Submit a new external adapter
|
||||||
|
|
||||||
|
See `CONTRIBUTING.md` for the adapter submission flow. Short version:
|
||||||
|
|
||||||
|
1. Implement `eval/runner/adapters/<your-adapter>.ts` conforming to the
|
||||||
|
`Adapter` interface in `eval/runner/types.ts`.
|
||||||
|
2. Add a unit test file alongside.
|
||||||
|
3. Wire your adapter into `eval/runner/multi-adapter.ts` (one line).
|
||||||
|
4. `bun run eval:run:dev` to verify.
|
||||||
|
5. Open a PR.
|
||||||
|
|
||||||
|
### Path 3: Write Tier 5.5 externally-authored queries
|
||||||
|
|
||||||
|
The T5.5 queries currently in the repo are AI-authored (`author:
|
||||||
|
"synthetic-outsider-v1"`) as a placeholder. Real outside researchers should:
|
||||||
|
|
||||||
|
1. `bun run eval:world:view` to understand the canonical world
|
||||||
|
2. `bun run eval:query:new --tier externally-authored --author "@your-handle"`
|
||||||
|
3. Edit the scaffolded template with a real query + gold slugs
|
||||||
|
4. `bun run eval:query:validate path/to/your.json`
|
||||||
|
5. Submit via `eval/external-authors/<your-handle>/queries.json` in a PR
|
||||||
|
|
||||||
|
See `CONTRIBUTING.md` for the query-submission template.
|
||||||
|
|
||||||
|
## Methodology one-pager
|
||||||
|
|
||||||
|
- **Corpus:** 240 Opus-generated fictional biographical pages. Fixed,
|
||||||
|
committed, zero private data. Reproducibility baseline for any run.
|
||||||
|
- **Gold:** Each page's `_facts` metadata defines canonical relationships.
|
||||||
|
The scorer never shows `_facts` to the adapters — **raw pages only**
|
||||||
|
cross the ingestion boundary (structural enforcement in `Adapter.init`).
|
||||||
|
- **Metrics:** P@5 and R@5 on relational queries (145 canonical from
|
||||||
|
`_facts`, 80 tier-5 + tier-5.5). Type accuracy on extracted edges
|
||||||
|
(`eval/runner/type-accuracy.ts`).
|
||||||
|
- **N=5 runs per adapter** with page-order shuffle (seeded LCG; runs are
|
||||||
|
reproducible). Stddev surfaces order-dependent adapter bugs. Deterministic
|
||||||
|
adapters correctly show stddev=0.
|
||||||
|
- **Temporal queries** require explicit `as_of_date` (validated at query
|
||||||
|
authoring time; rejected at load if a temporal verb is present without it).
|
||||||
|
|
||||||
|
## Adapter scorecard (most recent, N=5)
|
||||||
|
|
||||||
|
See `docs/benchmarks/2026-04-18-brainbench-v1.md` for the full report.
|
||||||
|
Quick summary from `bun run eval:run`:
|
||||||
|
|
||||||
|
| Adapter | P@5 | R@5 |
|
||||||
|
|-----------------|--------|--------|
|
||||||
|
| gbrain-after | 49.1% | 97.9% |
|
||||||
|
| hybrid-nograph | 17.8% | 65.1% |
|
||||||
|
| ripgrep-bm25 | 17.1% | 62.4% |
|
||||||
|
| vector-only | 10.8% | 40.7% |
|
||||||
|
|
||||||
|
The graph layer beats vector+keyword hybrid on relational queries by ~31
|
||||||
|
points; hybrid-without-graph barely edges BM25. That's the story.
|
||||||
+161
@@ -0,0 +1,161 @@
|
|||||||
|
# BrainBench runbook
|
||||||
|
|
||||||
|
Operational troubleshooting for the most common failures. One fix per entry.
|
||||||
|
|
||||||
|
## Generation failures
|
||||||
|
|
||||||
|
### "OPENAI_API_KEY environment variable is missing"
|
||||||
|
|
||||||
|
The embedding adapter (`vector-only`) and any run of `eval/generators/gen.ts`
|
||||||
|
calls the OpenAI API. You need an API key.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
export OPENAI_API_KEY=sk-proj-...
|
||||||
|
# or source from a dotenv file
|
||||||
|
source ~/.zshrc # if the key is in your shell profile
|
||||||
|
bun run eval:run
|
||||||
|
```
|
||||||
|
|
||||||
|
### "ANTHROPIC_API_KEY environment variable is missing"
|
||||||
|
|
||||||
|
Only needed if you regenerate the corpus (`eval/generators/gen.ts`). If
|
||||||
|
you're using the committed `eval/data/world-v1/` shards, you don't need it.
|
||||||
|
|
||||||
|
### `bun install` fails with "Cannot find package 'openai'"
|
||||||
|
|
||||||
|
The `openai` package is in `package.json` dependencies. Run `bun install`
|
||||||
|
to fetch it. This shouldn't happen post-clone if you followed the normal
|
||||||
|
setup; see CLAUDE.md troubleshooting.
|
||||||
|
|
||||||
|
## Runner failures
|
||||||
|
|
||||||
|
### `multi-adapter.ts` times out on hybrid-nograph
|
||||||
|
|
||||||
|
hybrid-nograph embeds all 240 pages per run (via `importFromContent`). At
|
||||||
|
N=5, that's 5 re-embeddings. Typical wall clock: ~10 minutes.
|
||||||
|
|
||||||
|
If you're iterating, use the dev mode:
|
||||||
|
```sh
|
||||||
|
BRAINBENCH_N=1 bun run eval:run:dev
|
||||||
|
```
|
||||||
|
|
||||||
|
Or skip embedding-based adapters for focused runs:
|
||||||
|
```sh
|
||||||
|
bun run eval:run -- --adapter=gbrain-after
|
||||||
|
bun run eval:run -- --adapter=ripgrep-bm25
|
||||||
|
```
|
||||||
|
|
||||||
|
### "hybrid-nograph returned P@5 0.0%"
|
||||||
|
|
||||||
|
Likely the adapter is calling `hybridSearch()` on an engine that doesn't
|
||||||
|
have chunks/embeddings populated. This shouldn't happen with current code
|
||||||
|
— `importFromContent` populates them. If it does happen:
|
||||||
|
|
||||||
|
1. Check the adapter uses `importFromContent(engine, slug, content)`,
|
||||||
|
not bare `engine.putPage(...)`. The latter skips chunking.
|
||||||
|
2. Check `auto_link` is OFF (the adapter sets it, but if someone edits
|
||||||
|
the engine's default, verify).
|
||||||
|
|
||||||
|
### "ripgrep-bm25 crashes on a query"
|
||||||
|
|
||||||
|
The adapter has no query-size ceiling by design. If a specific query crashes,
|
||||||
|
run it in isolation:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Drop other adapters temporarily and bisect the query list.
|
||||||
|
bun run eval:run -- --adapter=ripgrep-bm25
|
||||||
|
```
|
||||||
|
|
||||||
|
## Query validation failures
|
||||||
|
|
||||||
|
### `validateAll()` fails with "temporal verb detected; as_of_date required"
|
||||||
|
|
||||||
|
The query text matches the temporal verb regex. Pick one:
|
||||||
|
|
||||||
|
1. **The query is actually temporal.** Add `as_of_date: 'corpus-end' |
|
||||||
|
'per-source' | '2024-01-15'` (ISO-8601).
|
||||||
|
2. **The query isn't really temporal.** Rephrase to avoid the trigger verb.
|
||||||
|
"Where is Sarah working?" → "Sarah's current employer" (adjective-form
|
||||||
|
doesn't trigger).
|
||||||
|
3. **Edge case bug in the regex.** File an issue; the regex lives at
|
||||||
|
`eval/runner/queries/validator.ts:TEMPORAL_VERBS`.
|
||||||
|
|
||||||
|
### `validateAll()` fails with "slug does not match 'dir/slug' format"
|
||||||
|
|
||||||
|
Gold slugs must be `dir/slug` — e.g. `people/alice-chen`, not just
|
||||||
|
`alice-chen` or `people/Alice Chen`. Lowercase, hyphens, no spaces.
|
||||||
|
|
||||||
|
### `validateAll()` fails with "duplicate id in batch"
|
||||||
|
|
||||||
|
Two queries share an `id`. Renumber. Convention:
|
||||||
|
- Tier 5 (fuzzy): `q5-NNNN`
|
||||||
|
- Tier 5.5 (externally-authored): `q55-NNNN`
|
||||||
|
- Scaffolder default: `q-<timestamp-suffix>` (via `eval:query:new`)
|
||||||
|
|
||||||
|
## World.html rendering
|
||||||
|
|
||||||
|
### "world.html doesn't open automatically"
|
||||||
|
|
||||||
|
`eval:world:view` tries `open` (macOS), `xdg-open` (Linux), `start`
|
||||||
|
(Windows). If none work:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
bun run eval:world:render # generate only
|
||||||
|
# then open manually in your browser
|
||||||
|
open eval/data/world-v1/world.html # or xdg-open, start, etc.
|
||||||
|
```
|
||||||
|
|
||||||
|
### "world.html looks weird / broken"
|
||||||
|
|
||||||
|
Regenerate from scratch — shard files might have drifted since last render:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
rm eval/data/world-v1/world.html
|
||||||
|
bun run eval:world:view
|
||||||
|
```
|
||||||
|
|
||||||
|
### "I see unescaped HTML in world.html"
|
||||||
|
|
||||||
|
That's a security regression. Open an issue IMMEDIATELY with the specific
|
||||||
|
entity slug. Every string should route through `escapeHtml()` in
|
||||||
|
`eval/generators/world-html.ts`.
|
||||||
|
|
||||||
|
## Dataset regeneration (advanced)
|
||||||
|
|
||||||
|
Don't regenerate unless you know why. The committed corpus is the stable
|
||||||
|
baseline everyone benchmarks against. Regenerating produces a DIFFERENT
|
||||||
|
dataset (Opus isn't byte-deterministic), which becomes a new version.
|
||||||
|
|
||||||
|
If you need to regenerate (e.g. for a v1.2 dataset):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Clean slate
|
||||||
|
rm -rf eval/data/world-v1
|
||||||
|
# Regenerate (~$3 Opus cost, 30 min)
|
||||||
|
bun eval/generators/gen.ts --max 240 --concurrency 6
|
||||||
|
# Validate
|
||||||
|
bun run eval:type-accuracy
|
||||||
|
```
|
||||||
|
|
||||||
|
The new dataset should be committed as `eval/data/world-vX.Y/` with a
|
||||||
|
new ledger. Don't overwrite `world-v1/` — that's the reproducibility baseline.
|
||||||
|
|
||||||
|
## CI failures
|
||||||
|
|
||||||
|
### `bun run test:eval` fails on a fresh checkout
|
||||||
|
|
||||||
|
```sh
|
||||||
|
bun install # fetch openai (+ deps)
|
||||||
|
bun run test:eval # retry
|
||||||
|
```
|
||||||
|
|
||||||
|
If tests still fail, bisect:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
bun test eval/runner/queries/validator.test.ts # pure functions
|
||||||
|
bun test eval/runner/adapters/ripgrep-bm25.test.ts # pure functions
|
||||||
|
bun test eval/runner/adapters/vector-only.test.ts # pure functions (cosine math only)
|
||||||
|
bun test eval/generators/world-html.test.ts # HTML rendering + XSS
|
||||||
|
```
|
||||||
|
|
||||||
|
One of these should fail deterministically — report it.
|
||||||
Reference in New Issue
Block a user