From 679a3f9a3c9d2112be01385be2a3e6e1bd6fa616 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sun, 19 Apr 2026 08:34:28 +0800 Subject: [PATCH] docs(bench): 2026-04-19 gbrain v0.11.1 vs v0.12.1 regression comparison Runs the same eval harness against two gbrain src/ trees on the same 240-page corpus and 145 queries. Patches the v0.11 copy's gbrain-after adapter to use getLinks/getBacklinks (v0.11 has no traversePaths) with identical direction+linkType semantics. gbrain-after P@5 22.1% -> 49.1% (+27 pts); R@5 54.6% -> 97.9% (+43 pts); correct-in-top-5 99 -> 248 (+149). hybrid-nograph flat at 17.8% / 65.1% on both (v0.12 didn't touch hybridSearch / chunking). Driver is extraction quality, not graph presence: v0.12 emits 499 typed links (v0.11: 136, x3.7) and 2,208 timeline entries (v0.11: 27, x82) on the same 240 pages. Sharpens the April-18 "graph layer does the work" claim -- on v0.11 that architecture only beat hybrid-nograph by 4.3 points; the 31-point lead in the multi-adapter scorecard comes from graph + high-quality extract in combination. --- .../2026-04-19-brainbench-v0_11-vs-v0_12.md | 133 ++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 docs/benchmarks/2026-04-19-brainbench-v0_11-vs-v0_12.md diff --git a/docs/benchmarks/2026-04-19-brainbench-v0_11-vs-v0_12.md b/docs/benchmarks/2026-04-19-brainbench-v0_11-vs-v0_12.md new file mode 100644 index 000000000..3bcc3c3fa --- /dev/null +++ b/docs/benchmarks/2026-04-19-brainbench-v0_11-vs-v0_12.md @@ -0,0 +1,133 @@ +# BrainBench — gbrain v0.11.1 vs v0.12.1 (2026-04-19) + +Historical regression comparison. Same harness, same corpus, same 145 queries +— only the gbrain `src/` tree varies. Answers the question "did the v0.12 work +make retrieval better, or are the external-adapter numbers the whole story?" + +**Short answer:** v0.12.1 moves gbrain-after from **P@5 22.1% → 49.1%** and +**R@5 54.6% → 97.9%** on identical inputs. The v0.12 extract upgrades alone +explain most of the multi-adapter gap. + +## Setup + +| Slot | SHA | Dated | Version label | +|-----------------|-----------|--------------|----------------| +| BEFORE | `d861336` | 2026-04-18 | v0.11.1 (Minions + canonical migration) | +| AFTER (HEAD) | `b81373d` | 2026-04-19 | v0.12.1 base + eval harness Phase 3 | + +Method: +1. `git worktree add ../gbrain-eval-v0.11 d861336` — old `src/` tree in isolation +2. Copy current `eval/` (harness, corpus, queries) into the worktree so both + runs score the identical benchmark +3. Patch the worktree's `gbrain-after` adapter to call `getLinks`/`getBacklinks` + (v0.11 graph API) with the same linkType filter + direction semantics as + `traversePaths` (v0.12). Same ranking logic, different underlying primitives. +4. Run `bun eval/runner/multi-adapter.ts --adapter=gbrain-after` at N=5 on both. + +The external baselines (`ripgrep-bm25`, `vector-only`) share no code with +gbrain's `src/`, so their numbers are invariant across the two SHAs. Included +below for context only. + +## Headline + +| Adapter (config) | BEFORE v0.11.1 | AFTER v0.12.1 | Δ | +|-------------------------|----------------|----------------|---------------| +| **gbrain-after — P@5** | 22.1% | **49.1%** | **+27.0 pts** | +| **gbrain-after — R@5** | 54.6% | **97.9%** | **+43.3 pts** | +| Correct in top-5 (run 1)| 99 / 261 | **248 / 261** | **+149** | +| hybrid-nograph — P@5 | 17.8% | 17.8% | — | +| hybrid-nograph — R@5 | 65.1% | 65.1% | — | + +Stddev = 0 on both versions — both adapter codepaths are deterministic over +ingestion order. The entire movement is on `gbrain-after`; `hybrid-nograph` +holds flat because v0.12 didn't change `hybridSearch`, chunking, or embedding. + +## Where the gain came from + +`runExtract` is the hinge. Same 240 raw pages in, very different graph out: + +| What got extracted | v0.11.1 | v0.12.1 | Δ | +|--------------------------|-------------|-------------|------------| +| Pages with extractable links | 124 / 240 | 240 / 240 | +116 pages | +| Typed links created | 136 | 499 | **×3.7** | +| Timeline entries created | 27 | 2,208 | **×82** | + +Three shipped fixes account for the jump (all on master between the two SHAs): +1. **`inferLinkType` rewrite** (PR #188 five-part patch) — `invested_in`, + `works_at`, `founded`, `advises` regexes extended to the narrative verbs + Opus-generated prose actually uses ("led the Series A", "early investor", + "the founder", "joined as partner"). Context window 80 → 240 chars. +2. **Auto-link on `put_page`** (v0.12.0) — typed edges get extracted on every + write instead of only when the user runs `extract` manually. +3. **Timeline extraction in `extract --source db`** (v0.12.0) — walks the + whole brain, pulls dated lines into structured entries. v0.11 only did + this on filesystem sync, so DB-only ingestion paths (like this benchmark) + saw almost no timeline data. + +## What this means for the multi-adapter scorecard + +The April-18 multi-adapter scorecard shows gbrain-after beating +hybrid-nograph by 31 points P@5. This comparison explains the shape of that +gap: on v0.11.1 the same architecture only beats hybrid-nograph by **4.3 +points P@5** (22.1% vs 17.8%). The 27-point extra lift came from v0.12's +extract quality, not the graph layer being present vs absent. + +That's a useful refinement of the "the graph layer does the work" claim from +the April-18 benchmark. Sharper version: + +> **The typed-edge graph + high-quality extraction together do the work.** +> Either piece alone only moves the needle a few points. Both pieces in +> combination account for the +31 P@5 gap. + +## Reproducibility + +```sh +# From providence/, current HEAD (b81373d) +bun run eval:run --adapter=gbrain-after +# gbrain-after N=5: P@5 49.1% ±0, R@5 97.9% ±0 + +# Historical side +git worktree add -f ../gbrain-eval-v0.11 d861336 +cp -r eval ../gbrain-eval-v0.11/ +ln -s $PWD/node_modules ../gbrain-eval-v0.11/node_modules +# Patch: gbrain-after uses getLinks/getBacklinks instead of traversePaths +# (v0.11 doesn't have traversePaths). Same direction + linkType filter +# semantics, different primitive. See the perl one-liner in the +# session commit message for the exact diff. +cd ../gbrain-eval-v0.11 +bun eval/runner/multi-adapter.ts --adapter=gbrain-after +# gbrain-after N=5: P@5 22.1% ±0, R@5 54.6% ±0 +``` + +## Methodology notes + +- The v0.11 shim swaps `traversePaths(seed, {depth:1, direction, linkType})` + for `getLinks(seed)` / `getBacklinks(seed)` filtered in-memory by + `link_type`. At depth=1 this is semantically identical; it would diverge if + the query asked for depth>=2 (none here do). So the reported delta is + attributable to gbrain's extraction + storage, not to differences in how + the adapter interprets the graph at query time. +- External baselines would be identical on both SHAs by construction. + Re-running them adds no signal. If we later add a baseline that shares + gbrain code (a hybrid variant, say), we'd need to re-run it on both sides. +- `pages with extractable links` is the count `extract --source db` logs + after walking the brain. On v0.11 the filtering was narrower, so only + 124/240 pages contributed any typed edge. On v0.12 every page contributes + at least one. +- The exit-99 fix on the multi-adapter runner (teardown of PGLite engines) + was applied to both sides before running, so neither run spuriously + returns a failing status to CI. + +## What this does not test + +- **Other retrieval-adjacent v0.12 work** (sync quality, publish, lint, + integrations). BrainBench is scoped to retrieval. Per-feature tests still + live in `test/`. +- **Real prose beyond the 240-page fictional corpus.** The extract regex + wins on Opus-generated biographical prose. Real brain pages have their + own vocabulary quirks — future work is corpus diversity (tracked in + `eval/README.md` three-contributor-paths section). +- **Wall-clock or token cost.** v0.12's extract is slightly slower + (auto-link on every put_page, 2,208 timeline entries vs 27), but we + haven't benchmarked the difference. If that ever matters for autopilot, + it needs a separate pass.