docs(bench): 2026-04-19 gbrain v0.11.1 vs v0.12.1 regression comparison

Runs the same eval harness against two gbrain src/ trees on the same 240-page corpus and 145 queries. Patches the v0.11 copy's gbrain-after adapter to use getLinks/getBacklinks (v0.11 has no traversePaths) with identical direction+linkType semantics. gbrain-after P@5 22.1% -> 49.1% (+27 pts); R@5 54.6% -> 97.9% (+43 pts); correct-in-top-5 99 -> 248 (+149). hybrid-nograph flat at 17.8% / 65.1% on both (v0.12 didn't touch hybridSearch / chunking). Driver is extraction quality, not graph presence: v0.12 emits 499 typed links (v0.11: 136, x3.7) and 2,208 timeline entries (v0.11: 27, x82) on the same 240 pages. Sharpens the April-18 "graph layer does the work" claim -- on v0.11 that architecture only beat hybrid-nograph by 4.3 points; the 31-point lead in the multi-adapter scorecard comes from graph + high-quality extract in combination.
2026-07-31 04:07:52 +00:00 · 2026-04-19 08:34:28 +08:00
parent 4c885d01d9
commit 679a3f9a3c
1 changed files with 133 additions and 0 deletions
@@ -0,0 +1,133 @@
+# BrainBench — gbrain v0.11.1 vs v0.12.1 (2026-04-19)
+
+Historical regression comparison. Same harness, same corpus, same 145 queries
+— only the gbrain `src/` tree varies. Answers the question "did the v0.12 work
+make retrieval better, or are the external-adapter numbers the whole story?"
+
+**Short answer:** v0.12.1 moves gbrain-after from **P@5 22.1% → 49.1%** and
+**R@5 54.6% → 97.9%** on identical inputs. The v0.12 extract upgrades alone
+explain most of the multi-adapter gap.
+
+## Setup
+
+| Slot            | SHA       | Dated        | Version label |
+|-----------------|-----------|--------------|----------------|
+| BEFORE          | `d861336` | 2026-04-18   | v0.11.1 (Minions + canonical migration) |
+| AFTER (HEAD)    | `b81373d` | 2026-04-19   | v0.12.1 base + eval harness Phase 3    |
+
+Method:
+1. `git worktree add ../gbrain-eval-v0.11 d861336` — old `src/` tree in isolation
+2. Copy current `eval/` (harness, corpus, queries) into the worktree so both
+   runs score the identical benchmark
+3. Patch the worktree's `gbrain-after` adapter to call `getLinks`/`getBacklinks`
+   (v0.11 graph API) with the same linkType filter + direction semantics as
+   `traversePaths` (v0.12). Same ranking logic, different underlying primitives.
+4. Run `bun eval/runner/multi-adapter.ts --adapter=gbrain-after` at N=5 on both.
+
+The external baselines (`ripgrep-bm25`, `vector-only`) share no code with
+gbrain's `src/`, so their numbers are invariant across the two SHAs. Included
+below for context only.
+
+## Headline
+
+| Adapter (config)        | BEFORE v0.11.1 | AFTER v0.12.1 | Δ             |
+|-------------------------|----------------|----------------|---------------|
+| **gbrain-after — P@5**  | 22.1%          | **49.1%**      | **+27.0 pts** |
+| **gbrain-after — R@5**  | 54.6%          | **97.9%**      | **+43.3 pts** |
+| Correct in top-5 (run 1)| 99 / 261       | **248 / 261**  | **+149**      |
+| hybrid-nograph — P@5    | 17.8%          | 17.8%          | —             |
+| hybrid-nograph — R@5    | 65.1%          | 65.1%          | —             |
+
+Stddev = 0 on both versions — both adapter codepaths are deterministic over
+ingestion order. The entire movement is on `gbrain-after`; `hybrid-nograph`
+holds flat because v0.12 didn't change `hybridSearch`, chunking, or embedding.
+
+## Where the gain came from
+
+`runExtract` is the hinge. Same 240 raw pages in, very different graph out:
+
+| What got extracted       | v0.11.1     | v0.12.1     | Δ          |
+|--------------------------|-------------|-------------|------------|
+| Pages with extractable links | 124 / 240  | 240 / 240   | +116 pages |
+| Typed links created      | 136         | 499         | **×3.7**   |
+| Timeline entries created | 27          | 2,208       | **×82**    |
+
+Three shipped fixes account for the jump (all on master between the two SHAs):
+1. **`inferLinkType` rewrite** (PR #188 five-part patch) — `invested_in`,
+   `works_at`, `founded`, `advises` regexes extended to the narrative verbs
+   Opus-generated prose actually uses ("led the Series A", "early investor",
+   "the founder", "joined as partner"). Context window 80 → 240 chars.
+2. **Auto-link on `put_page`** (v0.12.0) — typed edges get extracted on every
+   write instead of only when the user runs `extract` manually.
+3. **Timeline extraction in `extract --source db`** (v0.12.0) — walks the
+   whole brain, pulls dated lines into structured entries. v0.11 only did
+   this on filesystem sync, so DB-only ingestion paths (like this benchmark)
+   saw almost no timeline data.
+
+## What this means for the multi-adapter scorecard
+
+The April-18 multi-adapter scorecard shows gbrain-after beating
+hybrid-nograph by 31 points P@5. This comparison explains the shape of that
+gap: on v0.11.1 the same architecture only beats hybrid-nograph by **4.3
+points P@5** (22.1% vs 17.8%). The 27-point extra lift came from v0.12's
+extract quality, not the graph layer being present vs absent.
+
+That's a useful refinement of the "the graph layer does the work" claim from
+the April-18 benchmark. Sharper version:
+
+> **The typed-edge graph + high-quality extraction together do the work.**
+> Either piece alone only moves the needle a few points. Both pieces in
+> combination account for the +31 P@5 gap.
+
+## Reproducibility
+
+```sh
+# From providence/, current HEAD (b81373d)
+bun run eval:run --adapter=gbrain-after
+#   gbrain-after N=5: P@5 49.1% ±0, R@5 97.9% ±0
+
+# Historical side
+git worktree add -f ../gbrain-eval-v0.11 d861336
+cp -r eval ../gbrain-eval-v0.11/
+ln -s $PWD/node_modules ../gbrain-eval-v0.11/node_modules
+# Patch: gbrain-after uses getLinks/getBacklinks instead of traversePaths
+#        (v0.11 doesn't have traversePaths). Same direction + linkType filter
+#        semantics, different primitive. See the perl one-liner in the
+#        session commit message for the exact diff.
+cd ../gbrain-eval-v0.11
+bun eval/runner/multi-adapter.ts --adapter=gbrain-after
+#   gbrain-after N=5: P@5 22.1% ±0, R@5 54.6% ±0
+```
+
+## Methodology notes
+
+- The v0.11 shim swaps `traversePaths(seed, {depth:1, direction, linkType})`
+  for `getLinks(seed)` / `getBacklinks(seed)` filtered in-memory by
+  `link_type`. At depth=1 this is semantically identical; it would diverge if
+  the query asked for depth>=2 (none here do). So the reported delta is
+  attributable to gbrain's extraction + storage, not to differences in how
+  the adapter interprets the graph at query time.
+- External baselines would be identical on both SHAs by construction.
+  Re-running them adds no signal. If we later add a baseline that shares
+  gbrain code (a hybrid variant, say), we'd need to re-run it on both sides.
+- `pages with extractable links` is the count `extract --source db` logs
+  after walking the brain. On v0.11 the filtering was narrower, so only
+  124/240 pages contributed any typed edge. On v0.12 every page contributes
+  at least one.
+- The exit-99 fix on the multi-adapter runner (teardown of PGLite engines)
+  was applied to both sides before running, so neither run spuriously
+  returns a failing status to CI.
+
+## What this does not test
+
+- **Other retrieval-adjacent v0.12 work** (sync quality, publish, lint,
+  integrations). BrainBench is scoped to retrieval. Per-feature tests still
+  live in `test/`.
+- **Real prose beyond the 240-page fictional corpus.** The extract regex
+  wins on Opus-generated biographical prose. Real brain pages have their
+  own vocabulary quirks — future work is corpus diversity (tracked in
+  `eval/README.md` three-contributor-paths section).
+- **Wall-clock or token cost.** v0.12's extract is slightly slower
+  (auto-link on every put_page, 2,208 timeline entries vs 27), but we
+  haven't benchmarked the difference. If that ever matters for autopilot,
+  it needs a separate pass.