mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.35.1.0: embedder shootout prereqs (pricing + gateway export + --resume-from) (#1055)
* docs(designs): 2026-05 embedder shootout eval plan Adds docs/designs/2026_05_EVAL_PLAN.md — the approved plan + 6 Conductor session briefs for the OpenAI vs Voyage vs ZeroEntropy embedder comparison. Why: produce a publishable comparison report for v0.35.x release notes pinning "which embedder wins, and does zerank-2 carry the win for ZeroEntropy" against public LongMemEval + in-house BrainBench. Each session brief is self-contained — repo, branch, commits, verify, ship, deliverable, hand-off. Stewardable one section per Conductor session. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(pricing): add voyage-4-large + zembed-1 to EMBEDDING_PRICING v0.35.0.0 shipped ZeroEntropy zembed-1 + zerank-2 reranker support and expanded the Voyage allow-list to include voyage-4-large. The pricing table missed both, so `gbrain upgrade`'s post-upgrade reembed prompt silently fell back to "estimate unavailable" for users on these models. - voyage:voyage-4-large @ $0.18/MTok (same as voyage-3-large) - zeroentropyai:zembed-1 @ $0.05/MTok New test file pins both entries plus the openai/voyage-3-large baselines, case-insensitive provider matching, bare-model openai-default fallback, table integrity (lowercase providers, finite non-negative prices), and the estimateCostFromChars approximation. 11 cases, 46 expect() calls. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(exports): expose gbrain/ai/gateway with canary test Adds ./ai/gateway to the package.json exports map so external eval consumers (notably gbrain-evals, the sibling repo running the embedder shootout in docs/designs/2026_05_EVAL_PLAN.md) can call configureGateway directly to swap embedding providers per cell. Why: pre-v0.35.1.0, gbrain-evals adapters hardcoded gbrain/embedding, which means every retrieval adapter was OpenAI-only. The newly-exposed gateway lets adapters route through Voyage and ZeroEntropy without forking gbrain or duplicating the recipe wiring. - package.json: add "./ai/gateway" -> "./src/core/ai/gateway.ts" - scripts/check-exports-count.sh: bump expected count 17 -> 18 - test/public-exports.test.ts: add canary pinning configureGateway + embed, bump expected count assertion Pre-existing import-resolution failures in this test file (16 on master) are unrelated to this change — they're a longstanding Bun package self-import behavior. The count + EXPECTED_EXPORTS list-match assertions both pass cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): add --resume-from <jsonl> to gbrain eval longmemeval Multi-cell embedder shootouts spend $50+/cell on the gpt-4o judge after gbrain emits hypotheses. A mid-run abort (rate-limit, cost-cap, OS interrupt, SIGKILL) previously meant re-paying the full cell. This flag makes those aborts cheap: re-invoke with --resume-from pointed at the partial JSONL and only the unanswered question_ids re-run. Behavior: - Read question_ids from the file; skip them on this run. - Rows with non-empty hypothesis count as done. - Rows with hypothesis="" AND an error field are NOT skipped (retry case for per-question failures recorded by the existing try/catch). - Corrupt trailing lines (SIGKILL'd writer mid-line) are silently skipped with a stderr warn. - When --resume-from path == --output path, the output emitter opens the file in append mode instead of truncating, so the existing rows survive. - Empty resume case (all questions already done) returns immediately without spinning up the brain or calling the client. New exported helper loadResumeSet() makes the parser unit-testable. 6 new test cases pinning: - File-not-found returns empty set - Well-formed JSONL load - Error-row retry semantics (empty hypothesis + error -> not in set) - Truncated final line recovery - End-to-end resume against the 5-question mini fixture - All-done early-return (stub client must NOT be invoked) All 18 cases in test/eval-longmemeval.test.ts green; bun run typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: v0.35.1.0 Bumps VERSION + package.json + CHANGELOG entry for the embedder-shootout prereq release. Three additive changes from the prior 4 commits: - pricing: voyage-4-large + zembed-1 entries - exports: gbrain/ai/gateway is now public - eval: gbrain eval longmemeval --resume-from <jsonl> Each commit on this branch is independently bisect-friendly and CI-green; the CHANGELOG entry is the user-facing rollup. No migrations, no breaking changes — the gateway export expands the surface, the resume-from flag is additive, the pricing patch only changes "estimate unavailable" -> a real dollar figure for two specific models. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
baf1a47798
commit
3933eb6a79
@@ -2,6 +2,51 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.35.1.0] - 2026-05-15
|
||||
|
||||
**Embedder shootout prereqs: pricing, public gateway export, and resume-from for long eval runs.**
|
||||
|
||||
A focused infrastructure release setting up the upcoming OpenAI vs Voyage vs ZeroEntropy comparison documented in `docs/designs/2026_05_EVAL_PLAN.md`. Three changes, each independently useful: `gbrain upgrade` now estimates costs correctly for `voyage:voyage-4-large` and `zeroentropyai:zembed-1` (previously fell through to "estimate unavailable"); external eval consumers can swap embedding providers per cell via the newly-public `gbrain/ai/gateway` subpath; multi-hour LongMemEval runs survive mid-run aborts via `--resume-from`.
|
||||
|
||||
### What you can now do
|
||||
|
||||
**See real cost estimates for Voyage 4 Large and ZeroEntropy zembed-1.** Before this release, `gbrain upgrade`'s post-upgrade reembed prompt silently fell back to "estimate unavailable" for these two models, even though both shipped with first-class recipe support in v0.35.0.0. Now: `voyage:voyage-4-large` resolves at $0.18/MTok (matching voyage-3-large) and `zeroentropyai:zembed-1` at $0.05/MTok. The lookup is case-insensitive on the provider name and falls back cleanly on unknown providers — no fabricated numbers.
|
||||
|
||||
**Drive gbrain's embedding gateway from outside the binary.** `package.json` exports gain `gbrain/ai/gateway` so external consumers (`gbrain-evals`, custom eval harnesses, third-party integrations) can call `configureGateway({embedding_model, embedding_dimensions, reranker_model})` directly instead of forking gbrain or duplicating the recipe wiring. This unblocks per-cell provider swapping in eval matrices without a brain DB. The exports surface count goes 17→18, locked by the canary contract test.
|
||||
|
||||
**Resume a half-finished LongMemEval run instead of re-paying $50.** `gbrain eval longmemeval --resume-from <jsonl>` skips question_ids already present in the file and continues writing the remaining questions in append mode. Rows whose `hypothesis` is empty AND have an `error` field (per-question failures from a prior run's try/catch) are NOT skipped — those retry. Corrupt trailing lines from a SIGKILL'd writer are silently dropped with a stderr warn. Empty-resume case (every question already answered) returns immediately without spinning up the brain or calling the LLM.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `src/core/embedding-pricing.ts` adds `voyage:voyage-4-large` ($0.18/MTok) and `zeroentropyai:zembed-1` ($0.05/MTok). New test file `test/embedding-pricing.test.ts` pins both entries, case-insensitive provider matching, bare-model openai-default fallback, table integrity (lowercase providers, finite non-negative prices), and the `estimateCostFromChars` approximation — 11 cases, 46 expect() calls.
|
||||
- `package.json` exports map adds `"./ai/gateway": "./src/core/ai/gateway.ts"`. `scripts/check-exports-count.sh` bumps `EXPECTED_COUNT` 17→18. `test/public-exports.test.ts` adds canary entries for `configureGateway` and `embed` symbols and bumps the inline count assertion. The pre-existing import-resolution failures in this test file are unchanged (longstanding Bun package self-import behavior, not introduced or worsened by this release).
|
||||
- `src/commands/eval-longmemeval.ts` adds `--resume-from <path>` CLI flag. New exported helper `loadResumeSet(path)` is the parser (file-not-found → empty set; corrupt lines silently skipped; error-rows retry). `makeEmitter()` takes a second `append` arg; runner sets it true when `--resume-from path === --output path`. 6 new test cases in `test/eval-longmemeval.test.ts` covering file-not-found, well-formed load, retry semantics, SIGKILL-recovery corrupt-line tolerance, end-to-end append-mode resume against the 5-question mini fixture, and all-done early-return (stub client must NOT be invoked).
|
||||
|
||||
## To take advantage of v0.35.1.0
|
||||
|
||||
`gbrain upgrade` handles the pricing-table refresh transparently. The new gateway export and `--resume-from` flag are additive — no migration required, nothing breaks if you don't use them.
|
||||
|
||||
1. **Run the orchestrator:**
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
2. **(Eval consumers only) Use the new gateway export.** External code can now:
|
||||
```ts
|
||||
import { configureGateway, embed } from 'gbrain/ai/gateway';
|
||||
configureGateway({ embedding_model: 'voyage:voyage-4-large', embedding_dimensions: 2048 });
|
||||
const vectors = await embed(['hello']);
|
||||
```
|
||||
3. **(Long eval runs only) Use `--resume-from` to recover from mid-run aborts:**
|
||||
```bash
|
||||
# First run aborts at question 312:
|
||||
gbrain eval longmemeval dataset.jsonl --output results.jsonl
|
||||
|
||||
# Resume:
|
||||
gbrain eval longmemeval dataset.jsonl --output results.jsonl --resume-from results.jsonl
|
||||
```
|
||||
4. **If anything looks off,** `gbrain doctor` should be clean. File an issue at
|
||||
https://github.com/garrytan/gbrain/issues with `gbrain doctor` output if not.
|
||||
|
||||
## [0.35.0.0] - 2026-05-15
|
||||
|
||||
**ZeroEntropy in the box: zembed-1 embeddings + zerank-2 cross-encoder reranking, on by default for tokenmax mode.**
|
||||
|
||||
@@ -0,0 +1,580 @@
|
||||
# Embedder Shootout — May 2026 Eval Plan
|
||||
|
||||
**Status:** approved, ready to execute
|
||||
**Owner:** Garry
|
||||
**Plan source:** `~/.claude/plans/system-instruction-you-are-working-linear-origami.md` (review log)
|
||||
**Target wallclock:** ~2 weeks
|
||||
**Target API spend:** ~$525 (hard cap $700)
|
||||
|
||||
## What this is
|
||||
|
||||
A head-to-head A/B/C comparison of three embedding providers under v0.35.0.0's new
|
||||
multi-vendor gateway routing:
|
||||
|
||||
- **OpenAI** `text-embedding-3-large` @ 1536 dims
|
||||
- **Voyage** `voyage-4-large` @ 2048 dims
|
||||
- **ZeroEntropy** `zembed-1` @ 2560 dims (also 1280 in a Matryoshka ablation)
|
||||
|
||||
Each tested with and without the `zerank-2` reranker. Two corpora: public LongMemEval
|
||||
(500q) and BrainBench in-house (145 relational queries + 50 newly-curated Cat 13
|
||||
embedder-sensitive queries).
|
||||
|
||||
The goal: produce a publishable comparison report that answers "which embedder wins,
|
||||
and does zerank-2 carry the win for ZeroEntropy" with bootstrap p-values, suitable
|
||||
for a v0.35.2.0 release-note headline.
|
||||
|
||||
## Why this design
|
||||
|
||||
Locked decisions from the planning review (see plan file + `GSTACK REVIEW REPORT` at
|
||||
the bottom of the linked plan):
|
||||
|
||||
- **Synthetic-only** — LongMemEval (public) + BrainBench (in-house). No `~/.gbrain` data.
|
||||
- **Answer-gen mode** — `gbrain eval longmemeval` runs the default answer-gen path
|
||||
(Anthropic Sonnet), then feeds the resulting hypothesis JSONL to LongMemEval's
|
||||
published `evaluate_qa.py` (OpenAI gpt-4o judge) for real correctness numbers.
|
||||
`--retrieval-only` is NOT used (would produce an attackable headline; the judge
|
||||
expects answer text, not retrieval text).
|
||||
- **`tokenmax` search mode** pinned across all cells (expansion + reranker slot active).
|
||||
- **Serial execution** in one workspace. Clean rate-limit profile; first-contact run on
|
||||
ZE wants debuggable signal.
|
||||
- **7-cell matrix** (no matched-dim cross-vendor row — no shared dim exists across
|
||||
all three vendors; honest framing is "each vendor at marketed sweet spot").
|
||||
|
||||
## Architectural facts that constrain the plan
|
||||
|
||||
- `content_chunks.embedding vector(N)` dim is fixed per brain. Per-question PGLite in
|
||||
LongMemEval makes this free; BrainBench needs separate brain per cell.
|
||||
- pgvector HNSW caps at **2000 dims** (`PGVECTOR_HNSW_VECTOR_MAX_DIMS` in
|
||||
`src/core/vector-index.ts:19`). Voyage 2048 and ZE 2560 fall back to exact vector
|
||||
scan. Helps quality (no HNSW approximation) but adds latency. Footnoted in writeup.
|
||||
- Reranker disable key is **`search.reranker.enabled false`**, NOT `reranker_model none`.
|
||||
`tokenmax` mode defaults reranker=true.
|
||||
- `gbrain/ai/gateway` is NOT exported in v0.35.0.0. PR α exposes it.
|
||||
|
||||
## Matrix
|
||||
|
||||
| Cell | Embedder | Dim | HNSW | Reranker | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| A0 | `openai:text-embedding-3-large` | 1536 | yes | none | OpenAI baseline |
|
||||
| A1 | `openai:text-embedding-3-large` | 1536 | yes | `zerank-2` | mixed-vendor |
|
||||
| B0 | `voyage:voyage-4-large` | 2048 | no (exact) | none | Voyage solo |
|
||||
| B1 | `voyage:voyage-4-large` | 2048 | no (exact) | `zerank-2` | mixed-vendor |
|
||||
| C0 | `zeroentropyai:zembed-1` | 2560 | no (exact) | none | ZE embedder solo |
|
||||
| C1 | `zeroentropyai:zembed-1` | 2560 | no (exact) | `zerank-2` | **ZE full stack** |
|
||||
| C2 | `zeroentropyai:zembed-1` | 1280 | yes | `zerank-2` | ZE-Matryoshka ablation |
|
||||
|
||||
## PR structure — as few as possible
|
||||
|
||||
**PR α — gbrain repo: v0.35.1.0 infra.** All gbrain changes bundled. Lands first.
|
||||
Bisect-friendly commits inside, ship at the very end.
|
||||
|
||||
**PR β — gbrain-evals repo: adapter + smoke + curation + eval receipts + writeup.** The
|
||||
big one. Includes the full eval-run output committed alongside the code that produced
|
||||
it, plus the comparison writeup. Lands when everything is done.
|
||||
|
||||
**PR γ (optional) — gbrain repo: v0.35.2.0 release** that cross-links the gbrain-evals
|
||||
benchmark in CHANGELOG. Small commit; no code changes.
|
||||
|
||||
Total: 2 substantive PRs + 1 optional release commit. **No mid-stream ships.**
|
||||
|
||||
## Conductor sessions
|
||||
|
||||
Each section below is a self-contained brief. Copy-paste into a fresh Conductor session
|
||||
to hand off. Each session ends with a clean deliverable.
|
||||
|
||||
---
|
||||
|
||||
## Session 1 — PR α: gbrain infra (v0.35.1.0)
|
||||
|
||||
**Repo:** `/Users/garrytan/conductor/workspaces/gbrain/<NEW-WORKSPACE>` (fresh from `master`)
|
||||
**Branch:** `garrytan/v0.35.1.0-infra`
|
||||
**Wallclock:** ~2h
|
||||
**API spend:** $0
|
||||
|
||||
### What this session ships
|
||||
Three changes in one PR, bundled so the embedder shootout in gbrain-evals (PR β) has a
|
||||
clean prereq baseline:
|
||||
|
||||
1. Add `voyage:voyage-4-large` ($0.18/M) and `zeroentropyai:zembed-1` ($0.05/M) to the
|
||||
embedding pricing table. Patch the `gbrain models doctor` cost estimator + test.
|
||||
2. Expose `gbrain/ai/gateway` in `package.json` exports map so the gbrain-evals
|
||||
adapters can call `configureGateway({embedding_model, embedding_dimensions, reranker_model})`
|
||||
from outside the gbrain process.
|
||||
3. Add `--resume-from <jsonl>` to `gbrain eval longmemeval` so a mid-run abort
|
||||
(rate-limit, cost-cap, OS interrupt) doesn't lose the cells we already paid for.
|
||||
|
||||
Ships at the end as v0.35.1.0.
|
||||
|
||||
### Prereqs (verify before starting)
|
||||
- On gbrain master at v0.35.0.0 baseline. `cat VERSION` shows `0.35.0.0`.
|
||||
- `bun test` and `bun run verify` both pass on master.
|
||||
|
||||
### Commits (bisect-friendly, one feature per commit)
|
||||
|
||||
```
|
||||
1. feat(pricing): add voyage-4-large + zembed-1 to EMBEDDING_PRICING
|
||||
- src/core/embedding-pricing.ts: add both entries
|
||||
- test/embedding-pricing.test.ts: pin both with $0.18 and $0.05
|
||||
- Verify: bun test test/embedding-pricing.test.ts
|
||||
|
||||
2. feat(exports): expose gbrain/ai/gateway with canary test
|
||||
- package.json: add "./ai/gateway" to exports map
|
||||
- test/public-exports.test.ts: add canary for configureGateway + embed
|
||||
- scripts/check-exports-count.sh: 17 -> 18
|
||||
- Verify: bun run verify
|
||||
|
||||
3. feat(eval): add --resume-from <jsonl> to longmemeval
|
||||
- src/commands/eval-longmemeval.ts: parse flag, skip questions already in input JSONL
|
||||
- test/eval-longmemeval.test.ts: simulated mid-run abort + resume regression
|
||||
- Verify: bun test test/eval-longmemeval.test.ts
|
||||
|
||||
4. chore: v0.35.1.0
|
||||
- VERSION: 0.35.1.0
|
||||
- package.json: 0.35.1.0
|
||||
- CHANGELOG.md: new entry
|
||||
- bun install (refresh lockfile)
|
||||
```
|
||||
|
||||
### Verify before /ship
|
||||
```bash
|
||||
bun run typecheck
|
||||
bun run verify
|
||||
bun test test/embedding-pricing.test.ts test/public-exports.test.ts test/eval-longmemeval.test.ts
|
||||
```
|
||||
|
||||
### Ship
|
||||
```bash
|
||||
/ship
|
||||
```
|
||||
|
||||
### Deliverable
|
||||
- `master` of gbrain at v0.35.1.0
|
||||
- `gbrain/ai/gateway` reachable from external consumers (verified by canary test)
|
||||
- `git tag eval-run-v0.35.1.0-baseline` (annotated, names this exact commit)
|
||||
- `gbrain --version` prints `0.35.1.0`
|
||||
|
||||
### Hand-off to Session 2
|
||||
- gbrain-evals can now `bun update gbrain` to v0.35.1.0
|
||||
- The tag preserves the exact commit for any future reproducibility need
|
||||
|
||||
---
|
||||
|
||||
## Session 2 — PR β setup: gbrain-evals adapter + smoke + subset flag
|
||||
|
||||
**Repo:** `/Users/garrytan/git/gbrain-evals` (or a fresh Conductor workspace cloned from it)
|
||||
**Branch:** `garrytan/embedder-shootout`
|
||||
**Wallclock:** ~3-4h
|
||||
**API spend:** ~$0.10 (smoke verification calls only)
|
||||
|
||||
### What this session ships into PR β (does NOT merge yet)
|
||||
Wire the harness to drive 3 embedding providers via the newly-exposed gbrain gateway:
|
||||
|
||||
1. New typed `EvalAdapterConfig {embedder, dim, reranker?}` passed into each adapter.
|
||||
2. Rewrite `vector.ts` + `hybrid-rrf.ts` to call `configureGateway()` from
|
||||
`gbrain/ai/gateway` instead of the hardcoded `gbrain/embedding` import.
|
||||
3. Critical: hybrid adapter must also route `search.reranker.enabled` (true/false) and
|
||||
`search.mode` (tokenmax) — codex flagged that the existing hybrid never sets these.
|
||||
4. New 3-phase smoke harness: wiring (5 queries × embed roundtrip + dim check) +
|
||||
long-haystack (1 query × 50K-token synthetic haystack) + rerank-payload (1 query
|
||||
× `topNIn=30`). Exit code is the gate.
|
||||
5. New `--include-subset <name>` flag on the BrainBench runner (Cat 13 wiring; subset
|
||||
itself comes in Session 3).
|
||||
|
||||
### Prereqs
|
||||
- Session 1 done. gbrain master at v0.35.1.0.
|
||||
- API keys present: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`,
|
||||
`ZEROENTROPY_API_KEY`. Smoke fails-loud on missing key.
|
||||
|
||||
### Commits
|
||||
|
||||
```
|
||||
1. chore(deps): bump gbrain pin to v0.35.1.0
|
||||
- package.json + bun.lock
|
||||
- Verify: bun install && bun run typecheck
|
||||
|
||||
2. feat(adapter): typed EvalAdapterConfig + gateway swap
|
||||
- NEW: eval/runner/eval-adapter-config.ts (the type)
|
||||
- eval/runner/adapters/vector.ts: constructor takes EvalAdapterConfig,
|
||||
calls configureGateway({embedding_model, embedding_dimensions})
|
||||
- Drop hardcoded gbrain/embedding import
|
||||
- Verify: existing vector adapter unit tests still pass
|
||||
|
||||
3. feat(adapter): hybrid-rrf wires reranker_enabled + search.mode
|
||||
- eval/runner/adapters/hybrid-rrf.ts: constructor takes EvalAdapterConfig,
|
||||
plumbs search.reranker.enabled + search.mode = tokenmax through
|
||||
- Verify: bun test eval/
|
||||
|
||||
4. feat(smoke): 3-phase smoke harness
|
||||
- NEW: eval/runner/smoke.ts (CLI entry: bun run eval:smoke -- --embedder X --dim Y [--reranker Z])
|
||||
- Phase 1: 5 queries × embed roundtrip, assert vector dim matches config
|
||||
- Phase 2: 1 query × synthetic 50K-token haystack, assert no token-limit error
|
||||
- Phase 3: 1 query × topNIn=30 documents, assert no 5MB payload cap hit
|
||||
- Non-zero exit on any failure
|
||||
- Verify: bun run eval:smoke -- --embedder openai:text-embedding-3-large --dim 1536
|
||||
|
||||
5. feat(runner): --include-subset flag for BrainBench
|
||||
- eval/runner/multi-adapter.ts: parse flag, filter queries by subset tag
|
||||
- Subset itself comes in next commit (Session 3)
|
||||
- Verify: bun run eval:run -- --include-subset cat13-embedder (errors politely because subset file doesn't exist yet)
|
||||
```
|
||||
|
||||
### Smoke verification (run manually before opening PR)
|
||||
```bash
|
||||
bun run eval:smoke -- --embedder openai:text-embedding-3-large --dim 1536
|
||||
bun run eval:smoke -- --embedder voyage:voyage-4-large --dim 2048
|
||||
bun run eval:smoke -- --embedder zeroentropyai:zembed-1 --dim 2560
|
||||
bun run eval:smoke -- --embedder zeroentropyai:zembed-1 --dim 2560 --reranker zeroentropyai:zerank-2
|
||||
```
|
||||
|
||||
All four MUST exit 0. Reports should print the observed vector dim, matching the
|
||||
configured dim.
|
||||
|
||||
### Open PR β
|
||||
```bash
|
||||
gh pr create --base main --title "feat: embedder shootout (adapter + smoke + Cat 13 + eval receipts)" --body "$(cat <<'EOF'
|
||||
## Summary
|
||||
v0.35.0.0 shipped ZeroEntropy zembed-1 + zerank-2 reranker support. This PR runs a head-to-head A/B/C comparison across OpenAI, Voyage, and ZeroEntropy under the new gateway routing.
|
||||
|
||||
This first commit batch lands the harness. Cat 13 curation, Phase 1+2 evals, and the
|
||||
writeup follow in subsequent commits to this same PR.
|
||||
|
||||
## Test plan
|
||||
- [x] Adapter unit tests pass
|
||||
- [x] Smoke harness exits 0 against all 3 providers
|
||||
- [ ] Cat 13 subset committed (Session 3)
|
||||
- [ ] LongMemEval x 7 cells run (Session 4)
|
||||
- [ ] BrainBench x 7 cells run (Session 5)
|
||||
- [ ] Writeup committed (Session 5)
|
||||
|
||||
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
### Deliverable
|
||||
- PR β open against gbrain-evals `main`, green CI
|
||||
- Smoke verified against all 3 providers (paste the smoke output in the PR body)
|
||||
- Branch ready for Session 3 (Cat 13 curation)
|
||||
|
||||
### Hand-off to Session 3
|
||||
- Branch `garrytan/embedder-shootout` exists on origin
|
||||
- The `--include-subset cat13-embedder` flag is wired but the subset file doesn't exist
|
||||
yet — that's Session 3
|
||||
|
||||
---
|
||||
|
||||
## Session 3 — PR β: Cat 13 conceptual-recall curation
|
||||
|
||||
**Repo:** `/Users/garrytan/git/gbrain-evals`, branch `garrytan/embedder-shootout` (same as Session 2)
|
||||
**Wallclock:** ~3-4h (heavily user-interactive; AI proposes, you review each)
|
||||
**API spend:** $0
|
||||
|
||||
### What this session ships into PR β
|
||||
Hand-curated 50 embedder-sensitive queries from BrainBench's Cat 13 (conceptual recall)
|
||||
corpus. These are the queries where a graph/keyword adapter would likely miss but a
|
||||
semantic adapter would find.
|
||||
|
||||
Codex flagged the existing 145-query relational corpus as graph/keyword-dominated and
|
||||
weak for embedder claims. Cat 13 is closer to the embedder-sensitive workload but
|
||||
needs hand-selection.
|
||||
|
||||
### Prereqs
|
||||
- Session 2 done. PR β open with adapter + smoke + subset flag.
|
||||
|
||||
### Workflow
|
||||
Interactive: Claude proposes queries in batches of 10, you accept/reject/edit each.
|
||||
|
||||
1. Claude reads the existing Cat 13 raw query pool:
|
||||
```bash
|
||||
ls eval/data/raw/ | grep -i cat13
|
||||
cat eval/data/raw/cat13-*.json | jq '.'
|
||||
```
|
||||
2. Claude proposes 10 candidate queries per batch, each tagged with the inclusion
|
||||
reasoning ("would a graph adapter miss this?")
|
||||
3. User accepts/rejects/edits inline. Target: 50 queries × ~5 batches.
|
||||
4. Claude commits to `eval/data/gold/brainbench-cat13-embedder-subset.json`:
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"subset": "cat13-embedder",
|
||||
"queries": [
|
||||
{
|
||||
"id": "cat13-emb-001",
|
||||
"query": "...",
|
||||
"relevant_chunk_ids": ["..."],
|
||||
"inclusion_reason": "paraphrase relationship; graph adapter wouldn't catch the synonym"
|
||||
}
|
||||
// ... 49 more
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Commit
|
||||
|
||||
```
|
||||
feat(eval): curate Cat 13 conceptual-recall subset (50 embedder-sensitive queries)
|
||||
- NEW: eval/data/gold/brainbench-cat13-embedder-subset.json
|
||||
- Each query tagged with inclusion_reason for future audit
|
||||
```
|
||||
|
||||
### Spot-check before commit
|
||||
- Pick 5 random queries, run them against a hypothetical graph adapter (e.g. grep on
|
||||
the relevant terms) and verify they would NOT surface the right chunk.
|
||||
- Run the same 5 against the existing hybrid adapter and verify they DO.
|
||||
|
||||
### Deliverable
|
||||
- `eval/data/gold/brainbench-cat13-embedder-subset.json` committed to PR β
|
||||
- Exactly 50 queries
|
||||
- Spot-check evidence in the commit message
|
||||
|
||||
### Hand-off to Session 4
|
||||
- PR β now has: adapter + smoke + Cat 13 subset
|
||||
- Ready for the actual eval runs
|
||||
|
||||
---
|
||||
|
||||
## Session 4 — PR β Phase 1: LongMemEval × 7 cells (overnight)
|
||||
|
||||
**Repo:** Same gbrain-evals branch
|
||||
**Wallclock:** ~10.5h (mostly hands-off, kick off and walk away)
|
||||
**API spend:** ~$476 (LongMemEval-heavy; 7 × $68/cell)
|
||||
|
||||
### What this session ships into PR β
|
||||
7 LongMemEval scored receipts (one per matrix cell). Each is a JSONL of 500
|
||||
hypotheses + a JSON file of correctness scores from `evaluate_qa.py`.
|
||||
|
||||
### Prereqs
|
||||
- Sessions 1+2+3 done. PR β has adapter + smoke + Cat 13.
|
||||
- LongMemEval dataset downloaded (gated HuggingFace; one-time setup).
|
||||
- `evaluate_qa.py` checked out somewhere (from
|
||||
https://github.com/xiaowu0162/LongMemEval) with its own venv set up.
|
||||
- API keys: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `VOYAGE_API_KEY`,
|
||||
`ZEROENTROPY_API_KEY`.
|
||||
|
||||
### Wrapper script
|
||||
Claude writes `scripts/run-shootout-phase1.sh` in the gbrain-evals branch. Single
|
||||
entry point that loops the 7 cells serially with smoke gating + cost-cap aborts.
|
||||
|
||||
```
|
||||
NEW: scripts/run-shootout-phase1.sh
|
||||
- Per cell: gbrain config set (embedder, dim, reranker, search.reranker.enabled, search.mode=tokenmax)
|
||||
- Per cell: bun run eval:smoke (abort cell on non-zero)
|
||||
- Per cell: gbrain eval longmemeval ... --output results/longmemeval-{cell}.jsonl
|
||||
- Per cell: cost-cap check ($90/cell hard stop)
|
||||
- Per cell: --resume-from existing results/longmemeval-{cell}.jsonl if present
|
||||
- Logs to results/phase1-run-log.txt
|
||||
```
|
||||
|
||||
### Run
|
||||
```bash
|
||||
# Kick off in background; check back in 10-12h
|
||||
bash scripts/run-shootout-phase1.sh 2>&1 | tee results/phase1-run-log.txt &
|
||||
```
|
||||
|
||||
Use `run_in_background: true` if running through Claude. Check back periodically.
|
||||
|
||||
### Scoring (after all 7 cells done)
|
||||
```bash
|
||||
for cell in A0 A1 B0 B1 C0 C1 C2; do
|
||||
python evaluate_qa.py \
|
||||
--input results/longmemeval-${cell}.jsonl \
|
||||
--output results/longmemeval-${cell}-scored.json
|
||||
done
|
||||
```
|
||||
|
||||
Each scored file has correctness %.
|
||||
|
||||
### Commits
|
||||
|
||||
```
|
||||
1. feat(scripts): Phase 1 LongMemEval wrapper with smoke gating + cost cap
|
||||
- NEW: scripts/run-shootout-phase1.sh
|
||||
|
||||
2. data(phase1): 7 LongMemEval cells (raw hypothesis JSONL)
|
||||
- results/longmemeval-{A0,A1,B0,B1,C0,C1,C2}.jsonl
|
||||
- results/phase1-run-log.txt (run timing + cost ledger)
|
||||
|
||||
3. data(phase1): evaluate_qa.py scoring results
|
||||
- results/longmemeval-{cell}-scored.json × 7
|
||||
```
|
||||
|
||||
### Verify
|
||||
- Each `longmemeval-{cell}.jsonl` has exactly 500 lines
|
||||
- Each `hypothesis` field is non-empty AND is actual answer text (NOT retrieval text)
|
||||
- Each `scored.json` has a `correctness_score` field
|
||||
|
||||
### Deliverable
|
||||
- 7 scored LongMemEval receipts committed to PR β
|
||||
- Real cost ledger committed alongside (compare against estimate)
|
||||
|
||||
### Hand-off to Session 5
|
||||
- Phase 1 done. Phase 2 (BrainBench, ~3.5h) and writeup remaining.
|
||||
|
||||
---
|
||||
|
||||
## Session 5 — PR β Phase 2 + writeup + ship
|
||||
|
||||
**Repo:** Same gbrain-evals branch
|
||||
**Wallclock:** ~7h (3.5h BrainBench + 3h writeup + /ship)
|
||||
**API spend:** ~$56 (BrainBench is cheap)
|
||||
|
||||
### What this session ships into PR β
|
||||
- 7 BrainBench cells (relational corpus + Cat 13 subset)
|
||||
- Final comparison writeup
|
||||
- PR β merged
|
||||
|
||||
### Prereqs
|
||||
- Session 4 done. PR β has Phase 1 receipts.
|
||||
|
||||
### Phase 2 wrapper script
|
||||
```
|
||||
NEW: scripts/run-shootout-phase2.sh
|
||||
- Per cell: configure provider (same as Phase 1)
|
||||
- Per cell: bun run eval:run -- --N 10 --include-subset cat13-embedder
|
||||
--output docs/benchmarks/2026-05-22-{cell}.md
|
||||
- Cost-cap check
|
||||
```
|
||||
|
||||
### Run
|
||||
```bash
|
||||
bash scripts/run-shootout-phase2.sh 2>&1 | tee results/phase2-run-log.txt
|
||||
```
|
||||
|
||||
### Writeup
|
||||
`docs/benchmarks/2026-05-22-embedder-shootout.md`. Structure:
|
||||
|
||||
1. **Headline table** — 7 cells × {LongMemEval correctness %, BrainBench relational MRR + P@5, Cat 13 correctness %, total cost}
|
||||
2. **Two questions answered:**
|
||||
- Which embedder wins solo? (A0 vs B0 vs C0)
|
||||
- Does zerank-2 carry ZE's win? (C0 vs C1 vs A1 vs B1)
|
||||
- Bonus: does dim matter for ZE? (C1 vs C2)
|
||||
3. **Paired-bootstrap p-values** per headline pair (methodology in
|
||||
`gbrain/docs/eval/SEARCH_MODE_METHODOLOGY.md`)
|
||||
4. **HNSW footnote** — Voyage 2048 and ZE 2560 used exact vector scan; OpenAI 1536
|
||||
and ZE 1280 used HNSW. Quality is primary, latency is secondary
|
||||
5. **What this does NOT prove** — synthetic-only, tokenmax-only, no real-brain replay
|
||||
6. **Recommendation:** explicit NON-recommendation to change `gbrain init` default;
|
||||
defer to a v0.36.x evidence pass with real-brain replay data
|
||||
|
||||
### Commits
|
||||
|
||||
```
|
||||
1. feat(scripts): Phase 2 BrainBench wrapper
|
||||
- NEW: scripts/run-shootout-phase2.sh
|
||||
|
||||
2. data(phase2): 7 BrainBench cells
|
||||
- docs/benchmarks/2026-05-22-{cell}.md × 7
|
||||
|
||||
3. docs(benchmark): embedder shootout comparison writeup
|
||||
- NEW: docs/benchmarks/2026-05-22-embedder-shootout.md
|
||||
- Bootstrap p-values, HNSW footnote, NOT-in-scope section
|
||||
```
|
||||
|
||||
### Ship
|
||||
```bash
|
||||
# Merge PR β to gbrain-evals main
|
||||
gh pr merge --squash --auto
|
||||
# Or non-auto if reviewing one more time:
|
||||
gh pr merge --squash
|
||||
```
|
||||
|
||||
### Deliverable
|
||||
- PR β merged to gbrain-evals `main`
|
||||
- Comparison report public at
|
||||
`gbrain-evals/docs/benchmarks/2026-05-22-embedder-shootout.md`
|
||||
|
||||
### Hand-off to Session 6 (optional)
|
||||
- gbrain-evals master has the full data + writeup
|
||||
- Ready for a v0.35.2.0 gbrain release that cross-links it
|
||||
|
||||
---
|
||||
|
||||
## Session 6 (optional) — PR γ: gbrain v0.35.2.0 release
|
||||
|
||||
**Repo:** `/Users/garrytan/conductor/workspaces/gbrain/<NEW-WORKSPACE>` (fresh from master)
|
||||
**Branch:** `garrytan/v0.35.2.0-benchmark-release`
|
||||
**Wallclock:** ~30min
|
||||
**API spend:** $0
|
||||
|
||||
### What this session ships
|
||||
A release-notes-only PR that bumps gbrain to v0.35.2.0 with a CHANGELOG entry
|
||||
cross-linking the embedder shootout benchmark. Optional — could be folded into the
|
||||
next routine release if no rush.
|
||||
|
||||
### Prereqs
|
||||
- Session 5 done. gbrain-evals merged with the comparison writeup.
|
||||
|
||||
### Commits
|
||||
|
||||
```
|
||||
1. docs(benchmark): mirror embedder shootout summary
|
||||
- NEW: docs/benchmarks/2026-05-22-embedder-shootout.md (slim mirror)
|
||||
- Cross-link to gbrain-evals canonical version
|
||||
|
||||
2. chore: v0.35.2.0
|
||||
- VERSION: 0.35.2.0
|
||||
- package.json: 0.35.2.0
|
||||
- CHANGELOG.md: new entry with the GStack-voice release summary
|
||||
+ "numbers that matter" table from the benchmark
|
||||
```
|
||||
|
||||
### Ship
|
||||
```bash
|
||||
/ship
|
||||
```
|
||||
|
||||
### Deliverable
|
||||
- gbrain v0.35.2.0 on master
|
||||
- CHANGELOG entry that drives the release-note headline
|
||||
|
||||
---
|
||||
|
||||
## Cost ledger (revised, post-review)
|
||||
|
||||
| Component | Per cell | × 7 cells |
|
||||
|---|---|---|
|
||||
| LongMemEval embed | <$0.05 | <$0.35 |
|
||||
| LongMemEval Sonnet answer-gen (500q × 2K tokens × $3/M) | $18 | $126 |
|
||||
| LongMemEval gpt-4o judge (500q × $0.10/q) | $50 | $350 |
|
||||
| BrainBench relational embed | $0.05-0.18 | <$1 |
|
||||
| BrainBench Cat 13 answer-gen + judge (50q × $0.14) | $7 | $49 |
|
||||
| Smoke harness (30 calls/cell) | <$0.10 | <$1 |
|
||||
| **Total** | **~$75/cell** | **~$525** |
|
||||
|
||||
**Hard cap: $700.** Per-cell hard cap: $90 (wrapper aborts cell if exceeded; partial
|
||||
JSONL preserved for resume).
|
||||
|
||||
## Failure modes and recovery
|
||||
|
||||
| Failure | Recovery |
|
||||
|---|---|
|
||||
| Voyage/ZE 429 rate-limit mid-cell | `gateway._shrinkState` halves safety_factor and retries. Cell continues. |
|
||||
| ZE 5MB rerank payload cap hit | `applyReranker` fail-opens, returns un-reranked results. Stderr warn. |
|
||||
| Mid-cell OS interrupt / cost-cap abort | Re-run with `gbrain eval longmemeval --resume-from results/longmemeval-{cell}.jsonl`. Picks up where it left off. |
|
||||
| `evaluate_qa.py` auth fail | OPENAI_API_KEY check in wrapper aborts before any spend. |
|
||||
| Adapter typo (bad dim) | `EvalAdapterConfig` runtime assertion at constructor throws AIConfigError. Cell aborts before API call. |
|
||||
|
||||
## NOT in scope (deliberate)
|
||||
|
||||
- **Real `~/.gbrain` replay** — adds 6-12h wallclock + $40-80 embed. Filed as v0.36.x.
|
||||
- **All 3 search modes** — pinned to tokenmax. `conservative` + `balanced` are v0.35.3.0
|
||||
follow-ups if reviewers push back.
|
||||
- **Matched-dim cross-vendor row** — no shared dim exists across all 3 vendors.
|
||||
Permanently out.
|
||||
- **`gbrain eval whoknows` / `cross-modal` / `takes-quality`** — embedding-invariant;
|
||||
rerunning across embedders produces noise.
|
||||
- **`gbrain eval code-retrieval`** — code corpus, separate concern.
|
||||
- **`gbrain eval suspected-contradictions`** — wants a real brain.
|
||||
- **`gbrain init --recommended` default change** — codex correctly flagged the evidence
|
||||
base as insufficient. Defer to v0.36.x with real-brain replay data.
|
||||
|
||||
## What already exists (reused, not rebuilt)
|
||||
|
||||
- `gbrain eval longmemeval` CLI (in-tree, answer-gen mode default)
|
||||
- gbrain-evals BrainBench runner (`eval:run`) — needs adapter parameterization but
|
||||
per-cell test plumbing is reused
|
||||
- Gateway routing for Voyage + ZE (shipped v0.35.0.0)
|
||||
- Reranker pipeline (`src/core/search/rerank.ts`, fail-open)
|
||||
- Pricing table (extended, not rebuilt)
|
||||
- Paired-bootstrap methodology (`docs/eval/SEARCH_MODE_METHODOLOGY.md`)
|
||||
- LongMemEval published `evaluate_qa.py` (invoked externally, not bundled)
|
||||
+2
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.35.0.0",
|
||||
"version": "0.35.1.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -24,6 +24,7 @@
|
||||
"./backoff": "./src/core/backoff.ts",
|
||||
"./search/hybrid": "./src/core/search/hybrid.ts",
|
||||
"./search/expansion": "./src/core/search/expansion.ts",
|
||||
"./ai/gateway": "./src/core/ai/gateway.ts",
|
||||
"./extract": "./src/commands/extract.ts"
|
||||
},
|
||||
"scripts": {
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
EXPECTED_COUNT=17
|
||||
EXPECTED_COUNT=18
|
||||
|
||||
# Count top-level keys in the exports object. `node -e` parses JSON
|
||||
# reliably without needing jq (which isn't in every CI environment).
|
||||
|
||||
@@ -38,6 +38,14 @@ interface ParsedArgs {
|
||||
outputPath?: string;
|
||||
/** v0.32.3 — search-lite mode to evaluate under. Resolves through resolveSearchMode. */
|
||||
mode?: 'conservative' | 'balanced' | 'tokenmax';
|
||||
/**
|
||||
* v0.35.1.0 — path to a previous run's hypothesis JSONL. Question IDs
|
||||
* already present in the file are skipped on this run; the run resumes
|
||||
* with the remaining questions. Typically set to the same path as
|
||||
* --output so a re-run continues writing to the same file in append mode.
|
||||
* Recovery path for mid-run aborts (rate-limit, cost-cap, OS interrupt).
|
||||
*/
|
||||
resumeFromPath?: string;
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): ParsedArgs {
|
||||
@@ -58,6 +66,7 @@ function parseArgs(args: string[]): ParsedArgs {
|
||||
if (a === '--model') { out.model = args[++i]; continue; }
|
||||
if (a === '--top-k') { out.topK = Number(args[++i]); continue; }
|
||||
if (a === '--output') { out.outputPath = args[++i]; continue; }
|
||||
if (a === '--resume-from') { out.resumeFromPath = args[++i]; continue; }
|
||||
if (a === '--mode') {
|
||||
const v = args[++i];
|
||||
if (v === 'conservative' || v === 'balanced' || v === 'tokenmax') {
|
||||
@@ -93,6 +102,10 @@ function printHelp(): void {
|
||||
` behavior matches what production gets under that mode.\n` +
|
||||
` --mode tokenmax implies --expansion unless overridden.\n` +
|
||||
` --output FILE Write JSONL to FILE instead of stdout.\n` +
|
||||
` --resume-from FILE Skip question_ids already present in FILE; resume the\n` +
|
||||
` remaining questions. Typically the same path as --output\n` +
|
||||
` so the run continues writing in append mode. Recovery for\n` +
|
||||
` mid-run aborts (rate-limit, cost-cap, OS interrupt).\n` +
|
||||
` -h, --help Show this help.\n\n` +
|
||||
`Note: a full 500-question run takes ~20-60 minutes depending on flags. Use\n` +
|
||||
`--limit during development.\n`,
|
||||
@@ -104,7 +117,7 @@ interface JsonlEmitter {
|
||||
close(): void;
|
||||
}
|
||||
|
||||
function makeEmitter(outputPath?: string): JsonlEmitter {
|
||||
function makeEmitter(outputPath?: string, append: boolean = false): JsonlEmitter {
|
||||
if (!outputPath) {
|
||||
return {
|
||||
emit(obj) {
|
||||
@@ -115,7 +128,10 @@ function makeEmitter(outputPath?: string): JsonlEmitter {
|
||||
close() { /* stdout stays open */ },
|
||||
};
|
||||
}
|
||||
const fd = openSync(outputPath, 'w');
|
||||
// v0.35.1.0: append mode used by --resume-from when output path overlaps the
|
||||
// resume file. Truncating ('w') would erase the already-answered questions
|
||||
// we just loaded into resumeSet.
|
||||
const fd = openSync(outputPath, append ? 'a' : 'w');
|
||||
return {
|
||||
emit(obj) {
|
||||
const json = JSON.stringify(obj);
|
||||
@@ -126,6 +142,42 @@ function makeEmitter(outputPath?: string): JsonlEmitter {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.35.1.0: Load the set of question_ids already present in `resumePath`.
|
||||
*
|
||||
* One row per line; we only care about the `question_id` field. Rows whose
|
||||
* `hypothesis` is empty AND have an `error` field are NOT skipped — those
|
||||
* are previous-run failures that should be retried, not preserved. A row
|
||||
* with non-empty `hypothesis` (regardless of mode) counts as "done."
|
||||
*
|
||||
* Returns an empty Set if the file doesn't exist (first run with the flag
|
||||
* acts identically to no flag).
|
||||
*/
|
||||
export function loadResumeSet(resumePath: string): Set<string> {
|
||||
const done = new Set<string>();
|
||||
if (!existsSync(resumePath)) return done;
|
||||
const raw = readFileSync(resumePath, 'utf8');
|
||||
let lineNo = 0;
|
||||
for (const line of raw.split('\n')) {
|
||||
lineNo++;
|
||||
if (!line.trim()) continue;
|
||||
let row: { question_id?: string; hypothesis?: string; error?: string };
|
||||
try {
|
||||
row = JSON.parse(line);
|
||||
} catch {
|
||||
// Corrupt line — log to stderr and continue; a partial JSONL from a
|
||||
// SIGKILL'd writer is the normal recovery case.
|
||||
process.stderr.write(`[longmemeval] resume: skipping corrupt line ${lineNo}\n`);
|
||||
continue;
|
||||
}
|
||||
if (typeof row.question_id !== 'string') continue;
|
||||
// Skip rows that recorded an error with no hypothesis — retry these.
|
||||
if (row.error && (!row.hypothesis || row.hypothesis === '')) continue;
|
||||
done.add(row.question_id);
|
||||
}
|
||||
return done;
|
||||
}
|
||||
|
||||
function loadDataset(datasetPath: string): LongMemEvalQuestion[] {
|
||||
if (!existsSync(datasetPath)) {
|
||||
throw new Error(
|
||||
@@ -270,6 +322,24 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
return;
|
||||
}
|
||||
|
||||
// v0.35.1.0 --resume-from: filter out already-answered question_ids before
|
||||
// any model/brain setup so a no-op resume costs ~zero. Append-mode emitter
|
||||
// is only triggered when resume and output point at the same file.
|
||||
let appendOutput = false;
|
||||
if (opts.resumeFromPath) {
|
||||
const done = loadResumeSet(opts.resumeFromPath);
|
||||
const before = questions.length;
|
||||
questions = questions.filter(q => !done.has(q.question_id));
|
||||
process.stderr.write(`[longmemeval] resume: ${done.size} already done; ${questions.length}/${before} remaining\n`);
|
||||
if (opts.outputPath && opts.resumeFromPath === opts.outputPath) {
|
||||
appendOutput = true;
|
||||
}
|
||||
if (questions.length === 0) {
|
||||
process.stderr.write(`[longmemeval] resume: nothing to do (all questions already answered).\n`);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const model = await resolveModel(null, {
|
||||
cliFlag: opts.model,
|
||||
configKey: 'models.eval.longmemeval',
|
||||
@@ -288,7 +358,7 @@ export async function runEvalLongMemEval(args: string[], runOpts: RunOpts = {}):
|
||||
process.stderr.write(`[longmemeval] connecting in-memory brain...\n`);
|
||||
process.stderr.write(`[longmemeval] starting (questions: ${questions.length}, model: ${model}, expansion: ${opts.expansion ? 'on' : 'off'}${opts.mode ? `, mode: ${opts.mode}` : ''})\n`);
|
||||
|
||||
const emitter = makeEmitter(opts.outputPath);
|
||||
const emitter = makeEmitter(opts.outputPath, appendOutput);
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('eval.longmemeval', questions.length);
|
||||
|
||||
|
||||
@@ -31,9 +31,12 @@ export const EMBEDDING_PRICING: Record<string, EmbeddingPricing> = {
|
||||
'openai:text-embedding-3-small': { pricePerMTok: 0.02 },
|
||||
// Legacy OpenAI ada (still common in older brains)
|
||||
'openai:text-embedding-ada-002': { pricePerMTok: 0.10 },
|
||||
// Voyage (https://www.voyageai.com/pricing — voyage-3-large default)
|
||||
// Voyage (https://www.voyageai.com/pricing)
|
||||
'voyage:voyage-3-large': { pricePerMTok: 0.18 },
|
||||
'voyage:voyage-3': { pricePerMTok: 0.06 },
|
||||
'voyage:voyage-4-large': { pricePerMTok: 0.18 },
|
||||
// ZeroEntropy (https://zeroentropy.dev/pricing — zembed-1)
|
||||
'zeroentropyai:zembed-1': { pricePerMTok: 0.05 },
|
||||
};
|
||||
|
||||
export type PriceLookupResult =
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
/**
|
||||
* Pricing table contract — Voyage + ZeroEntropy coverage gate.
|
||||
*
|
||||
* The post-upgrade reembed cost prompt in `gbrain upgrade` falls back to
|
||||
* "estimate unavailable" on unknown providers, which is fine for safety
|
||||
* but bad UX if the provider IS in the recipe registry. These tests pin
|
||||
* the providers that v0.35.x officially supports as first-class.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
EMBEDDING_PRICING,
|
||||
lookupEmbeddingPrice,
|
||||
estimateCostFromChars,
|
||||
} from '../src/core/embedding-pricing.ts';
|
||||
|
||||
describe('lookupEmbeddingPrice — first-class providers', () => {
|
||||
test('OpenAI text-embedding-3-large at $0.13/MTok', () => {
|
||||
const r = lookupEmbeddingPrice('openai:text-embedding-3-large');
|
||||
expect(r.kind).toBe('known');
|
||||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.13);
|
||||
});
|
||||
|
||||
test('Voyage voyage-3-large at $0.18/MTok', () => {
|
||||
const r = lookupEmbeddingPrice('voyage:voyage-3-large');
|
||||
expect(r.kind).toBe('known');
|
||||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.18);
|
||||
});
|
||||
|
||||
test('Voyage voyage-4-large at $0.18/MTok (v0.35.1.0+)', () => {
|
||||
const r = lookupEmbeddingPrice('voyage:voyage-4-large');
|
||||
expect(r.kind).toBe('known');
|
||||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.18);
|
||||
});
|
||||
|
||||
test('ZeroEntropy zembed-1 at $0.05/MTok (v0.35.1.0+)', () => {
|
||||
const r = lookupEmbeddingPrice('zeroentropyai:zembed-1');
|
||||
expect(r.kind).toBe('known');
|
||||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.05);
|
||||
});
|
||||
});
|
||||
|
||||
describe('lookupEmbeddingPrice — fall-through behavior', () => {
|
||||
test('returns unknown for bogus provider', () => {
|
||||
const r = lookupEmbeddingPrice('madeup:model-9000');
|
||||
expect(r.kind).toBe('unknown');
|
||||
if (r.kind === 'unknown') {
|
||||
expect(r.provider).toBe('madeup');
|
||||
expect(r.model).toBe('model-9000');
|
||||
}
|
||||
});
|
||||
|
||||
test('bare model strings default to openai', () => {
|
||||
const r = lookupEmbeddingPrice('text-embedding-3-small');
|
||||
expect(r.kind).toBe('known');
|
||||
if (r.kind === 'known') expect(r.key).toBe('openai:text-embedding-3-small');
|
||||
});
|
||||
|
||||
test('provider name is case-insensitive', () => {
|
||||
const r = lookupEmbeddingPrice('ZeroEntropyAI:zembed-1');
|
||||
expect(r.kind).toBe('known');
|
||||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.05);
|
||||
});
|
||||
});
|
||||
|
||||
describe('EMBEDDING_PRICING — table integrity', () => {
|
||||
test('all entries have pricePerMTok as a non-negative finite number', () => {
|
||||
for (const [key, val] of Object.entries(EMBEDDING_PRICING)) {
|
||||
expect(Number.isFinite(val.pricePerMTok)).toBe(true);
|
||||
expect(val.pricePerMTok).toBeGreaterThanOrEqual(0);
|
||||
expect(key).toContain(':');
|
||||
}
|
||||
});
|
||||
|
||||
test('keys use lowercase provider names', () => {
|
||||
for (const key of Object.keys(EMBEDDING_PRICING)) {
|
||||
const provider = key.split(':')[0];
|
||||
expect(provider).toBe(provider.toLowerCase());
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('estimateCostFromChars', () => {
|
||||
test('returns 0 for 0 chars', () => {
|
||||
expect(estimateCostFromChars(0, 0.13)).toBe(0);
|
||||
});
|
||||
|
||||
test('100M chars @ $0.13/MTok ≈ $3.71 (100M / 3.5 ≈ 28.57M tokens × 0.13)', () => {
|
||||
const c = estimateCostFromChars(100_000_000, 0.13);
|
||||
expect(c).toBeGreaterThan(3.7);
|
||||
expect(c).toBeLessThan(3.8);
|
||||
});
|
||||
});
|
||||
@@ -21,7 +21,7 @@ import {
|
||||
withBenchmarkBrain,
|
||||
} from '../src/eval/longmemeval/harness.ts';
|
||||
import { haystackToPages, type LongMemEvalQuestion } from '../src/eval/longmemeval/adapter.ts';
|
||||
import { runEvalLongMemEval } from '../src/commands/eval-longmemeval.ts';
|
||||
import { runEvalLongMemEval, loadResumeSet } from '../src/commands/eval-longmemeval.ts';
|
||||
import { importFromContent } from '../src/core/import-file.ts';
|
||||
import { DEFAULT_SOURCE_BOOSTS } from '../src/core/search/source-boost.ts';
|
||||
import type { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
@@ -466,3 +466,152 @@ describe('per-question failure handling', () => {
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// 13. v0.35.1.0: --resume-from
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('loadResumeSet (v0.35.1.0)', () => {
|
||||
test('returns empty set when path does not exist', () => {
|
||||
const set = loadResumeSet('/nonexistent/path/never/exists.jsonl');
|
||||
expect(set.size).toBe(0);
|
||||
});
|
||||
|
||||
test('reads question_ids from a well-formed JSONL', async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'lme-resume-'));
|
||||
const p = join(tmp, 'partial.jsonl');
|
||||
const { writeFileSync } = await import('fs');
|
||||
try {
|
||||
writeFileSync(
|
||||
p,
|
||||
[
|
||||
JSON.stringify({ question_id: 'a', hypothesis: 'one' }),
|
||||
JSON.stringify({ question_id: 'b', hypothesis: 'two' }),
|
||||
].join('\n') + '\n',
|
||||
'utf8',
|
||||
);
|
||||
const set = loadResumeSet(p);
|
||||
expect(set.size).toBe(2);
|
||||
expect(set.has('a')).toBe(true);
|
||||
expect(set.has('b')).toBe(true);
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('skips rows whose hypothesis is empty AND error is set (retry case)', async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'lme-resume-'));
|
||||
const p = join(tmp, 'with-errors.jsonl');
|
||||
const { writeFileSync } = await import('fs');
|
||||
try {
|
||||
writeFileSync(
|
||||
p,
|
||||
[
|
||||
JSON.stringify({ question_id: 'good', hypothesis: 'real-answer' }),
|
||||
JSON.stringify({ question_id: 'bad', hypothesis: '', error: 'rate-limit' }),
|
||||
JSON.stringify({ question_id: 'recovered', hypothesis: 'second-try', error: 'old-error' }),
|
||||
].join('\n') + '\n',
|
||||
'utf8',
|
||||
);
|
||||
const set = loadResumeSet(p);
|
||||
// 'bad' is retried; 'good' and 'recovered' are kept (hypothesis non-empty).
|
||||
expect(set.size).toBe(2);
|
||||
expect(set.has('good')).toBe(true);
|
||||
expect(set.has('bad')).toBe(false);
|
||||
expect(set.has('recovered')).toBe(true);
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('tolerates a truncated/corrupt final line (SIGKILL recovery case)', async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'lme-resume-'));
|
||||
const p = join(tmp, 'truncated.jsonl');
|
||||
const { writeFileSync } = await import('fs');
|
||||
try {
|
||||
writeFileSync(
|
||||
p,
|
||||
JSON.stringify({ question_id: 'a', hypothesis: 'one' }) + '\n' +
|
||||
'{"question_id":"b","hypothesis":"two-trunc' /* no closing brace, no LF */,
|
||||
'utf8',
|
||||
);
|
||||
const set = loadResumeSet(p);
|
||||
// First line counts; second is silently skipped (stderr warn).
|
||||
expect(set.size).toBe(1);
|
||||
expect(set.has('a')).toBe(true);
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('runEvalLongMemEval --resume-from (v0.35.1.0)', () => {
|
||||
test('skips already-answered questions and appends to the same output file', async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'lme-resume-'));
|
||||
const outPath = join(tmp, 'hypothesis.jsonl');
|
||||
try {
|
||||
// Simulate prior run: 2 questions already answered, written to the file
|
||||
// with hypothesis set. The fixture has 5 questions total.
|
||||
const { writeFileSync } = await import('fs');
|
||||
const fixture = readFileSync(FIXTURE_PATH, 'utf8')
|
||||
.split('\n').filter(l => l.length > 0).map(l => JSON.parse(l));
|
||||
writeFileSync(
|
||||
outPath,
|
||||
[
|
||||
JSON.stringify({ question_id: fixture[0].question_id, hypothesis: 'prior-1' }),
|
||||
JSON.stringify({ question_id: fixture[1].question_id, hypothesis: 'prior-2' }),
|
||||
].join('\n') + '\n',
|
||||
'utf8',
|
||||
);
|
||||
|
||||
const { client } = makeStubClient('resumed-answer');
|
||||
await runEvalLongMemEval(
|
||||
[FIXTURE_PATH, '--keyword-only', '--limit', '5', '--top-k', '3',
|
||||
'--output', outPath, '--resume-from', outPath],
|
||||
{ client },
|
||||
);
|
||||
|
||||
const text = readFileSync(outPath, 'utf8');
|
||||
const lines = text.split('\n').filter(l => l.length > 0).map(l => JSON.parse(l));
|
||||
// 2 prior rows + 3 new rows = 5 total
|
||||
expect(lines.length).toBe(5);
|
||||
// First two preserve their prior hypothesis (proves append, not truncate).
|
||||
expect(lines[0].hypothesis).toBe('prior-1');
|
||||
expect(lines[1].hypothesis).toBe('prior-2');
|
||||
// Newly-answered three carry the canned stub.
|
||||
for (let i = 2; i < 5; i++) {
|
||||
expect(lines[i].hypothesis).toContain('resumed-answer');
|
||||
}
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test('all questions already done -> early return, no client calls', async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'lme-resume-'));
|
||||
const outPath = join(tmp, 'all-done.jsonl');
|
||||
try {
|
||||
const { writeFileSync } = await import('fs');
|
||||
const fixture = readFileSync(FIXTURE_PATH, 'utf8')
|
||||
.split('\n').filter(l => l.length > 0).map(l => JSON.parse(l)).slice(0, 5);
|
||||
writeFileSync(
|
||||
outPath,
|
||||
fixture.map(q => JSON.stringify({ question_id: q.question_id, hypothesis: 'done' })).join('\n') + '\n',
|
||||
'utf8',
|
||||
);
|
||||
const { client, calls } = makeStubClient('should-not-be-called');
|
||||
await runEvalLongMemEval(
|
||||
[FIXTURE_PATH, '--keyword-only', '--limit', '5',
|
||||
'--output', outPath, '--resume-from', outPath],
|
||||
{ client },
|
||||
);
|
||||
// The client must not have been invoked at all — every question was skipped.
|
||||
expect(calls.length).toBe(0);
|
||||
// The output file is untouched (no new lines appended).
|
||||
const lines = readFileSync(outPath, 'utf8').split('\n').filter(l => l.length > 0);
|
||||
expect(lines.length).toBe(5);
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
@@ -49,6 +49,7 @@ const EXPECTED_EXPORTS: ExpectedExport[] = [
|
||||
{ subpath: 'gbrain/backoff', canary: [] },
|
||||
{ subpath: 'gbrain/search/hybrid', canary: ['hybridSearch', 'rrfFusion'] },
|
||||
{ subpath: 'gbrain/search/expansion', canary: ['expandQuery'] },
|
||||
{ subpath: 'gbrain/ai/gateway', canary: ['configureGateway', 'embed'] },
|
||||
{ subpath: 'gbrain/extract', canary: [] },
|
||||
];
|
||||
|
||||
@@ -65,7 +66,7 @@ describe('public exports — package.json exports map', () => {
|
||||
// Adding new exports: increment this + add to EXPECTED_EXPORTS below.
|
||||
// Removing exports: see CLAUDE.md "Removing any of these is a
|
||||
// breaking change going forward" — bump minor and update this count.
|
||||
expect(count).toBe(17);
|
||||
expect(count).toBe(18);
|
||||
});
|
||||
|
||||
test('EXPECTED_EXPORTS list matches the exports map exactly (no drift)', () => {
|
||||
|
||||
Reference in New Issue
Block a user