* docs(designs): 2026-05 embedder shootout eval plan
Adds docs/designs/2026_05_EVAL_PLAN.md — the approved plan + 6 Conductor session
briefs for the OpenAI vs Voyage vs ZeroEntropy embedder comparison.
Why: produce a publishable comparison report for v0.35.x release notes pinning
"which embedder wins, and does zerank-2 carry the win for ZeroEntropy" against
public LongMemEval + in-house BrainBench.
Each session brief is self-contained — repo, branch, commits, verify, ship,
deliverable, hand-off. Stewardable one section per Conductor session.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(pricing): add voyage-4-large + zembed-1 to EMBEDDING_PRICING
v0.35.0.0 shipped ZeroEntropy zembed-1 + zerank-2 reranker support and
expanded the Voyage allow-list to include voyage-4-large. The pricing
table missed both, so `gbrain upgrade`'s post-upgrade reembed prompt
silently fell back to "estimate unavailable" for users on these models.
- voyage:voyage-4-large @ $0.18/MTok (same as voyage-3-large)
- zeroentropyai:zembed-1 @ $0.05/MTok
New test file pins both entries plus the openai/voyage-3-large baselines,
case-insensitive provider matching, bare-model openai-default fallback,
table integrity (lowercase providers, finite non-negative prices), and
the estimateCostFromChars approximation. 11 cases, 46 expect() calls.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(exports): expose gbrain/ai/gateway with canary test
Adds ./ai/gateway to the package.json exports map so external eval
consumers (notably gbrain-evals, the sibling repo running the embedder
shootout in docs/designs/2026_05_EVAL_PLAN.md) can call configureGateway
directly to swap embedding providers per cell.
Why: pre-v0.35.1.0, gbrain-evals adapters hardcoded gbrain/embedding,
which means every retrieval adapter was OpenAI-only. The newly-exposed
gateway lets adapters route through Voyage and ZeroEntropy without
forking gbrain or duplicating the recipe wiring.
- package.json: add "./ai/gateway" -> "./src/core/ai/gateway.ts"
- scripts/check-exports-count.sh: bump expected count 17 -> 18
- test/public-exports.test.ts: add canary pinning configureGateway + embed,
bump expected count assertion
Pre-existing import-resolution failures in this test file (16 on master)
are unrelated to this change — they're a longstanding Bun package
self-import behavior. The count + EXPECTED_EXPORTS list-match assertions
both pass cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(eval): add --resume-from <jsonl> to gbrain eval longmemeval
Multi-cell embedder shootouts spend $50+/cell on the gpt-4o judge after
gbrain emits hypotheses. A mid-run abort (rate-limit, cost-cap, OS
interrupt, SIGKILL) previously meant re-paying the full cell. This flag
makes those aborts cheap: re-invoke with --resume-from pointed at the
partial JSONL and only the unanswered question_ids re-run.
Behavior:
- Read question_ids from the file; skip them on this run.
- Rows with non-empty hypothesis count as done.
- Rows with hypothesis="" AND an error field are NOT skipped (retry case
for per-question failures recorded by the existing try/catch).
- Corrupt trailing lines (SIGKILL'd writer mid-line) are silently skipped
with a stderr warn.
- When --resume-from path == --output path, the output emitter opens the
file in append mode instead of truncating, so the existing rows survive.
- Empty resume case (all questions already done) returns immediately
without spinning up the brain or calling the client.
New exported helper loadResumeSet() makes the parser unit-testable.
6 new test cases pinning:
- File-not-found returns empty set
- Well-formed JSONL load
- Error-row retry semantics (empty hypothesis + error -> not in set)
- Truncated final line recovery
- End-to-end resume against the 5-question mini fixture
- All-done early-return (stub client must NOT be invoked)
All 18 cases in test/eval-longmemeval.test.ts green; bun run typecheck
clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: v0.35.1.0
Bumps VERSION + package.json + CHANGELOG entry for the embedder-shootout
prereq release. Three additive changes from the prior 4 commits:
- pricing: voyage-4-large + zembed-1 entries
- exports: gbrain/ai/gateway is now public
- eval: gbrain eval longmemeval --resume-from <jsonl>
Each commit on this branch is independently bisect-friendly and CI-green;
the CHANGELOG entry is the user-facing rollup. No migrations, no breaking
changes — the gateway export expands the surface, the resume-from flag is
additive, the pricing patch only changes "estimate unavailable" -> a real
dollar figure for two specific models.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(eval): longmemeval adapter handles _s split + sanitizes session_id slugs
Three tightly-coupled bugs blocked `gbrain eval longmemeval` against the
public LongMemEval _s split from HuggingFace (the dataset every shootout
cell needs):
1. HAYSTACK SHAPE: the _s split serializes haystack_sessions as
LongMemEvalTurn[][] (each inner array is one session's turns directly)
plus a parallel `haystack_session_ids: string[]` field. The
pre-v0.35.1.1 adapter expected only the oracle `{session_id, turns}`
shape and crashed with `session.turns is undefined` on every question.
Fix: new `normalizeSessions` helper accepts both shapes, mirroring the
proven `normalizeSessions` in gbrain-evals/eval/runner/longmemeval.ts.
2. SLUG VALIDATOR: the _s split's session_ids look like
`sharegpt_yywfIrx_0` — underscored and mixed-case. The v0.32.7 CJK
wave's `validatePageSlug` rejects both (allowed set is `[a-z0-9-]`
case-insensitive, slash-separated). Fix: `sanitizeSessionIdForSlug`
lowercases and replaces `_` + `.` + any other non-[a-z0-9-] character
with `-`. The frontmatter `session_id:` keeps the original verbatim
for downstream JSONL emit; only the SLUG is rewritten.
3. INTERFACE: `LongMemEvalQuestion.haystack_sessions` typed as a union
of `LongMemEvalSession[] | LongMemEvalTurn[][]` so TypeScript callers
see both shapes are accepted. New `haystack_session_ids?: string[]`
field documented as parallel to the array-of-turns shape.
Pre-v0.35.1.1 caught by a fresh smoke pre-spend (3 questions × ZE @ 2560
→ 3 errors). Post-fix: 3/3 OK with non-empty hypotheses, single-session
recall measured (low on a 3-question sample but the pipeline runs).
2 new regression test cases pinning:
- _s split shape normalizes (slugs sanitized + frontmatter preserves
original session_id + dates flow through)
- _s split with missing haystack_session_ids synthesizes
`lme_<question_id>_<i>` ids
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): configure AI gateway before running gbrain eval longmemeval
v0.28.8 skipped connectEngine() for `gbrain eval longmemeval` so the
subcommand could run on machines without a configured brain. Side
effect (silent until v0.35.1.0 made it observable via the embedder
shootout): the gateway was never configureGateway()'d either, so the
first embed call inside importFromContent crashed with "AI gateway is
not configured. Call configureGateway() during engine connect."
Fix: call configureGateway() before runEvalLongMemEval, mirroring the
connectEngine() path. Reads `~/.gbrain/config.json` when present; falls
back to env vars (GBRAIN_EMBEDDING_MODEL, GBRAIN_EMBEDDING_DIMENSIONS,
OPENAI_API_KEY, etc.) when there's no config — preserving the v0.28.8
"runs on fresh machine" property.
Gated on the --help short-circuit so `gbrain eval longmemeval --help`
still works without spinning up the gateway.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: v0.35.1.1
Bumps VERSION + package.json + CHANGELOG entry for the longmemeval fix
wave. Three commits this branch:
1. fix(eval): adapter handles _s split + sanitizes session_id slugs
2. fix(cli): configure AI gateway before running gbrain eval longmemeval
3. chore: v0.35.1.1
Each commit independently bisects; CHANGELOG entry is the user-facing
rollup. No schema migration; no breaking change.
Caught pre-spend by smoking Phase 1 of the embedder shootout — would
otherwise have wasted ~$476 in judge tokens across 7 cells.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* ci: retrigger workflows
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
