Files
gbrain/test/embedding-pricing.test.ts
T
3933eb6a79 v0.35.1.0: embedder shootout prereqs (pricing + gateway export + --resume-from) (#1055)
* docs(designs): 2026-05 embedder shootout eval plan

Adds docs/designs/2026_05_EVAL_PLAN.md — the approved plan + 6 Conductor session
briefs for the OpenAI vs Voyage vs ZeroEntropy embedder comparison.

Why: produce a publishable comparison report for v0.35.x release notes pinning
"which embedder wins, and does zerank-2 carry the win for ZeroEntropy" against
public LongMemEval + in-house BrainBench.

Each session brief is self-contained — repo, branch, commits, verify, ship,
deliverable, hand-off. Stewardable one section per Conductor session.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(pricing): add voyage-4-large + zembed-1 to EMBEDDING_PRICING

v0.35.0.0 shipped ZeroEntropy zembed-1 + zerank-2 reranker support and
expanded the Voyage allow-list to include voyage-4-large. The pricing
table missed both, so `gbrain upgrade`'s post-upgrade reembed prompt
silently fell back to "estimate unavailable" for users on these models.

- voyage:voyage-4-large @ $0.18/MTok (same as voyage-3-large)
- zeroentropyai:zembed-1 @ $0.05/MTok

New test file pins both entries plus the openai/voyage-3-large baselines,
case-insensitive provider matching, bare-model openai-default fallback,
table integrity (lowercase providers, finite non-negative prices), and
the estimateCostFromChars approximation. 11 cases, 46 expect() calls.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(exports): expose gbrain/ai/gateway with canary test

Adds ./ai/gateway to the package.json exports map so external eval
consumers (notably gbrain-evals, the sibling repo running the embedder
shootout in docs/designs/2026_05_EVAL_PLAN.md) can call configureGateway
directly to swap embedding providers per cell.

Why: pre-v0.35.1.0, gbrain-evals adapters hardcoded gbrain/embedding,
which means every retrieval adapter was OpenAI-only. The newly-exposed
gateway lets adapters route through Voyage and ZeroEntropy without
forking gbrain or duplicating the recipe wiring.

- package.json: add "./ai/gateway" -> "./src/core/ai/gateway.ts"
- scripts/check-exports-count.sh: bump expected count 17 -> 18
- test/public-exports.test.ts: add canary pinning configureGateway + embed,
  bump expected count assertion

Pre-existing import-resolution failures in this test file (16 on master)
are unrelated to this change — they're a longstanding Bun package
self-import behavior. The count + EXPECTED_EXPORTS list-match assertions
both pass cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): add --resume-from <jsonl> to gbrain eval longmemeval

Multi-cell embedder shootouts spend $50+/cell on the gpt-4o judge after
gbrain emits hypotheses. A mid-run abort (rate-limit, cost-cap, OS
interrupt, SIGKILL) previously meant re-paying the full cell. This flag
makes those aborts cheap: re-invoke with --resume-from pointed at the
partial JSONL and only the unanswered question_ids re-run.

Behavior:
- Read question_ids from the file; skip them on this run.
- Rows with non-empty hypothesis count as done.
- Rows with hypothesis="" AND an error field are NOT skipped (retry case
  for per-question failures recorded by the existing try/catch).
- Corrupt trailing lines (SIGKILL'd writer mid-line) are silently skipped
  with a stderr warn.
- When --resume-from path == --output path, the output emitter opens the
  file in append mode instead of truncating, so the existing rows survive.
- Empty resume case (all questions already done) returns immediately
  without spinning up the brain or calling the client.

New exported helper loadResumeSet() makes the parser unit-testable.

6 new test cases pinning:
- File-not-found returns empty set
- Well-formed JSONL load
- Error-row retry semantics (empty hypothesis + error -> not in set)
- Truncated final line recovery
- End-to-end resume against the 5-question mini fixture
- All-done early-return (stub client must NOT be invoked)

All 18 cases in test/eval-longmemeval.test.ts green; bun run typecheck
clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: v0.35.1.0

Bumps VERSION + package.json + CHANGELOG entry for the embedder-shootout
prereq release. Three additive changes from the prior 4 commits:

- pricing: voyage-4-large + zembed-1 entries
- exports: gbrain/ai/gateway is now public
- eval: gbrain eval longmemeval --resume-from <jsonl>

Each commit on this branch is independently bisect-friendly and CI-green;
the CHANGELOG entry is the user-facing rollup. No migrations, no breaking
changes — the gateway export expands the surface, the resume-from flag is
additive, the pricing patch only changes "estimate unavailable" -> a real
dollar figure for two specific models.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 18:54:36 -07:00

93 lines
3.2 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* Pricing table contract — Voyage + ZeroEntropy coverage gate.
*
* The post-upgrade reembed cost prompt in `gbrain upgrade` falls back to
* "estimate unavailable" on unknown providers, which is fine for safety
* but bad UX if the provider IS in the recipe registry. These tests pin
* the providers that v0.35.x officially supports as first-class.
*/
import { describe, test, expect } from 'bun:test';
import {
EMBEDDING_PRICING,
lookupEmbeddingPrice,
estimateCostFromChars,
} from '../src/core/embedding-pricing.ts';
describe('lookupEmbeddingPrice — first-class providers', () => {
test('OpenAI text-embedding-3-large at $0.13/MTok', () => {
const r = lookupEmbeddingPrice('openai:text-embedding-3-large');
expect(r.kind).toBe('known');
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.13);
});
test('Voyage voyage-3-large at $0.18/MTok', () => {
const r = lookupEmbeddingPrice('voyage:voyage-3-large');
expect(r.kind).toBe('known');
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.18);
});
test('Voyage voyage-4-large at $0.18/MTok (v0.35.1.0+)', () => {
const r = lookupEmbeddingPrice('voyage:voyage-4-large');
expect(r.kind).toBe('known');
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.18);
});
test('ZeroEntropy zembed-1 at $0.05/MTok (v0.35.1.0+)', () => {
const r = lookupEmbeddingPrice('zeroentropyai:zembed-1');
expect(r.kind).toBe('known');
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.05);
});
});
describe('lookupEmbeddingPrice — fall-through behavior', () => {
test('returns unknown for bogus provider', () => {
const r = lookupEmbeddingPrice('madeup:model-9000');
expect(r.kind).toBe('unknown');
if (r.kind === 'unknown') {
expect(r.provider).toBe('madeup');
expect(r.model).toBe('model-9000');
}
});
test('bare model strings default to openai', () => {
const r = lookupEmbeddingPrice('text-embedding-3-small');
expect(r.kind).toBe('known');
if (r.kind === 'known') expect(r.key).toBe('openai:text-embedding-3-small');
});
test('provider name is case-insensitive', () => {
const r = lookupEmbeddingPrice('ZeroEntropyAI:zembed-1');
expect(r.kind).toBe('known');
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.05);
});
});
describe('EMBEDDING_PRICING — table integrity', () => {
test('all entries have pricePerMTok as a non-negative finite number', () => {
for (const [key, val] of Object.entries(EMBEDDING_PRICING)) {
expect(Number.isFinite(val.pricePerMTok)).toBe(true);
expect(val.pricePerMTok).toBeGreaterThanOrEqual(0);
expect(key).toContain(':');
}
});
test('keys use lowercase provider names', () => {
for (const key of Object.keys(EMBEDDING_PRICING)) {
const provider = key.split(':')[0];
expect(provider).toBe(provider.toLowerCase());
}
});
});
describe('estimateCostFromChars', () => {
test('returns 0 for 0 chars', () => {
expect(estimateCostFromChars(0, 0.13)).toBe(0);
});
test('100M chars @ $0.13/MTok ≈ $3.71 (100M / 3.5 ≈ 28.57M tokens × 0.13)', () => {
const c = estimateCostFromChars(100_000_000, 0.13);
expect(c).toBeGreaterThan(3.7);
expect(c).toBeLessThan(3.8);
});
});