mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* docs(designs): 2026-05 embedder shootout eval plan Adds docs/designs/2026_05_EVAL_PLAN.md — the approved plan + 6 Conductor session briefs for the OpenAI vs Voyage vs ZeroEntropy embedder comparison. Why: produce a publishable comparison report for v0.35.x release notes pinning "which embedder wins, and does zerank-2 carry the win for ZeroEntropy" against public LongMemEval + in-house BrainBench. Each session brief is self-contained — repo, branch, commits, verify, ship, deliverable, hand-off. Stewardable one section per Conductor session. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(pricing): add voyage-4-large + zembed-1 to EMBEDDING_PRICING v0.35.0.0 shipped ZeroEntropy zembed-1 + zerank-2 reranker support and expanded the Voyage allow-list to include voyage-4-large. The pricing table missed both, so `gbrain upgrade`'s post-upgrade reembed prompt silently fell back to "estimate unavailable" for users on these models. - voyage:voyage-4-large @ $0.18/MTok (same as voyage-3-large) - zeroentropyai:zembed-1 @ $0.05/MTok New test file pins both entries plus the openai/voyage-3-large baselines, case-insensitive provider matching, bare-model openai-default fallback, table integrity (lowercase providers, finite non-negative prices), and the estimateCostFromChars approximation. 11 cases, 46 expect() calls. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(exports): expose gbrain/ai/gateway with canary test Adds ./ai/gateway to the package.json exports map so external eval consumers (notably gbrain-evals, the sibling repo running the embedder shootout in docs/designs/2026_05_EVAL_PLAN.md) can call configureGateway directly to swap embedding providers per cell. Why: pre-v0.35.1.0, gbrain-evals adapters hardcoded gbrain/embedding, which means every retrieval adapter was OpenAI-only. The newly-exposed gateway lets adapters route through Voyage and ZeroEntropy without forking gbrain or duplicating the recipe wiring. - package.json: add "./ai/gateway" -> "./src/core/ai/gateway.ts" - scripts/check-exports-count.sh: bump expected count 17 -> 18 - test/public-exports.test.ts: add canary pinning configureGateway + embed, bump expected count assertion Pre-existing import-resolution failures in this test file (16 on master) are unrelated to this change — they're a longstanding Bun package self-import behavior. The count + EXPECTED_EXPORTS list-match assertions both pass cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): add --resume-from <jsonl> to gbrain eval longmemeval Multi-cell embedder shootouts spend $50+/cell on the gpt-4o judge after gbrain emits hypotheses. A mid-run abort (rate-limit, cost-cap, OS interrupt, SIGKILL) previously meant re-paying the full cell. This flag makes those aborts cheap: re-invoke with --resume-from pointed at the partial JSONL and only the unanswered question_ids re-run. Behavior: - Read question_ids from the file; skip them on this run. - Rows with non-empty hypothesis count as done. - Rows with hypothesis="" AND an error field are NOT skipped (retry case for per-question failures recorded by the existing try/catch). - Corrupt trailing lines (SIGKILL'd writer mid-line) are silently skipped with a stderr warn. - When --resume-from path == --output path, the output emitter opens the file in append mode instead of truncating, so the existing rows survive. - Empty resume case (all questions already done) returns immediately without spinning up the brain or calling the client. New exported helper loadResumeSet() makes the parser unit-testable. 6 new test cases pinning: - File-not-found returns empty set - Well-formed JSONL load - Error-row retry semantics (empty hypothesis + error -> not in set) - Truncated final line recovery - End-to-end resume against the 5-question mini fixture - All-done early-return (stub client must NOT be invoked) All 18 cases in test/eval-longmemeval.test.ts green; bun run typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: v0.35.1.0 Bumps VERSION + package.json + CHANGELOG entry for the embedder-shootout prereq release. Three additive changes from the prior 4 commits: - pricing: voyage-4-large + zembed-1 entries - exports: gbrain/ai/gateway is now public - eval: gbrain eval longmemeval --resume-from <jsonl> Each commit on this branch is independently bisect-friendly and CI-green; the CHANGELOG entry is the user-facing rollup. No migrations, no breaking changes — the gateway export expands the surface, the resume-from flag is additive, the pricing patch only changes "estimate unavailable" -> a real dollar figure for two specific models. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
93 lines
3.2 KiB
TypeScript
93 lines
3.2 KiB
TypeScript
/**
|
||
* Pricing table contract — Voyage + ZeroEntropy coverage gate.
|
||
*
|
||
* The post-upgrade reembed cost prompt in `gbrain upgrade` falls back to
|
||
* "estimate unavailable" on unknown providers, which is fine for safety
|
||
* but bad UX if the provider IS in the recipe registry. These tests pin
|
||
* the providers that v0.35.x officially supports as first-class.
|
||
*/
|
||
import { describe, test, expect } from 'bun:test';
|
||
import {
|
||
EMBEDDING_PRICING,
|
||
lookupEmbeddingPrice,
|
||
estimateCostFromChars,
|
||
} from '../src/core/embedding-pricing.ts';
|
||
|
||
describe('lookupEmbeddingPrice — first-class providers', () => {
|
||
test('OpenAI text-embedding-3-large at $0.13/MTok', () => {
|
||
const r = lookupEmbeddingPrice('openai:text-embedding-3-large');
|
||
expect(r.kind).toBe('known');
|
||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.13);
|
||
});
|
||
|
||
test('Voyage voyage-3-large at $0.18/MTok', () => {
|
||
const r = lookupEmbeddingPrice('voyage:voyage-3-large');
|
||
expect(r.kind).toBe('known');
|
||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.18);
|
||
});
|
||
|
||
test('Voyage voyage-4-large at $0.18/MTok (v0.35.1.0+)', () => {
|
||
const r = lookupEmbeddingPrice('voyage:voyage-4-large');
|
||
expect(r.kind).toBe('known');
|
||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.18);
|
||
});
|
||
|
||
test('ZeroEntropy zembed-1 at $0.05/MTok (v0.35.1.0+)', () => {
|
||
const r = lookupEmbeddingPrice('zeroentropyai:zembed-1');
|
||
expect(r.kind).toBe('known');
|
||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.05);
|
||
});
|
||
});
|
||
|
||
describe('lookupEmbeddingPrice — fall-through behavior', () => {
|
||
test('returns unknown for bogus provider', () => {
|
||
const r = lookupEmbeddingPrice('madeup:model-9000');
|
||
expect(r.kind).toBe('unknown');
|
||
if (r.kind === 'unknown') {
|
||
expect(r.provider).toBe('madeup');
|
||
expect(r.model).toBe('model-9000');
|
||
}
|
||
});
|
||
|
||
test('bare model strings default to openai', () => {
|
||
const r = lookupEmbeddingPrice('text-embedding-3-small');
|
||
expect(r.kind).toBe('known');
|
||
if (r.kind === 'known') expect(r.key).toBe('openai:text-embedding-3-small');
|
||
});
|
||
|
||
test('provider name is case-insensitive', () => {
|
||
const r = lookupEmbeddingPrice('ZeroEntropyAI:zembed-1');
|
||
expect(r.kind).toBe('known');
|
||
if (r.kind === 'known') expect(r.pricePerMTok).toBe(0.05);
|
||
});
|
||
});
|
||
|
||
describe('EMBEDDING_PRICING — table integrity', () => {
|
||
test('all entries have pricePerMTok as a non-negative finite number', () => {
|
||
for (const [key, val] of Object.entries(EMBEDDING_PRICING)) {
|
||
expect(Number.isFinite(val.pricePerMTok)).toBe(true);
|
||
expect(val.pricePerMTok).toBeGreaterThanOrEqual(0);
|
||
expect(key).toContain(':');
|
||
}
|
||
});
|
||
|
||
test('keys use lowercase provider names', () => {
|
||
for (const key of Object.keys(EMBEDDING_PRICING)) {
|
||
const provider = key.split(':')[0];
|
||
expect(provider).toBe(provider.toLowerCase());
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('estimateCostFromChars', () => {
|
||
test('returns 0 for 0 chars', () => {
|
||
expect(estimateCostFromChars(0, 0.13)).toBe(0);
|
||
});
|
||
|
||
test('100M chars @ $0.13/MTok ≈ $3.71 (100M / 3.5 ≈ 28.57M tokens × 0.13)', () => {
|
||
const c = estimateCostFromChars(100_000_000, 0.13);
|
||
expect(c).toBeGreaterThan(3.7);
|
||
expect(c).toBeLessThan(3.8);
|
||
});
|
||
});
|