diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f9d14174..11365ae68 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,35 @@ All notable changes to GBrain will be documented in this file. +## [0.42.25.0] - 2026-06-03 + +**Cost caps and budget gates now fire on Opus 4.8 — and every model price lives in one place, so they can't silently drift again.** If you pointed a gbrain tier at Opus 4.8 (`models.aliases.opus`), the cost guardrails were quietly not working: there was no price on file for 4.8, so the `gbrain dream` budget meter let runs proceed unbounded (it warns `BUDGET_METER_NO_PRICING` and skips the gate for unpriced models), and `gbrain skillopt --max-cost-usd` fell back to a cheaper tier's rate and refused too late. This release adds Opus 4.8 pricing ($5 in / $25 out per 1M tokens, same as 4.7) and the caps enforce again. + +While fixing that, the deeper problem surfaced: model prices were hand-copied across five separate tables and had already drifted apart. One eval's budget gate priced Opus 4.7 at a stale $15/$75 (3x too high), and Gemini 2.0 Flash disagreed with itself between two tables. All chat-model prices now live in one canonical table (`src/core/model-pricing.ts`) and every other table derives from it — so cross-table price drift is structurally impossible, not just discouraged. + +Nothing to configure. `gbrain upgrade` and your caps start enforcing on 4.8. + +### Added +- Opus 4.8 pricing ($5 in / $25 out per 1M tokens) so `--max-cost-usd` and the dream-cycle budget meter enforce on 4.8 runs. + +### Changed +- All chat-model pricing unified into one canonical source. The Anthropic bare-key table, the takes-quality eval allowlist, the contradictions cost-tracker, the cross-modal eval panel, and the skillopt preflight estimate all derive their numbers from it instead of carrying their own copies. +- skillopt preflight now resolves bare, colon, and slash model ids through the shared parser (previously bare-only), so provider-prefixed 4.8 ids price correctly. + +### Fixed +- Stale Opus 4.7 price in the takes-quality eval budget gate ($15/$75 → $5/$25), which over-estimated cost ~3x. +- Gemini 2.0 Flash price reconciled to $0.10/$0.40 across the budget-gating tables (the coarse per-provider baselines shown by `gbrain providers` are a separate display layer, unchanged here). +- Brainstorm and brain-score cost estimates now price provider-prefixed model ids (e.g. `anthropic:claude-opus-4-8`) instead of silently falling back to a default rate. + +### To take advantage of v0.42.25.0 + +`gbrain upgrade` applies this automatically — no migrations, no config. To confirm: + +1. With Opus 4.8 as your deep tier, run a capped skillopt dry-run and watch the estimate refuse above the cap: + ```bash + gbrain skillopt --bootstrap-from-skill --dry-run --max-cost-usd 1 + ``` +2. The dream-cycle budget meter no longer prints `BUDGET_METER_NO_PRICING` for Opus 4.8. ## [0.42.24.0] - 2026-06-03 **Minion workers no longer silently wedge mid-job on a Supabase brain.** The background worker that runs your cron jobs, enrich fan-out, and autopilot cycle holds a lock on each job and heartbeats it every couple of seconds to say "still working." On a Supabase brain that heartbeat was running on the transaction-mode pooler (the high-traffic 6543 port), which recycles its connections per transaction. A lock is held open for minutes, so the pooler would periodically drop the socket mid-heartbeat. The worker read that dropped socket as "the lock expired," force-evicted its own in-flight job, and then sat in a claim loop holding nothing — process alive, no errors in the log, just quietly doing no work. It showed up most under heavy `enrich` load. diff --git a/CLAUDE.md b/CLAUDE.md index d20a7af19..cbdb58c52 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -74,6 +74,15 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`. Postgres; plain `CREATE INDEX` on PGLite via `sqlFor.pglite`). - **Multi-source.** Slug uniqueness is `(source_id, slug)`, not slug. Key batch ops and reverse-writes on the composite key; `validateSourceId` before any `source_id` path join. +- **One canonical chat-pricing table.** All paid-cloud chat/completion prices live ONCE in + `src/core/model-pricing.ts` (`CANONICAL_PRICING` + `canonicalLookup`). Every other table + (`anthropic-pricing.ts`'s `ANTHROPIC_PRICING`, `takes-quality-eval/pricing.ts`'s + `MODEL_PRICING`, the contradictions/cross-modal/skillopt cost views) is a DERIVED view, never + a hand-copied duplicate — so cross-table price drift is structurally impossible. Update a + price in `model-pricing.ts` only; each consumer keeps its own key allowlist + miss policy + (fail-closed vs warn-only vs null), not its own numbers. Pinned by `test/model-pricing.test.ts` + (drift guard asserts each view equals canonical). Embeddings price separately in + `embedding-pricing.ts` (different unit). ## Reference map (load on demand) diff --git a/TODOS.md b/TODOS.md index 9d45c5d1a..2e94167dd 100644 --- a/TODOS.md +++ b/TODOS.md @@ -657,9 +657,23 @@ all are latent-debt cleanup. - [ ] **Config-write normalization.** Whenever a user writes `gbrain config set models.tier.deep anthropic/claude-opus-4-7` we silently store the slash form. v0.41.22.1 centralized the read-side via `splitProviderModelId`, but config writes still preserve whatever shape the user typed. Canonical form should be colon (`anthropic:claude-opus-4-7`). Fix: rewrite at config-write time in `src/core/config.ts`. Breaks existing config files that explicitly hold the slash form — defer to a v0.42+ config-migration wave that also handles the rewrite + once-per-process deprecation warn. Files: `src/core/config.ts`, `src/core/model-config.ts:saveConfig` path. Priority: P3 (latent, not user-visible). -- [ ] **Non-Anthropic pricing tables.** `src/core/anthropic-pricing.ts` is the only pricing surface gbrain ships. Brainstorm + LSD users routing through OpenAI / Gemini / OpenRouter get `BUDGET_TRACKER_NO_PRICING` warn-once + bypass-gate (without `--max-cost`) OR `no_pricing` hard-fail (with `--max-cost`). The right shape: rename to `provider-pricing.ts`, add OpenAI / Gemini / OpenRouter tables, route `lookupPricing` through provider-routed table selection. OpenRouter is a special case (period-vs-dash key mismatch: their `claude-sonnet-4.6` won't match our `claude-sonnet-4-6` either way). Files: `src/core/anthropic-pricing.ts` (rename + extend), `src/core/budget/budget-tracker.ts`, `src/core/eval-contradictions/cost-tracker.ts`. Priority: P2 (real user pain when running brainstorm against non-Anthropic). +- [ ] **Non-Anthropic budget-tracker pricing.** PARTIALLY ADDRESSED by v0.42.25.0: `src/core/model-pricing.ts` is now the canonical multi-provider table (OpenAI / Google / Together / DeepSeek entries exist alongside Anthropic), and cross-modal-eval + takes-quality already price non-Anthropic models from it. REMAINING: `src/core/budget/budget-tracker.ts:lookupPricing` still routes only through the bare-keyed `ANTHROPIC_PRICING` view, so brainstorm + LSD users running budget gates against OpenAI / Gemini / OpenRouter still get `BUDGET_TRACKER_NO_PRICING` warn-once + bypass-gate (without `--max-cost`) OR `no_pricing` hard-fail (with `--max-cost`). Right fix: route `lookupPricing` through `canonicalLookup`. OpenRouter stays a special case (period-vs-dash key mismatch: their `claude-sonnet-4.6` won't match our `claude-sonnet-4-6`, and it intentionally misses to avoid pricing markup as native). Files: `src/core/budget/budget-tracker.ts`, `src/core/model-pricing.ts`. Priority: P2 (real user pain when running brainstorm against non-Anthropic). -- [ ] **Eval-contradictions duplicate ANTHROPIC_PRICING consolidation.** `src/core/eval-contradictions/cost-tracker.ts:28-38` ships its OWN copy of the Anthropic pricing table with different keys (both bare and `anthropic:`-prefixed forms) and a silent-Haiku fallback on unknown. v0.41.22.1 routed both tables' lookups through `splitProviderModelId` but left the duplication. Right fix: delete the local table, import from `src/core/anthropic-pricing.ts`. Either (a) preserve the silent-Haiku-fallback semantic with an explicit `?? canonicalPricing['claude-haiku-4-5']` at the call site, or (b) tighten to warn-once on unknown (which changes the eval-contradictions soft-ceiling `--budget-usd` contract — coordinate with that subsystem). Files: `src/core/eval-contradictions/cost-tracker.ts`, `src/core/anthropic-pricing.ts`, `test/eval-contradictions/cost-tracker-slash.test.ts` (the legacy-Haiku-fallback pin would need updating). Priority: P3 (DRY cleanup, no user-visible impact). +- [x] **Eval-contradictions duplicate ANTHROPIC_PRICING consolidation.** **Completed:** v0.42.25.0 (2026-06-03). Deleted the local duplicate table in `src/core/eval-contradictions/cost-tracker.ts`; it now imports the canonical-derived `ANTHROPIC_PRICING` view and `pricingFor` preserves the silent-Haiku fallback (pinned by `test/eval-contradictions/cost-tracker-slash.test.ts`). Closed as part of the wider model-pricing unification. + +## v0.42.25.0 pricing-unification follow-ups (v0.42+) + +Filed from the v0.42.25.0 ship review (Claude + Codex adversarial + pre-landing). +All latent / hardening — none are user-reported bugs. The unification landed a +single canonical `src/core/model-pricing.ts` with `canonicalLookup`. + +- [ ] **`canonicalLookup` is case-sensitive (silent-miss undercount).** `src/core/model-pricing.ts:canonicalLookup` does exact-key + `splitProviderModelId` lookups with no lowercasing, so `ANTHROPIC:claude-opus-4-8` or `anthropic:CLAUDE-OPUS-4-8` return `undefined` → consumers that treat a miss as zero-cost (cross-modal runner note, cost-tracker silent-Haiku, skillopt Sonnet fallback) silently mis-budget. Latent today (recipe/CLI paths emit lowercase), but the fail-mode is a silent undercount, not a throw. Fix: lowercase provider+model before lookup in `canonicalLookup`. Add a mixed-case test. Priority: P3. + +- [ ] **takes-quality `getPricing` is exact-key only.** `src/core/takes-quality-eval/pricing.ts:getPricing` does a raw `MODEL_PRICING[modelId]` lookup. A user passing a bare/slash/dotted form of an allowlisted model (e.g. `google:gemini-2.0-flash` when the allowlist holds `google:gemini-2-flash`, or `anthropic/claude-opus-4-8`) hits `PricingNotFoundError` even though canonical prices it. Safe direction (fail-closed) but a usability regression. Fix: normalize the lookup key through `canonicalLookup`/`splitProviderModelId` before the allowlist check, keeping fail-closed for genuinely-unsupported models. Priority: P3. + +- [ ] **No negative-path test for the takes-quality module-load throw.** `src/core/takes-quality-eval/pricing.ts` throws at import if a `SUPPORTED_MODELS` id is absent from canonical (good fail-fast), but nothing tests it (awkward to test a module-load-time throw in-process). Add a small harness/fixture test. Priority: P3 (programmer-error guard). + +- [ ] **Recipe display-layer pricing is stale and unconsolidated.** Each `src/core/ai/recipes/*.ts` carries coarse per-provider `cost_per_1m_input_usd`/`cost_per_1m_output_usd` baselines (e.g. `google.ts` chat = `$0.30/$1.20`, `price_last_verified: 2026-04-20`) read only by `gbrain providers` for display — NOT by any budget gate. They've drifted (google chat baseline predates the Gemini 2.0 Flash `$0.10/$0.40` reconciliation; codex flagged OpenAI baselines too). These are intentionally a separate coarse layer from the per-model `model-pricing.ts` budget tables, so consolidating is non-trivial (one-number-per-provider vs per-model). Options: (a) refresh the `price_last_verified` baselines, or (b) have `gbrain providers` show per-model rates from canonical where available and fall back to the recipe baseline. Flagged by the v0.42.25.0 ship Codex adversarial pass. Priority: P3 (display-only, no budget-gating impact). ## v0.41.21.0 ops-fix-wave follow-ups (v0.41.22+) diff --git a/VERSION b/VERSION index 6997a4e12..44cf481ac 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.24.0 \ No newline at end of file +0.42.25.0 diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index f450d8a4a..f6fca7085 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -107,7 +107,9 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/search/autocut.ts` (default ON in reranked modes) — Weaviate-style autocut: score-discontinuity result-sizing on the cross-encoder rerank separatrix. `applyAutocut(results, scoreOf, cfg)` normalizes the reranker scores, finds the largest consecutive gap, and cuts there when it clears `jumpRatio` (default 0.20); robust to unsorted provider output (cuts on a sorted copy, keeps items in INPUT order via a score threshold), guards `top<=0`/non-finite, never returns empty, and no-ops when <2 results carry a finite `rerank_score` (covers the reranker fail-open path). WHY rerank_score and NOT RRF/cosine: gbrain measured (see `return-policy.ts`) that the RRF rank1→rank2 gap is ~flat whether rank-1 is right or wrong — not a separatrix; the cross-encoder score IS. So autocut runs ONLY where the reranker ran (the floor reaches `balanced`+`tokenmax`; `conservative` is a documented no-op). Exports `AutocutConfig`, `DEFAULT_AUTOCUT` (frozen: `enabled=true, jumpRatio=0.20, minKeep=1`), `AutocutDecision` (`{applied, signal:'rerank'|'none', cut, kept, total, gapRatio}`), `AutocutInput`, `autocutFromConfig`, `resolveAutocut`, `applyAutocut`. Cache-key integration (clean path, not the adaptive-return cache-skip hack): enable+sensitivity flow through `ModeBundle` → `ResolvedSearchKnobs` → `knobsHash` exactly like `graph_signals`. `mode.ts` adds `autocut`/`autocut_jump` (conservative false, balanced/tokenmax true@0.20) AND sets `reranker_top_n_in = searchLimit` for reranked modes (so the reranker scores the full returned set; there is no un-scored tail for autocut to wrongly drop — closes the load-bearing recall finding). `KNOBS_HASH_VERSION` is 8 (title_boost claimed 7; autocut appends 8 — one-time global cache cold-miss on upgrade). Wired into `hybridSearch` AFTER adaptive-return, BEFORE the limit slice, first page only; emits `HybridSearchMeta.autocut`. BOTH the cache-miss `finalMeta` and cache-HIT `cachedMeta` rebuilds carry `autocut`+`adaptive_return`+`mode`+`embedding_column`. Preserves alias-hop exact matches: `applyAutocut` takes an optional `preserve` predicate; hybrid passes `r => r.alias_hit === true` so a canonical page injected by `applyAliasHop` after reranking (no `rerank_score`) is never cut. Agent surface: `query` op `autocut` boolean (ceiling override — `false` forces full top-K); `SearchOpts.autocut`; `--explain` shows per-result `rerank_score`, `formatAutocutSummary` renders the decision when search meta is threaded; `gbrain search modes` attribution; metric glossary `autocut.signal`/`autocut.gap_ratio`. Config: `search.autocut`, `search.autocut_jump`. Default-ON backed by an in-repo eval gate — `test/search/autocut-eval.test.ts` (also `bun run eval:autocut`) measures precision-lift-without-recall-regression over labeled qrels fixtures with modeled cross-encoder distributions (no API key, no sibling repo; runs in CI): mean precision 0.33→0.94, recall 1.00→0.95, ZERO recall regression on enumeration queries. Env-overridable floors. Pinned by `test/search/autocut.test.ts` (pure-fn), `test/search/query-op-autocut.test.ts` (agent surface), `test/search/autocut-integration.serial.test.ts` (IRON-RULE behavioral via `rerankerFn` DI seam: cliff trims, flat doesn't, no-reranker no-ops, `autocut:false` ceiling, composes with adaptive-return), `test/search/autocut-eval.test.ts` (the precision/recall gate), and the v=8 knobsHash assertions in `test/search-mode.test.ts`. - `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. Declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio), avoiding the backfill loop where tiktoken-grounded budgeting undercounted Voyage's actual token usage. Declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. Recipe docstring at `:7-16` names the seven hosted flexible-dim models that accept `output_dimension` (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and notes `voyage-4-nano` is the open-weight variant fixed at 1024-dim that does NOT accept the parameter (negative regression assertion in `test/ai/gateway.test.ts`: `dimsProviderOptions` returns `undefined` for `voyage-4-nano`). `voyage-code-3` is the recommended embedding model for gstack per-worktree code brains (Topology 3 in `docs/architecture/topologies.md`); discoverability surfaces: decision-tree branch in `docs/integrations/embedding-providers.md`, Topology 3 "Recommended embedding model" subsection, runtime nudge from `gbrain reindex --code` against non-code-tuned models. Recipe-shape regression pinned by `test/ai/voyage-code-3-recipe.test.ts`. - `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). Canonical id is `claude-sonnet-4-6` (no date suffix); a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize`). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts`. -- `src/core/anthropic-pricing.ts` — single source of truth for Anthropic model pricing (per-MTok input/output). Opus 4.7 is `$5/$25` (Opus 4.6 also corrected). Consumed by `src/core/budget-meter.ts` and `src/core/cross-modal-eval/runner.ts` — the cross-modal estimator reads `ANTHROPIC_PRICING` for Anthropic models instead of duplicating the table. +- `src/core/model-pricing.ts` — single source of truth for paid-cloud CHAT/completion model pricing (USD per 1M tokens, input | output). `CANONICAL_PRICING` is a `provider:model`-keyed table (Anthropic Opus 4.8/4.7/4.6 `$5/$25`, Sonnet 4.6 `$3/$15`, Haiku 4.5 `$1/$5` both dateless + dated, plus OpenAI / Google / Together / DeepSeek panel models). `canonicalLookup(modelId)` resolves bare (`claude-opus-4-8`), colon (`anthropic:claude-opus-4-8`), and slash (`anthropic/...`) forms — bare ids default to the `anthropic:` provider; nested OpenRouter ids (`openrouter:anthropic/...`) intentionally MISS so OpenRouter markup isn't repriced as the inner vendor. Every other chat-pricing table is a DERIVED view of this one (NOT a hand-copied duplicate), so cross-table price drift is structurally impossible. Embeddings live separately in `embedding-pricing.ts` (different unit). Pinned by `test/model-pricing.test.ts` whose drift guard asserts each derived view equals canonical and that the cross-modal panel models are all present. +- `src/core/anthropic-pricing.ts` — bare-keyed Anthropic VIEW of `model-pricing.ts` (the `anthropic:` canonical entries with the prefix stripped). Kept distinct because many callers look up by bare Claude id and because `estimateMaxCostUsd(modelId, inTokens, maxOutTokens)` carries the null-on-miss contract the dream-cycle budget gate depends on (non-Anthropic ids return null → caller warns `BUDGET_METER_NO_PRICING` once and runs unbounded). `estimateMaxCostUsd` routes bare/colon/slash ids through `splitProviderModelId`. Do NOT hand-edit prices here — the map is derived from canonical, so it cannot drift. `ANTHROPIC_PRICING` is consumed by `budget/budget-tracker.ts`, `minions/batch-projection.ts`, and `cycle/budget-meter.ts`. +- `src/core/takes-quality-eval/pricing.ts` — fail-closed budget pricing for `eval takes-quality run --budget-usd N`. `MODEL_PRICING` is a curated `provider:model` allowlist (default panel + likely overrides) whose VALUES are derived from `model-pricing.ts` via `canonicalLookup`; an allowlisted id missing from canonical throws at module load. Schema is `{input_per_1m, output_per_1m}`. A model NOT on the allowlist aborts the run with an actionable error rather than guessing (distinct from `cross-modal-eval/runner.ts`, which silently estimates zero on unknown models — both now source numbers from canonical). - `src/core/budget/budget-tracker.ts` — keystone primitive for the brainstorm cost-cathedral wave. One typed error (`BudgetExhausted` with `reason: 'cost' | 'runtime' | 'no_pricing'`), one schema-stable audit JSONL at `~/.gbrain/audit/budget-YYYY-Www.jsonl`. Contracts: `record()` throws when cumulative spend exceeds cap (the cap is a real ceiling, not a suggestion); `reserve()` hard-fails with `reason: 'no_pricing'` when `maxCostUsd` is set AND the model is missing from pricing maps (warn-once preserved when cap is unset); `extractUsageFromError(err, fallback)` returns `err.usage` when the SDK provides it, else the pessimistic fallback (caller passes `maxOutputTokens`, not the optimistic pre-call estimate). `onExhausted(cb)` fires once synchronously BEFORE the throw propagates so callers can persist checkpoints. Replaces three parallel copies (inline brainstorm class, cycle/budget-meter, eval-contradictions). Adapts the old `BudgetMeter` (public shape preserved + `schema_version: 1` stamped on every dream-budget audit line). Pinned by 18 unit cases. - `src/core/audit-week-file.ts` — single source of truth for ISO-week audit JSONL filename math. Exports `isoWeek(d)`, `isoWeekFilename(prefix, now?)`, `resolveAuditDir()` (honors `GBRAIN_AUDIT_DIR`). Year-boundary correctness pinned by tests at 2020-W53 (the 53-week year), 2025-W01 rolling in from 2024-12-30 (Monday), 2026-W01. Four call sites migrated: `src/core/minions/handlers/shell-audit.ts`, `src/core/facts/phantom-audit.ts`, `src/core/audit-slug-fallback.ts`, `src/core/cycle/budget-meter.ts`. Each keeps its `computeAuditFilename` thin wrapper for back-compat with existing tests. - `src/core/ai/gateway.ts:withBudgetTracker` — gateway-layer enforcement via `AsyncLocalStorage`. `withBudgetTracker(tracker, fn)` installs the tracker on the module-internal store; every `gateway.chat / embed / rerank` call inside the scope auto-composes (reserve before, record in try/finally). Outside-scope calls are budget no-ops. Nested scopes restore the outer tracker on exit. `getCurrentBudgetTracker()` is the test seam. The chat path uses the pessimistic fallback on error paths; the embed path estimates input tokens from char count × recipe's `chars_per_token` because the AI SDK doesn't surface per-batch embed token usage; the rerank path estimates char count of query+docs. Pinned by 6 unit cases. diff --git a/docs/eval/SEARCH_MODE_METHODOLOGY.md b/docs/eval/SEARCH_MODE_METHODOLOGY.md index b56d1ea1f..6093a4770 100644 --- a/docs/eval/SEARCH_MODE_METHODOLOGY.md +++ b/docs/eval/SEARCH_MODE_METHODOLOGY.md @@ -160,7 +160,7 @@ The mode-picker prompt at `gbrain init` and the CLAUDE.md `## Search Mode` table - Your agent's system prompt + reasoning tokens add input that gbrain doesn't see. - Compaction reduces input over a long session. - Most agents make 1-5 searches per turn; cost-per-turn is what bills you, not cost-per-query. -- The model price column drifts as providers reprice; pin the rate via `src/core/anthropic-pricing.ts` for a current snapshot. +- The model price column drifts as providers reprice; pin the rate via `src/core/model-pricing.ts` (the canonical chat-pricing table) for a current snapshot. The picker copy + CLAUDE.md table are the canonical user-facing source. Update them in lockstep when the underlying chunker size or default `searchLimit` changes. diff --git a/llms-full.txt b/llms-full.txt index aaa5e682d..e90fef474 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -222,6 +222,15 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`. Postgres; plain `CREATE INDEX` on PGLite via `sqlFor.pglite`). - **Multi-source.** Slug uniqueness is `(source_id, slug)`, not slug. Key batch ops and reverse-writes on the composite key; `validateSourceId` before any `source_id` path join. +- **One canonical chat-pricing table.** All paid-cloud chat/completion prices live ONCE in + `src/core/model-pricing.ts` (`CANONICAL_PRICING` + `canonicalLookup`). Every other table + (`anthropic-pricing.ts`'s `ANTHROPIC_PRICING`, `takes-quality-eval/pricing.ts`'s + `MODEL_PRICING`, the contradictions/cross-modal/skillopt cost views) is a DERIVED view, never + a hand-copied duplicate — so cross-table price drift is structurally impossible. Update a + price in `model-pricing.ts` only; each consumer keeps its own key allowlist + miss policy + (fail-closed vs warn-only vs null), not its own numbers. Pinned by `test/model-pricing.test.ts` + (drift guard asserts each view equals canonical). Embeddings price separately in + `embedding-pricing.ts` (different unit). ## Reference map (load on demand) diff --git a/package.json b/package.json index 9fd5009e6..19034af85 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.24.0" + "version": "0.42.25.0" } diff --git a/src/core/anthropic-pricing.ts b/src/core/anthropic-pricing.ts index 46aac2f45..668444ed3 100644 --- a/src/core/anthropic-pricing.ts +++ b/src/core/anthropic-pricing.ts @@ -1,45 +1,38 @@ /** - * v0.28: Anthropic model pricing constants for the dream-cycle budget meter. + * Anthropic chat pricing — a bare-keyed VIEW of the canonical pricing table + * (`src/core/model-pricing.ts`). * - * Prices in USD per 1M tokens (input | output). Numbers reflect Anthropic's - * published pricing as of 2026-05-01. Update when Anthropic publishes new - * pricing — the JSON in `~/.gbrain/audit/dream-budget-*.jsonl` carries the - * snapshot per call so historical estimates stay reproducible. + * Kept as a distinct export because many callers look up by bare Claude id + * (`claude-opus-4-7`) and because `estimateMaxCostUsd` carries the + * null-on-miss contract the dream-cycle budget gate depends on. The dollar + * numbers live in model-pricing.ts — DO NOT hand-edit prices here; this map is + * derived from the `anthropic:` canonical entries (prefix stripped), so it + * cannot drift from the other pricing views. (Pre-unification this map and + * takes-quality-eval/pricing.ts duplicated the numbers and drifted: Opus 4.7 + * read $15/$75 in one and $5/$25 in the other.) * - * Codex P1 #10 fold: non-Anthropic models (gemini, gpt, anything not in - * this map) bypass the budget gate with a `BUDGET_METER_NO_PRICING` warn - * once per process. The cycle still runs unbounded for those models. - * Future: per-provider pricing modules. + * Codex P1 #10 fold: non-Anthropic models (gemini, gpt, anything not in this + * map) bypass the budget gate with a `BUDGET_METER_NO_PRICING` warn once per + * process. The cycle still runs unbounded for those models. */ -export interface ModelPricing { - /** USD per 1M input tokens. */ - input: number; - /** USD per 1M output tokens. */ - output: number; -} - -/** Map of Anthropic model id → pricing. Aliases (opus/sonnet/haiku) resolve via DEFAULT_ALIASES. */ -export const ANTHROPIC_PRICING: Record = { - // Claude 4.7 generation (current) - // Opus 4.7 dropped from $15/$75 (Opus 4) to $5/$25 per - // https://platform.claude.com/docs/en/about-claude/models/overview (verified 2026-05-10). - 'claude-opus-4-7': { input: 5.00, output: 25.00 }, - 'claude-sonnet-4-6': { input: 3.00, output: 15.00 }, - // Both the dateless canonical id (TIER_DEFAULTS / aliases / every caller uses - // this) AND the dated snapshot. The dateless entry was missing pre-v0.42.9.0, - // so a budget-capped Haiku run (skillopt with --max-cost, eval harnesses) hit - // BudgetTracker no_pricing and every rollout silently scored 0. - 'claude-haiku-4-5': { input: 1.00, output: 5.00 }, - 'claude-haiku-4-5-20251001': { input: 1.00, output: 5.00 }, - // Older but still frequently aliased - 'claude-opus-4-6': { input: 5.00, output: 25.00 }, - 'claude-3-5-sonnet-20241022': { input: 3.00, output: 15.00 }, - 'claude-3-5-haiku-20241022': { input: 0.80, output: 4.00 }, -}; - +import { CANONICAL_PRICING, type ModelPricing } from './model-pricing.ts'; import { splitProviderModelId } from './model-id.ts'; +export type { ModelPricing }; + +/** + * Bare-keyed Anthropic view, derived from the canonical table. Both the + * dateless ids (`claude-haiku-4-5`, used by aliases / TIER_DEFAULTS / most + * callers) and the dated snapshots (`claude-haiku-4-5-20251001`) are present + * because canonical carries both. + */ +export const ANTHROPIC_PRICING: Record = Object.fromEntries( + Object.entries(CANONICAL_PRICING) + .filter(([key]) => key.startsWith('anthropic:')) + .map(([key, pricing]) => [key.slice('anthropic:'.length), pricing]), +); + /** * Estimate the upper-bound USD cost of a single submit. * Uses (estimatedInputTokens × inputRate) + (maxOutputTokens × outputRate). diff --git a/src/core/brain-score-recommendations.ts b/src/core/brain-score-recommendations.ts index 74806d686..810b43bed 100644 --- a/src/core/brain-score-recommendations.ts +++ b/src/core/brain-score-recommendations.ts @@ -1,6 +1,6 @@ import { createHash } from 'crypto'; import type { BrainHealth } from './types.ts'; -import { ANTHROPIC_PRICING } from './anthropic-pricing.ts'; +import { canonicalLookup } from './model-pricing.ts'; import { lookupEmbeddingPrice, estimateCostFromChars } from './embedding-pricing.ts'; import { getRecipe } from './ai/recipes/index.ts'; import { parseModelId } from './ai/model-resolver.ts'; @@ -433,7 +433,7 @@ export function estimateAnthropicCost( estInputTokensPerCall = 5_000, estOutputTokensPerCall = 1_000, ): number { - const pricing = ANTHROPIC_PRICING[modelId]; + const pricing = canonicalLookup(modelId); if (!pricing) return 0; const inputCost = (estInputTokensPerCall * estCallsPerInvocation / 1_000_000) * pricing.input; const outputCost = (estOutputTokensPerCall * estCallsPerInvocation / 1_000_000) * pricing.output; diff --git a/src/core/brainstorm/orchestrator.ts b/src/core/brainstorm/orchestrator.ts index 7a4b2f5dc..9d3b85f1d 100644 --- a/src/core/brainstorm/orchestrator.ts +++ b/src/core/brainstorm/orchestrator.ts @@ -46,7 +46,7 @@ import { type JudgeConfig, type ChatFn, } from './judges.ts'; -import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts'; +import { canonicalLookup } from '../model-pricing.ts'; // --------------------------------------------------------------------------- // BudgetExhausted is the canonical typed error (Q2) used by every cost @@ -264,7 +264,7 @@ export function estimateCost(profile: BrainstormProfile, model: string): number const judgeIn = ideas * 350; const judgeOut = ideas * 200; - const pricing = ANTHROPIC_PRICING[model] ?? { input: 3, output: 15 }; + const pricing = canonicalLookup(model) ?? { input: 3, output: 15 }; const inCost = ((inTokens + judgeIn) / 1_000_000) * pricing.input; const outCost = ((outTokens + judgeOut) / 1_000_000) * pricing.output; return inCost + outCost; @@ -771,7 +771,7 @@ async function _runBrainstormInner( crossModel = result.model; // Mid-run cost guard: if running spend already exceeds the projected // ceiling or the strict-budget multiplier, abort the remaining crosses. - const runningPricing = ANTHROPIC_PRICING[result.model] ?? { input: 3, output: 15 }; + const runningPricing = canonicalLookup(result.model) ?? { input: 3, output: 15 }; const runningUsd = (totalUsage.input_tokens / 1_000_000) * runningPricing.input + (totalUsage.output_tokens / 1_000_000) * runningPricing.output; @@ -897,7 +897,7 @@ async function _runBrainstormInner( // Cost actuals (codex r2 #10). const totalIn = totalUsage.input_tokens + judgeUsage.input_tokens; const totalOut = totalUsage.output_tokens + judgeUsage.output_tokens; - const pricing = ANTHROPIC_PRICING[crossModel] ?? { input: 3, output: 15 }; + const pricing = canonicalLookup(crossModel) ?? { input: 3, output: 15 }; const actual = (totalIn / 1_000_000) * pricing.input + (totalOut / 1_000_000) * pricing.output; stderr(`[${profile.label}] actual cost: ${fmtUsd(actual)} (estimated ${fmtUsd(estimate)}) — in=${totalIn} out=${totalOut} tokens\n`); diff --git a/src/core/cross-modal-eval/runner.ts b/src/core/cross-modal-eval/runner.ts index b6efa894e..9fdaa4920 100644 --- a/src/core/cross-modal-eval/runner.ts +++ b/src/core/cross-modal-eval/runner.ts @@ -22,7 +22,7 @@ import type { AggregateResult, SlotResult } from './aggregate.ts'; import { parseModelJSON } from './json-repair.ts'; import { receiptName, sha8 } from './receipt-name.ts'; import { writeReceipt } from './receipt-write.ts'; -import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts'; +import { canonicalLookup } from '../model-pricing.ts'; export const RECEIPT_SCHEMA_VERSION = 1; @@ -322,37 +322,23 @@ export function estimateCost(slots: SlotConfig[], cycles: number, maxTokens: num // Per-call cost = (input_tokens × input_price + output_tokens × output_price) / 1e6. // Without knowing prompt size, estimate input ~5k tokens (a SKILL.md + scoring rubric). // - // Anthropic prices read from ANTHROPIC_PRICING (single source of truth — fixes - // the drift trap Codex flagged in v0.31.12 plan review: this map and - // anthropic-pricing.ts duplicated Anthropic prices, with stale values diverging). - // Non-Anthropic models still live inline until OPENAI_PRICING / GOOGLE_PRICING - // tables exist. + // All prices (anthropic + openai + google + together + deepseek) come from the + // canonical table via canonicalLookup (src/core/model-pricing.ts) — single + // source of truth. This finishes the de-duplication the v0.31.12 plan started + // for Anthropic; OpenAI/Google/Together/DeepSeek panel models no longer carry + // inline rates here. Slots with no canonical entry fall to the "no pricing on + // file" note (cost estimate may be low), preserving prior behavior. const ESTIMATED_INPUT_TOKENS = 5000; - const anthropicPrice = (modelId: string): { in: number; out: number } | undefined => { - const p = ANTHROPIC_PRICING[modelId]; - return p ? { in: p.input, out: p.output } : undefined; - }; - const PRICING: Record = { - 'openai:gpt-4o': { in: 2.5, out: 10.0 }, - 'openai:gpt-4o-mini': { in: 0.15, out: 0.6 }, - 'anthropic:claude-opus-4-7': anthropicPrice('claude-opus-4-7'), - 'anthropic:claude-sonnet-4-6': anthropicPrice('claude-sonnet-4-6'), - 'anthropic:claude-haiku-4-5-20251001': anthropicPrice('claude-haiku-4-5-20251001'), - 'google:gemini-1.5-pro': { in: 1.25, out: 5.0 }, - 'google:gemini-2.0-flash': { in: 0.1, out: 0.4 }, - 'together:meta-llama/Llama-3.3-70B-Instruct-Turbo': { in: 0.88, out: 0.88 }, - 'deepseek:deepseek-chat': { in: 0.14, out: 0.28 }, - }; const notes: string[] = []; let perCycle = 0; for (const slot of slots) { - const p = PRICING[slot.model]; + const p = canonicalLookup(slot.model); if (!p) { notes.push(`(${slot.model}): no pricing on file; cost estimate may be low`); continue; } - const cost = (ESTIMATED_INPUT_TOKENS * p.in + maxTokens * p.out) / 1_000_000; + const cost = (ESTIMATED_INPUT_TOKENS * p.input + maxTokens * p.output) / 1_000_000; perCycle += cost; } return { diff --git a/src/core/eval-contradictions/cost-tracker.ts b/src/core/eval-contradictions/cost-tracker.ts index fb8c64dfd..8be3bd4b6 100644 --- a/src/core/eval-contradictions/cost-tracker.ts +++ b/src/core/eval-contradictions/cost-tracker.ts @@ -21,28 +21,16 @@ import type { CostBreakdown } from './types.ts'; import { splitProviderModelId } from '../model-id.ts'; +import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts'; /** - * Per-million-token prices (USD). Update when models bump. These are - * approximate — provider accounting after the call is authoritative. - * - * NOTE: duplicate of the canonical `src/core/anthropic-pricing.ts` table. - * Slated for consolidation (TODOS.md #3 from v0.41.20.0 plan); keys differ - * (this table uses both bare and `anthropic:`-prefixed forms; canonical - * is bare-only). For now we route lookup through `parseModelId` so the - * slash-prefix bug class is closed at this site too. + * Chat prices come from the canonical table via the bare-keyed + * `ANTHROPIC_PRICING` view (`src/core/anthropic-pricing.ts` → `model-pricing.ts`). + * This site used to carry its own duplicate (TODOS.md #3); folding it in closes + * that consolidation. `pricingFor` still routes through `splitProviderModelId` + * so colon/slash forms hit, and keeps the legacy silent-Haiku fallback for + * genuinely-unknown models (pinned by test/eval-contradictions/cost-tracker-slash.test.ts). */ -const ANTHROPIC_PRICING: Record = { - // Haiku 4.5: ~$1/Mtok in, $5/Mtok out (current as of 2026-05). - 'claude-haiku-4-5': { input: 1.0, output: 5.0 }, - 'anthropic:claude-haiku-4-5': { input: 1.0, output: 5.0 }, - // Sonnet 4.6: ~$3/Mtok in, $15/Mtok out. - 'claude-sonnet-4-6': { input: 3.0, output: 15.0 }, - 'anthropic:claude-sonnet-4-6': { input: 3.0, output: 15.0 }, - // Opus 4.7: ~$5/Mtok in, $25/Mtok out. - 'claude-opus-4-7': { input: 5.0, output: 25.0 }, - 'anthropic:claude-opus-4-7': { input: 5.0, output: 25.0 }, -}; /** OpenAI text-embedding-3-large: ~$0.13/Mtok (current as of 2026-05). */ const OPENAI_EMBEDDING_PRICE_PER_MTOK = 0.13; diff --git a/src/core/model-pricing.ts b/src/core/model-pricing.ts new file mode 100644 index 000000000..db1e8854e --- /dev/null +++ b/src/core/model-pricing.ts @@ -0,0 +1,114 @@ +/** + * model-pricing.ts — single source of truth for paid cloud CHAT/completion + * model pricing (USD per 1M tokens, input | output). + * + * Every chat-pricing site in the codebase derives its numbers from this table: + * - anthropic-pricing.ts (bare-keyed Anthropic view + estimateMaxCostUsd) + * - takes-quality-eval/pricing.ts (curated fail-closed allowlist) + * - eval-contradictions/cost-tracker.ts (silent-Haiku-fallback view) + * - cross-modal-eval/runner.ts (multi-provider eval panel) + * - skillopt/preflight.ts (Sonnet-fallback warn-only estimate) + * The bare-keyed `ANTHROPIC_PRICING` view is itself consumed by budget/budget-tracker.ts, + * minions/batch-projection.ts, and cycle/budget-meter.ts — so those inherit canonical too. + * + * The dollar amounts live HERE ONCE — update prices in this file only. Each + * consumer keeps its own key allowlist and miss-handling policy (fail-closed + * vs warn-only vs null); this module owns the values, not the policy. Because + * every other table is DERIVED from this one (not a hand-copied duplicate), + * cross-table price drift — the kind that left Opus 4.7 at $15/$75 in one table + * for months — is structurally impossible. test/model-pricing.test.ts pins that: + * its "drift guard" asserts each derived view still equals canonical (a + * regression trip-wire if anyone later re-hardcodes a view back into a duplicate) + * and that the cross-modal panel models are all present in canonical. + * + * Prices verified 2026-06-03 against published provider pricing: + * - Anthropic: https://platform.claude.com/docs/en/about-claude/models/overview + * - OpenAI: https://openai.com/api/pricing + * - Google: https://ai.google.dev/gemini-api/docs/pricing + * The dream-budget audit JSONL snapshots the rate per call, so historical + * estimates stay reproducible even after this table changes. + * + * Scope: PAID CLOUD chat models only. Free/local providers (llama-server, + * zero-cost rerankers) are intentionally absent — callers treat those as + * zero-cost elsewhere. Embeddings live in embedding-pricing.ts (different unit: + * per-MTok, char-based). + */ + +import { splitProviderModelId } from './model-id.ts'; + +export interface ModelPricing { + /** USD per 1M input tokens. */ + input: number; + /** USD per 1M output tokens. */ + output: number; +} + +/** + * Canonical price table. Keys are provider-prefixed (`provider:model`), + * matching the exact id strings consumers pass. One physical model may carry + * more than one key when a provider ships multiple id spellings (e.g. + * `google:gemini-2.0-flash` plus the legacy `google:gemini-2-flash` alias) — + * keep aliases in lockstep; the drift guard asserts they agree. + */ +export const CANONICAL_PRICING: Record = { + // ── Anthropic ────────────────────────────────────────────────────────── + // Opus 4.x: $5 in / $25 out. 4.8 (released 2026-05-28) shares 4.7's + // per-token rate — closes gbrain#1819. + 'anthropic:claude-opus-4-8': { input: 5.00, output: 25.00 }, + 'anthropic:claude-opus-4-7': { input: 5.00, output: 25.00 }, + 'anthropic:claude-opus-4-6': { input: 5.00, output: 25.00 }, + 'anthropic:claude-sonnet-4-6': { input: 3.00, output: 15.00 }, + // Haiku 4.5 — both the dateless canonical id and the dated snapshot. + 'anthropic:claude-haiku-4-5': { input: 1.00, output: 5.00 }, + 'anthropic:claude-haiku-4-5-20251001': { input: 1.00, output: 5.00 }, + 'anthropic:claude-3-5-sonnet-20241022': { input: 3.00, output: 15.00 }, + 'anthropic:claude-3-5-haiku-20241022': { input: 0.80, output: 4.00 }, + + // ── OpenAI ───────────────────────────────────────────────────────────── + 'openai:gpt-4o': { input: 2.50, output: 10.00 }, + 'openai:gpt-4o-mini': { input: 0.15, output: 0.60 }, + 'openai:gpt-5': { input: 5.00, output: 20.00 }, + 'openai:gpt-5.5': { input: 4.00, output: 16.00 }, + + // ── Google ───────────────────────────────────────────────────────────── + 'google:gemini-1.5-pro': { input: 1.25, output: 5.00 }, + // Gemini 2.0 Flash: $0.10 in / $0.40 out (verified 2026-06-03). Reconciled + // from a stale $0.30/$1.20 entry that had drifted in takes-quality-eval. + // `gemini-2-flash` kept as an alias for the legacy id spelling. + 'google:gemini-2.0-flash': { input: 0.10, output: 0.40 }, + 'google:gemini-2-flash': { input: 0.10, output: 0.40 }, + + // ── Together / DeepSeek (cross-modal-eval panel) ─────────────────────── + 'together:meta-llama/Llama-3.3-70B-Instruct-Turbo': { input: 0.88, output: 0.88 }, + 'deepseek:deepseek-chat': { input: 0.14, output: 0.28 }, +}; + +/** + * Resolve a model id to its canonical pricing, or `undefined` on miss. + * + * Accepts bare (`claude-opus-4-8`), colon (`anthropic:claude-opus-4-8`), and + * slash (`anthropic/claude-opus-4-8`) forms. Bare ids default to the + * `anthropic:` provider (matching the historical bare-key Anthropic tables); + * non-Anthropic bare ids therefore miss, preserving the prior null-return + * contract for ids like `gpt-5`. + * + * Nested OpenRouter ids (`openrouter:anthropic/claude-...`) intentionally MISS: + * splitProviderModelId yields provider `openrouter`, model + * `anthropic/claude-...`, and `openrouter:anthropic/claude-...` is not a + * canonical key. OpenRouter markup ≠ native pricing, so we never reprice it as + * the inner vendor. + */ +export function canonicalLookup( + modelId: string | null | undefined, +): ModelPricing | undefined { + if (!modelId) return undefined; + // 1. Exact key — colon form, already-canonical ids, and slash-bearing model + // tails carried verbatim as keys (e.g. together:.../Llama-3.3-70B-...). + const direct = CANONICAL_PRICING[modelId]; + if (direct) return direct; + // 2. Normalize bare/slash via the shared splitter (colon-first precedence). + const { provider, model } = splitProviderModelId(modelId); + if (!model) return undefined; + const key = provider ? `${provider}:${model}` : `anthropic:${model}`; + return CANONICAL_PRICING[key]; +} diff --git a/src/core/skillopt/preflight.ts b/src/core/skillopt/preflight.ts index 5bae98834..9f2e9dcc0 100644 --- a/src/core/skillopt/preflight.ts +++ b/src/core/skillopt/preflight.ts @@ -20,12 +20,13 @@ * - epochs × 1 slow-update reflect call (if no improvement that epoch) * - 1× final test eval on D_test * - * Prices come from existing per-model pricing tables (Anthropic + - * embedding-pricing). For unknown providers we fail-loud — same posture - * as BudgetTracker's TX2 contract. + * Prices come from the canonical pricing table (model-pricing.ts) via + * canonicalLookup. For unknown providers `lookupPrice` returns a warn-only + * Sonnet-tier fallback (preflight estimates, never gates) — the actual + * fail-loud gate is BudgetTracker's TX2 contract at run time. */ -import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts'; +import { canonicalLookup } from '../model-pricing.ts'; import { VALIDATION_RUNS_PER_TASK } from './types.ts'; /** Conservative per-rollout token estimates (input + output). */ @@ -188,10 +189,10 @@ export function preflight(opts: PreflightOpts): PreflightResult { } function lookupPrice(model: string): { input: number; output: number } { - // Anthropic models — strip provider prefix. - const bare = model.startsWith('anthropic:') ? model.slice('anthropic:'.length) : model; - const anth = (ANTHROPIC_PRICING as Record)[bare]; - if (anth) return anth; + // Canonical lookup handles bare/colon/slash forms (fixes this site's prior + // bare-only limitation, which mispriced `anthropic/...` slash ids). + const p = canonicalLookup(model); + if (p) return p; // Conservative fallback: assume Sonnet-tier pricing for unknown providers. // Don't throw — preflight is for warning, not gating. The actual budget // tracker (BudgetTracker TX2) will fail-loud at run time if pricing is diff --git a/src/core/takes-quality-eval/pricing.ts b/src/core/takes-quality-eval/pricing.ts index b1dac6c6d..b8305a417 100644 --- a/src/core/takes-quality-eval/pricing.ts +++ b/src/core/takes-quality-eval/pricing.ts @@ -1,20 +1,24 @@ /** - * takes-quality-eval/pricing — fail-closed model pricing table for budget + * takes-quality-eval/pricing — fail-closed model pricing for budget * enforcement. * - * Per-1M-token rates in USD. Drifts as providers update prices; refresh - * alongside model-family bumps. The list is intentionally small — only - * the default 3-model panel and a handful of likely overrides. If you - * pass a model not in this table to `eval takes-quality run --budget-usd N`, - * the runner aborts with an actionable error rather than guessing - * (codex review #4 fail-closed posture vs cross-modal-eval/runner.ts - * which silently estimates zero on unknown models). + * The KEY SET here is an intentional allowlist — only the default panel and a + * handful of likely overrides. Passing a model NOT in this list to + * `eval takes-quality run --budget-usd N` aborts with an actionable error + * rather than guessing (codex review #4 fail-closed posture vs + * cross-modal-eval/runner.ts which silently estimates zero on unknown models). * - * Schema is `{model_id: {input_per_1m, output_per_1m}}` so callers can - * compute estimated cost as + * The VALUES come from the canonical table (`src/core/model-pricing.ts`) — do + * NOT hand-edit rates here; update canonical and they flow through. This keeps + * the allowlist's fail-closed curation while removing the duplicated numbers + * that let Opus 4.7 drift to a stale $15/$75 here. + * + * Schema is `{model_id: {input_per_1m, output_per_1m}}` so callers can compute * (in_tokens * input_per_1m + out_tokens * output_per_1m) / 1_000_000. */ +import { canonicalLookup } from '../model-pricing.ts'; + export interface ModelPricing { /** USD per 1M input tokens. */ input_per_1m: number; @@ -22,28 +26,44 @@ export interface ModelPricing { output_per_1m: number; } -export const MODEL_PRICING: Record = { - // OpenAI (refreshed 2026-05; verify before relying for budget gating) - 'openai:gpt-4o': { input_per_1m: 2.5, output_per_1m: 10.0 }, - 'openai:gpt-5': { input_per_1m: 5.0, output_per_1m: 20.0 }, - 'openai:gpt-5.5': { input_per_1m: 4.0, output_per_1m: 16.0 }, +/** + * The curated allowlist of models takes-quality will budget-gate. Each must + * exist in CANONICAL_PRICING; the map below fails fast at module load if one + * is missing (a programmer error caught immediately, not at run time). + */ +const SUPPORTED_MODELS = [ + 'openai:gpt-4o', + 'openai:gpt-5', + 'openai:gpt-5.5', + 'anthropic:claude-opus-4-8', + 'anthropic:claude-opus-4-7', + 'anthropic:claude-sonnet-4-6', + 'anthropic:claude-haiku-4-5', + 'google:gemini-1.5-pro', + 'google:gemini-2-flash', +] as const; - // Anthropic - 'anthropic:claude-opus-4-7': { input_per_1m: 15.0, output_per_1m: 75.0 }, - 'anthropic:claude-sonnet-4-6': { input_per_1m: 3.0, output_per_1m: 15.0 }, - 'anthropic:claude-haiku-4-5': { input_per_1m: 0.8, output_per_1m: 4.0 }, - - // Google - 'google:gemini-1.5-pro': { input_per_1m: 1.25, output_per_1m: 5.0 }, - 'google:gemini-2-flash': { input_per_1m: 0.30, output_per_1m: 1.20 }, -}; +export const MODEL_PRICING: Record = Object.fromEntries( + SUPPORTED_MODELS.map((id) => { + const p = canonicalLookup(id); + if (!p) { + throw new Error( + `takes-quality allowlist model "${id}" is missing from CANONICAL_PRICING ` + + `(src/core/model-pricing.ts). Add it there.`, + ); + } + return [id, { input_per_1m: p.input, output_per_1m: p.output }]; + }), +); export class PricingNotFoundError extends Error { constructor(public readonly modelId: string) { super( - `Model "${modelId}" has no pricing entry in src/core/takes-quality-eval/pricing.ts. ` + - `Add an entry for the model and re-run, OR pass --budget-usd 0 to disable budget ` + - `enforcement (you'll still see the cost printed to stderr but the runner won't abort).`, + `Model "${modelId}" has no pricing entry. Add it to CANONICAL_PRICING in ` + + `src/core/model-pricing.ts AND to the SUPPORTED_MODELS allowlist in ` + + `src/core/takes-quality-eval/pricing.ts, OR pass --budget-usd 0 to disable ` + + `budget enforcement (you'll still see the cost printed to stderr but the ` + + `runner won't abort).`, ); this.name = 'PricingNotFoundError'; } diff --git a/test/anthropic-pricing.test.ts b/test/anthropic-pricing.test.ts index c56af7918..dd85913a1 100644 --- a/test/anthropic-pricing.test.ts +++ b/test/anthropic-pricing.test.ts @@ -41,6 +41,14 @@ describe('estimateMaxCostUsd', () => { expect(cost).toBeCloseTo(1.75, 5); }); + test('opus 4.8 priced same as 4.7 ($5/$25) — closes #1819', () => { + // 100K in + 50K out = 0.1*5 + 0.05*25 = 0.5 + 1.25 = 1.75. Pre-fix this + // model was absent from the pricing table → estimateMaxCostUsd returned + // null and the dream-cycle budget gate silently no-op'd on 4.8 runs. + const cost = estimateMaxCostUsd('anthropic:claude-opus-4-8', 100_000, 50_000); + expect(cost).toBeCloseTo(1.75, 5); + }); + test('unknown model → returns null (caller warn-once + bypass)', () => { expect(estimateMaxCostUsd('mistral:medium', 1_000, 1_000)).toBeNull(); expect(estimateMaxCostUsd('gpt-5', 1_000, 1_000)).toBeNull(); diff --git a/test/eval-takes-quality-pricing.test.ts b/test/eval-takes-quality-pricing.test.ts index 7782593a5..95eaf55e6 100644 --- a/test/eval-takes-quality-pricing.test.ts +++ b/test/eval-takes-quality-pricing.test.ts @@ -20,6 +20,18 @@ describe('getPricing — fail-closed contract', () => { expect(() => getPricing('unknown:gpt-99')).toThrow(PricingNotFoundError); }); + test('opus 4.7 priced at $5/$25 (regression: was a stale $15/$75) — gbrain#1819', () => { + const p = getPricing('anthropic:claude-opus-4-7'); + expect(p.input_per_1m).toBeCloseTo(5.0, 5); + expect(p.output_per_1m).toBeCloseTo(25.0, 5); + }); + + test('opus 4.8 is supported and priced $5/$25 — gbrain#1819', () => { + const p = getPricing('anthropic:claude-opus-4-8'); + expect(p.input_per_1m).toBeCloseTo(5.0, 5); + expect(p.output_per_1m).toBeCloseTo(25.0, 5); + }); + test('error message names the model AND points to the file', () => { try { getPricing('foo:bar'); diff --git a/test/model-pricing.test.ts b/test/model-pricing.test.ts new file mode 100644 index 000000000..40be89818 --- /dev/null +++ b/test/model-pricing.test.ts @@ -0,0 +1,145 @@ +/** + * model-pricing — canonical table + canonicalLookup + the drift guard. + * + * Because the other pricing tables are DERIVED from CANONICAL_PRICING (not + * hand-copied), cross-table price drift — the kind that left Opus 4.7 at + * $15/$75 in takes-quality-eval while anthropic-pricing.ts had $5/$25 + * (gbrain#1819) — is structurally impossible. The "drift guard" below is a + * regression trip-wire: it asserts each derived view still equals canonical, so + * if anyone later re-hardcodes a view back into a duplicate, CI catches it. The + * cross-modal panel check is genuinely load-bearing — it asserts canonical + * actually carries every model the runner prices. + */ +import { describe, test, expect } from 'bun:test'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { + CANONICAL_PRICING, + canonicalLookup, +} from '../src/core/model-pricing.ts'; +import { ANTHROPIC_PRICING } from '../src/core/anthropic-pricing.ts'; +import { MODEL_PRICING } from '../src/core/takes-quality-eval/pricing.ts'; +import { estimateAnthropicCost } from '../src/core/brain-score-recommendations.ts'; + +describe('CANONICAL_PRICING — table integrity', () => { + test('every entry has finite positive rates and a provider-prefixed key', () => { + for (const [key, p] of Object.entries(CANONICAL_PRICING)) { + expect(Number.isFinite(p.input)).toBe(true); + expect(Number.isFinite(p.output)).toBe(true); + expect(p.input).toBeGreaterThan(0); + expect(p.output).toBeGreaterThan(0); + // Provider-prefixed key (sanity guard against a bare key sneaking in). + // NOTE: deliberately NO output>=input invariant — symmetric pricing is + // legitimate (e.g. together:...Llama-3.3 is 0.88/0.88). + expect(key).toContain(':'); + } + }); + + test('Opus 4.8 present at $5/$25 (closes gbrain#1819)', () => { + expect(CANONICAL_PRICING['anthropic:claude-opus-4-8']).toEqual({ input: 5.0, output: 25.0 }); + }); + + test('Opus 4.7 at $5/$25 (not the stale $15/$75)', () => { + expect(CANONICAL_PRICING['anthropic:claude-opus-4-7']).toEqual({ input: 5.0, output: 25.0 }); + }); + + test('Gemini 2.0 Flash reconciled to $0.10/$0.40; legacy alias agrees', () => { + expect(CANONICAL_PRICING['google:gemini-2.0-flash']).toEqual({ input: 0.1, output: 0.4 }); + expect(CANONICAL_PRICING['google:gemini-2-flash']).toEqual( + CANONICAL_PRICING['google:gemini-2.0-flash'], + ); + }); +}); + +describe('canonicalLookup — id normalization', () => { + test('bare anthropic id → hit (defaults to anthropic provider)', () => { + expect(canonicalLookup('claude-opus-4-8')).toEqual({ input: 5.0, output: 25.0 }); + }); + + test('colon form → hit', () => { + expect(canonicalLookup('anthropic:claude-opus-4-8')).toEqual({ input: 5.0, output: 25.0 }); + }); + + test('slash form → hit', () => { + expect(canonicalLookup('anthropic/claude-opus-4-8')).toEqual({ input: 5.0, output: 25.0 }); + }); + + test('non-anthropic bare id → miss (preserves prior null contract)', () => { + expect(canonicalLookup('gpt-5')).toBeUndefined(); + }); + + test('nested OpenRouter id → MISS (markup ≠ native pricing)', () => { + expect(canonicalLookup('openrouter:anthropic/claude-sonnet-4-6')).toBeUndefined(); + }); + + test('slash-bearing model tail kept as exact key (together Llama)', () => { + expect(canonicalLookup('together:meta-llama/Llama-3.3-70B-Instruct-Turbo')).toEqual({ + input: 0.88, + output: 0.88, + }); + }); + + test('null / empty → undefined (no throw)', () => { + expect(canonicalLookup(null)).toBeUndefined(); + expect(canonicalLookup(undefined)).toBeUndefined(); + expect(canonicalLookup('')).toBeUndefined(); + }); +}); + +describe('DRIFT GUARD — derived views stay equal to canonical (re-hardcode trip-wire)', () => { + test('ANTHROPIC_PRICING (bare) equals canonical anthropic: entries', () => { + for (const [key, p] of Object.entries(CANONICAL_PRICING)) { + if (!key.startsWith('anthropic:')) continue; + const bare = key.slice('anthropic:'.length); + expect(ANTHROPIC_PRICING[bare]).toEqual(p); + } + }); + + test('takes-quality MODEL_PRICING equals canonical for every allowlisted key', () => { + for (const [key, p] of Object.entries(MODEL_PRICING)) { + const c = canonicalLookup(key); + expect(c).toBeDefined(); + expect(p.input_per_1m).toBe(c!.input); + expect(p.output_per_1m).toBe(c!.output); + } + }); + + test('cross-modal panel models are all priced from canonical', () => { + // The runner now calls canonicalLookup(slot.model) directly, so presence + // here = the runner prices these. Mirrors the panel it used to inline. + for (const id of [ + 'openai:gpt-4o', + 'openai:gpt-4o-mini', + 'anthropic:claude-opus-4-7', + 'anthropic:claude-sonnet-4-6', + 'google:gemini-1.5-pro', + 'google:gemini-2.0-flash', + 'together:meta-llama/Llama-3.3-70B-Instruct-Turbo', + 'deepseek:deepseek-chat', + ]) { + expect(canonicalLookup(id)).toBeDefined(); + } + }); +}); + +describe('S1A — raw-index consumers price provider-prefixed ids', () => { + test('estimateAnthropicCost prices anthropic:claude-opus-4-8 (was zero pre-fix)', () => { + // 1 call, 1M in, 1M out → 1*5 + 1*25 = $30. Pre-fix the bare-key index + // missed on the provider-prefixed id and returned 0. + const cost = estimateAnthropicCost('anthropic:claude-opus-4-8', 1, 1_000_000, 1_000_000); + expect(cost).toBeCloseTo(30.0, 2); + }); +}); + +describe('no heavy import (cycle guard)', () => { + test('model-pricing.ts imports only model-id.ts (relative)', () => { + const src = readFileSync( + fileURLToPath(new URL('../src/core/model-pricing.ts', import.meta.url)), + 'utf8', + ); + const relImports = [...src.matchAll(/^\s*import\s.*from\s+['"](\.[^'"]+)['"]/gm)].map( + (m) => m[1], + ); + expect(relImports).toEqual(['./model-id.ts']); + }); +});