mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.42.25.0 fix(pricing): unify chat-model pricing into one canonical source; add Opus 4.8 (#1819) (#1827)
* fix(pricing): unify chat-model pricing into one canonical source; add Opus 4.8 (#1819) Single canonical CANONICAL_PRICING table (src/core/model-pricing.ts) with canonicalLookup; ANTHROPIC_PRICING and takes-quality MODEL_PRICING become derived views. cost-tracker, cross-modal runner, skillopt preflight, brainstorm orchestrator, and brain-score all source from it. Adds Opus 4.8 ($5/$25) so --max-cost-usd and the dream-cycle budget meter enforce on 4.8 runs; fixes a stale Opus 4.7 $15/$75 in the takes-quality gate and reconciles Gemini 2.0 Flash to $0.10/$0.40. Because every table derives from canonical, cross-table price drift is structurally impossible. * chore: bump version and changelog (v0.42.25.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document canonical chat-pricing table for v0.42.25.0 Add KEY_FILES.md entries for src/core/model-pricing.ts (canonical CANONICAL_PRICING + canonicalLookup) and refresh the now-derived anthropic-pricing.ts + takes-quality-eval/pricing.ts entries to current-state. Add the "one canonical chat-pricing table" cross-cutting invariant to CLAUDE.md. Fix the stale model-price snapshot pointer in SEARCH_MODE_METHODOLOGY.md (anthropic-pricing.ts -> model-pricing.ts). Regenerate llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
f868257405
commit
9a0bae8d62
@@ -2,6 +2,35 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.25.0] - 2026-06-03
|
||||
|
||||
**Cost caps and budget gates now fire on Opus 4.8 — and every model price lives in one place, so they can't silently drift again.** If you pointed a gbrain tier at Opus 4.8 (`models.aliases.opus`), the cost guardrails were quietly not working: there was no price on file for 4.8, so the `gbrain dream` budget meter let runs proceed unbounded (it warns `BUDGET_METER_NO_PRICING` and skips the gate for unpriced models), and `gbrain skillopt --max-cost-usd` fell back to a cheaper tier's rate and refused too late. This release adds Opus 4.8 pricing ($5 in / $25 out per 1M tokens, same as 4.7) and the caps enforce again.
|
||||
|
||||
While fixing that, the deeper problem surfaced: model prices were hand-copied across five separate tables and had already drifted apart. One eval's budget gate priced Opus 4.7 at a stale $15/$75 (3x too high), and Gemini 2.0 Flash disagreed with itself between two tables. All chat-model prices now live in one canonical table (`src/core/model-pricing.ts`) and every other table derives from it — so cross-table price drift is structurally impossible, not just discouraged.
|
||||
|
||||
Nothing to configure. `gbrain upgrade` and your caps start enforcing on 4.8.
|
||||
|
||||
### Added
|
||||
- Opus 4.8 pricing ($5 in / $25 out per 1M tokens) so `--max-cost-usd` and the dream-cycle budget meter enforce on 4.8 runs.
|
||||
|
||||
### Changed
|
||||
- All chat-model pricing unified into one canonical source. The Anthropic bare-key table, the takes-quality eval allowlist, the contradictions cost-tracker, the cross-modal eval panel, and the skillopt preflight estimate all derive their numbers from it instead of carrying their own copies.
|
||||
- skillopt preflight now resolves bare, colon, and slash model ids through the shared parser (previously bare-only), so provider-prefixed 4.8 ids price correctly.
|
||||
|
||||
### Fixed
|
||||
- Stale Opus 4.7 price in the takes-quality eval budget gate ($15/$75 → $5/$25), which over-estimated cost ~3x.
|
||||
- Gemini 2.0 Flash price reconciled to $0.10/$0.40 across the budget-gating tables (the coarse per-provider baselines shown by `gbrain providers` are a separate display layer, unchanged here).
|
||||
- Brainstorm and brain-score cost estimates now price provider-prefixed model ids (e.g. `anthropic:claude-opus-4-8`) instead of silently falling back to a default rate.
|
||||
|
||||
### To take advantage of v0.42.25.0
|
||||
|
||||
`gbrain upgrade` applies this automatically — no migrations, no config. To confirm:
|
||||
|
||||
1. With Opus 4.8 as your deep tier, run a capped skillopt dry-run and watch the estimate refuse above the cap:
|
||||
```bash
|
||||
gbrain skillopt <skill> --bootstrap-from-skill --dry-run --max-cost-usd 1
|
||||
```
|
||||
2. The dream-cycle budget meter no longer prints `BUDGET_METER_NO_PRICING` for Opus 4.8.
|
||||
## [0.42.24.0] - 2026-06-03
|
||||
|
||||
**Minion workers no longer silently wedge mid-job on a Supabase brain.** The background worker that runs your cron jobs, enrich fan-out, and autopilot cycle holds a lock on each job and heartbeats it every couple of seconds to say "still working." On a Supabase brain that heartbeat was running on the transaction-mode pooler (the high-traffic 6543 port), which recycles its connections per transaction. A lock is held open for minutes, so the pooler would periodically drop the socket mid-heartbeat. The worker read that dropped socket as "the lock expired," force-evicted its own in-flight job, and then sat in a claim loop holding nothing — process alive, no errors in the log, just quietly doing no work. It showed up most under heavy `enrich` load.
|
||||
|
||||
@@ -74,6 +74,15 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
|
||||
Postgres; plain `CREATE INDEX` on PGLite via `sqlFor.pglite`).
|
||||
- **Multi-source.** Slug uniqueness is `(source_id, slug)`, not slug. Key batch ops and
|
||||
reverse-writes on the composite key; `validateSourceId` before any `source_id` path join.
|
||||
- **One canonical chat-pricing table.** All paid-cloud chat/completion prices live ONCE in
|
||||
`src/core/model-pricing.ts` (`CANONICAL_PRICING` + `canonicalLookup`). Every other table
|
||||
(`anthropic-pricing.ts`'s `ANTHROPIC_PRICING`, `takes-quality-eval/pricing.ts`'s
|
||||
`MODEL_PRICING`, the contradictions/cross-modal/skillopt cost views) is a DERIVED view, never
|
||||
a hand-copied duplicate — so cross-table price drift is structurally impossible. Update a
|
||||
price in `model-pricing.ts` only; each consumer keeps its own key allowlist + miss policy
|
||||
(fail-closed vs warn-only vs null), not its own numbers. Pinned by `test/model-pricing.test.ts`
|
||||
(drift guard asserts each view equals canonical). Embeddings price separately in
|
||||
`embedding-pricing.ts` (different unit).
|
||||
|
||||
|
||||
## Reference map (load on demand)
|
||||
|
||||
@@ -657,9 +657,23 @@ all are latent-debt cleanup.
|
||||
|
||||
- [ ] **Config-write normalization.** Whenever a user writes `gbrain config set models.tier.deep anthropic/claude-opus-4-7` we silently store the slash form. v0.41.22.1 centralized the read-side via `splitProviderModelId`, but config writes still preserve whatever shape the user typed. Canonical form should be colon (`anthropic:claude-opus-4-7`). Fix: rewrite at config-write time in `src/core/config.ts`. Breaks existing config files that explicitly hold the slash form — defer to a v0.42+ config-migration wave that also handles the rewrite + once-per-process deprecation warn. Files: `src/core/config.ts`, `src/core/model-config.ts:saveConfig` path. Priority: P3 (latent, not user-visible).
|
||||
|
||||
- [ ] **Non-Anthropic pricing tables.** `src/core/anthropic-pricing.ts` is the only pricing surface gbrain ships. Brainstorm + LSD users routing through OpenAI / Gemini / OpenRouter get `BUDGET_TRACKER_NO_PRICING` warn-once + bypass-gate (without `--max-cost`) OR `no_pricing` hard-fail (with `--max-cost`). The right shape: rename to `provider-pricing.ts`, add OpenAI / Gemini / OpenRouter tables, route `lookupPricing` through provider-routed table selection. OpenRouter is a special case (period-vs-dash key mismatch: their `claude-sonnet-4.6` won't match our `claude-sonnet-4-6` either way). Files: `src/core/anthropic-pricing.ts` (rename + extend), `src/core/budget/budget-tracker.ts`, `src/core/eval-contradictions/cost-tracker.ts`. Priority: P2 (real user pain when running brainstorm against non-Anthropic).
|
||||
- [ ] **Non-Anthropic budget-tracker pricing.** PARTIALLY ADDRESSED by v0.42.25.0: `src/core/model-pricing.ts` is now the canonical multi-provider table (OpenAI / Google / Together / DeepSeek entries exist alongside Anthropic), and cross-modal-eval + takes-quality already price non-Anthropic models from it. REMAINING: `src/core/budget/budget-tracker.ts:lookupPricing` still routes only through the bare-keyed `ANTHROPIC_PRICING` view, so brainstorm + LSD users running budget gates against OpenAI / Gemini / OpenRouter still get `BUDGET_TRACKER_NO_PRICING` warn-once + bypass-gate (without `--max-cost`) OR `no_pricing` hard-fail (with `--max-cost`). Right fix: route `lookupPricing` through `canonicalLookup`. OpenRouter stays a special case (period-vs-dash key mismatch: their `claude-sonnet-4.6` won't match our `claude-sonnet-4-6`, and it intentionally misses to avoid pricing markup as native). Files: `src/core/budget/budget-tracker.ts`, `src/core/model-pricing.ts`. Priority: P2 (real user pain when running brainstorm against non-Anthropic).
|
||||
|
||||
- [ ] **Eval-contradictions duplicate ANTHROPIC_PRICING consolidation.** `src/core/eval-contradictions/cost-tracker.ts:28-38` ships its OWN copy of the Anthropic pricing table with different keys (both bare and `anthropic:`-prefixed forms) and a silent-Haiku fallback on unknown. v0.41.22.1 routed both tables' lookups through `splitProviderModelId` but left the duplication. Right fix: delete the local table, import from `src/core/anthropic-pricing.ts`. Either (a) preserve the silent-Haiku-fallback semantic with an explicit `?? canonicalPricing['claude-haiku-4-5']` at the call site, or (b) tighten to warn-once on unknown (which changes the eval-contradictions soft-ceiling `--budget-usd` contract — coordinate with that subsystem). Files: `src/core/eval-contradictions/cost-tracker.ts`, `src/core/anthropic-pricing.ts`, `test/eval-contradictions/cost-tracker-slash.test.ts` (the legacy-Haiku-fallback pin would need updating). Priority: P3 (DRY cleanup, no user-visible impact).
|
||||
- [x] **Eval-contradictions duplicate ANTHROPIC_PRICING consolidation.** **Completed:** v0.42.25.0 (2026-06-03). Deleted the local duplicate table in `src/core/eval-contradictions/cost-tracker.ts`; it now imports the canonical-derived `ANTHROPIC_PRICING` view and `pricingFor` preserves the silent-Haiku fallback (pinned by `test/eval-contradictions/cost-tracker-slash.test.ts`). Closed as part of the wider model-pricing unification.
|
||||
|
||||
## v0.42.25.0 pricing-unification follow-ups (v0.42+)
|
||||
|
||||
Filed from the v0.42.25.0 ship review (Claude + Codex adversarial + pre-landing).
|
||||
All latent / hardening — none are user-reported bugs. The unification landed a
|
||||
single canonical `src/core/model-pricing.ts` with `canonicalLookup`.
|
||||
|
||||
- [ ] **`canonicalLookup` is case-sensitive (silent-miss undercount).** `src/core/model-pricing.ts:canonicalLookup` does exact-key + `splitProviderModelId` lookups with no lowercasing, so `ANTHROPIC:claude-opus-4-8` or `anthropic:CLAUDE-OPUS-4-8` return `undefined` → consumers that treat a miss as zero-cost (cross-modal runner note, cost-tracker silent-Haiku, skillopt Sonnet fallback) silently mis-budget. Latent today (recipe/CLI paths emit lowercase), but the fail-mode is a silent undercount, not a throw. Fix: lowercase provider+model before lookup in `canonicalLookup`. Add a mixed-case test. Priority: P3.
|
||||
|
||||
- [ ] **takes-quality `getPricing` is exact-key only.** `src/core/takes-quality-eval/pricing.ts:getPricing` does a raw `MODEL_PRICING[modelId]` lookup. A user passing a bare/slash/dotted form of an allowlisted model (e.g. `google:gemini-2.0-flash` when the allowlist holds `google:gemini-2-flash`, or `anthropic/claude-opus-4-8`) hits `PricingNotFoundError` even though canonical prices it. Safe direction (fail-closed) but a usability regression. Fix: normalize the lookup key through `canonicalLookup`/`splitProviderModelId` before the allowlist check, keeping fail-closed for genuinely-unsupported models. Priority: P3.
|
||||
|
||||
- [ ] **No negative-path test for the takes-quality module-load throw.** `src/core/takes-quality-eval/pricing.ts` throws at import if a `SUPPORTED_MODELS` id is absent from canonical (good fail-fast), but nothing tests it (awkward to test a module-load-time throw in-process). Add a small harness/fixture test. Priority: P3 (programmer-error guard).
|
||||
|
||||
- [ ] **Recipe display-layer pricing is stale and unconsolidated.** Each `src/core/ai/recipes/*.ts` carries coarse per-provider `cost_per_1m_input_usd`/`cost_per_1m_output_usd` baselines (e.g. `google.ts` chat = `$0.30/$1.20`, `price_last_verified: 2026-04-20`) read only by `gbrain providers` for display — NOT by any budget gate. They've drifted (google chat baseline predates the Gemini 2.0 Flash `$0.10/$0.40` reconciliation; codex flagged OpenAI baselines too). These are intentionally a separate coarse layer from the per-model `model-pricing.ts` budget tables, so consolidating is non-trivial (one-number-per-provider vs per-model). Options: (a) refresh the `price_last_verified` baselines, or (b) have `gbrain providers` show per-model rates from canonical where available and fall back to the recipe baseline. Flagged by the v0.42.25.0 ship Codex adversarial pass. Priority: P3 (display-only, no budget-gating impact).
|
||||
|
||||
## v0.41.21.0 ops-fix-wave follow-ups (v0.41.22+)
|
||||
|
||||
|
||||
@@ -107,7 +107,9 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/search/autocut.ts` (default ON in reranked modes) — Weaviate-style autocut: score-discontinuity result-sizing on the cross-encoder rerank separatrix. `applyAutocut(results, scoreOf, cfg)` normalizes the reranker scores, finds the largest consecutive gap, and cuts there when it clears `jumpRatio` (default 0.20); robust to unsorted provider output (cuts on a sorted copy, keeps items in INPUT order via a score threshold), guards `top<=0`/non-finite, never returns empty, and no-ops when <2 results carry a finite `rerank_score` (covers the reranker fail-open path). WHY rerank_score and NOT RRF/cosine: gbrain measured (see `return-policy.ts`) that the RRF rank1→rank2 gap is ~flat whether rank-1 is right or wrong — not a separatrix; the cross-encoder score IS. So autocut runs ONLY where the reranker ran (the floor reaches `balanced`+`tokenmax`; `conservative` is a documented no-op). Exports `AutocutConfig`, `DEFAULT_AUTOCUT` (frozen: `enabled=true, jumpRatio=0.20, minKeep=1`), `AutocutDecision` (`{applied, signal:'rerank'|'none', cut, kept, total, gapRatio}`), `AutocutInput`, `autocutFromConfig`, `resolveAutocut`, `applyAutocut`. Cache-key integration (clean path, not the adaptive-return cache-skip hack): enable+sensitivity flow through `ModeBundle` → `ResolvedSearchKnobs` → `knobsHash` exactly like `graph_signals`. `mode.ts` adds `autocut`/`autocut_jump` (conservative false, balanced/tokenmax true@0.20) AND sets `reranker_top_n_in = searchLimit` for reranked modes (so the reranker scores the full returned set; there is no un-scored tail for autocut to wrongly drop — closes the load-bearing recall finding). `KNOBS_HASH_VERSION` is 8 (title_boost claimed 7; autocut appends 8 — one-time global cache cold-miss on upgrade). Wired into `hybridSearch` AFTER adaptive-return, BEFORE the limit slice, first page only; emits `HybridSearchMeta.autocut`. BOTH the cache-miss `finalMeta` and cache-HIT `cachedMeta` rebuilds carry `autocut`+`adaptive_return`+`mode`+`embedding_column`. Preserves alias-hop exact matches: `applyAutocut` takes an optional `preserve` predicate; hybrid passes `r => r.alias_hit === true` so a canonical page injected by `applyAliasHop` after reranking (no `rerank_score`) is never cut. Agent surface: `query` op `autocut` boolean (ceiling override — `false` forces full top-K); `SearchOpts.autocut`; `--explain` shows per-result `rerank_score`, `formatAutocutSummary` renders the decision when search meta is threaded; `gbrain search modes` attribution; metric glossary `autocut.signal`/`autocut.gap_ratio`. Config: `search.autocut`, `search.autocut_jump`. Default-ON backed by an in-repo eval gate — `test/search/autocut-eval.test.ts` (also `bun run eval:autocut`) measures precision-lift-without-recall-regression over labeled qrels fixtures with modeled cross-encoder distributions (no API key, no sibling repo; runs in CI): mean precision 0.33→0.94, recall 1.00→0.95, ZERO recall regression on enumeration queries. Env-overridable floors. Pinned by `test/search/autocut.test.ts` (pure-fn), `test/search/query-op-autocut.test.ts` (agent surface), `test/search/autocut-integration.serial.test.ts` (IRON-RULE behavioral via `rerankerFn` DI seam: cliff trims, flat doesn't, no-reranker no-ops, `autocut:false` ceiling, composes with adaptive-return), `test/search/autocut-eval.test.ts` (the precision/recall gate), and the v=8 knobsHash assertions in `test/search-mode.test.ts`.
|
||||
- `src/core/ai/recipes/voyage.ts` — Voyage AI openai-compatible recipe. Declares `chars_per_token=1` + `safety_factor=0.5` so the gateway pre-splits Voyage batches at a 60K-character budget (50% of 120K-token cap with the dense-tokenizer ratio), avoiding the backfill loop where tiktoken-grounded budgeting undercounted Voyage's actual token usage. Declares `multimodal_models: ['voyage-multimodal-3']` so the gateway rejects text-only Voyage models pointed at the multimodal endpoint with a clear `AIConfigError` instead of waiting for Voyage's HTTP 400. Recipe docstring at `:7-16` names the seven hosted flexible-dim models that accept `output_dimension` (`voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3`) and notes `voyage-4-nano` is the open-weight variant fixed at 1024-dim that does NOT accept the parameter (negative regression assertion in `test/ai/gateway.test.ts`: `dimsProviderOptions` returns `undefined` for `voyage-4-nano`). `voyage-code-3` is the recommended embedding model for gstack per-worktree code brains (Topology 3 in `docs/architecture/topologies.md`); discoverability surfaces: decision-tree branch in `docs/integrations/embedding-providers.md`, Topology 3 "Recommended embedding model" subsection, runtime nudge from `gbrain reindex --code` against non-code-tuned models. Recipe-shape regression pinned by `test/ai/voyage-code-3-recipe.test.ts`.
|
||||
- `src/core/ai/recipes/anthropic.ts` — Anthropic recipe (chat + expansion touchpoints). Canonical id is `claude-sonnet-4-6` (no date suffix); a reverse alias `claude-sonnet-4-6-20250929 → claude-sonnet-4-6` keeps stale user configs working (rescues `facts.extraction_model` and `models.dream.synthesize`). Recipe-shape regression pinned by `test/anthropic-model-ids.test.ts`.
|
||||
- `src/core/anthropic-pricing.ts` — single source of truth for Anthropic model pricing (per-MTok input/output). Opus 4.7 is `$5/$25` (Opus 4.6 also corrected). Consumed by `src/core/budget-meter.ts` and `src/core/cross-modal-eval/runner.ts` — the cross-modal estimator reads `ANTHROPIC_PRICING` for Anthropic models instead of duplicating the table.
|
||||
- `src/core/model-pricing.ts` — single source of truth for paid-cloud CHAT/completion model pricing (USD per 1M tokens, input | output). `CANONICAL_PRICING` is a `provider:model`-keyed table (Anthropic Opus 4.8/4.7/4.6 `$5/$25`, Sonnet 4.6 `$3/$15`, Haiku 4.5 `$1/$5` both dateless + dated, plus OpenAI / Google / Together / DeepSeek panel models). `canonicalLookup(modelId)` resolves bare (`claude-opus-4-8`), colon (`anthropic:claude-opus-4-8`), and slash (`anthropic/...`) forms — bare ids default to the `anthropic:` provider; nested OpenRouter ids (`openrouter:anthropic/...`) intentionally MISS so OpenRouter markup isn't repriced as the inner vendor. Every other chat-pricing table is a DERIVED view of this one (NOT a hand-copied duplicate), so cross-table price drift is structurally impossible. Embeddings live separately in `embedding-pricing.ts` (different unit). Pinned by `test/model-pricing.test.ts` whose drift guard asserts each derived view equals canonical and that the cross-modal panel models are all present.
|
||||
- `src/core/anthropic-pricing.ts` — bare-keyed Anthropic VIEW of `model-pricing.ts` (the `anthropic:` canonical entries with the prefix stripped). Kept distinct because many callers look up by bare Claude id and because `estimateMaxCostUsd(modelId, inTokens, maxOutTokens)` carries the null-on-miss contract the dream-cycle budget gate depends on (non-Anthropic ids return null → caller warns `BUDGET_METER_NO_PRICING` once and runs unbounded). `estimateMaxCostUsd` routes bare/colon/slash ids through `splitProviderModelId`. Do NOT hand-edit prices here — the map is derived from canonical, so it cannot drift. `ANTHROPIC_PRICING` is consumed by `budget/budget-tracker.ts`, `minions/batch-projection.ts`, and `cycle/budget-meter.ts`.
|
||||
- `src/core/takes-quality-eval/pricing.ts` — fail-closed budget pricing for `eval takes-quality run --budget-usd N`. `MODEL_PRICING` is a curated `provider:model` allowlist (default panel + likely overrides) whose VALUES are derived from `model-pricing.ts` via `canonicalLookup`; an allowlisted id missing from canonical throws at module load. Schema is `{input_per_1m, output_per_1m}`. A model NOT on the allowlist aborts the run with an actionable error rather than guessing (distinct from `cross-modal-eval/runner.ts`, which silently estimates zero on unknown models — both now source numbers from canonical).
|
||||
- `src/core/budget/budget-tracker.ts` — keystone primitive for the brainstorm cost-cathedral wave. One typed error (`BudgetExhausted` with `reason: 'cost' | 'runtime' | 'no_pricing'`), one schema-stable audit JSONL at `~/.gbrain/audit/budget-YYYY-Www.jsonl`. Contracts: `record()` throws when cumulative spend exceeds cap (the cap is a real ceiling, not a suggestion); `reserve()` hard-fails with `reason: 'no_pricing'` when `maxCostUsd` is set AND the model is missing from pricing maps (warn-once preserved when cap is unset); `extractUsageFromError(err, fallback)` returns `err.usage` when the SDK provides it, else the pessimistic fallback (caller passes `maxOutputTokens`, not the optimistic pre-call estimate). `onExhausted(cb)` fires once synchronously BEFORE the throw propagates so callers can persist checkpoints. Replaces three parallel copies (inline brainstorm class, cycle/budget-meter, eval-contradictions). Adapts the old `BudgetMeter` (public shape preserved + `schema_version: 1` stamped on every dream-budget audit line). Pinned by 18 unit cases.
|
||||
- `src/core/audit-week-file.ts` — single source of truth for ISO-week audit JSONL filename math. Exports `isoWeek(d)`, `isoWeekFilename(prefix, now?)`, `resolveAuditDir()` (honors `GBRAIN_AUDIT_DIR`). Year-boundary correctness pinned by tests at 2020-W53 (the 53-week year), 2025-W01 rolling in from 2024-12-30 (Monday), 2026-W01. Four call sites migrated: `src/core/minions/handlers/shell-audit.ts`, `src/core/facts/phantom-audit.ts`, `src/core/audit-slug-fallback.ts`, `src/core/cycle/budget-meter.ts`. Each keeps its `compute<X>AuditFilename` thin wrapper for back-compat with existing tests.
|
||||
- `src/core/ai/gateway.ts:withBudgetTracker` — gateway-layer enforcement via `AsyncLocalStorage<BudgetTracker>`. `withBudgetTracker(tracker, fn)` installs the tracker on the module-internal store; every `gateway.chat / embed / rerank` call inside the scope auto-composes (reserve before, record in try/finally). Outside-scope calls are budget no-ops. Nested scopes restore the outer tracker on exit. `getCurrentBudgetTracker()` is the test seam. The chat path uses the pessimistic fallback on error paths; the embed path estimates input tokens from char count × recipe's `chars_per_token` because the AI SDK doesn't surface per-batch embed token usage; the rerank path estimates char count of query+docs. Pinned by 6 unit cases.
|
||||
|
||||
@@ -160,7 +160,7 @@ The mode-picker prompt at `gbrain init` and the CLAUDE.md `## Search Mode` table
|
||||
- Your agent's system prompt + reasoning tokens add input that gbrain doesn't see.
|
||||
- Compaction reduces input over a long session.
|
||||
- Most agents make 1-5 searches per turn; cost-per-turn is what bills you, not cost-per-query.
|
||||
- The model price column drifts as providers reprice; pin the rate via `src/core/anthropic-pricing.ts` for a current snapshot.
|
||||
- The model price column drifts as providers reprice; pin the rate via `src/core/model-pricing.ts` (the canonical chat-pricing table) for a current snapshot.
|
||||
|
||||
The picker copy + CLAUDE.md table are the canonical user-facing source. Update them in lockstep when the underlying chunker size or default `searchLimit` changes.
|
||||
|
||||
|
||||
@@ -222,6 +222,15 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
|
||||
Postgres; plain `CREATE INDEX` on PGLite via `sqlFor.pglite`).
|
||||
- **Multi-source.** Slug uniqueness is `(source_id, slug)`, not slug. Key batch ops and
|
||||
reverse-writes on the composite key; `validateSourceId` before any `source_id` path join.
|
||||
- **One canonical chat-pricing table.** All paid-cloud chat/completion prices live ONCE in
|
||||
`src/core/model-pricing.ts` (`CANONICAL_PRICING` + `canonicalLookup`). Every other table
|
||||
(`anthropic-pricing.ts`'s `ANTHROPIC_PRICING`, `takes-quality-eval/pricing.ts`'s
|
||||
`MODEL_PRICING`, the contradictions/cross-modal/skillopt cost views) is a DERIVED view, never
|
||||
a hand-copied duplicate — so cross-table price drift is structurally impossible. Update a
|
||||
price in `model-pricing.ts` only; each consumer keeps its own key allowlist + miss policy
|
||||
(fail-closed vs warn-only vs null), not its own numbers. Pinned by `test/model-pricing.test.ts`
|
||||
(drift guard asserts each view equals canonical). Embeddings price separately in
|
||||
`embedding-pricing.ts` (different unit).
|
||||
|
||||
|
||||
## Reference map (load on demand)
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.24.0"
|
||||
"version": "0.42.25.0"
|
||||
}
|
||||
|
||||
@@ -1,45 +1,38 @@
|
||||
/**
|
||||
* v0.28: Anthropic model pricing constants for the dream-cycle budget meter.
|
||||
* Anthropic chat pricing — a bare-keyed VIEW of the canonical pricing table
|
||||
* (`src/core/model-pricing.ts`).
|
||||
*
|
||||
* Prices in USD per 1M tokens (input | output). Numbers reflect Anthropic's
|
||||
* published pricing as of 2026-05-01. Update when Anthropic publishes new
|
||||
* pricing — the JSON in `~/.gbrain/audit/dream-budget-*.jsonl` carries the
|
||||
* snapshot per call so historical estimates stay reproducible.
|
||||
* Kept as a distinct export because many callers look up by bare Claude id
|
||||
* (`claude-opus-4-7`) and because `estimateMaxCostUsd` carries the
|
||||
* null-on-miss contract the dream-cycle budget gate depends on. The dollar
|
||||
* numbers live in model-pricing.ts — DO NOT hand-edit prices here; this map is
|
||||
* derived from the `anthropic:` canonical entries (prefix stripped), so it
|
||||
* cannot drift from the other pricing views. (Pre-unification this map and
|
||||
* takes-quality-eval/pricing.ts duplicated the numbers and drifted: Opus 4.7
|
||||
* read $15/$75 in one and $5/$25 in the other.)
|
||||
*
|
||||
* Codex P1 #10 fold: non-Anthropic models (gemini, gpt, anything not in
|
||||
* this map) bypass the budget gate with a `BUDGET_METER_NO_PRICING` warn
|
||||
* once per process. The cycle still runs unbounded for those models.
|
||||
* Future: per-provider pricing modules.
|
||||
* Codex P1 #10 fold: non-Anthropic models (gemini, gpt, anything not in this
|
||||
* map) bypass the budget gate with a `BUDGET_METER_NO_PRICING` warn once per
|
||||
* process. The cycle still runs unbounded for those models.
|
||||
*/
|
||||
|
||||
export interface ModelPricing {
|
||||
/** USD per 1M input tokens. */
|
||||
input: number;
|
||||
/** USD per 1M output tokens. */
|
||||
output: number;
|
||||
}
|
||||
|
||||
/** Map of Anthropic model id → pricing. Aliases (opus/sonnet/haiku) resolve via DEFAULT_ALIASES. */
|
||||
export const ANTHROPIC_PRICING: Record<string, ModelPricing> = {
|
||||
// Claude 4.7 generation (current)
|
||||
// Opus 4.7 dropped from $15/$75 (Opus 4) to $5/$25 per
|
||||
// https://platform.claude.com/docs/en/about-claude/models/overview (verified 2026-05-10).
|
||||
'claude-opus-4-7': { input: 5.00, output: 25.00 },
|
||||
'claude-sonnet-4-6': { input: 3.00, output: 15.00 },
|
||||
// Both the dateless canonical id (TIER_DEFAULTS / aliases / every caller uses
|
||||
// this) AND the dated snapshot. The dateless entry was missing pre-v0.42.9.0,
|
||||
// so a budget-capped Haiku run (skillopt with --max-cost, eval harnesses) hit
|
||||
// BudgetTracker no_pricing and every rollout silently scored 0.
|
||||
'claude-haiku-4-5': { input: 1.00, output: 5.00 },
|
||||
'claude-haiku-4-5-20251001': { input: 1.00, output: 5.00 },
|
||||
// Older but still frequently aliased
|
||||
'claude-opus-4-6': { input: 5.00, output: 25.00 },
|
||||
'claude-3-5-sonnet-20241022': { input: 3.00, output: 15.00 },
|
||||
'claude-3-5-haiku-20241022': { input: 0.80, output: 4.00 },
|
||||
};
|
||||
|
||||
import { CANONICAL_PRICING, type ModelPricing } from './model-pricing.ts';
|
||||
import { splitProviderModelId } from './model-id.ts';
|
||||
|
||||
export type { ModelPricing };
|
||||
|
||||
/**
|
||||
* Bare-keyed Anthropic view, derived from the canonical table. Both the
|
||||
* dateless ids (`claude-haiku-4-5`, used by aliases / TIER_DEFAULTS / most
|
||||
* callers) and the dated snapshots (`claude-haiku-4-5-20251001`) are present
|
||||
* because canonical carries both.
|
||||
*/
|
||||
export const ANTHROPIC_PRICING: Record<string, ModelPricing> = Object.fromEntries(
|
||||
Object.entries(CANONICAL_PRICING)
|
||||
.filter(([key]) => key.startsWith('anthropic:'))
|
||||
.map(([key, pricing]) => [key.slice('anthropic:'.length), pricing]),
|
||||
);
|
||||
|
||||
/**
|
||||
* Estimate the upper-bound USD cost of a single submit.
|
||||
* Uses (estimatedInputTokens × inputRate) + (maxOutputTokens × outputRate).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { createHash } from 'crypto';
|
||||
import type { BrainHealth } from './types.ts';
|
||||
import { ANTHROPIC_PRICING } from './anthropic-pricing.ts';
|
||||
import { canonicalLookup } from './model-pricing.ts';
|
||||
import { lookupEmbeddingPrice, estimateCostFromChars } from './embedding-pricing.ts';
|
||||
import { getRecipe } from './ai/recipes/index.ts';
|
||||
import { parseModelId } from './ai/model-resolver.ts';
|
||||
@@ -433,7 +433,7 @@ export function estimateAnthropicCost(
|
||||
estInputTokensPerCall = 5_000,
|
||||
estOutputTokensPerCall = 1_000,
|
||||
): number {
|
||||
const pricing = ANTHROPIC_PRICING[modelId];
|
||||
const pricing = canonicalLookup(modelId);
|
||||
if (!pricing) return 0;
|
||||
const inputCost = (estInputTokensPerCall * estCallsPerInvocation / 1_000_000) * pricing.input;
|
||||
const outputCost = (estOutputTokensPerCall * estCallsPerInvocation / 1_000_000) * pricing.output;
|
||||
|
||||
@@ -46,7 +46,7 @@ import {
|
||||
type JudgeConfig,
|
||||
type ChatFn,
|
||||
} from './judges.ts';
|
||||
import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts';
|
||||
import { canonicalLookup } from '../model-pricing.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// BudgetExhausted is the canonical typed error (Q2) used by every cost
|
||||
@@ -264,7 +264,7 @@ export function estimateCost(profile: BrainstormProfile, model: string): number
|
||||
const judgeIn = ideas * 350;
|
||||
const judgeOut = ideas * 200;
|
||||
|
||||
const pricing = ANTHROPIC_PRICING[model] ?? { input: 3, output: 15 };
|
||||
const pricing = canonicalLookup(model) ?? { input: 3, output: 15 };
|
||||
const inCost = ((inTokens + judgeIn) / 1_000_000) * pricing.input;
|
||||
const outCost = ((outTokens + judgeOut) / 1_000_000) * pricing.output;
|
||||
return inCost + outCost;
|
||||
@@ -771,7 +771,7 @@ async function _runBrainstormInner(
|
||||
crossModel = result.model;
|
||||
// Mid-run cost guard: if running spend already exceeds the projected
|
||||
// ceiling or the strict-budget multiplier, abort the remaining crosses.
|
||||
const runningPricing = ANTHROPIC_PRICING[result.model] ?? { input: 3, output: 15 };
|
||||
const runningPricing = canonicalLookup(result.model) ?? { input: 3, output: 15 };
|
||||
const runningUsd =
|
||||
(totalUsage.input_tokens / 1_000_000) * runningPricing.input +
|
||||
(totalUsage.output_tokens / 1_000_000) * runningPricing.output;
|
||||
@@ -897,7 +897,7 @@ async function _runBrainstormInner(
|
||||
// Cost actuals (codex r2 #10).
|
||||
const totalIn = totalUsage.input_tokens + judgeUsage.input_tokens;
|
||||
const totalOut = totalUsage.output_tokens + judgeUsage.output_tokens;
|
||||
const pricing = ANTHROPIC_PRICING[crossModel] ?? { input: 3, output: 15 };
|
||||
const pricing = canonicalLookup(crossModel) ?? { input: 3, output: 15 };
|
||||
const actual = (totalIn / 1_000_000) * pricing.input + (totalOut / 1_000_000) * pricing.output;
|
||||
stderr(`[${profile.label}] actual cost: ${fmtUsd(actual)} (estimated ${fmtUsd(estimate)}) — in=${totalIn} out=${totalOut} tokens\n`);
|
||||
|
||||
|
||||
@@ -22,7 +22,7 @@ import type { AggregateResult, SlotResult } from './aggregate.ts';
|
||||
import { parseModelJSON } from './json-repair.ts';
|
||||
import { receiptName, sha8 } from './receipt-name.ts';
|
||||
import { writeReceipt } from './receipt-write.ts';
|
||||
import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts';
|
||||
import { canonicalLookup } from '../model-pricing.ts';
|
||||
|
||||
export const RECEIPT_SCHEMA_VERSION = 1;
|
||||
|
||||
@@ -322,37 +322,23 @@ export function estimateCost(slots: SlotConfig[], cycles: number, maxTokens: num
|
||||
// Per-call cost = (input_tokens × input_price + output_tokens × output_price) / 1e6.
|
||||
// Without knowing prompt size, estimate input ~5k tokens (a SKILL.md + scoring rubric).
|
||||
//
|
||||
// Anthropic prices read from ANTHROPIC_PRICING (single source of truth — fixes
|
||||
// the drift trap Codex flagged in v0.31.12 plan review: this map and
|
||||
// anthropic-pricing.ts duplicated Anthropic prices, with stale values diverging).
|
||||
// Non-Anthropic models still live inline until OPENAI_PRICING / GOOGLE_PRICING
|
||||
// tables exist.
|
||||
// All prices (anthropic + openai + google + together + deepseek) come from the
|
||||
// canonical table via canonicalLookup (src/core/model-pricing.ts) — single
|
||||
// source of truth. This finishes the de-duplication the v0.31.12 plan started
|
||||
// for Anthropic; OpenAI/Google/Together/DeepSeek panel models no longer carry
|
||||
// inline rates here. Slots with no canonical entry fall to the "no pricing on
|
||||
// file" note (cost estimate may be low), preserving prior behavior.
|
||||
const ESTIMATED_INPUT_TOKENS = 5000;
|
||||
const anthropicPrice = (modelId: string): { in: number; out: number } | undefined => {
|
||||
const p = ANTHROPIC_PRICING[modelId];
|
||||
return p ? { in: p.input, out: p.output } : undefined;
|
||||
};
|
||||
const PRICING: Record<string, { in: number; out: number } | undefined> = {
|
||||
'openai:gpt-4o': { in: 2.5, out: 10.0 },
|
||||
'openai:gpt-4o-mini': { in: 0.15, out: 0.6 },
|
||||
'anthropic:claude-opus-4-7': anthropicPrice('claude-opus-4-7'),
|
||||
'anthropic:claude-sonnet-4-6': anthropicPrice('claude-sonnet-4-6'),
|
||||
'anthropic:claude-haiku-4-5-20251001': anthropicPrice('claude-haiku-4-5-20251001'),
|
||||
'google:gemini-1.5-pro': { in: 1.25, out: 5.0 },
|
||||
'google:gemini-2.0-flash': { in: 0.1, out: 0.4 },
|
||||
'together:meta-llama/Llama-3.3-70B-Instruct-Turbo': { in: 0.88, out: 0.88 },
|
||||
'deepseek:deepseek-chat': { in: 0.14, out: 0.28 },
|
||||
};
|
||||
|
||||
const notes: string[] = [];
|
||||
let perCycle = 0;
|
||||
for (const slot of slots) {
|
||||
const p = PRICING[slot.model];
|
||||
const p = canonicalLookup(slot.model);
|
||||
if (!p) {
|
||||
notes.push(`(${slot.model}): no pricing on file; cost estimate may be low`);
|
||||
continue;
|
||||
}
|
||||
const cost = (ESTIMATED_INPUT_TOKENS * p.in + maxTokens * p.out) / 1_000_000;
|
||||
const cost = (ESTIMATED_INPUT_TOKENS * p.input + maxTokens * p.output) / 1_000_000;
|
||||
perCycle += cost;
|
||||
}
|
||||
return {
|
||||
|
||||
@@ -21,28 +21,16 @@
|
||||
|
||||
import type { CostBreakdown } from './types.ts';
|
||||
import { splitProviderModelId } from '../model-id.ts';
|
||||
import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts';
|
||||
|
||||
/**
|
||||
* Per-million-token prices (USD). Update when models bump. These are
|
||||
* approximate — provider accounting after the call is authoritative.
|
||||
*
|
||||
* NOTE: duplicate of the canonical `src/core/anthropic-pricing.ts` table.
|
||||
* Slated for consolidation (TODOS.md #3 from v0.41.20.0 plan); keys differ
|
||||
* (this table uses both bare and `anthropic:`-prefixed forms; canonical
|
||||
* is bare-only). For now we route lookup through `parseModelId` so the
|
||||
* slash-prefix bug class is closed at this site too.
|
||||
* Chat prices come from the canonical table via the bare-keyed
|
||||
* `ANTHROPIC_PRICING` view (`src/core/anthropic-pricing.ts` → `model-pricing.ts`).
|
||||
* This site used to carry its own duplicate (TODOS.md #3); folding it in closes
|
||||
* that consolidation. `pricingFor` still routes through `splitProviderModelId`
|
||||
* so colon/slash forms hit, and keeps the legacy silent-Haiku fallback for
|
||||
* genuinely-unknown models (pinned by test/eval-contradictions/cost-tracker-slash.test.ts).
|
||||
*/
|
||||
const ANTHROPIC_PRICING: Record<string, { input: number; output: number }> = {
|
||||
// Haiku 4.5: ~$1/Mtok in, $5/Mtok out (current as of 2026-05).
|
||||
'claude-haiku-4-5': { input: 1.0, output: 5.0 },
|
||||
'anthropic:claude-haiku-4-5': { input: 1.0, output: 5.0 },
|
||||
// Sonnet 4.6: ~$3/Mtok in, $15/Mtok out.
|
||||
'claude-sonnet-4-6': { input: 3.0, output: 15.0 },
|
||||
'anthropic:claude-sonnet-4-6': { input: 3.0, output: 15.0 },
|
||||
// Opus 4.7: ~$5/Mtok in, $25/Mtok out.
|
||||
'claude-opus-4-7': { input: 5.0, output: 25.0 },
|
||||
'anthropic:claude-opus-4-7': { input: 5.0, output: 25.0 },
|
||||
};
|
||||
|
||||
/** OpenAI text-embedding-3-large: ~$0.13/Mtok (current as of 2026-05). */
|
||||
const OPENAI_EMBEDDING_PRICE_PER_MTOK = 0.13;
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
/**
|
||||
* model-pricing.ts — single source of truth for paid cloud CHAT/completion
|
||||
* model pricing (USD per 1M tokens, input | output).
|
||||
*
|
||||
* Every chat-pricing site in the codebase derives its numbers from this table:
|
||||
* - anthropic-pricing.ts (bare-keyed Anthropic view + estimateMaxCostUsd)
|
||||
* - takes-quality-eval/pricing.ts (curated fail-closed allowlist)
|
||||
* - eval-contradictions/cost-tracker.ts (silent-Haiku-fallback view)
|
||||
* - cross-modal-eval/runner.ts (multi-provider eval panel)
|
||||
* - skillopt/preflight.ts (Sonnet-fallback warn-only estimate)
|
||||
* The bare-keyed `ANTHROPIC_PRICING` view is itself consumed by budget/budget-tracker.ts,
|
||||
* minions/batch-projection.ts, and cycle/budget-meter.ts — so those inherit canonical too.
|
||||
*
|
||||
* The dollar amounts live HERE ONCE — update prices in this file only. Each
|
||||
* consumer keeps its own key allowlist and miss-handling policy (fail-closed
|
||||
* vs warn-only vs null); this module owns the values, not the policy. Because
|
||||
* every other table is DERIVED from this one (not a hand-copied duplicate),
|
||||
* cross-table price drift — the kind that left Opus 4.7 at $15/$75 in one table
|
||||
* for months — is structurally impossible. test/model-pricing.test.ts pins that:
|
||||
* its "drift guard" asserts each derived view still equals canonical (a
|
||||
* regression trip-wire if anyone later re-hardcodes a view back into a duplicate)
|
||||
* and that the cross-modal panel models are all present in canonical.
|
||||
*
|
||||
* Prices verified 2026-06-03 against published provider pricing:
|
||||
* - Anthropic: https://platform.claude.com/docs/en/about-claude/models/overview
|
||||
* - OpenAI: https://openai.com/api/pricing
|
||||
* - Google: https://ai.google.dev/gemini-api/docs/pricing
|
||||
* The dream-budget audit JSONL snapshots the rate per call, so historical
|
||||
* estimates stay reproducible even after this table changes.
|
||||
*
|
||||
* Scope: PAID CLOUD chat models only. Free/local providers (llama-server,
|
||||
* zero-cost rerankers) are intentionally absent — callers treat those as
|
||||
* zero-cost elsewhere. Embeddings live in embedding-pricing.ts (different unit:
|
||||
* per-MTok, char-based).
|
||||
*/
|
||||
|
||||
import { splitProviderModelId } from './model-id.ts';
|
||||
|
||||
export interface ModelPricing {
|
||||
/** USD per 1M input tokens. */
|
||||
input: number;
|
||||
/** USD per 1M output tokens. */
|
||||
output: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Canonical price table. Keys are provider-prefixed (`provider:model`),
|
||||
* matching the exact id strings consumers pass. One physical model may carry
|
||||
* more than one key when a provider ships multiple id spellings (e.g.
|
||||
* `google:gemini-2.0-flash` plus the legacy `google:gemini-2-flash` alias) —
|
||||
* keep aliases in lockstep; the drift guard asserts they agree.
|
||||
*/
|
||||
export const CANONICAL_PRICING: Record<string, ModelPricing> = {
|
||||
// ── Anthropic ──────────────────────────────────────────────────────────
|
||||
// Opus 4.x: $5 in / $25 out. 4.8 (released 2026-05-28) shares 4.7's
|
||||
// per-token rate — closes gbrain#1819.
|
||||
'anthropic:claude-opus-4-8': { input: 5.00, output: 25.00 },
|
||||
'anthropic:claude-opus-4-7': { input: 5.00, output: 25.00 },
|
||||
'anthropic:claude-opus-4-6': { input: 5.00, output: 25.00 },
|
||||
'anthropic:claude-sonnet-4-6': { input: 3.00, output: 15.00 },
|
||||
// Haiku 4.5 — both the dateless canonical id and the dated snapshot.
|
||||
'anthropic:claude-haiku-4-5': { input: 1.00, output: 5.00 },
|
||||
'anthropic:claude-haiku-4-5-20251001': { input: 1.00, output: 5.00 },
|
||||
'anthropic:claude-3-5-sonnet-20241022': { input: 3.00, output: 15.00 },
|
||||
'anthropic:claude-3-5-haiku-20241022': { input: 0.80, output: 4.00 },
|
||||
|
||||
// ── OpenAI ─────────────────────────────────────────────────────────────
|
||||
'openai:gpt-4o': { input: 2.50, output: 10.00 },
|
||||
'openai:gpt-4o-mini': { input: 0.15, output: 0.60 },
|
||||
'openai:gpt-5': { input: 5.00, output: 20.00 },
|
||||
'openai:gpt-5.5': { input: 4.00, output: 16.00 },
|
||||
|
||||
// ── Google ─────────────────────────────────────────────────────────────
|
||||
'google:gemini-1.5-pro': { input: 1.25, output: 5.00 },
|
||||
// Gemini 2.0 Flash: $0.10 in / $0.40 out (verified 2026-06-03). Reconciled
|
||||
// from a stale $0.30/$1.20 entry that had drifted in takes-quality-eval.
|
||||
// `gemini-2-flash` kept as an alias for the legacy id spelling.
|
||||
'google:gemini-2.0-flash': { input: 0.10, output: 0.40 },
|
||||
'google:gemini-2-flash': { input: 0.10, output: 0.40 },
|
||||
|
||||
// ── Together / DeepSeek (cross-modal-eval panel) ───────────────────────
|
||||
'together:meta-llama/Llama-3.3-70B-Instruct-Turbo': { input: 0.88, output: 0.88 },
|
||||
'deepseek:deepseek-chat': { input: 0.14, output: 0.28 },
|
||||
};
|
||||
|
||||
/**
|
||||
* Resolve a model id to its canonical pricing, or `undefined` on miss.
|
||||
*
|
||||
* Accepts bare (`claude-opus-4-8`), colon (`anthropic:claude-opus-4-8`), and
|
||||
* slash (`anthropic/claude-opus-4-8`) forms. Bare ids default to the
|
||||
* `anthropic:` provider (matching the historical bare-key Anthropic tables);
|
||||
* non-Anthropic bare ids therefore miss, preserving the prior null-return
|
||||
* contract for ids like `gpt-5`.
|
||||
*
|
||||
* Nested OpenRouter ids (`openrouter:anthropic/claude-...`) intentionally MISS:
|
||||
* splitProviderModelId yields provider `openrouter`, model
|
||||
* `anthropic/claude-...`, and `openrouter:anthropic/claude-...` is not a
|
||||
* canonical key. OpenRouter markup ≠ native pricing, so we never reprice it as
|
||||
* the inner vendor.
|
||||
*/
|
||||
export function canonicalLookup(
|
||||
modelId: string | null | undefined,
|
||||
): ModelPricing | undefined {
|
||||
if (!modelId) return undefined;
|
||||
// 1. Exact key — colon form, already-canonical ids, and slash-bearing model
|
||||
// tails carried verbatim as keys (e.g. together:.../Llama-3.3-70B-...).
|
||||
const direct = CANONICAL_PRICING[modelId];
|
||||
if (direct) return direct;
|
||||
// 2. Normalize bare/slash via the shared splitter (colon-first precedence).
|
||||
const { provider, model } = splitProviderModelId(modelId);
|
||||
if (!model) return undefined;
|
||||
const key = provider ? `${provider}:${model}` : `anthropic:${model}`;
|
||||
return CANONICAL_PRICING[key];
|
||||
}
|
||||
@@ -20,12 +20,13 @@
|
||||
* - epochs × 1 slow-update reflect call (if no improvement that epoch)
|
||||
* - 1× final test eval on D_test
|
||||
*
|
||||
* Prices come from existing per-model pricing tables (Anthropic +
|
||||
* embedding-pricing). For unknown providers we fail-loud — same posture
|
||||
* as BudgetTracker's TX2 contract.
|
||||
* Prices come from the canonical pricing table (model-pricing.ts) via
|
||||
* canonicalLookup. For unknown providers `lookupPrice` returns a warn-only
|
||||
* Sonnet-tier fallback (preflight estimates, never gates) — the actual
|
||||
* fail-loud gate is BudgetTracker's TX2 contract at run time.
|
||||
*/
|
||||
|
||||
import { ANTHROPIC_PRICING } from '../anthropic-pricing.ts';
|
||||
import { canonicalLookup } from '../model-pricing.ts';
|
||||
import { VALIDATION_RUNS_PER_TASK } from './types.ts';
|
||||
|
||||
/** Conservative per-rollout token estimates (input + output). */
|
||||
@@ -188,10 +189,10 @@ export function preflight(opts: PreflightOpts): PreflightResult {
|
||||
}
|
||||
|
||||
function lookupPrice(model: string): { input: number; output: number } {
|
||||
// Anthropic models — strip provider prefix.
|
||||
const bare = model.startsWith('anthropic:') ? model.slice('anthropic:'.length) : model;
|
||||
const anth = (ANTHROPIC_PRICING as Record<string, { input: number; output: number }>)[bare];
|
||||
if (anth) return anth;
|
||||
// Canonical lookup handles bare/colon/slash forms (fixes this site's prior
|
||||
// bare-only limitation, which mispriced `anthropic/...` slash ids).
|
||||
const p = canonicalLookup(model);
|
||||
if (p) return p;
|
||||
// Conservative fallback: assume Sonnet-tier pricing for unknown providers.
|
||||
// Don't throw — preflight is for warning, not gating. The actual budget
|
||||
// tracker (BudgetTracker TX2) will fail-loud at run time if pricing is
|
||||
|
||||
@@ -1,20 +1,24 @@
|
||||
/**
|
||||
* takes-quality-eval/pricing — fail-closed model pricing table for budget
|
||||
* takes-quality-eval/pricing — fail-closed model pricing for budget
|
||||
* enforcement.
|
||||
*
|
||||
* Per-1M-token rates in USD. Drifts as providers update prices; refresh
|
||||
* alongside model-family bumps. The list is intentionally small — only
|
||||
* the default 3-model panel and a handful of likely overrides. If you
|
||||
* pass a model not in this table to `eval takes-quality run --budget-usd N`,
|
||||
* the runner aborts with an actionable error rather than guessing
|
||||
* (codex review #4 fail-closed posture vs cross-modal-eval/runner.ts
|
||||
* which silently estimates zero on unknown models).
|
||||
* The KEY SET here is an intentional allowlist — only the default panel and a
|
||||
* handful of likely overrides. Passing a model NOT in this list to
|
||||
* `eval takes-quality run --budget-usd N` aborts with an actionable error
|
||||
* rather than guessing (codex review #4 fail-closed posture vs
|
||||
* cross-modal-eval/runner.ts which silently estimates zero on unknown models).
|
||||
*
|
||||
* Schema is `{model_id: {input_per_1m, output_per_1m}}` so callers can
|
||||
* compute estimated cost as
|
||||
* The VALUES come from the canonical table (`src/core/model-pricing.ts`) — do
|
||||
* NOT hand-edit rates here; update canonical and they flow through. This keeps
|
||||
* the allowlist's fail-closed curation while removing the duplicated numbers
|
||||
* that let Opus 4.7 drift to a stale $15/$75 here.
|
||||
*
|
||||
* Schema is `{model_id: {input_per_1m, output_per_1m}}` so callers can compute
|
||||
* (in_tokens * input_per_1m + out_tokens * output_per_1m) / 1_000_000.
|
||||
*/
|
||||
|
||||
import { canonicalLookup } from '../model-pricing.ts';
|
||||
|
||||
export interface ModelPricing {
|
||||
/** USD per 1M input tokens. */
|
||||
input_per_1m: number;
|
||||
@@ -22,28 +26,44 @@ export interface ModelPricing {
|
||||
output_per_1m: number;
|
||||
}
|
||||
|
||||
export const MODEL_PRICING: Record<string, ModelPricing> = {
|
||||
// OpenAI (refreshed 2026-05; verify before relying for budget gating)
|
||||
'openai:gpt-4o': { input_per_1m: 2.5, output_per_1m: 10.0 },
|
||||
'openai:gpt-5': { input_per_1m: 5.0, output_per_1m: 20.0 },
|
||||
'openai:gpt-5.5': { input_per_1m: 4.0, output_per_1m: 16.0 },
|
||||
/**
|
||||
* The curated allowlist of models takes-quality will budget-gate. Each must
|
||||
* exist in CANONICAL_PRICING; the map below fails fast at module load if one
|
||||
* is missing (a programmer error caught immediately, not at run time).
|
||||
*/
|
||||
const SUPPORTED_MODELS = [
|
||||
'openai:gpt-4o',
|
||||
'openai:gpt-5',
|
||||
'openai:gpt-5.5',
|
||||
'anthropic:claude-opus-4-8',
|
||||
'anthropic:claude-opus-4-7',
|
||||
'anthropic:claude-sonnet-4-6',
|
||||
'anthropic:claude-haiku-4-5',
|
||||
'google:gemini-1.5-pro',
|
||||
'google:gemini-2-flash',
|
||||
] as const;
|
||||
|
||||
// Anthropic
|
||||
'anthropic:claude-opus-4-7': { input_per_1m: 15.0, output_per_1m: 75.0 },
|
||||
'anthropic:claude-sonnet-4-6': { input_per_1m: 3.0, output_per_1m: 15.0 },
|
||||
'anthropic:claude-haiku-4-5': { input_per_1m: 0.8, output_per_1m: 4.0 },
|
||||
|
||||
// Google
|
||||
'google:gemini-1.5-pro': { input_per_1m: 1.25, output_per_1m: 5.0 },
|
||||
'google:gemini-2-flash': { input_per_1m: 0.30, output_per_1m: 1.20 },
|
||||
};
|
||||
export const MODEL_PRICING: Record<string, ModelPricing> = Object.fromEntries(
|
||||
SUPPORTED_MODELS.map((id) => {
|
||||
const p = canonicalLookup(id);
|
||||
if (!p) {
|
||||
throw new Error(
|
||||
`takes-quality allowlist model "${id}" is missing from CANONICAL_PRICING ` +
|
||||
`(src/core/model-pricing.ts). Add it there.`,
|
||||
);
|
||||
}
|
||||
return [id, { input_per_1m: p.input, output_per_1m: p.output }];
|
||||
}),
|
||||
);
|
||||
|
||||
export class PricingNotFoundError extends Error {
|
||||
constructor(public readonly modelId: string) {
|
||||
super(
|
||||
`Model "${modelId}" has no pricing entry in src/core/takes-quality-eval/pricing.ts. ` +
|
||||
`Add an entry for the model and re-run, OR pass --budget-usd 0 to disable budget ` +
|
||||
`enforcement (you'll still see the cost printed to stderr but the runner won't abort).`,
|
||||
`Model "${modelId}" has no pricing entry. Add it to CANONICAL_PRICING in ` +
|
||||
`src/core/model-pricing.ts AND to the SUPPORTED_MODELS allowlist in ` +
|
||||
`src/core/takes-quality-eval/pricing.ts, OR pass --budget-usd 0 to disable ` +
|
||||
`budget enforcement (you'll still see the cost printed to stderr but the ` +
|
||||
`runner won't abort).`,
|
||||
);
|
||||
this.name = 'PricingNotFoundError';
|
||||
}
|
||||
|
||||
@@ -41,6 +41,14 @@ describe('estimateMaxCostUsd', () => {
|
||||
expect(cost).toBeCloseTo(1.75, 5);
|
||||
});
|
||||
|
||||
test('opus 4.8 priced same as 4.7 ($5/$25) — closes #1819', () => {
|
||||
// 100K in + 50K out = 0.1*5 + 0.05*25 = 0.5 + 1.25 = 1.75. Pre-fix this
|
||||
// model was absent from the pricing table → estimateMaxCostUsd returned
|
||||
// null and the dream-cycle budget gate silently no-op'd on 4.8 runs.
|
||||
const cost = estimateMaxCostUsd('anthropic:claude-opus-4-8', 100_000, 50_000);
|
||||
expect(cost).toBeCloseTo(1.75, 5);
|
||||
});
|
||||
|
||||
test('unknown model → returns null (caller warn-once + bypass)', () => {
|
||||
expect(estimateMaxCostUsd('mistral:medium', 1_000, 1_000)).toBeNull();
|
||||
expect(estimateMaxCostUsd('gpt-5', 1_000, 1_000)).toBeNull();
|
||||
|
||||
@@ -20,6 +20,18 @@ describe('getPricing — fail-closed contract', () => {
|
||||
expect(() => getPricing('unknown:gpt-99')).toThrow(PricingNotFoundError);
|
||||
});
|
||||
|
||||
test('opus 4.7 priced at $5/$25 (regression: was a stale $15/$75) — gbrain#1819', () => {
|
||||
const p = getPricing('anthropic:claude-opus-4-7');
|
||||
expect(p.input_per_1m).toBeCloseTo(5.0, 5);
|
||||
expect(p.output_per_1m).toBeCloseTo(25.0, 5);
|
||||
});
|
||||
|
||||
test('opus 4.8 is supported and priced $5/$25 — gbrain#1819', () => {
|
||||
const p = getPricing('anthropic:claude-opus-4-8');
|
||||
expect(p.input_per_1m).toBeCloseTo(5.0, 5);
|
||||
expect(p.output_per_1m).toBeCloseTo(25.0, 5);
|
||||
});
|
||||
|
||||
test('error message names the model AND points to the file', () => {
|
||||
try {
|
||||
getPricing('foo:bar');
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
/**
|
||||
* model-pricing — canonical table + canonicalLookup + the drift guard.
|
||||
*
|
||||
* Because the other pricing tables are DERIVED from CANONICAL_PRICING (not
|
||||
* hand-copied), cross-table price drift — the kind that left Opus 4.7 at
|
||||
* $15/$75 in takes-quality-eval while anthropic-pricing.ts had $5/$25
|
||||
* (gbrain#1819) — is structurally impossible. The "drift guard" below is a
|
||||
* regression trip-wire: it asserts each derived view still equals canonical, so
|
||||
* if anyone later re-hardcodes a view back into a duplicate, CI catches it. The
|
||||
* cross-modal panel check is genuinely load-bearing — it asserts canonical
|
||||
* actually carries every model the runner prices.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import {
|
||||
CANONICAL_PRICING,
|
||||
canonicalLookup,
|
||||
} from '../src/core/model-pricing.ts';
|
||||
import { ANTHROPIC_PRICING } from '../src/core/anthropic-pricing.ts';
|
||||
import { MODEL_PRICING } from '../src/core/takes-quality-eval/pricing.ts';
|
||||
import { estimateAnthropicCost } from '../src/core/brain-score-recommendations.ts';
|
||||
|
||||
describe('CANONICAL_PRICING — table integrity', () => {
|
||||
test('every entry has finite positive rates and a provider-prefixed key', () => {
|
||||
for (const [key, p] of Object.entries(CANONICAL_PRICING)) {
|
||||
expect(Number.isFinite(p.input)).toBe(true);
|
||||
expect(Number.isFinite(p.output)).toBe(true);
|
||||
expect(p.input).toBeGreaterThan(0);
|
||||
expect(p.output).toBeGreaterThan(0);
|
||||
// Provider-prefixed key (sanity guard against a bare key sneaking in).
|
||||
// NOTE: deliberately NO output>=input invariant — symmetric pricing is
|
||||
// legitimate (e.g. together:...Llama-3.3 is 0.88/0.88).
|
||||
expect(key).toContain(':');
|
||||
}
|
||||
});
|
||||
|
||||
test('Opus 4.8 present at $5/$25 (closes gbrain#1819)', () => {
|
||||
expect(CANONICAL_PRICING['anthropic:claude-opus-4-8']).toEqual({ input: 5.0, output: 25.0 });
|
||||
});
|
||||
|
||||
test('Opus 4.7 at $5/$25 (not the stale $15/$75)', () => {
|
||||
expect(CANONICAL_PRICING['anthropic:claude-opus-4-7']).toEqual({ input: 5.0, output: 25.0 });
|
||||
});
|
||||
|
||||
test('Gemini 2.0 Flash reconciled to $0.10/$0.40; legacy alias agrees', () => {
|
||||
expect(CANONICAL_PRICING['google:gemini-2.0-flash']).toEqual({ input: 0.1, output: 0.4 });
|
||||
expect(CANONICAL_PRICING['google:gemini-2-flash']).toEqual(
|
||||
CANONICAL_PRICING['google:gemini-2.0-flash'],
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('canonicalLookup — id normalization', () => {
|
||||
test('bare anthropic id → hit (defaults to anthropic provider)', () => {
|
||||
expect(canonicalLookup('claude-opus-4-8')).toEqual({ input: 5.0, output: 25.0 });
|
||||
});
|
||||
|
||||
test('colon form → hit', () => {
|
||||
expect(canonicalLookup('anthropic:claude-opus-4-8')).toEqual({ input: 5.0, output: 25.0 });
|
||||
});
|
||||
|
||||
test('slash form → hit', () => {
|
||||
expect(canonicalLookup('anthropic/claude-opus-4-8')).toEqual({ input: 5.0, output: 25.0 });
|
||||
});
|
||||
|
||||
test('non-anthropic bare id → miss (preserves prior null contract)', () => {
|
||||
expect(canonicalLookup('gpt-5')).toBeUndefined();
|
||||
});
|
||||
|
||||
test('nested OpenRouter id → MISS (markup ≠ native pricing)', () => {
|
||||
expect(canonicalLookup('openrouter:anthropic/claude-sonnet-4-6')).toBeUndefined();
|
||||
});
|
||||
|
||||
test('slash-bearing model tail kept as exact key (together Llama)', () => {
|
||||
expect(canonicalLookup('together:meta-llama/Llama-3.3-70B-Instruct-Turbo')).toEqual({
|
||||
input: 0.88,
|
||||
output: 0.88,
|
||||
});
|
||||
});
|
||||
|
||||
test('null / empty → undefined (no throw)', () => {
|
||||
expect(canonicalLookup(null)).toBeUndefined();
|
||||
expect(canonicalLookup(undefined)).toBeUndefined();
|
||||
expect(canonicalLookup('')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('DRIFT GUARD — derived views stay equal to canonical (re-hardcode trip-wire)', () => {
|
||||
test('ANTHROPIC_PRICING (bare) equals canonical anthropic: entries', () => {
|
||||
for (const [key, p] of Object.entries(CANONICAL_PRICING)) {
|
||||
if (!key.startsWith('anthropic:')) continue;
|
||||
const bare = key.slice('anthropic:'.length);
|
||||
expect(ANTHROPIC_PRICING[bare]).toEqual(p);
|
||||
}
|
||||
});
|
||||
|
||||
test('takes-quality MODEL_PRICING equals canonical for every allowlisted key', () => {
|
||||
for (const [key, p] of Object.entries(MODEL_PRICING)) {
|
||||
const c = canonicalLookup(key);
|
||||
expect(c).toBeDefined();
|
||||
expect(p.input_per_1m).toBe(c!.input);
|
||||
expect(p.output_per_1m).toBe(c!.output);
|
||||
}
|
||||
});
|
||||
|
||||
test('cross-modal panel models are all priced from canonical', () => {
|
||||
// The runner now calls canonicalLookup(slot.model) directly, so presence
|
||||
// here = the runner prices these. Mirrors the panel it used to inline.
|
||||
for (const id of [
|
||||
'openai:gpt-4o',
|
||||
'openai:gpt-4o-mini',
|
||||
'anthropic:claude-opus-4-7',
|
||||
'anthropic:claude-sonnet-4-6',
|
||||
'google:gemini-1.5-pro',
|
||||
'google:gemini-2.0-flash',
|
||||
'together:meta-llama/Llama-3.3-70B-Instruct-Turbo',
|
||||
'deepseek:deepseek-chat',
|
||||
]) {
|
||||
expect(canonicalLookup(id)).toBeDefined();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('S1A — raw-index consumers price provider-prefixed ids', () => {
|
||||
test('estimateAnthropicCost prices anthropic:claude-opus-4-8 (was zero pre-fix)', () => {
|
||||
// 1 call, 1M in, 1M out → 1*5 + 1*25 = $30. Pre-fix the bare-key index
|
||||
// missed on the provider-prefixed id and returned 0.
|
||||
const cost = estimateAnthropicCost('anthropic:claude-opus-4-8', 1, 1_000_000, 1_000_000);
|
||||
expect(cost).toBeCloseTo(30.0, 2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('no heavy import (cycle guard)', () => {
|
||||
test('model-pricing.ts imports only model-id.ts (relative)', () => {
|
||||
const src = readFileSync(
|
||||
fileURLToPath(new URL('../src/core/model-pricing.ts', import.meta.url)),
|
||||
'utf8',
|
||||
);
|
||||
const relImports = [...src.matchAll(/^\s*import\s.*from\s+['"](\.[^'"]+)['"]/gm)].map(
|
||||
(m) => m[1],
|
||||
);
|
||||
expect(relImports).toEqual(['./model-id.ts']);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user