Files
gbrain/test/budget-meter.test.ts
T
29961811a4 v0.31.12 fix: canonical Anthropic model IDs + tier routing surface + gbrain models CLI (#844)
* fix: canonical Anthropic model IDs + reverse alias + Opus 4.7 pricing

Replace claude-sonnet-4-6-20250929 with claude-sonnet-4-6 everywhere it
appears as a model ID. Starting with Claude 4.6, Anthropic API IDs are
dateless and pinned — the date suffix was carried forward from Sonnet 4.5
by mistake, producing a phantom ID that 404'd on every call.

Production impact in v0.31.6: isAvailable("chat") returned false in every
code path that loaded the recipe's model list, and extractFactsFromTurn
silently returned []. The headline real-time facts extraction feature
was a no-op on the happy path.

- gateway.ts:46 DEFAULT_CHAT_MODEL -> anthropic:claude-sonnet-4-6
- recipes/anthropic.ts: chat + expansion model lists drop date suffix;
  remove wrong-direction alias (claude-sonnet-4-6 -> -20250929);
  add reverse alias (-20250929 -> claude-sonnet-4-6) so stale user
  configs in models.dream.synthesize etc. keep working
- facts/extract.ts: routes through resolveModel; both fallbacks corrected
- anthropic-pricing.ts: Opus 4.7 corrected $15/$75 -> $5/$25 per
  Anthropic docs (the $15/$75 was Opus 4.0 pricing)
- cross-modal-eval/runner.ts: PRICING now reads from ANTHROPIC_PRICING
  for Anthropic models instead of duplicating the map (single source of
  truth — fixes the drift trap that motivated this whole patch)

Tests: cherry-pick PR #830's test/anthropic-model-ids.test.ts verbatim
(6 recipe-shape guardrails). Update gateway-chat tests to assert reverse
alias resolves correctly. Update budget-meter test for new Opus pricing.

Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: model tier system + recipe-models merge + async reconfigure hook

Add 4-tier model routing (utility/reasoning/deep/subagent) so users can
swap defaults with one config key. Each tier maps to a class of work;
override globally via models.default or per-tier via models.tier.<tier>.

Codex flagged three real architecture issues in the v0.31.12 plan review;
this commit addresses each.

F3 — sync/async timing of configureGateway:
  - buildGatewayConfig stays synchronous (pre-engine-connect callers
    keep working)
  - New reconfigureGatewayWithEngine(engine) async function re-resolves
    expansion + chat defaults through resolveModel after engine.connect()
  - cli.ts wires the re-stamp into the post-connect path

F4/F5 — softening assertTouchpoint was too broad:
  - Earlier plan was to flip native-recipe validation from throw to warn,
    affecting gateway.chat AND gateway.expand AND gateway.embed
  - Instead: per-gateway-instance recipe-models merge. assertTouchpoint
    gets an optional extendedModels Set; when the user opted into a model
    via config, it bypasses the throw. Source-code typos still fail fast.
  - Existing contract test (test/ai/gateway-chat.test.ts:106) preserved

Tier defaults are TIER_DEFAULTS in model-config.ts. Resolution chain
inserts at step 5 (between models.default and env var). Each existing
resolveModel call site gains a tier: arg — think (deep), cycle/synthesize
(reasoning + utility for verdict), patterns/drift (reasoning), auto-think
(deep), facts/extract (reasoning).

Plus 10 new tests pinning tier precedence, subagent-tier fallback when
models.default is non-Anthropic, and the F6 alias-chain conflict case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: subagent runtime enforcement for non-Anthropic models (3 layers)

The subagent loop uses Anthropic's Messages API with prompt caching on
system + tools. OpenAI/Google have different shapes. Setting
models.default = openai:gpt-5.5 and routing the subagent there silently
breaks the loop.

Codex F1+F2+F13 in the v0.31.12 plan review pointed out that "warn at
doctor" wasn't enough — handlers/subagent.ts:148 still did
`const model = data.model ?? DEFAULT_MODEL` and called Anthropic directly,
so a job submitted with data.model = openai:gpt-5.5 bypassed any tier
logic and failed at runtime with a confusing provider error.

Three layers of enforcement, defense in depth:

Layer 1 (queue.ts:add) — submit-time guard. When name === 'subagent'
and data.model is set, validate the provider. Non-Anthropic rejects
before the job enters the queue.

Layer 2 (handlers/subagent.ts) — tier-resolution fallback. The handler
routes through resolveModel({ tier: 'subagent' }). If the chain resolves
to a non-Anthropic provider (via models.default or models.tier.subagent),
the resolver warns + falls back to TIER_DEFAULTS.subagent
(claude-sonnet-4-6).

Layer 3 (doctor.ts:checkSubagentProvider) — surfacing layer. Warns when
models.tier.subagent or models.default is explicitly set to a
non-Anthropic provider, with a paste-ready fix command. Lets users see
config drift before submitting a job.

Tests: 3 new cases in test/agent-cli.test.ts asserting the queue-level
guard rejects non-Anthropic data.model. Existing test/subagent-handler
suite still passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: gbrain models CLI + doctor probe + silent-no-op regression test

New gbrain models CLI gives the agent and user visibility into routing.
Read mode prints the tier table, current overrides, per-task config,
and aliases with source-of-truth attribution per row. Doctor subcommand
fires a 1-token probe to each configured chat/expansion model and
classifies failures (model_not_found / auth / rate_limit / network /
unknown) so config-time invalid IDs surface without waiting for a
production call that silently degrades.

Per Codex F11 — no specific dollar cost claim in either the help text
or the CHANGELOG (providers have minimum-output billing and prompt-cache
rounding that vary). Probe is opt-in (gbrain doctor --probe-models),
never auto-runs. --skip=<provider> narrows the matrix for cost-sensitive
operators.

Per Codex F7+F8+F15 (the structural regression gap): new
test/facts-extract-silent-no-op.test.ts is THE regression test for the
bug class that motivated v0.31.12. Five cases including the smoking-gun:
when chat IS available, extractFactsFromTurn MUST actually call the chat
transport, not silently return []. Uses the gateway's
__setChatTransportForTests seam so it runs in every shard with no API key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.31.12)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: document v0.31.12 model tier system + gbrain models CLI

Add CLAUDE.md Key Files annotations for the v0.31.12 work:
src/core/model-config.ts (tier system + isAnthropicProvider + TIER_DEFAULTS),
src/core/ai/model-resolver.ts (assertTouchpoint extendedModels arg),
src/core/ai/gateway.ts (reconfigureGatewayWithEngine + extended-models registry),
src/core/minions/queue.ts (subagent submit-time guard, layer 1 of 3),
src/commands/models.ts (new gbrain models CLI + doctor probe),
src/commands/doctor.ts (subagent_provider check, layer 3 of 3),
src/core/ai/recipes/anthropic.ts (canonical model IDs + reverse alias),
src/core/anthropic-pricing.ts (Opus 4.7 corrected to \$5/\$25).

Add CLAUDE.md commands section for gbrain models + gbrain models doctor
+ power-user config recipes. Add README.md command-table rows for the
same. Regenerate llms-full.txt so the bundled docs stay in sync.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: scrub --probe-models reference (flag not actually wired)

The v0.31.12 CHANGELOG and skills/conventions/model-routing.md both
referenced `gbrain doctor --probe-models` as an integrated probe entry
point. The flag was never implemented — only `gbrain models doctor`
landed as the probe surface. Caught by /document-release subagent.

Drop the references rather than wire an untested flag at the last minute.
The probe is reachable via `gbrain models doctor`; users who want it
in doctor's output run that command separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 20:06:31 -07:00

82 lines
4.1 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { describe, test, expect, beforeEach } from 'bun:test';
import { mkdtempSync, rmSync, readFileSync, existsSync } from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { BudgetMeter, _resetBudgetMeterWarningsForTest, ANTHROPIC_PRICING } from '../src/core/cycle/budget-meter.ts';
let tmpDir: string;
let auditPath: string;
beforeEach(() => {
tmpDir = mkdtempSync(join(tmpdir(), 'budget-meter-'));
auditPath = join(tmpDir, 'budget.jsonl');
_resetBudgetMeterWarningsForTest();
});
function readLedger(): Array<Record<string, unknown>> {
if (!existsSync(auditPath)) return [];
return readFileSync(auditPath, 'utf-8').trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
}
describe('BudgetMeter', () => {
test('Anthropic pricing map covers the alias resolution targets', () => {
expect(ANTHROPIC_PRICING['claude-opus-4-7']).toBeDefined();
expect(ANTHROPIC_PRICING['claude-sonnet-4-6']).toBeDefined();
expect(ANTHROPIC_PRICING['claude-haiku-4-5-20251001']).toBeDefined();
});
test('first submit is allowed when within budget', () => {
const meter = new BudgetMeter({ budgetUsd: 1.0, phase: 'auto_think', auditPath });
const r = meter.check({ modelId: 'claude-haiku-4-5-20251001', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'test' });
expect(r.allowed).toBe(true);
expect(r.estimatedCostUsd).toBeGreaterThan(0);
expect(r.cumulativeCostUsd).toBe(r.estimatedCostUsd);
});
test('cumulative cost denies the second submit when budget exhausted', () => {
const meter = new BudgetMeter({ budgetUsd: 0.50, phase: 'auto_think', auditPath });
// Opus 4.7: $5 in / $25 out per 1M. Per call: 5000×5/1M + 10000×25/1M = $0.025 + $0.25 = $0.275
const big = { modelId: 'claude-opus-4-7', estimatedInputTokens: 5000, maxOutputTokens: 10000, label: 'big' };
const r1 = meter.check(big); // $0.275 cumulative — allowed
const r2 = meter.check(big); // $0.55 cumulative — exceeds $0.50 → DENY
expect(r1.allowed).toBe(true);
expect(r2.allowed).toBe(false);
expect(r2.reason).toContain('BUDGET_EXHAUSTED');
});
test('budget=0 disables the gate (cycle runs unbounded)', () => {
const meter = new BudgetMeter({ budgetUsd: 0, phase: 'drift', auditPath });
const r = meter.check({ modelId: 'claude-opus-4-7', estimatedInputTokens: 100_000, maxOutputTokens: 100_000, label: 'huge' });
expect(r.allowed).toBe(true);
});
test('non-Anthropic model bypasses gate with warn-once + ledger entry', () => {
const meter = new BudgetMeter({ budgetUsd: 0.001, phase: 'auto_think', auditPath });
const r1 = meter.check({ modelId: 'gemini-3-pro', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'gem1' });
const r2 = meter.check({ modelId: 'gemini-3-pro', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'gem2' });
expect(r1.allowed).toBe(true);
expect(r1.unpriced).toBe(true);
expect(r2.allowed).toBe(true);
expect(meter.unpricedSubmits).toBe(2);
});
test('ledger captures every submit (allowed + denied + unpriced)', () => {
const meter = new BudgetMeter({ budgetUsd: 0.001, phase: 'auto_think', auditPath });
meter.check({ modelId: 'claude-opus-4-7', estimatedInputTokens: 5000, maxOutputTokens: 4000, label: 'a' });
meter.check({ modelId: 'claude-opus-4-7', estimatedInputTokens: 5000, maxOutputTokens: 4000, label: 'b-denied' });
meter.check({ modelId: 'gpt-5', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'c-unpriced' });
const lines = readLedger();
expect(lines).toHaveLength(3);
expect(lines[0].event).toBe('submit_denied'); // first opus call exceeds the $0.001 cap
expect(lines[1].event).toBe('submit_denied');
expect(lines[2].event).toBe('submit_unpriced');
});
test('ledger uses ISO-week filename when auditPath not overridden', () => {
// Implicit path branch — just verify it doesn't throw and writes somewhere reasonable.
const meter = new BudgetMeter({ budgetUsd: 1.0, phase: 'drift' });
const r = meter.check({ modelId: 'claude-haiku-4-5-20251001', estimatedInputTokens: 100, maxOutputTokens: 100, label: 'wk' });
expect(r.allowed).toBe(true);
});
});