mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* fix: canonical Anthropic model IDs + reverse alias + Opus 4.7 pricing
Replace claude-sonnet-4-6-20250929 with claude-sonnet-4-6 everywhere it
appears as a model ID. Starting with Claude 4.6, Anthropic API IDs are
dateless and pinned — the date suffix was carried forward from Sonnet 4.5
by mistake, producing a phantom ID that 404'd on every call.
Production impact in v0.31.6: isAvailable("chat") returned false in every
code path that loaded the recipe's model list, and extractFactsFromTurn
silently returned []. The headline real-time facts extraction feature
was a no-op on the happy path.
- gateway.ts:46 DEFAULT_CHAT_MODEL -> anthropic:claude-sonnet-4-6
- recipes/anthropic.ts: chat + expansion model lists drop date suffix;
remove wrong-direction alias (claude-sonnet-4-6 -> -20250929);
add reverse alias (-20250929 -> claude-sonnet-4-6) so stale user
configs in models.dream.synthesize etc. keep working
- facts/extract.ts: routes through resolveModel; both fallbacks corrected
- anthropic-pricing.ts: Opus 4.7 corrected $15/$75 -> $5/$25 per
Anthropic docs (the $15/$75 was Opus 4.0 pricing)
- cross-modal-eval/runner.ts: PRICING now reads from ANTHROPIC_PRICING
for Anthropic models instead of duplicating the map (single source of
truth — fixes the drift trap that motivated this whole patch)
Tests: cherry-pick PR #830's test/anthropic-model-ids.test.ts verbatim
(6 recipe-shape guardrails). Update gateway-chat tests to assert reverse
alias resolves correctly. Update budget-meter test for new Opus pricing.
Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: model tier system + recipe-models merge + async reconfigure hook
Add 4-tier model routing (utility/reasoning/deep/subagent) so users can
swap defaults with one config key. Each tier maps to a class of work;
override globally via models.default or per-tier via models.tier.<tier>.
Codex flagged three real architecture issues in the v0.31.12 plan review;
this commit addresses each.
F3 — sync/async timing of configureGateway:
- buildGatewayConfig stays synchronous (pre-engine-connect callers
keep working)
- New reconfigureGatewayWithEngine(engine) async function re-resolves
expansion + chat defaults through resolveModel after engine.connect()
- cli.ts wires the re-stamp into the post-connect path
F4/F5 — softening assertTouchpoint was too broad:
- Earlier plan was to flip native-recipe validation from throw to warn,
affecting gateway.chat AND gateway.expand AND gateway.embed
- Instead: per-gateway-instance recipe-models merge. assertTouchpoint
gets an optional extendedModels Set; when the user opted into a model
via config, it bypasses the throw. Source-code typos still fail fast.
- Existing contract test (test/ai/gateway-chat.test.ts:106) preserved
Tier defaults are TIER_DEFAULTS in model-config.ts. Resolution chain
inserts at step 5 (between models.default and env var). Each existing
resolveModel call site gains a tier: arg — think (deep), cycle/synthesize
(reasoning + utility for verdict), patterns/drift (reasoning), auto-think
(deep), facts/extract (reasoning).
Plus 10 new tests pinning tier precedence, subagent-tier fallback when
models.default is non-Anthropic, and the F6 alias-chain conflict case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: subagent runtime enforcement for non-Anthropic models (3 layers)
The subagent loop uses Anthropic's Messages API with prompt caching on
system + tools. OpenAI/Google have different shapes. Setting
models.default = openai:gpt-5.5 and routing the subagent there silently
breaks the loop.
Codex F1+F2+F13 in the v0.31.12 plan review pointed out that "warn at
doctor" wasn't enough — handlers/subagent.ts:148 still did
`const model = data.model ?? DEFAULT_MODEL` and called Anthropic directly,
so a job submitted with data.model = openai:gpt-5.5 bypassed any tier
logic and failed at runtime with a confusing provider error.
Three layers of enforcement, defense in depth:
Layer 1 (queue.ts:add) — submit-time guard. When name === 'subagent'
and data.model is set, validate the provider. Non-Anthropic rejects
before the job enters the queue.
Layer 2 (handlers/subagent.ts) — tier-resolution fallback. The handler
routes through resolveModel({ tier: 'subagent' }). If the chain resolves
to a non-Anthropic provider (via models.default or models.tier.subagent),
the resolver warns + falls back to TIER_DEFAULTS.subagent
(claude-sonnet-4-6).
Layer 3 (doctor.ts:checkSubagentProvider) — surfacing layer. Warns when
models.tier.subagent or models.default is explicitly set to a
non-Anthropic provider, with a paste-ready fix command. Lets users see
config drift before submitting a job.
Tests: 3 new cases in test/agent-cli.test.ts asserting the queue-level
guard rejects non-Anthropic data.model. Existing test/subagent-handler
suite still passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: gbrain models CLI + doctor probe + silent-no-op regression test
New gbrain models CLI gives the agent and user visibility into routing.
Read mode prints the tier table, current overrides, per-task config,
and aliases with source-of-truth attribution per row. Doctor subcommand
fires a 1-token probe to each configured chat/expansion model and
classifies failures (model_not_found / auth / rate_limit / network /
unknown) so config-time invalid IDs surface without waiting for a
production call that silently degrades.
Per Codex F11 — no specific dollar cost claim in either the help text
or the CHANGELOG (providers have minimum-output billing and prompt-cache
rounding that vary). Probe is opt-in (gbrain doctor --probe-models),
never auto-runs. --skip=<provider> narrows the matrix for cost-sensitive
operators.
Per Codex F7+F8+F15 (the structural regression gap): new
test/facts-extract-silent-no-op.test.ts is THE regression test for the
bug class that motivated v0.31.12. Five cases including the smoking-gun:
when chat IS available, extractFactsFromTurn MUST actually call the chat
transport, not silently return []. Uses the gateway's
__setChatTransportForTests seam so it runs in every shard with no API key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.31.12)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: document v0.31.12 model tier system + gbrain models CLI
Add CLAUDE.md Key Files annotations for the v0.31.12 work:
src/core/model-config.ts (tier system + isAnthropicProvider + TIER_DEFAULTS),
src/core/ai/model-resolver.ts (assertTouchpoint extendedModels arg),
src/core/ai/gateway.ts (reconfigureGatewayWithEngine + extended-models registry),
src/core/minions/queue.ts (subagent submit-time guard, layer 1 of 3),
src/commands/models.ts (new gbrain models CLI + doctor probe),
src/commands/doctor.ts (subagent_provider check, layer 3 of 3),
src/core/ai/recipes/anthropic.ts (canonical model IDs + reverse alias),
src/core/anthropic-pricing.ts (Opus 4.7 corrected to \$5/\$25).
Add CLAUDE.md commands section for gbrain models + gbrain models doctor
+ power-user config recipes. Add README.md command-table rows for the
same. Regenerate llms-full.txt so the bundled docs stay in sync.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: scrub --probe-models reference (flag not actually wired)
The v0.31.12 CHANGELOG and skills/conventions/model-routing.md both
referenced `gbrain doctor --probe-models` as an integrated probe entry
point. The flag was never implemented — only `gbrain models doctor`
landed as the probe surface. Caught by /document-release subagent.
Drop the references rather than wire an untested flag at the last minute.
The probe is reachable via `gbrain models doctor`; users who want it
in doctor's output run that command separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
82 lines
4.1 KiB
TypeScript
82 lines
4.1 KiB
TypeScript
import { describe, test, expect, beforeEach } from 'bun:test';
|
||
import { mkdtempSync, rmSync, readFileSync, existsSync } from 'node:fs';
|
||
import { join } from 'node:path';
|
||
import { tmpdir } from 'node:os';
|
||
import { BudgetMeter, _resetBudgetMeterWarningsForTest, ANTHROPIC_PRICING } from '../src/core/cycle/budget-meter.ts';
|
||
|
||
let tmpDir: string;
|
||
let auditPath: string;
|
||
|
||
beforeEach(() => {
|
||
tmpDir = mkdtempSync(join(tmpdir(), 'budget-meter-'));
|
||
auditPath = join(tmpDir, 'budget.jsonl');
|
||
_resetBudgetMeterWarningsForTest();
|
||
});
|
||
|
||
function readLedger(): Array<Record<string, unknown>> {
|
||
if (!existsSync(auditPath)) return [];
|
||
return readFileSync(auditPath, 'utf-8').trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
|
||
}
|
||
|
||
describe('BudgetMeter', () => {
|
||
test('Anthropic pricing map covers the alias resolution targets', () => {
|
||
expect(ANTHROPIC_PRICING['claude-opus-4-7']).toBeDefined();
|
||
expect(ANTHROPIC_PRICING['claude-sonnet-4-6']).toBeDefined();
|
||
expect(ANTHROPIC_PRICING['claude-haiku-4-5-20251001']).toBeDefined();
|
||
});
|
||
|
||
test('first submit is allowed when within budget', () => {
|
||
const meter = new BudgetMeter({ budgetUsd: 1.0, phase: 'auto_think', auditPath });
|
||
const r = meter.check({ modelId: 'claude-haiku-4-5-20251001', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'test' });
|
||
expect(r.allowed).toBe(true);
|
||
expect(r.estimatedCostUsd).toBeGreaterThan(0);
|
||
expect(r.cumulativeCostUsd).toBe(r.estimatedCostUsd);
|
||
});
|
||
|
||
test('cumulative cost denies the second submit when budget exhausted', () => {
|
||
const meter = new BudgetMeter({ budgetUsd: 0.50, phase: 'auto_think', auditPath });
|
||
// Opus 4.7: $5 in / $25 out per 1M. Per call: 5000×5/1M + 10000×25/1M = $0.025 + $0.25 = $0.275
|
||
const big = { modelId: 'claude-opus-4-7', estimatedInputTokens: 5000, maxOutputTokens: 10000, label: 'big' };
|
||
const r1 = meter.check(big); // $0.275 cumulative — allowed
|
||
const r2 = meter.check(big); // $0.55 cumulative — exceeds $0.50 → DENY
|
||
expect(r1.allowed).toBe(true);
|
||
expect(r2.allowed).toBe(false);
|
||
expect(r2.reason).toContain('BUDGET_EXHAUSTED');
|
||
});
|
||
|
||
test('budget=0 disables the gate (cycle runs unbounded)', () => {
|
||
const meter = new BudgetMeter({ budgetUsd: 0, phase: 'drift', auditPath });
|
||
const r = meter.check({ modelId: 'claude-opus-4-7', estimatedInputTokens: 100_000, maxOutputTokens: 100_000, label: 'huge' });
|
||
expect(r.allowed).toBe(true);
|
||
});
|
||
|
||
test('non-Anthropic model bypasses gate with warn-once + ledger entry', () => {
|
||
const meter = new BudgetMeter({ budgetUsd: 0.001, phase: 'auto_think', auditPath });
|
||
const r1 = meter.check({ modelId: 'gemini-3-pro', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'gem1' });
|
||
const r2 = meter.check({ modelId: 'gemini-3-pro', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'gem2' });
|
||
expect(r1.allowed).toBe(true);
|
||
expect(r1.unpriced).toBe(true);
|
||
expect(r2.allowed).toBe(true);
|
||
expect(meter.unpricedSubmits).toBe(2);
|
||
});
|
||
|
||
test('ledger captures every submit (allowed + denied + unpriced)', () => {
|
||
const meter = new BudgetMeter({ budgetUsd: 0.001, phase: 'auto_think', auditPath });
|
||
meter.check({ modelId: 'claude-opus-4-7', estimatedInputTokens: 5000, maxOutputTokens: 4000, label: 'a' });
|
||
meter.check({ modelId: 'claude-opus-4-7', estimatedInputTokens: 5000, maxOutputTokens: 4000, label: 'b-denied' });
|
||
meter.check({ modelId: 'gpt-5', estimatedInputTokens: 1000, maxOutputTokens: 1000, label: 'c-unpriced' });
|
||
const lines = readLedger();
|
||
expect(lines).toHaveLength(3);
|
||
expect(lines[0].event).toBe('submit_denied'); // first opus call exceeds the $0.001 cap
|
||
expect(lines[1].event).toBe('submit_denied');
|
||
expect(lines[2].event).toBe('submit_unpriced');
|
||
});
|
||
|
||
test('ledger uses ISO-week filename when auditPath not overridden', () => {
|
||
// Implicit path branch — just verify it doesn't throw and writes somewhere reasonable.
|
||
const meter = new BudgetMeter({ budgetUsd: 1.0, phase: 'drift' });
|
||
const r = meter.check({ modelId: 'claude-haiku-4-5-20251001', estimatedInputTokens: 100, maxOutputTokens: 100, label: 'wk' });
|
||
expect(r.allowed).toBe(true);
|
||
});
|
||
});
|