Files
gbrain/test/agent-cli.test.ts
T
29961811a4 v0.31.12 fix: canonical Anthropic model IDs + tier routing surface + gbrain models CLI (#844)
* fix: canonical Anthropic model IDs + reverse alias + Opus 4.7 pricing

Replace claude-sonnet-4-6-20250929 with claude-sonnet-4-6 everywhere it
appears as a model ID. Starting with Claude 4.6, Anthropic API IDs are
dateless and pinned — the date suffix was carried forward from Sonnet 4.5
by mistake, producing a phantom ID that 404'd on every call.

Production impact in v0.31.6: isAvailable("chat") returned false in every
code path that loaded the recipe's model list, and extractFactsFromTurn
silently returned []. The headline real-time facts extraction feature
was a no-op on the happy path.

- gateway.ts:46 DEFAULT_CHAT_MODEL -> anthropic:claude-sonnet-4-6
- recipes/anthropic.ts: chat + expansion model lists drop date suffix;
  remove wrong-direction alias (claude-sonnet-4-6 -> -20250929);
  add reverse alias (-20250929 -> claude-sonnet-4-6) so stale user
  configs in models.dream.synthesize etc. keep working
- facts/extract.ts: routes through resolveModel; both fallbacks corrected
- anthropic-pricing.ts: Opus 4.7 corrected $15/$75 -> $5/$25 per
  Anthropic docs (the $15/$75 was Opus 4.0 pricing)
- cross-modal-eval/runner.ts: PRICING now reads from ANTHROPIC_PRICING
  for Anthropic models instead of duplicating the map (single source of
  truth — fixes the drift trap that motivated this whole patch)

Tests: cherry-pick PR #830's test/anthropic-model-ids.test.ts verbatim
(6 recipe-shape guardrails). Update gateway-chat tests to assert reverse
alias resolves correctly. Update budget-meter test for new Opus pricing.

Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: model tier system + recipe-models merge + async reconfigure hook

Add 4-tier model routing (utility/reasoning/deep/subagent) so users can
swap defaults with one config key. Each tier maps to a class of work;
override globally via models.default or per-tier via models.tier.<tier>.

Codex flagged three real architecture issues in the v0.31.12 plan review;
this commit addresses each.

F3 — sync/async timing of configureGateway:
  - buildGatewayConfig stays synchronous (pre-engine-connect callers
    keep working)
  - New reconfigureGatewayWithEngine(engine) async function re-resolves
    expansion + chat defaults through resolveModel after engine.connect()
  - cli.ts wires the re-stamp into the post-connect path

F4/F5 — softening assertTouchpoint was too broad:
  - Earlier plan was to flip native-recipe validation from throw to warn,
    affecting gateway.chat AND gateway.expand AND gateway.embed
  - Instead: per-gateway-instance recipe-models merge. assertTouchpoint
    gets an optional extendedModels Set; when the user opted into a model
    via config, it bypasses the throw. Source-code typos still fail fast.
  - Existing contract test (test/ai/gateway-chat.test.ts:106) preserved

Tier defaults are TIER_DEFAULTS in model-config.ts. Resolution chain
inserts at step 5 (between models.default and env var). Each existing
resolveModel call site gains a tier: arg — think (deep), cycle/synthesize
(reasoning + utility for verdict), patterns/drift (reasoning), auto-think
(deep), facts/extract (reasoning).

Plus 10 new tests pinning tier precedence, subagent-tier fallback when
models.default is non-Anthropic, and the F6 alias-chain conflict case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: subagent runtime enforcement for non-Anthropic models (3 layers)

The subagent loop uses Anthropic's Messages API with prompt caching on
system + tools. OpenAI/Google have different shapes. Setting
models.default = openai:gpt-5.5 and routing the subagent there silently
breaks the loop.

Codex F1+F2+F13 in the v0.31.12 plan review pointed out that "warn at
doctor" wasn't enough — handlers/subagent.ts:148 still did
`const model = data.model ?? DEFAULT_MODEL` and called Anthropic directly,
so a job submitted with data.model = openai:gpt-5.5 bypassed any tier
logic and failed at runtime with a confusing provider error.

Three layers of enforcement, defense in depth:

Layer 1 (queue.ts:add) — submit-time guard. When name === 'subagent'
and data.model is set, validate the provider. Non-Anthropic rejects
before the job enters the queue.

Layer 2 (handlers/subagent.ts) — tier-resolution fallback. The handler
routes through resolveModel({ tier: 'subagent' }). If the chain resolves
to a non-Anthropic provider (via models.default or models.tier.subagent),
the resolver warns + falls back to TIER_DEFAULTS.subagent
(claude-sonnet-4-6).

Layer 3 (doctor.ts:checkSubagentProvider) — surfacing layer. Warns when
models.tier.subagent or models.default is explicitly set to a
non-Anthropic provider, with a paste-ready fix command. Lets users see
config drift before submitting a job.

Tests: 3 new cases in test/agent-cli.test.ts asserting the queue-level
guard rejects non-Anthropic data.model. Existing test/subagent-handler
suite still passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: gbrain models CLI + doctor probe + silent-no-op regression test

New gbrain models CLI gives the agent and user visibility into routing.
Read mode prints the tier table, current overrides, per-task config,
and aliases with source-of-truth attribution per row. Doctor subcommand
fires a 1-token probe to each configured chat/expansion model and
classifies failures (model_not_found / auth / rate_limit / network /
unknown) so config-time invalid IDs surface without waiting for a
production call that silently degrades.

Per Codex F11 — no specific dollar cost claim in either the help text
or the CHANGELOG (providers have minimum-output billing and prompt-cache
rounding that vary). Probe is opt-in (gbrain doctor --probe-models),
never auto-runs. --skip=<provider> narrows the matrix for cost-sensitive
operators.

Per Codex F7+F8+F15 (the structural regression gap): new
test/facts-extract-silent-no-op.test.ts is THE regression test for the
bug class that motivated v0.31.12. Five cases including the smoking-gun:
when chat IS available, extractFactsFromTurn MUST actually call the chat
transport, not silently return []. Uses the gateway's
__setChatTransportForTests seam so it runs in every shard with no API key.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.31.12)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: document v0.31.12 model tier system + gbrain models CLI

Add CLAUDE.md Key Files annotations for the v0.31.12 work:
src/core/model-config.ts (tier system + isAnthropicProvider + TIER_DEFAULTS),
src/core/ai/model-resolver.ts (assertTouchpoint extendedModels arg),
src/core/ai/gateway.ts (reconfigureGatewayWithEngine + extended-models registry),
src/core/minions/queue.ts (subagent submit-time guard, layer 1 of 3),
src/commands/models.ts (new gbrain models CLI + doctor probe),
src/commands/doctor.ts (subagent_provider check, layer 3 of 3),
src/core/ai/recipes/anthropic.ts (canonical model IDs + reverse alias),
src/core/anthropic-pricing.ts (Opus 4.7 corrected to \$5/\$25).

Add CLAUDE.md commands section for gbrain models + gbrain models doctor
+ power-user config recipes. Add README.md command-table rows for the
same. Regenerate llms-full.txt so the bundled docs stay in sync.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: scrub --probe-models reference (flag not actually wired)

The v0.31.12 CHANGELOG and skills/conventions/model-routing.md both
referenced `gbrain doctor --probe-models` as an integrated probe entry
point. The flag was never implemented — only `gbrain models doctor`
landed as the probe surface. Caught by /document-release subagent.

Drop the references rather than wire an untested flag at the last minute.
The probe is reachable via `gbrain models doctor`; users who want it
in doctor's output run that command separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 20:06:31 -07:00

260 lines
9.4 KiB
TypeScript

/**
* `gbrain agent` CLI tests. Covers arg parsing, --since parser, and the
* submit path end-to-end against PGLite so we verify trusted submission,
* protected-name guard, and fan-out wiring.
*
* The full handler-run loop is NOT exercised here (tested in subagent-
* handler.test.ts). This file checks the CLI's submission + orchestration
* glue.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
import * as os from 'node:os';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { MinionQueue } from '../src/core/minions/queue.ts';
import { __testing as agentTesting } from '../src/commands/agent.ts';
import { parseSince } from '../src/commands/agent-logs.ts';
import { isProtectedJobName, PROTECTED_JOB_NAMES } from '../src/core/minions/protected-names.ts';
let engine: PGLiteEngine;
let queue: MinionQueue;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({ database_url: '' });
await engine.initSchema();
queue = new MinionQueue(engine);
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await engine.executeRaw('DELETE FROM minion_jobs');
});
describe('parseRunFlags', () => {
test('follow defaults off when stdout is non-TTY (test env)', () => {
const { flags, rest } = agentTesting.parseRunFlags(['hello', 'world']);
expect(flags.follow).toBe(process.stdout.isTTY === true);
expect(rest).toEqual(['hello', 'world']);
});
test('flags before prompt are parsed, unknown token ends flag parsing', () => {
const { flags, rest } = agentTesting.parseRunFlags([
'--model', 'claude-opus-4-7', '--max-turns', '30', 'summarize', 'everything',
]);
expect(flags.model).toBe('claude-opus-4-7');
expect(flags.maxTurns).toBe(30);
expect(rest).toEqual(['summarize', 'everything']);
});
test('--tools comma-split', () => {
const { flags } = agentTesting.parseRunFlags(['--tools', 'brain_search, brain_get_page', 'prompt']);
expect(flags.tools).toEqual(['brain_search', 'brain_get_page']);
});
test('--detach implies !follow', () => {
const { flags } = agentTesting.parseRunFlags(['--detach', 'x']);
expect(flags.detach).toBe(true);
expect(flags.follow).toBe(false);
});
test('double-dash ends flag parsing explicitly', () => {
const { flags, rest } = agentTesting.parseRunFlags(['--model', 'm', '--', '--not-a-flag']);
expect(flags.model).toBe('m');
expect(rest).toEqual(['--not-a-flag']);
});
test('unknown flag throws', () => {
expect(() => agentTesting.parseRunFlags(['--what', 'x'])).toThrow(/unknown flag/);
});
test('--subagent-def + --timeout-ms parsed', () => {
const { flags } = agentTesting.parseRunFlags([
'--subagent-def', 'researcher', '--timeout-ms', '60000', 'hello',
]);
expect(flags.subagentDef).toBe('researcher');
expect(flags.timeoutMs).toBe(60000);
});
test('--fanout-manifest parsed', () => {
const { flags } = agentTesting.parseRunFlags(['--fanout-manifest', '/tmp/m.json']);
expect(flags.fanoutManifest).toBe('/tmp/m.json');
});
});
describe('parseSince', () => {
test('returns undefined on empty input', () => {
expect(parseSince(undefined)).toBeUndefined();
expect(parseSince('')).toBeUndefined();
});
test('parses ISO-8601 timestamps', () => {
const iso = '2026-04-20T12:00:00.000Z';
expect(parseSince(iso)).toBe(iso);
});
test('parses relative 5m', () => {
const out = parseSince('5m')!;
const parsed = new Date(out).getTime();
const now = Date.now();
expect(now - parsed).toBeGreaterThanOrEqual(5 * 60 * 1000 - 1000);
expect(now - parsed).toBeLessThan(5 * 60 * 1000 + 1000);
});
test('parses relative 2h', () => {
const out = parseSince('2h')!;
const delta = Date.now() - new Date(out).getTime();
expect(delta).toBeGreaterThanOrEqual(2 * 3600 * 1000 - 1000);
});
test('parses relative 1d', () => {
const out = parseSince('1d')!;
const delta = Date.now() - new Date(out).getTime();
expect(delta).toBeGreaterThanOrEqual(86_400_000 - 1000);
});
test('throws on unparseable input', () => {
expect(() => parseSince('not-a-date')).toThrow(/could not parse/);
});
});
describe('protected-name guard includes subagent + aggregator', () => {
test('shell stays protected', () => {
expect(isProtectedJobName('shell')).toBe(true);
expect(PROTECTED_JOB_NAMES.has('shell')).toBe(true);
});
test('subagent is protected (v0.15)', () => {
expect(isProtectedJobName('subagent')).toBe(true);
});
test('subagent_aggregator is protected (v0.15)', () => {
expect(isProtectedJobName('subagent_aggregator')).toBe(true);
});
test('a random non-protected name is not protected', () => {
expect(isProtectedJobName('sync')).toBe(false);
});
test('trim normalization still blocks " subagent "', () => {
expect(isProtectedJobName(' subagent ')).toBe(true);
});
});
describe('queue.add trusted-submit gate for subagent', () => {
test('subagent without allowProtectedSubmit throws', async () => {
await expect(queue.add('subagent', { prompt: 'hi' })).rejects.toThrow();
});
test('subagent with allowProtectedSubmit succeeds', async () => {
const job = await queue.add('subagent', { prompt: 'hi' }, {}, { allowProtectedSubmit: true });
expect(job.name).toBe('subagent');
expect(job.status).toBe('waiting');
});
test('subagent_aggregator gated the same way', async () => {
await expect(queue.add('subagent_aggregator', { children_ids: [] })).rejects.toThrow();
const ok = await queue.add('subagent_aggregator', { children_ids: [1] }, {}, {
allowProtectedSubmit: true,
});
expect(ok.name).toBe('subagent_aggregator');
});
test('v0.31.12: subagent with non-Anthropic data.model is rejected at submit time (Layer 1)', async () => {
// Codex F1 in v0.31.12 plan review: the subagent loop is Anthropic Messages
// API + prompt caching. A job submitted with `data.model = openai:gpt-5.5`
// would silently fail at runtime with a confusing provider error. The
// submit-time guard rejects BEFORE the job enters the queue.
await expect(
queue.add('subagent', { prompt: 'hi', model: 'openai:gpt-5.5' }, {}, { allowProtectedSubmit: true }),
).rejects.toThrow(/non-Anthropic/i);
});
test('v0.31.12: subagent with Anthropic data.model still succeeds', async () => {
const job = await queue.add(
'subagent',
{ prompt: 'hi', model: 'anthropic:claude-opus-4-7' },
{},
{ allowProtectedSubmit: true },
);
expect(job.name).toBe('subagent');
});
test('v0.31.12: subagent with bare claude- model id passes (provider-prefix optional)', async () => {
// isAnthropicProvider accepts both `anthropic:claude-foo` and bare `claude-foo`.
const job = await queue.add(
'subagent',
{ prompt: 'hi', model: 'claude-sonnet-4-6' },
{},
{ allowProtectedSubmit: true },
);
expect(job.name).toBe('subagent');
});
});
describe('fan-out manifest shape (integration)', () => {
test('fanout-manifest with 3 entries creates 3 subagent children + 1 aggregator', async () => {
// Manually replicate what runAgentRun does for --fanout-manifest > 1.
// We don't invoke runAgentRun (it calls process.exit on error) — we
// assert that the plumbing works via direct queue calls with the
// same flags it uses.
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'fanout-'));
try {
const manifestPath = path.join(tmp, 'm.json');
fs.writeFileSync(manifestPath, JSON.stringify([
{ prompt: 'chunk 1' }, { prompt: 'chunk 2' }, { prompt: 'chunk 3' },
]));
// Aggregator first.
const agg = await queue.add(
'subagent_aggregator',
{ children_ids: [] },
{ max_stalled: 3 },
{ allowProtectedSubmit: true },
);
const kids: number[] = [];
for (const p of ['chunk 1', 'chunk 2', 'chunk 3']) {
const c = await queue.add(
'subagent',
{ prompt: p },
{ parent_job_id: agg.id, on_child_fail: 'continue', max_stalled: 3 },
{ allowProtectedSubmit: true },
);
kids.push(c.id);
}
await engine.executeRaw(
`UPDATE minion_jobs SET data = jsonb_set(data, '{children_ids}', $1::jsonb) WHERE id = $2`,
[JSON.stringify(kids), agg.id],
);
// Aggregator should be in waiting-children since kids were submitted
// with parent_job_id = agg.id (Lane 1B behavior).
const aggNow = await queue.getJob(agg.id);
expect(aggNow?.status).toBe('waiting-children');
// Aggregator's data.children_ids reflects the spawned children.
const dataRow = await engine.executeRaw<{ data: unknown }>(
`SELECT data FROM minion_jobs WHERE id = $1`, [agg.id],
);
const data = typeof dataRow[0]!.data === 'string'
? JSON.parse(dataRow[0]!.data as string)
: dataRow[0]!.data as Record<string, unknown>;
expect(data.children_ids).toEqual(kids);
// Each child should have on_child_fail = 'continue'.
const childRows = await engine.executeRaw<{ on_child_fail: string }>(
`SELECT on_child_fail FROM minion_jobs WHERE parent_job_id = $1`, [agg.id],
);
expect(childRows.length).toBe(3);
expect(childRows.every(r => r.on_child_fail === 'continue')).toBe(true);
} finally {
fs.rmSync(tmp, { recursive: true, force: true });
}
});
});