Files
gbrain/src/commands/models.ts
dab441f59f v0.41.4.0 wave: local providers + cross-platform stdin + gateway-routed dream judge (6 community PRs) (#1377)
* fix(cli): use fd 0 instead of '/dev/stdin' for cross-platform stdin reads

`readFileSync('/dev/stdin', 'utf-8')` works on Unix but fails on Windows
(Git Bash, PowerShell, cmd) with `ENOENT: no such file or directory,
open '/dev/stdin'`. Windows doesn't expose `/dev/stdin` as a filesystem
path.

Reading file descriptor 0 directly (`readFileSync(0, 'utf-8')`) is the
documented Node.js idiom and works on every platform. No behavior change
on Unix — same syscall path, same semantics.

Repro on Windows before the fix:
  echo "test" | gbrain put my-page
  ENOENT: no such file or directory, open '/dev/stdin'

After: round-trip put/search/delete works on Windows Git Bash.

* v0.40.6.1 feat: llama-server reranker — local Qwen3 / self-hosted ZE via llama.cpp

Adds local reranker support so users can point gbrain's reranker call at their
own llama.cpp server instead of ZeroEntropy's hosted API. One new recipe
(`llama-server-reranker`), a `path?: string` + `default_timeout_ms?: number`
extension on `RerankerTouchpoint`, env passthrough wiring, budget-tracker
`FREE_LOCAL_RERANK_PROVIDERS` set so `--max-cost` callers don't TX2 hard-fail on
local rerank, and a doctor-probe divergence fix (probe and live search now read
the same `search.reranker.model` path via `loadSearchModeConfig` + `resolveSearchMode`).

ZE-hosted users are unchanged. Voyage / Cohere / vLLM rerankers stay out of
scope — different wire shapes need adapter hooks designed against their actual
shapes in a follow-up plan.

Verification:
- `bun run verify` (typecheck + 13 pre-checks): clean
- `bun run check:all` (15 historical checks): clean
- 107/107 expect() calls pass across 5 affected test files
- /codex review against the full diff: GATE PASS (caught one [P2] /v1 path
  doubling bug pre-merge; fixed by changing recipe path to leaf `/rerank`)
- Claude adversarial subagent: 7 net-new findings filed as v0.40.7+ TODOs
  (none currently exploitable; hardening for future contributor traps)

Test surface (107 cases, 5 files):
- test/ai/rerank.test.ts: path override (exact URL match), default_timeout_ms
  honored, empty models[] accepts any id, ZE regression
- test/ai/recipe-llama-server-reranker.test.ts: recipe shape regression guard
  + base_url + path concat assertion (codex-caught /v1/v1/ regression)
- test/search-mode.test.ts: timeout precedence chain (per-call > config >
  recipe > bundle), ZE no-recipe-default regression, unknown provider fallthrough
- test/models-doctor-reranker.test.ts: divergence-fix helper across DB-plane
  read, mode default, disabled, override, DB-error graceful fallback
- test/core/budget/budget-tracker.test.ts: free-local rerank pricing + arbitrary
  model id + chat-kind TX2 hard-fail preserved

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: post-ship documentation sync

* docs: index docs/ai-providers/ in llms.txt (zeroentropy + llama-server-reranker)

The hand-curated llms-config.ts doc map never included docs/ai-providers/, so
both zeroentropy.md (since v0.35.0.0) and the new llama-server-reranker.md were
invisible to the AI-facing llms.txt / llms-full.txt index. Adds an "AI providers"
section with both. Marked includeInFull: false (setup walkthroughs belong in the
index but would push the single-fetch bundle past FULL_SIZE_BUDGET) — same
treatment CHANGELOG.md gets.

Caught by the /ship document-release subagent.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix: recipe-aware embedding-provider check for local providers

doctor --remediation-plan and autopilot both judged the embedding
provider with a hosted-only key check, so a brain on ollama: or
llama-server: was reported "blocked" on a missing API key it never
needed, contradicting doctor --json's 100%-coverage health.

Extract a shared embeddingProviderConfigured() helper into
brain-score-recommendations.ts: empty auth_env.required (local
providers) is configured with no key; hosted providers check their
OWN required key. Both producers (doctor, autopilot) call it,
killing the DRY violation that caused the bug. Hosted brains with a
missing key still block.

* fix(budget): price local embed providers at $0

A --max-cost-bounded embed/reindex job configured for ollama: or
llama-server: TX2 hard-failed with no_pricing because
lookupEmbeddingPrice has no entry for local models. Add
FREE_LOCAL_EMBED_PROVIDERS (sibling to FREE_LOCAL_RERANK_PROVIDERS)
so a pricing miss on a local-inference provider returns $0 instead
of null. lmstudio/litellm intentionally excluded.

* feat(models): embedding reachability probe in gbrain models doctor

A down/misconfigured local embed server was invisible until first
embed. Add probeEmbeddingReachability() (mirrors the reranker probe):
a 1-input embed with a 5s abort timeout, classified via classifyError,
under a new 'embedding_reachability' touchpoint, gated on the
zero-network config probe returning ok first.

* fix: don't count config-plane voyage/google keys as configured

codex review caught a false positive: HOSTED_EMBED_KEY_CONFIG mapped
VOYAGE_API_KEY/GOOGLE_GENERATIVE_AI_API_KEY to config fields, but
buildGatewayConfig only threads openai/anthropic/zeroentropy config
keys into the gateway env. A Voyage/Google brain with the key only in
config.json would be judged "configured" and dispatch an embed.stale
job that then fails auth at the gateway. Drop those two from the map so
the producer closures resolve them by env var only, matching what the
gateway can actually use. Pinned by a regression test.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat(dream): route significance judge through gateway.chat for multi-provider support

Replaces the hardcoded `new Anthropic()` client in the dream-cycle synthesize
phase with a gateway-routed JudgeClient adapter. Mirrors the v0.35.5.0 pattern
that closed #952 for runThink: construction-time provider/key probe returns null
on a clear miss (cheap pre-flight); the verdict loop wraps the chat call in
try/catch for AIConfigError mid-run.

Any provider with a registered gateway recipe (Anthropic, DeepSeek, OpenRouter,
Voyage, Ollama, llama-server, etc.) is now reachable via:

    gbrain config set models.dream.synthesize_verdict <provider>:<model>

The canonical config key `models.dream.synthesize_verdict` (per PER_TASK_KEYS
in src/core/model-config.ts) is used unchanged. The exported JudgeClient
interface signature is preserved for test-seam stability.

The original community PR (#1349) shipped a custom fetch adapter that
bypassed the gateway entirely. This reworked landing routes through the
canonical seam so future provider additions automatically benefit, and a
CI guard (T7) will land in this wave to prevent the bug class from
re-opening (the same one that bit src/core/think/index.ts before v0.35.5.0).

Co-Authored-By: justemu <206393437+justemu@users.noreply.github.com>

* test(dream): synthesize-gateway-adapter unit tests + R3 parsed-verdict parity

11 cases pin the gateway-routed JudgeClient adapter from T5:

- A1: makeJudgeClient returns null on missing Anthropic key (legacy short-circuit preserved)
- A2: returns a JudgeClient when chat provider is reachable
- A3: JudgeClient.create routes through gateway.chat (via __setChatTransportForTests)
- A4: ChatResult.text → Anthropic.Message.content[0].text mapping
- A5: empty text from gateway → graceful empty-text Anthropic.Message
- A6: non-AIConfigError from gateway propagates to caller (no swallow)
- A7: AIConfigError from gateway propagates as AIConfigError (caught per-transcript in production loop)
- A8: makeJudgeClient returns null on unknown provider prefix
- A9: returns a JudgeClient for non-anthropic providers without env-probing (delegates to gateway at call time)
- R3: parsed-verdict SEMANTIC parity — gateway-routed and legacy SDK-shape JudgeClients produce same {worth_processing, reasons} given identical canned LLM text
- R3 corollary: unparseable LLM output → both paths fall through to cheap-fallback verdict

Codex flagged byte-identical-Anthropic.Message as a meaningless gate; R3 is
parsed-verdict semantic parity instead. Mirror pattern of
test/think-gateway-adapter.test.ts for cross-site consistency with the
v0.35.5.0 runThink migration.

* ci: guard against direct Anthropic SDK construction in gateway-routed files

New scripts/check-gateway-routed-no-direct-anthropic.sh greps two guarded
files (src/core/cycle/synthesize.ts and src/core/think/index.ts) for
`new Anthropic()` constructor calls and runtime imports of @anthropic-ai/sdk.
Type-only imports (`import type Anthropic from '@anthropic-ai/sdk'`) stay
allowed because both files use Anthropic.Message / .MessageCreateParamsNonStreaming
as adapter types.

Comment lines (starting with `//` or ` *`) are excluded so historical
references in JSDoc don't false-fire. Negative test in this commit's
verification confirms: injecting `new Anthropic()` into synthesize.ts
makes the guard exit 1 with a clear error pointing at the gateway adapter
pattern; reverting restores the OK state.

Wired into both `bun run verify` and `bun run check:all`. Closes the bug
class that bit synthesize.ts in PR #1349 (which would have shipped a
parallel fetch stack instead of routing through the canonical gateway).
The same class previously bit think/index.ts and was fixed structurally
in v0.35.5.0; this guard prevents either file from regressing.

Extend GUARDED_FILES in the script when migrating another file off
direct SDK construction.

* docs(put_page): point Windows / pipe-buffer users at gbrain capture --file

Extends the put_page op description (surfaced by `gbrain put --help`) with a
one-line pointer to `gbrain capture --file PATH --slug SLUG` for the file-
as-input use case. Capture (v0.39.3.0) is the canonical Windows-pipe-buffer
escape route: reads files as a Buffer first, scans the first 8KB for NUL bytes
to refuse binary content, decodes to UTF-8 only after the safety check, and
adds provenance write-through.

Lands the user-facing value the closed PR #1365 was reaching for, without
duplicating the CLI surface. Credits the original contributor.

Co-Authored-By: ecat2010 <90021101+ecat2010@users.noreply.github.com>

* test: R1+R2+R4 critical regression pins for the community-PR-wave landing

Per the wave's eng-review plan (IRON RULE — mandatory):

  R1 — get_page handler accepts calls without `content` param. Pre-wave
       PR #1365 landed its `!p.content → throw` check in the WRONG handler
       (get_page instead of put_page), which would have broken every read
       in the system. Pin: get_page MUST NOT require content + the schema
       carries no `content` or `file` param.

  R2 — put_page schema content stays `required: true`. PR #1365 also
       flipped `content` from required→optional in the schema. Pin: the
       contract stays at `required: true` + the closed PR's `file` param
       is NOT in the schema.

  R4 — Cross-platform stdin via fd 0 (PR #1325 regression pin). Source-grep
       asserts src/cli.ts uses `readFileSync(0, ...)` and NOT the legacy
       `readFileSync('/dev/stdin', ...)`. Belt-and-suspenders pattern
       assertions confirm the parseOpArgs branch shape (cliHints.stdin
       check, 5MB cap, isTTY gate) hasn't drifted.

R3 (gateway-adapter parsed-verdict parity) lives in the sibling file
test/cycle/synthesize-gateway-adapter.test.ts.

* test(e2e): update dream-synthesize no-key reason text + harden hermeticity

After T5's gateway-adapter rework, the "no API key" verdict text changed from
'no ANTHROPIC_API_KEY for significance judge' to
'no configured provider for verdict model: <model>' (broader + names the
actual model so the user sees WHICH provider failed). Update both assertions
that check the old text.

Hermeticity bug fix in the same commit: `withoutAnthropicKey` previously only
cleared the env var. After the rework, `makeJudgeClient` ALSO checks
`loadConfig().anthropic_api_key` (same hasAnthropicKey() pattern think/index.ts
uses since v0.35.5.0). If the developer running the test has the key set in
~/.gbrain/config.json, the test would behave non-deterministically. Fix:
override GBRAIN_HOME to a fresh tmpdir for the duration of the body, restore
on return (even on throw).

* test(e2e): pin verdict-loop AIConfigError catch from T5 rework end-to-end

Drives runPhaseSynthesize against a real PGLite engine with the gateway
chat transport stubbed to throw AIConfigError on every call (simulates a
revoked/misconfigured provider surfacing mid-run). Asserts:

  - Phase does NOT crash; converts the throw to a per-transcript verdict
    with worth=false and reasons[0] matching "gateway error: ...".
  - status='ok' so subsequent transcripts in the loop would continue
    being judged (not visible in 1-transcript test, but the loop shape is
    proven not to abort).

Pre-rework (T5), this code path didn't exist — judgeSignificance threw
directly to runPhaseSynthesize and crashed the whole phase. Pin so a
future regression that removes the try/catch fires loudly.

* docs(claude.md): annotate v0.41+ community-PR-wave changes

Two additions to the Key files section:

- src/core/cycle/synthesize.ts — appends a v0.41+ paragraph documenting
  the gateway-adapter rework (makeJudgeClient + AIConfigError catch loop +
  canonical config key + JudgeClient interface preserved + CI guard
  reference + test file references).

- scripts/check-gateway-routed-no-direct-anthropic.sh — new entry
  documenting the CI guard's contract, scope, and how to extend
  GUARDED_FILES when migrating another file off direct SDK construction.

CLAUDE.md drives /sync-gbrain and llms.txt generation; both need the
wave's annotations to land BEFORE the llms regeneration step (T10).

* docs(llms): regenerate llms.txt + llms-full.txt for v0.41+ wave

Refreshes the auto-generated llms.txt bundles to pick up the CLAUDE.md
annotations landed earlier in this wave (gateway-adapter synthesize.ts
+ check-gateway-routed-no-direct-anthropic.sh + the cherry-picked
llama-server-reranker recipe). Pinned by test/build-llms.test.ts.

* fix(providers): dynamic-width id column accommodates llama-server-reranker

v0.40.6.1 introduced `llama-server-reranker` (21 chars), which overflowed
formatRecipeTable's static 14-char PROVIDER column. When the id is longer
than the column, padEnd is a no-op — the row starts with the tier name
directly, no space delimiter. test/providers.test.ts 'each recipe appears
at most once' iterates every recipe and asserts at least one row starts
with `${id} ` or `${id}  `; with no space after `llama-server-reranker`,
the assertion fails and the recipe appears effectively missing from the
human-readable list.

Fix: compute column width dynamically as `max(14, max(id.length) + 1)` so
every id is followed by at least one space, regardless of length. Also
widens the separator rule to match. 14 stays as the floor so the existing
short-id rows (openai 6, ollama 6, anthropic 9, ...) keep their familiar
layout when llama-server-reranker isn't in the active recipe set.

10/10 cases in test/providers.test.ts pass after the fix.

* chore: pre-landing review polish — refresh models doctor tip + file embed timeout TODO

Two pre-landing review absorptions:

- `src/commands/models.ts:154` — the help-text tip said `gbrain models doctor`
  "spends ~1 token per model" but the wave added an `embed(['probe'])` call
  AND a reranker probe. Generalize to "spends a minimal request per configured
  chat/embed/rerank surface" so the cost expectation matches reality.

- `TODOS.md` — file a follow-up to widen `default_timeout_ms` from
  RerankerTouchpoint to EmbeddingTouchpoint so `probeEmbeddingReachability`
  doesn't hardcode 5000ms while the sibling reranker probe reads the
  recipe's configured timeout. Local CPU embedding endpoints (llama-server)
  hit the same cold-start curve as Qwen3-Reranker-4B; workaround today is
  "re-run the probe" per the existing JSDoc.

Other informational findings from pre-landing review either match
established patterns (no behavioral test for `probeEmbeddingReachability`,
matching `probeRerankerReachability`), are intentional choices documented
in JSDoc (the `as unknown as Anthropic.Message` cast), or are micro-perf
in non-hot paths (autopilot's 4 sequential `getConfig` awaits per
5-minute tick). All non-blocking.

* ci: tighten gateway-routed guard against import bypass shapes + honest JSDoc

Adversarial review caught two soft spots in the wave's new contracts:

1. `scripts/check-gateway-routed-no-direct-anthropic.sh` only matched the
   default-import shape `import Anthropic from '@anthropic-ai/sdk'`. A future
   contributor (or, more realistically, a future refactor) could bypass with:
     - `import { Anthropic } from '@anthropic-ai/sdk'`
     - `import { Anthropic as A } from '@anthropic-ai/sdk'`
     - `import * as Anthropic from '@anthropic-ai/sdk'`
     - `const x = await import('@anthropic-ai/sdk')`
   Tightened the regex to match ANY value-shaped import from the SDK module
   (excluding only the explicit `import type ... from '@anthropic-ai/sdk'`
   form which the adapter's Anthropic.Message return type needs). Added a
   second grep for dynamic imports. Verified all four bypass shapes now
   trigger the guard against synthesize.ts; type-only import still passes.

2. `synthesize.ts:makeJudgeClient` JSDoc claimed the adapter "tolerates the
   array-of-blocks shape for future flexibility" — but the mapping flattens
   ONLY text blocks; `tool_use`, `tool_result`, image blocks silently
   become empty strings. Today only `judgeSignificance` calls this and it
   only sends string content, so no behavior bug. But the comment was
   marketing future flexibility the code doesn't deliver. Narrowed to call
   out the silent-drop and say to extend the mapping if a future caller
   wires non-text content through.

Both wave-scope: the CI guard was added by the wave, the JSDoc was added
by the wave's T5 rework. Adversarial review caught them before merge.

* fix(models doctor): reranker probe timeout matches live search precedence chain

Codex Pass-9 adversarial review caught a probe-vs-production divergence:
production `hybridSearch` resolves reranker timeout via the full chain
(per-call > config > recipe > bundle) by going through
`loadSearchModeConfig + resolveSearchMode`, but `probeRerankerReachability`
was reading ONLY the recipe's `default_timeout_ms` — so an operator who
set `search.reranker.timeout_ms=1000` would see doctor wait 30s and report
"reachable" while production search timed out at 1s and fail-opened.
A higher configured timeout produces the opposite false failure (probe
gives up at 5s when production would have waited longer).

Fix: extract `resolveLiveRerankerTimeoutMs(engine)` parallel to the
existing `resolveLiveRerankerModel(engine)` — same precedence chain,
same DB-plane consistency posture. The probe now reads the SAME timeout
live search reads, on the same lookup path.

The codex P1 finding about `FREE_LOCAL_*_PROVIDERS` zero-pricing being
bypassable via redirected `LLAMA_SERVER_BASE_URL` is filed as a TODO under
community-pr-wave follow-ups — couples with the existing
FREE_LOCAL_PROVIDERS unification TODO so both close in one v0.41+ PR.

* ci(guard): handle mixed type+value imports + macOS BSD sed POSIX classes

Codex structured review [P3] caught a bypass in the freshly-tightened
gateway-routed guard:

  import { type Message, Anthropic } from '@anthropic-ai/sdk';
  new Anthropic();

The previous regex `^\s*import\s+[^t][^y]*from ...` was meant to exclude
`import type ...` but stops at the `y` in `type` inside the brace list,
silently allowing the value-import `Anthropic` through. Two fixes:

1. Replace the brittle regex-based type-exclusion with a clause-level
   parse: extract the brace-list specifiers, allow the import iff EVERY
   non-empty specifier is `type`-prefixed. Catches mixed-import bypasses
   (`{ type Foo, Bar }`) while keeping all-type braces (`{ type Foo, type Bar }`)
   passing. Default + namespace imports remain always-value-shaped.

2. Replace `\s` with POSIX `[[:space:]]` in the sed extract — macOS BSD sed
   doesn't honor `\s` in extended-regex mode (it silently no-ops the pattern
   so `specifiers` comes back empty and the script falls through to the
   default/namespace branch's wrong error message).

Hermetic 7-shape regression matrix now verifies every TypeScript import
shape against the expected ALLOW/BLOCK verdict; all 7 pass:
- ALLOW: `import type Anthropic from '...'`
- ALLOW: `import type { Foo } from '...'`
- ALLOW: `import { type Message, type Foo } from '...'`
- BLOCK: `import { type Message, Anthropic } from '...'`
- BLOCK: `import { Anthropic } from '...'`
- BLOCK: `import Anthropic from '...'`
- BLOCK: `import * as A from '...'`

Subshell-trap fix in the same commit: the previous "exit 1 inside while-pipe"
pattern doesn't propagate to the outer `$?` because the pipe spawns a
subshell. Switched to a tmpfile-flagged sentinel so the verdict survives
the subshell boundary cleanly.

* chore: bump version and changelog (v0.41.4.0)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(audit-writer): route log() to file matching event ts, not real-now

CI failure surfaced a time-dependent test flake in
`test/audit/audit-writer.test.ts` "returns events from current week,
filtered by ts cutoff" (added in v0.40.4.0 PR #1300). The test pinned
synthetic `now = 2026-05-22T12:00:00Z` (ISO week 21), logged 3 events
with synthetic ts values, then called `readRecent(7, now)` expecting
to find 2 events in window.

Root cause: `log()` ignored the caller-supplied `ts` for filename
routing and ALWAYS wrote to the file matching real-time-now's ISO
week. When real CI time crossed into 2026-W22 (this Monday), the
events went to W22's file but `readRecent` walked W21 + W20 → 0 hits.

Fix:
- `log()` parses `event.ts` (when provided) and routes to the file
  matching that ts's ISO week. Falls back to real-now when ts is
  missing or unparseable.
- No behavior change for production callers — none of the 5 audit
  consumers pass `ts` explicitly (rerank-audit, audit-slug-fallback,
  content-sanity-audit, graph-signals, supervisor-audit). The writer
  stamps real-now → both ts and filename use real-now → same file
  as before.
- Sibling test "honors caller-supplied ts override" also pinned a
  fixed ts and would have broken from the opposite angle (test
  read from `computeFilename()` default = real-now). Updated to
  read from `computeFilename(new Date(fixedTs))` so it asserts the
  per-row file routing the wave now provides.

22/22 audit-writer cases pass. Production callers (5 sites) unchanged.

Pre-existing on master since v0.40.4.0; surfaced when real time
crossed into a different ISO week than the test's synthetic now.
NOT introduced by this PR (#1377 community-PR-wave) — audit-writer
files aren't touched by the wave.

---------

Co-authored-by: Tobias <34135750+tobbecokta@users.noreply.github.com>
Co-authored-by: kohai-ut <chris@tincreek.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: justemu <noreply@github.com>
Co-authored-by: justemu <206393437+justemu@users.noreply.github.com>
Co-authored-by: ecat2010 <90021101+ecat2010@users.noreply.github.com>
2026-05-25 10:39:09 -07:00

647 lines
26 KiB
TypeScript

/**
* v0.31.12 — `gbrain models` CLI.
*
* Two modes:
*
* `gbrain models` — read-only routing table. Prints the four
* tier defaults, the resolved value for each
* (after consulting models.default + models.tier.*),
* per-task overrides, alias map, and source-of-truth
* column (default / config / env).
*
* `gbrain models doctor` — opt-in probe. Fires a 1-token `gateway.chat()`
* call against each configured chat / expansion
* model and reports reachability with the
* provider's error string. Catches the bug class
* that motivated v0.31.12 (the v0.31.6 chat
* default 404'd silently against the Anthropic
* API).
*
* Flags:
* --json — JSON output (both modes)
* --skip=<provider> — narrow `doctor` probe to skip a provider
* (e.g. cost-sensitive operators with rate limits)
*
* Per Codex F11 in plan review: no specific dollar cost claim. Probe uses
* `max_tokens: 1` against each configured model; actual cost depends on
* provider billing minimums.
*/
import type { BrainEngine } from '../core/engine.ts';
import {
DEFAULT_ALIASES,
TIER_DEFAULTS,
resolveModel,
type ModelTier,
} from '../core/model-config.ts';
const TIERS: ModelTier[] = ['utility', 'reasoning', 'deep', 'subagent'];
const PER_TASK_KEYS: Array<{ key: string; tier: ModelTier; description: string }> = [
{ key: 'models.dream.synthesize', tier: 'reasoning', description: 'Dream synthesis (conversation → brain pages)' },
{ key: 'models.dream.synthesize_verdict', tier: 'utility', description: 'Dream synthesis verdict (Haiku judge)' },
{ key: 'models.dream.patterns', tier: 'reasoning', description: 'Pattern discovery (cross-take themes)' },
{ key: 'models.drift', tier: 'reasoning', description: 'Drift LLM judge (v0.29 scaffold)' },
{ key: 'models.auto_think', tier: 'deep', description: 'Auto-think question answering' },
{ key: 'models.think', tier: 'deep', description: '`gbrain think` synthesis op' },
{ key: 'models.subagent', tier: 'subagent', description: '`gbrain agent run` subagent loop' },
{ key: 'facts.extraction_model', tier: 'reasoning', description: 'Real-time facts extraction during sync' },
{ key: 'models.eval.longmemeval', tier: 'reasoning', description: 'LongMemEval benchmark answer-gen' },
{ key: 'models.eval.contradictions_judge', tier: 'utility', description: 'Contradiction probe judge (v0.34 temporal-aware)' },
{ key: 'models.expansion', tier: 'utility', description: 'Query expansion for hybrid search' },
{ key: 'models.chat', tier: 'reasoning', description: 'Default `gateway.chat()` model' },
];
interface ModelEntry {
tier: ModelTier;
resolved: string;
source: string; // "default" | "config: <key>" | "env: <VAR>"
}
interface ModelsReport {
schema_version: 1;
global_default: { value: string | null };
tiers: Record<ModelTier, ModelEntry>;
per_task: Array<{ key: string; tier: ModelTier; resolved: string; source: string; description: string }>;
aliases: { defaults: Record<string, string>; user: Record<string, string> };
}
async function probeSource(engine: BrainEngine, configKey: string, envVar: string): Promise<string | null> {
// For per-task probes, return the source the resolver USED (config / env /
// tier default / hardcoded). The resolver itself is the source of truth;
// we re-walk a subset of its precedence here to attribute the value.
const configVal = await engine.getConfig(configKey);
if (configVal && configVal.trim()) return `config: ${configKey}`;
if (process.env[envVar] && process.env[envVar]!.trim()) return `env: ${envVar}`;
return null;
}
async function buildReport(engine: BrainEngine): Promise<ModelsReport> {
const globalDefault = await engine.getConfig('models.default');
const tiers = {} as Record<ModelTier, ModelEntry>;
for (const t of TIERS) {
const tierOverride = await engine.getConfig(`models.tier.${t}`);
// What models.default beats tier — re-walk the chain to attribute properly.
let source: string;
if (globalDefault && globalDefault.trim()) {
source = 'config: models.default';
} else if (tierOverride && tierOverride.trim()) {
source = `config: models.tier.${t}`;
} else {
source = 'default';
}
const resolved = await resolveModel(engine, { tier: t, fallback: TIER_DEFAULTS[t] });
tiers[t] = { tier: t, resolved, source };
}
const per_task: ModelsReport['per_task'] = [];
for (const { key, tier, description } of PER_TASK_KEYS) {
const resolved = await resolveModel(engine, { configKey: key, tier, fallback: TIER_DEFAULTS[tier] });
const explicit = await probeSource(engine, key, 'GBRAIN_MODEL');
const source = explicit ?? `tier.${tier}`;
per_task.push({ key, tier, resolved, source, description });
}
// User-defined aliases (engine.getConfig is the source; we don't enumerate
// every possible alias key, just the common ones the docs mention).
const userAliases: Record<string, string> = {};
for (const name of ['opus', 'sonnet', 'haiku', 'gemini', 'gpt']) {
const v = await engine.getConfig(`models.aliases.${name}`);
if (v && v.trim()) userAliases[name] = v.trim();
}
return {
schema_version: 1,
global_default: { value: globalDefault?.trim() || null },
tiers,
per_task,
aliases: { defaults: { ...DEFAULT_ALIASES }, user: userAliases },
};
}
function formatText(report: ModelsReport): string {
const lines: string[] = [];
lines.push('Tier routing:');
for (const t of TIERS) {
const e = report.tiers[t];
lines.push(` tier.${t.padEnd(10)} ${e.resolved.padEnd(45)} [${e.source}]`);
}
lines.push('');
lines.push('Global default:');
lines.push(` models.default ${report.global_default.value ?? '(unset)'}`);
lines.push('');
lines.push('Per-task overrides:');
for (const t of report.per_task) {
lines.push(` ${t.key.padEnd(34)}${t.resolved.padEnd(45)} [${t.source}]`);
}
lines.push('');
lines.push('Aliases:');
for (const [k, v] of Object.entries(report.aliases.defaults)) {
const userOverride = report.aliases.user[k];
if (userOverride) {
lines.push(` ${k.padEnd(8)}${userOverride} (user override; default: ${v})`);
} else {
lines.push(` ${k.padEnd(8)}${v}`);
}
}
for (const [k, v] of Object.entries(report.aliases.user)) {
if (!(k in report.aliases.defaults)) {
lines.push(` ${k.padEnd(8)}${v} (user)`);
}
}
lines.push('');
lines.push('Tip: probe reachability with `gbrain models doctor` (opt-in; spends a minimal request per configured chat/embed/rerank surface).');
return lines.join('\n');
}
// ── Doctor (probe) mode ────────────────────────────────────────────
type ProbeStatus = 'ok' | 'model_not_found' | 'auth' | 'rate_limit' | 'network' | 'config' | 'unknown';
interface ProbeResult {
model: string;
touchpoint: 'chat' | 'expansion' | 'embedding_config' | 'embedding_reachability' | 'reranker_config';
status: ProbeStatus;
message: string;
elapsed_ms: number;
fix?: string;
}
function classifyError(err: unknown): { status: ProbeStatus; message: string } {
const msg = err instanceof Error ? err.message : String(err);
const lower = msg.toLowerCase();
if (/not_?found|does not exist|invalid_model|model.*invalid|404/.test(lower)) {
return { status: 'model_not_found', message: msg };
}
if (/auth|unauthor|401|403|api[_-]?key/.test(lower)) {
return { status: 'auth', message: msg };
}
if (/rate.?limit|429|too many/.test(lower)) {
return { status: 'rate_limit', message: msg };
}
if (/timeout|network|econn|fetch failed|enotfound/.test(lower)) {
return { status: 'network', message: msg };
}
return { status: 'unknown', message: msg };
}
/**
* Validate the configured embedding model + dims combo without spending tokens.
* Catches the bug class where a brain configured for Voyage with a missing or
* out-of-allowlist `embedding_dimensions` value would fail at first-embed with
* an opaque HTTP 400. Runs purely against local config + recipe metadata —
* zero network I/O.
*/
async function probeEmbeddingConfig(): Promise<ProbeResult> {
const start = Date.now();
const { getEmbeddingModel, getEmbeddingDimensions } = await import('../core/ai/gateway.ts');
const { parseModelId } = await import('../core/ai/model-resolver.ts');
const {
supportsVoyageOutputDimension, isValidVoyageOutputDim, VOYAGE_VALID_OUTPUT_DIMS,
supportsZeroEntropyDimension, isValidZeroEntropyDim, ZEROENTROPY_VALID_DIMS,
} = await import('../core/ai/dims.ts');
const modelStr = getEmbeddingModel();
const dims = getEmbeddingDimensions();
try {
const { providerId, modelId } = parseModelId(modelStr);
// Voyage flexible-dim check — the bug class that motivated this probe.
if (providerId === 'voyage' && supportsVoyageOutputDimension(modelId)) {
if (!isValidVoyageOutputDim(dims)) {
return {
model: modelStr,
touchpoint: 'embedding_config',
status: 'config',
message:
`embedding_dimensions=${dims} is not a valid Voyage output_dimension ` +
`for "${modelId}" (allowed: ${VOYAGE_VALID_OUTPUT_DIMS.join('/')}).`,
fix:
`gbrain config set embedding_dimensions <${VOYAGE_VALID_OUTPUT_DIMS.join('|')}>, ` +
`or switch to a fixed-dim Voyage model (e.g. voyage-3, voyage-3-lite).`,
elapsed_ms: Date.now() - start,
};
}
}
// ZeroEntropy zembed-1 flexible-dim check. Same bug class as Voyage:
// `embedding_model: zeroentropyai:zembed-1` configured without
// `embedding_dimensions` falls back to DEFAULT_EMBEDDING_DIMENSIONS=1536
// (an OpenAI default) which ZE doesn't accept.
if (providerId === 'zeroentropyai' && supportsZeroEntropyDimension(modelId)) {
if (!isValidZeroEntropyDim(dims)) {
return {
model: modelStr,
touchpoint: 'embedding_config',
status: 'config',
message:
`embedding_dimensions=${dims} is not a valid ZeroEntropy dimensions ` +
`for "${modelId}" (allowed: ${ZEROENTROPY_VALID_DIMS.join('/')}).`,
fix:
`gbrain config set embedding_dimensions <${ZEROENTROPY_VALID_DIMS.join('|')}>.`,
elapsed_ms: Date.now() - start,
};
}
}
return {
model: modelStr,
touchpoint: 'embedding_config',
status: 'ok',
message: `embedding_dimensions=${dims} ok for ${modelStr}`,
elapsed_ms: Date.now() - start,
};
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
const fix = err && typeof err === 'object' && 'fix' in err
? (err as { fix?: string }).fix
: undefined;
return {
model: modelStr,
touchpoint: 'embedding_config',
status: 'config',
message: msg,
fix,
elapsed_ms: Date.now() - start,
};
}
}
/**
* v0.40.6.1: resolve the reranker model the same way live search does, so
* doctor doesn't drift from the live path. Pre-v0.40.6.1 the probe read
* `getRerankerModel()` from the gateway, which is fed from
* `GBrainConfig.reranker_model` — a file-plane field nothing currently
* writes. Meanwhile live search resolves `search.reranker.model` via
* `resolveSearchMode()` (per-call > config-key > recipe > bundle default).
* The two paths could disagree silently: doctor says "not configured"
* while every `gbrain search` call is using a mode default. This helper
* walks the same chain live search does so doctor's verdict matches.
*
* Falls back to `getRerankerModel()` (gateway value) when the engine path
* fails, so doctor stays useful in degraded states.
*/
export async function resolveLiveRerankerModel(engine: BrainEngine): Promise<string | undefined> {
try {
const { loadSearchModeConfig, resolveSearchMode } = await import('../core/search/mode.ts');
const input = await loadSearchModeConfig(engine);
const resolved = resolveSearchMode(input);
return resolved.reranker_enabled ? resolved.reranker_model : undefined;
} catch {
const { getRerankerModel } = await import('../core/ai/gateway.ts');
return getRerankerModel();
}
}
/**
* Resolve the reranker timeout the same way live search does, via
* `loadSearchModeConfig` + `resolveSearchMode`. Precedence chain:
* per-call > `search.reranker.timeout_ms` config > recipe `default_timeout_ms` > mode bundle.
*
* Codex outside-voice (Pass 9 of the wave) caught the probe lying either way
* when the operator sets `search.reranker.timeout_ms`: the probe used the
* recipe default (30s for llama) while production search used the (lower)
* config value, so doctor reported reachable while production always
* timed out. Same fix shape as `resolveLiveRerankerModel`.
*/
export async function resolveLiveRerankerTimeoutMs(engine: BrainEngine): Promise<number> {
try {
const { loadSearchModeConfig, resolveSearchMode } = await import('../core/search/mode.ts');
const input = await loadSearchModeConfig(engine);
const resolved = resolveSearchMode(input);
return resolved.reranker_timeout_ms ?? 5000;
} catch {
return 5000;
}
}
/**
* v0.35.0.0+: zero-network reranker config probe. Validates that the
* configured reranker model resolves through the recipe registry, that the
* recipe declares a `reranker` touchpoint, and that the model is in the
* touchpoint's `models[]` allowlist.
*
* CDX2-F11: `assertTouchpoint()` does NOT enforce allowlists for
* openai-compatible recipes — the probe does it directly here. Without
* this, `search.reranker.model=zeroentropyai:made-up-name` would silently
* pass config probes and fail at first rerank call.
*
* v0.40.6.1: resolves via `resolveLiveRerankerModel(engine)` so probe and
* live search read the same value (closes the file-plane / DB-plane
* divergence flagged in plan review).
*
* Returns 'ok' when reranker is unconfigured (default state — opt-in
* feature). Surfaces `status: 'config'` with paste-ready fix hint when
* model is invalid.
*/
async function probeRerankerConfig(engine: BrainEngine): Promise<ProbeResult> {
const start = Date.now();
const { resolveRecipe } = await import('../core/ai/model-resolver.ts');
const modelStr = await resolveLiveRerankerModel(engine);
if (!modelStr) {
// Reranker not configured. Default state for fresh installs and any
// brain that hasn't opted in. Not an error; doctor reports 'ok' so the
// probe row is informational.
return {
model: '(none)',
touchpoint: 'reranker_config',
status: 'ok',
message: 'reranker not configured (set `gbrain config set search.reranker.model <provider:model>` and `search.reranker.enabled true`)',
elapsed_ms: Date.now() - start,
};
}
try {
const { parsed, recipe } = resolveRecipe(modelStr);
const tp = recipe.touchpoints.reranker;
if (!tp) {
return {
model: modelStr,
touchpoint: 'reranker_config',
status: 'config',
message: `Provider "${recipe.id}" does not declare a reranker touchpoint.`,
fix: 'Switch to a provider that does (e.g. zeroentropyai:zerank-2).',
elapsed_ms: Date.now() - start,
};
}
if (tp.models.length > 0 && !tp.models.includes(parsed.modelId)) {
return {
model: modelStr,
touchpoint: 'reranker_config',
status: 'config',
message: `Model "${parsed.modelId}" is not in ${recipe.name}'s reranker allowlist.`,
fix: `gbrain config set search.reranker.model ${recipe.id}:<one of ${tp.models.join('|')}>`,
elapsed_ms: Date.now() - start,
};
}
return {
model: modelStr,
touchpoint: 'reranker_config',
status: 'ok',
message: `reranker configured: ${modelStr}`,
elapsed_ms: Date.now() - start,
};
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
return {
model: modelStr,
touchpoint: 'reranker_config',
status: 'config',
message: msg,
elapsed_ms: Date.now() - start,
};
}
}
/**
* v0.35.0.0+: 1-doc reachability probe. Sends a real `POST <recipe path>`
* with `{query, documents: [doc]}` so the probe actually verifies the
* server is in reranking mode (not just alive). For llama.cpp specifically,
* `--reranking` is mutually exclusive with `--embeddings`, and a server in
* embedding mode would 404/501 the rerank path — which this probe catches
* via classifyError().
*
* Returns 'ok' silently when reranker is unconfigured (no probe needed) —
* probeRerankerConfig already surfaced the missing-config state.
*
* v0.40.6.1: uses the resolved live model (same path live search uses),
* and reads the per-call timeout from the recipe's `default_timeout_ms`
* when set — so a CPU-only local reranker's cold-start warmup doesn't
* cause the probe to false-fail with `network`/timeout.
*/
async function probeRerankerReachability(engine: BrainEngine): Promise<ProbeResult | null> {
const modelStr = await resolveLiveRerankerModel(engine);
if (!modelStr) return null;
// Use the same timeout resolution live search uses: per-call > config >
// recipe > bundle. Pre-fix the probe read only the recipe default, so an
// operator who set `search.reranker.timeout_ms=1000` would see doctor wait
// 30s and report reachable while production search timed out at 1s
// (codex Pass-9 finding). resolveLiveRerankerTimeoutMs reuses the full
// precedence chain via mode.ts.
const probeTimeoutMs = await resolveLiveRerankerTimeoutMs(engine);
const start = Date.now();
try {
const { rerank } = await import('../core/ai/gateway.ts');
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(new Error(`probe timed out after ${probeTimeoutMs}ms`)), probeTimeoutMs);
try {
await rerank({
model: modelStr,
query: 'probe',
documents: ['probe document'],
signal: controller.signal,
timeoutMs: probeTimeoutMs,
});
return {
model: modelStr,
touchpoint: 'reranker_config',
status: 'ok',
message: 'reachable',
elapsed_ms: Date.now() - start,
};
} finally {
clearTimeout(timeoutId);
}
} catch (err) {
const { status, message } = classifyError(err);
return {
model: modelStr,
touchpoint: 'reranker_config',
status,
message,
elapsed_ms: Date.now() - start,
};
}
}
/**
* v0.40.x: embedding reachability probe. Mirrors probeRerankerReachability —
* sends a real 1-input `embed(['probe'])` to verify the configured embedding
* provider actually answers (auth + URL + model loaded). probeEmbeddingConfig
* is zero-network and only validates dims/recipe shape; for LOCAL providers
* (ollama, llama-server) it can't tell whether the server is up, so a dead or
* embedding-mode-off endpoint was previously only discovered at first real
* embed. Caller gates this on probeEmbeddingConfig returning 'ok' so a config
* failure isn't reported twice.
*
* Cold-start note: a local CPU embedder loading a model on first call can take
* several seconds; the 5s timeout may trip on the very first probe. Re-run if so.
*/
async function probeEmbeddingReachability(): Promise<ProbeResult | null> {
const { getEmbeddingModel, embed } = await import('../core/ai/gateway.ts');
const modelStr = getEmbeddingModel();
if (!modelStr) return null;
const start = Date.now();
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(new Error('probe timed out after 5s')), 5000);
try {
await embed(['probe'], { inputType: 'query', abortSignal: controller.signal });
return {
model: modelStr,
touchpoint: 'embedding_reachability',
status: 'ok',
message: 'reachable',
elapsed_ms: Date.now() - start,
};
} catch (err) {
const { status, message } = classifyError(err);
return {
model: modelStr,
touchpoint: 'embedding_reachability',
status,
message,
elapsed_ms: Date.now() - start,
};
} finally {
clearTimeout(timeoutId);
}
}
async function probeModel(modelStr: string, touchpoint: 'chat' | 'expansion'): Promise<ProbeResult> {
const start = Date.now();
try {
const { chat } = await import('../core/ai/gateway.ts');
// Use AbortController so the 5s timeout doesn't hang on a stuck network.
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(new Error('probe timed out after 5s')), 5000);
try {
await chat({
model: modelStr,
messages: [{ role: 'user', content: '.' }],
maxTokens: 1,
abortSignal: controller.signal,
});
return { model: modelStr, touchpoint, status: 'ok', message: 'reachable', elapsed_ms: Date.now() - start };
} finally {
clearTimeout(timeoutId);
}
} catch (err) {
const { status, message } = classifyError(err);
return { model: modelStr, touchpoint, status, message, elapsed_ms: Date.now() - start };
}
}
function shouldSkipProvider(modelStr: string, skip: string[]): boolean {
if (skip.length === 0) return false;
const colon = modelStr.indexOf(':');
const provider = colon === -1 ? '' : modelStr.slice(0, colon).toLowerCase();
return skip.includes(provider);
}
export async function runModels(engine: BrainEngine, args: string[]): Promise<void> {
const json = args.includes('--json');
const sub = args[1] === 'doctor' ? 'doctor' : args[1] === 'help' || args.includes('--help') || args.includes('-h') ? 'help' : 'read';
if (sub === 'help') {
process.stdout.write(
`Usage:
gbrain models Show routing table (read-only)
gbrain models doctor [flags] Probe each configured model (~1 token each)
gbrain models --json Machine-readable output
Flags (doctor only):
--skip=<provider> Skip a provider (e.g. --skip=openai)
Repeatable: --skip=openai --skip=google
--json JSON output
Configure routing:
gbrain config set models.default <model> # global hammer
gbrain config set models.tier.<tier> <model> # per-tier (utility/reasoning/deep/subagent)
gbrain config set models.aliases.<name> <model> # custom alias
Tiers: utility (haiku-class) | reasoning (sonnet) | deep (opus) | subagent (Anthropic-only)
`);
return;
}
if (sub === 'read') {
const report = await buildReport(engine);
if (json) {
process.stdout.write(JSON.stringify(report, null, 2) + '\n');
} else {
process.stdout.write(formatText(report) + '\n');
}
return;
}
// doctor mode
const skipArgs = args.filter(a => a.startsWith('--skip='));
const skip = skipArgs.map(a => a.slice('--skip='.length).toLowerCase()).filter(Boolean);
const { getChatModel, getExpansionModel } = await import('../core/ai/gateway.ts');
const chatModel = getChatModel();
const expansionModel = getExpansionModel();
const results: ProbeResult[] = [];
// Config-only probe runs first: zero tokens, catches the bug class where a
// brain misconfigured for Voyage with the wrong embedding_dimensions would
// 400 on first embed. Fast feedback before we spend a single token.
const embeddingConfig = await probeEmbeddingConfig();
results.push(embeddingConfig);
// v0.35.0.0+ reranker config probe — same zero-network model as embedding.
// v0.40.6.1: takes the engine so it can read the same `search.reranker.*`
// config keys live search reads (closes file-plane / DB-plane divergence).
results.push(await probeRerankerConfig(engine));
for (const [modelStr, touchpoint] of [[chatModel, 'chat'], [expansionModel, 'expansion']] as const) {
if (shouldSkipProvider(modelStr, skip)) {
if (!json) process.stderr.write(`[skip] ${touchpoint}: ${modelStr} (provider in --skip)\n`);
continue;
}
results.push(await probeModel(modelStr, touchpoint));
}
// v0.40.x: embedding reachability — only when the config probe passed
// (codex #8: a config failure shouldn't be reported twice) AND the provider
// isn't in --skip. Catches a dead/misconfigured LOCAL embed server early.
if (embeddingConfig.status === 'ok' && !shouldSkipProvider(embeddingConfig.model, skip)) {
const er = await probeEmbeddingReachability();
if (er) results.push(er);
}
// v0.40.6.1: reranker reachability uses the live-search resolution path
// (file-plane / DB-plane divergence fix); only fires when reranker is
// actually enabled per the resolved mode bundle.
const liveRerankerModel = await resolveLiveRerankerModel(engine);
if (liveRerankerModel && !shouldSkipProvider(liveRerankerModel, skip)) {
const r = await probeRerankerReachability(engine);
if (r) results.push(r);
}
const report = {
schema_version: 1 as const,
probes: results,
summary: {
total: results.length,
ok: results.filter(r => r.status === 'ok').length,
failed: results.filter(r => r.status !== 'ok').length,
},
};
if (json) {
process.stdout.write(JSON.stringify(report, null, 2) + '\n');
} else {
process.stdout.write('Model reachability probe:\n');
for (const r of results) {
const icon = r.status === 'ok' ? '✔' : '✘';
process.stdout.write(` ${icon} ${r.touchpoint.padEnd(17)} ${r.model.padEnd(50)} ${r.status} (${r.elapsed_ms}ms)\n`);
if (r.status !== 'ok') {
process.stdout.write(` ${r.message}\n`);
if (r.fix) process.stdout.write(` fix: ${r.fix}\n`);
}
}
process.stdout.write(`\nSummary: ${report.summary.ok}/${report.summary.total} reachable.\n`);
}
if (report.summary.failed > 0) {
process.exit(1);
}
}