Files
gbrain/test/cli-multimodal-integration.test.ts
T
bfab1ded08 v0.28.11 feat: embedding_multimodal_model — separate model routing for multimodal embeddings (#719)
* feat: embedding_multimodal_model — separate model routing for multimodal embeddings

v0.28.9 shipped multimodal image embeddings via Voyage, but
embedMultimodal() hardcodes to the primary embedding_model. Brains
using OpenAI text-embedding-3-large (1536-dim) for text cannot use
Voyage voyage-multimodal-3 (1024-dim) for images without switching
their entire embedding pipeline.

This adds embedding_multimodal_model as a distinct config key that
embedMultimodal() prefers over embedding_model when set. The dual-
column schema (embedding vs embedding_image) already supports
different dimensions — this patch completes the routing.

Config surface:
  - gbrain config set embedding_multimodal_model voyage:voyage-multimodal-3
  - env: GBRAIN_EMBEDDING_MULTIMODAL_MODEL=voyage:voyage-multimodal-3

Files changed:
  - core/ai/types.ts: AIGatewayConfig gains embedding_multimodal_model
  - core/ai/gateway.ts: configureGateway stores it; embedMultimodal reads it
  - core/config.ts: GBrainConfig type + env loader + DB merge path
  - cli.ts: threads config into gateway; reconfigures after DB merge

Tested on a 96K-page brain with OpenAI text + Voyage multimodal
running side by side. Voyage returns 1024-dim vectors into
embedding_image column; text embeddings unchanged.

* refactor(cli): extract buildGatewayConfig + always re-config after DB merge

Two related changes co-located so the un-gate doesn't leave the duplicated
configureGateway shapes drifting:

1. Extract file-local `buildGatewayConfig(c: GBrainConfig): AIGatewayConfig`
   helper. Both configureGateway sites in connectEngine() now pass through
   it; future fields touch one place.

2. Drop the field-name-gated re-config trigger. The previous gate fired
   only when `merged.embedding_multimodal_model` was truthy, coupling the
   trigger to one field name. Future DB-mutable gateway fields would
   silently miss it. Re-config now always fires when loadConfigWithEngine
   returns non-null. One extra cache+shrinkState clear per startup is
   microseconds, no hot path.

Schema-sizing fields stay stable because loadConfigWithEngine respects
file/env first; merged.embedding_dimensions equals config.embedding_dimensions
when no DB override exists.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ai): model-level multimodal validation + getMultimodalModel accessor

Codex review of PR #719 (F1) caught a real footgun: the Voyage recipe
shares supports_multimodal: true across all 12 models in its embedding
touchpoint, of which only voyage-multimodal-3 is valid at
/multimodalembeddings. A user setting embedding_multimodal_model to a
text-only Voyage model (e.g. voyage-3-large) passes local validation and
fails at the endpoint with HTTP 400 — which gateway.ts:626 misclassifies
as transient (TODO: reclassify, tracked in TODOS.md).

Adds:
- EmbeddingTouchpoint.multimodal_models?: string[] (optional, model-level
  allow-list inside a recipe that mixes text-only + multimodal models).
- Voyage declares multimodal_models: ['voyage-multimodal-3'].
- embedMultimodal() validates parsed.modelId against the allow-list AFTER
  the existing recipe-level supports_multimodal check. Throws AIConfigError
  with the full multimodal_models list in the fix hint.
- getMultimodalModel() public accessor mirroring getEmbeddingModel /
  getChatModel — needed by the cli-multimodal-integration test and useful
  for future doctor checks.

Recipe-level fast-fail stays so non-multimodal providers (Anthropic /
OpenAI today) keep their AIConfigError path unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: cover embedding_multimodal_model precedence + gateway override + cli integration

PR #719 originally shipped zero tests for the new code paths. Closes
that gap with three layers:

1. test/loadConfig-merge.test.ts — extends the existing env > file > DB
   precedence pattern (which already covers embedding_image_ocr_model)
   with four cases for embedding_multimodal_model: DB-only fills in,
   file wins over DB, all-unset stays undefined, null/empty DB ignored.

2. test/voyage-multimodal.test.ts — four cases for embedMultimodal model
   resolution: prefers multimodal_model over embedding_model, falls back
   to embedding_model when unset (regression guard), AIConfigError on
   non-multimodal recipe, AIConfigError on Voyage text-only model
   (Codex F1 model-level validation).

3. test/cli-multimodal-integration.test.ts (NEW) — three PGLite-based
   integration tests for the cli.ts re-config glue itself (Codex F3:
   the actual bug site that "mechanical glue" claims hide). Drives the
   loadConfigWithEngine + buildGatewayConfig + configureGateway sequence
   connectEngine() runs and asserts the gateway observed the DB-set value.

11 new test cases total. All pass against the production code in this
PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(todos): follow-ups from PR #719 codex review

Three items surfaced during /codex outside-voice review of PR #719's plan
that are out of scope for the current PR but worth tracking:

- gbrain doctor: warn on misconfigured multimodal model (P2). Two checks:
  multimodal_model set without recipe API key; embedding_multimodal flag
  on without a multimodal-capable embedding_model.

- Reclassify Voyage HTTP 4xx as AIConfigError (P2, Codex F2). Today
  gateway.ts:626 throws AITransientError for any non-401/403 4xx, so
  permanent config bugs (malformed body, model not in multimodal_models)
  trigger retry storms. Aligns with normalizeAIError's contract.

- gbrain config unset <key> (P3, Codex F6). Once a user sets a key in DB
  there's no normal CLI path to clear it. Pre-existing UX gap; PR #719's
  new key surfaces it again.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.28.11)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v0.28.11 annotations for ai/types, ai/gateway, voyage recipe

Updates the Key Files section so the per-file annotations reflect the
multimodal_model routing + model-level validation that landed in #719.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 13:41:46 -07:00

125 lines
4.9 KiB
TypeScript

// v0.28.11 (PR #719, D5): cli connectEngine() DB→gateway plumbing.
//
// The unit tests for loadConfigWithEngine and embedMultimodal cover their
// own contracts but don't exercise the cli.ts glue that ties them together.
// Codex F3 flagged this as the actual bug site. This file drives the same
// merge + reconfigure sequence connectEngine() runs and asserts the gateway
// observed the DB-set value through buildGatewayConfig.
//
// PGLite-only: in-memory engine, no DATABASE_URL needed.
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { loadConfigWithEngine, type GBrainConfig } from '../src/core/config.ts';
import {
configureGateway,
getEmbeddingModel,
getMultimodalModel,
resetGateway,
} from '../src/core/ai/gateway.ts';
import type { AIGatewayConfig } from '../src/core/ai/types.ts';
// Mirror the cli.ts buildGatewayConfig helper exactly. Keeping a copy here
// (instead of exporting from cli.ts) is intentional: the test asserts the
// shape of the contract, not the helper's identity. If cli.ts drifts, the
// e2e behavior these tests care about (DB-set value lands in gateway) still
// holds, but a helper-shape test would also catch the drift in PR review.
function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
return {
embedding_model: c.embedding_model,
embedding_dimensions: c.embedding_dimensions,
embedding_multimodal_model: c.embedding_multimodal_model,
expansion_model: c.expansion_model,
chat_model: c.chat_model,
chat_fallback_chain: c.chat_fallback_chain,
base_urls: c.provider_base_urls,
env: { ...process.env },
};
}
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
resetGateway();
// Clear any prior config rows so tests are independent. setConfig with
// empty string is treated as undefined by loadConfigWithEngine (per
// dbStr semantics), so this is safe to call between tests.
await engine.setConfig('embedding_multimodal_model', '');
});
describe('cli connectEngine — embedding_multimodal_model DB→gateway plumbing', () => {
test('DB-set multimodal_model flows to gateway via merge + reconfigure', async () => {
await engine.setConfig('embedding_multimodal_model', 'voyage:voyage-multimodal-3');
const baseConfig: GBrainConfig = {
engine: 'pglite',
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: 1536,
};
// First call mirrors cli.ts:715 (pre-engine-connect).
configureGateway(buildGatewayConfig(baseConfig));
expect(getMultimodalModel()).toBeUndefined();
// Second call mirrors cli.ts:776 (post-DB-merge).
const merged = await loadConfigWithEngine(engine, baseConfig);
expect(merged).not.toBeNull();
configureGateway(buildGatewayConfig(merged!));
// Primary embedding_model stays put (file/env wins); multimodal_model
// arrived via DB.
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
expect(getMultimodalModel()).toBe('voyage:voyage-multimodal-3');
});
test('file value wins over DB value (env > file > DB precedence at gateway level)', async () => {
await engine.setConfig('embedding_multimodal_model', 'voyage:voyage-3-large');
const baseConfig: GBrainConfig = {
engine: 'pglite',
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: 1536,
embedding_multimodal_model: 'voyage:voyage-multimodal-3', // file plane
};
const merged = await loadConfigWithEngine(engine, baseConfig);
configureGateway(buildGatewayConfig(merged!));
expect(getMultimodalModel()).toBe('voyage:voyage-multimodal-3');
});
test('un-gated re-config: merged DB has no multimodal_model → gateway still gets re-configured', async () => {
// Codex F5 was about whether the un-gated re-config weakens an
// intentional contract. This test pins the actual behavior (D6 = B):
// re-config always fires when merge succeeds, even when no DB key
// changed. Schema-sizing fields stay stable because loadConfigWithEngine
// respects file/env first.
const baseConfig: GBrainConfig = {
engine: 'pglite',
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: 1536,
};
configureGateway(buildGatewayConfig(baseConfig));
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
const merged = await loadConfigWithEngine(engine, baseConfig);
configureGateway(buildGatewayConfig(merged!));
// Primary model unchanged (DB had no override); the re-config is a
// semantic no-op for these fields.
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
expect(getMultimodalModel()).toBeUndefined();
});
});