mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat: embedding_multimodal_model — separate model routing for multimodal embeddings v0.28.9 shipped multimodal image embeddings via Voyage, but embedMultimodal() hardcodes to the primary embedding_model. Brains using OpenAI text-embedding-3-large (1536-dim) for text cannot use Voyage voyage-multimodal-3 (1024-dim) for images without switching their entire embedding pipeline. This adds embedding_multimodal_model as a distinct config key that embedMultimodal() prefers over embedding_model when set. The dual- column schema (embedding vs embedding_image) already supports different dimensions — this patch completes the routing. Config surface: - gbrain config set embedding_multimodal_model voyage:voyage-multimodal-3 - env: GBRAIN_EMBEDDING_MULTIMODAL_MODEL=voyage:voyage-multimodal-3 Files changed: - core/ai/types.ts: AIGatewayConfig gains embedding_multimodal_model - core/ai/gateway.ts: configureGateway stores it; embedMultimodal reads it - core/config.ts: GBrainConfig type + env loader + DB merge path - cli.ts: threads config into gateway; reconfigures after DB merge Tested on a 96K-page brain with OpenAI text + Voyage multimodal running side by side. Voyage returns 1024-dim vectors into embedding_image column; text embeddings unchanged. * refactor(cli): extract buildGatewayConfig + always re-config after DB merge Two related changes co-located so the un-gate doesn't leave the duplicated configureGateway shapes drifting: 1. Extract file-local `buildGatewayConfig(c: GBrainConfig): AIGatewayConfig` helper. Both configureGateway sites in connectEngine() now pass through it; future fields touch one place. 2. Drop the field-name-gated re-config trigger. The previous gate fired only when `merged.embedding_multimodal_model` was truthy, coupling the trigger to one field name. Future DB-mutable gateway fields would silently miss it. Re-config now always fires when loadConfigWithEngine returns non-null. One extra cache+shrinkState clear per startup is microseconds, no hot path. Schema-sizing fields stay stable because loadConfigWithEngine respects file/env first; merged.embedding_dimensions equals config.embedding_dimensions when no DB override exists. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai): model-level multimodal validation + getMultimodalModel accessor Codex review of PR #719 (F1) caught a real footgun: the Voyage recipe shares supports_multimodal: true across all 12 models in its embedding touchpoint, of which only voyage-multimodal-3 is valid at /multimodalembeddings. A user setting embedding_multimodal_model to a text-only Voyage model (e.g. voyage-3-large) passes local validation and fails at the endpoint with HTTP 400 — which gateway.ts:626 misclassifies as transient (TODO: reclassify, tracked in TODOS.md). Adds: - EmbeddingTouchpoint.multimodal_models?: string[] (optional, model-level allow-list inside a recipe that mixes text-only + multimodal models). - Voyage declares multimodal_models: ['voyage-multimodal-3']. - embedMultimodal() validates parsed.modelId against the allow-list AFTER the existing recipe-level supports_multimodal check. Throws AIConfigError with the full multimodal_models list in the fix hint. - getMultimodalModel() public accessor mirroring getEmbeddingModel / getChatModel — needed by the cli-multimodal-integration test and useful for future doctor checks. Recipe-level fast-fail stays so non-multimodal providers (Anthropic / OpenAI today) keep their AIConfigError path unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cover embedding_multimodal_model precedence + gateway override + cli integration PR #719 originally shipped zero tests for the new code paths. Closes that gap with three layers: 1. test/loadConfig-merge.test.ts — extends the existing env > file > DB precedence pattern (which already covers embedding_image_ocr_model) with four cases for embedding_multimodal_model: DB-only fills in, file wins over DB, all-unset stays undefined, null/empty DB ignored. 2. test/voyage-multimodal.test.ts — four cases for embedMultimodal model resolution: prefers multimodal_model over embedding_model, falls back to embedding_model when unset (regression guard), AIConfigError on non-multimodal recipe, AIConfigError on Voyage text-only model (Codex F1 model-level validation). 3. test/cli-multimodal-integration.test.ts (NEW) — three PGLite-based integration tests for the cli.ts re-config glue itself (Codex F3: the actual bug site that "mechanical glue" claims hide). Drives the loadConfigWithEngine + buildGatewayConfig + configureGateway sequence connectEngine() runs and asserts the gateway observed the DB-set value. 11 new test cases total. All pass against the production code in this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(todos): follow-ups from PR #719 codex review Three items surfaced during /codex outside-voice review of PR #719's plan that are out of scope for the current PR but worth tracking: - gbrain doctor: warn on misconfigured multimodal model (P2). Two checks: multimodal_model set without recipe API key; embedding_multimodal flag on without a multimodal-capable embedding_model. - Reclassify Voyage HTTP 4xx as AIConfigError (P2, Codex F2). Today gateway.ts:626 throws AITransientError for any non-401/403 4xx, so permanent config bugs (malformed body, model not in multimodal_models) trigger retry storms. Aligns with normalizeAIError's contract. - gbrain config unset <key> (P3, Codex F6). Once a user sets a key in DB there's no normal CLI path to clear it. Pre-existing UX gap; PR #719's new key surfaces it again. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.28.11) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: v0.28.11 annotations for ai/types, ai/gateway, voyage recipe Updates the Key Files section so the per-file annotations reflect the multimodal_model routing + model-level validation that landed in #719. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
125 lines
4.9 KiB
TypeScript
125 lines
4.9 KiB
TypeScript
// v0.28.11 (PR #719, D5): cli connectEngine() DB→gateway plumbing.
|
|
//
|
|
// The unit tests for loadConfigWithEngine and embedMultimodal cover their
|
|
// own contracts but don't exercise the cli.ts glue that ties them together.
|
|
// Codex F3 flagged this as the actual bug site. This file drives the same
|
|
// merge + reconfigure sequence connectEngine() runs and asserts the gateway
|
|
// observed the DB-set value through buildGatewayConfig.
|
|
//
|
|
// PGLite-only: in-memory engine, no DATABASE_URL needed.
|
|
|
|
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
|
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
|
import { loadConfigWithEngine, type GBrainConfig } from '../src/core/config.ts';
|
|
import {
|
|
configureGateway,
|
|
getEmbeddingModel,
|
|
getMultimodalModel,
|
|
resetGateway,
|
|
} from '../src/core/ai/gateway.ts';
|
|
import type { AIGatewayConfig } from '../src/core/ai/types.ts';
|
|
|
|
// Mirror the cli.ts buildGatewayConfig helper exactly. Keeping a copy here
|
|
// (instead of exporting from cli.ts) is intentional: the test asserts the
|
|
// shape of the contract, not the helper's identity. If cli.ts drifts, the
|
|
// e2e behavior these tests care about (DB-set value lands in gateway) still
|
|
// holds, but a helper-shape test would also catch the drift in PR review.
|
|
function buildGatewayConfig(c: GBrainConfig): AIGatewayConfig {
|
|
return {
|
|
embedding_model: c.embedding_model,
|
|
embedding_dimensions: c.embedding_dimensions,
|
|
embedding_multimodal_model: c.embedding_multimodal_model,
|
|
expansion_model: c.expansion_model,
|
|
chat_model: c.chat_model,
|
|
chat_fallback_chain: c.chat_fallback_chain,
|
|
base_urls: c.provider_base_urls,
|
|
env: { ...process.env },
|
|
};
|
|
}
|
|
|
|
let engine: PGLiteEngine;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({});
|
|
await engine.initSchema();
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
resetGateway();
|
|
// Clear any prior config rows so tests are independent. setConfig with
|
|
// empty string is treated as undefined by loadConfigWithEngine (per
|
|
// dbStr semantics), so this is safe to call between tests.
|
|
await engine.setConfig('embedding_multimodal_model', '');
|
|
});
|
|
|
|
describe('cli connectEngine — embedding_multimodal_model DB→gateway plumbing', () => {
|
|
test('DB-set multimodal_model flows to gateway via merge + reconfigure', async () => {
|
|
await engine.setConfig('embedding_multimodal_model', 'voyage:voyage-multimodal-3');
|
|
|
|
const baseConfig: GBrainConfig = {
|
|
engine: 'pglite',
|
|
embedding_model: 'openai:text-embedding-3-large',
|
|
embedding_dimensions: 1536,
|
|
};
|
|
|
|
// First call mirrors cli.ts:715 (pre-engine-connect).
|
|
configureGateway(buildGatewayConfig(baseConfig));
|
|
expect(getMultimodalModel()).toBeUndefined();
|
|
|
|
// Second call mirrors cli.ts:776 (post-DB-merge).
|
|
const merged = await loadConfigWithEngine(engine, baseConfig);
|
|
expect(merged).not.toBeNull();
|
|
configureGateway(buildGatewayConfig(merged!));
|
|
|
|
// Primary embedding_model stays put (file/env wins); multimodal_model
|
|
// arrived via DB.
|
|
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
|
|
expect(getMultimodalModel()).toBe('voyage:voyage-multimodal-3');
|
|
});
|
|
|
|
test('file value wins over DB value (env > file > DB precedence at gateway level)', async () => {
|
|
await engine.setConfig('embedding_multimodal_model', 'voyage:voyage-3-large');
|
|
|
|
const baseConfig: GBrainConfig = {
|
|
engine: 'pglite',
|
|
embedding_model: 'openai:text-embedding-3-large',
|
|
embedding_dimensions: 1536,
|
|
embedding_multimodal_model: 'voyage:voyage-multimodal-3', // file plane
|
|
};
|
|
|
|
const merged = await loadConfigWithEngine(engine, baseConfig);
|
|
configureGateway(buildGatewayConfig(merged!));
|
|
|
|
expect(getMultimodalModel()).toBe('voyage:voyage-multimodal-3');
|
|
});
|
|
|
|
test('un-gated re-config: merged DB has no multimodal_model → gateway still gets re-configured', async () => {
|
|
// Codex F5 was about whether the un-gated re-config weakens an
|
|
// intentional contract. This test pins the actual behavior (D6 = B):
|
|
// re-config always fires when merge succeeds, even when no DB key
|
|
// changed. Schema-sizing fields stay stable because loadConfigWithEngine
|
|
// respects file/env first.
|
|
const baseConfig: GBrainConfig = {
|
|
engine: 'pglite',
|
|
embedding_model: 'openai:text-embedding-3-large',
|
|
embedding_dimensions: 1536,
|
|
};
|
|
|
|
configureGateway(buildGatewayConfig(baseConfig));
|
|
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
|
|
|
|
const merged = await loadConfigWithEngine(engine, baseConfig);
|
|
configureGateway(buildGatewayConfig(merged!));
|
|
|
|
// Primary model unchanged (DB had no override); the re-config is a
|
|
// semantic no-op for these fields.
|
|
expect(getEmbeddingModel()).toBe('openai:text-embedding-3-large');
|
|
expect(getMultimodalModel()).toBeUndefined();
|
|
});
|
|
});
|