mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.36.6.0 feat: cross-modal search wave (text↔image + unified column + LLM intent) (#1165)
* feat(cross-modal/0): batched multimodal + query helpers + SSRF helper
Commit 0 of the cross-modal search wave. Foundation for Phase 1-3:
- embedMultimodal accepts MultimodalInput text variant + EmbedMultimodalOpts
with inputType: 'document' | 'query' (D22-2). Default unchanged so
importImageFile keeps document-side embedding.
- embedQueryMultimodal(text) + embedQueryMultimodalImage(input) wrappers
for hybridSearch + searchByImage query paths.
- embedMultimodalSafe binary-search retry on transient batch failure +
failed_indices surfacing. Phase 3 reindex uses this so a single bad
chunk doesn't discard the 31 in-flight embeddings around it.
- Voyage path: text + image inputs in one batch via content arrays.
- openai-compat path: text + image inputs in one request per input.
- src/core/ssrf-validate.ts (D19): DNS-resolve-and-fetch-by-IP defense
for redirect chains. Closes the DNS-rebinding gap that url-safety.ts'
static check leaves open. Uses node:dns/promises with {all: true,
family: 0} to inspect every A and AAAA record before connecting.
fetchWithSSRFGuard helper validates per-redirect-hop and limits chain
depth (default 3).
- Re-exports from src/core/embedding.ts public seam.
Tests:
- test/embed-multimodal-batching.test.ts (13 cases): text variant, query
inputType discipline, mixed text+image batches, embedQueryMultimodal,
embedQueryMultimodalImage, embedMultimodalSafe happy/empty/all-fail/
mid-batch-recovery/permanent-misconfig.
- test/ssrf-validate.test.ts (20 cases): static rejections via
isInternalUrl, scheme + credentials rejection, DNS rebinding defense
(single-record + multi-record), public happy path, IPv6 literals,
malformed URLs.
No regression in existing voyage-multimodal.test.ts or
openai-compat-multimodal.test.ts (33 cases all pass).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cross-modal/1): Phase 1 text→image routing + knobsHash + RRF + backfill
Phase 1 of the cross-modal search wave. Wires the existing 1024d Voyage
multimodal embedding space (already populated for image chunks via
importImageFile) into the user-facing query path. Text queries that match
cross-modal intent regex route through Voyage multimodal-3 instead of the
text embedding model, then search content_chunks.embedding_image.
- query-intent.ts: new `suggestedModality: 'text' | 'image' | 'both'`
axis on `QuerySuggestions`. Module-scope CROSS_MODAL_PATTERNS regex
array (D15 — compiled once at module load). Conservative on purpose;
LLM intent escalation (Commit 4) catches genuinely ambiguous phrasings.
- query-intent.ts: new `isAmbiguousModalityQuery(query)` pure heuristic
for Commit 4's escalation gate. Returns true ONLY when regex misses
AND a visual noun + reference marker both fire.
- types.ts: `SearchOpts.crossModal: 'text' | 'image' | 'both' | 'auto'`
+ `SearchResult.modality: 'text' | 'image'` for downstream renderers.
- mode.ts: 7 new knobs in ModeBundle (D2): cross_modal_both_text_weight,
cross_modal_both_image_weight, image_query_text_refinement_weight,
image_query_image_refinement_weight, unified_multimodal,
unified_multimodal_only, cross_modal_llm_intent. All three mode
bundles default to the same values (cross-modal is opt-in).
- mode.ts: D2 cache-key fix — KNOBS_HASH_VERSION bumped 2→3, all 7 new
knobs participate in knobsHash so a text-mode cache hit can't be
served to an image-mode caller.
- mode.ts: D3 registry — all 7 keys land in SEARCH_MODE_CONFIG_KEYS so
`gbrain search modes` / `stats` / `tune` see them.
- hybrid.ts: routing branch at the embed step. Resolves effective
modality from (per-call opts → suggestions → 'text'). Image route:
embedQueryMultimodal + searchVector(embedding_image), skip expansion
+ keyword (D9 mode-bundle override). Both route: parallel text + image
vector searches merged via weighted RRF (D6) with cross_modal_both_*
weights. Fail-open: multimodal misconfigured → structured warn + text
fallback. 'auto' literal normalized to undefined (D22-1).
- operations.ts: thread `cross_modal` param through `query` op.
- backfill-registry.ts: new `modality` backfill kind. SQL filter requires
`chunk_source='image_asset'` (D22-7 defensive guard). Idempotent.
- doctor.ts: `cross_modal_modality_backfill` check surfaces unflagged
image-asset chunks with paste-ready `gbrain backfill modality` hint.
Tests:
- cross-modal-phase1.test.ts (45 cases): regex classification (positive
+ negative + plural-safe), isAmbiguousModalityQuery, D3 registry, D2
knobsHash diffs across all 7 new knobs, MODE_BUNDLES defaults,
resolveSearchMode precedence chain.
- cross-modal-hybrid-integration.test.ts (7 cases): PGLite + stubbed
gateway. Verifies image-modality calls Voyage and not OpenAI, text
calls OpenAI and not Voyage, 'auto' literal normalizes, 'both' mode
hits both endpoints, fail-open routes to text on multimodal misconfig.
- search-mode.test.ts: updated MODE_BUNDLES + KNOBS_HASH_VERSION
assertions (148 cross-suite tests still pass; no regression).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cross-modal/2): Phase 2 image-as-query + D18 path ban + D23-#6 spend cap
Phase 2 of the cross-modal search wave. Adds the `search_by_image` MCP op,
the SSRF-defended image loader, and the daily per-OAuth-client spend cap
on paid Voyage multimodal calls. D17 honest framing applied: Phase 2 ships
image→similar-images + image-OCR-text retrieval. True image→full-text-
knowledge requires Phase 3's unified column.
- src/core/search/image-loader.ts: loadImageInput accepts local path,
data: URI, or http(s):// URL. Magic-byte sniff for PNG/JPEG/WebP (no
other formats). Hard size cap (10MB local default, 2MB remote default).
http(s) path uses fetchWithSSRFGuard from Commit 0: every redirect hop
re-resolved via DNS lookup + every record checked against the internal
IP deny list. Max 3 redirect hops. 5s total fetch timeout. Pre-flight
Content-Length check + post-fetch size guard for lying servers.
- src/core/search/by-image.ts: searchByImage runs the image branch
always; D13 hybrid intersect runs a parallel text branch when
`query` is provided, merged via weighted RRF. Phase 3 will widen
the column routing to embedding_multimodal once that lands.
- src/core/operations.ts: new search_by_image op (scope: read, NOT
localOnly). D18 P0 — when ctx.remote === true AND image_path is set,
rejects with permission_denied at handler entry (validateParams would
catch it again at dispatch). D5 source-id thread via sourceScopeOpts.
D12 per-param length cap enforced via remote-vs-local maxBytes config
read at handler entry. D23-#6 pre-flight checkBudget + post-call
recordSpend (best-effort; failures don't block response).
- src/core/spend-log.ts: BudgetExceededError + checkBudget + recordSpend
+ getTodaySpendCents. UTC day-aligned aggregation so the cap rolls
over deterministically. Local CLI callers (no clientId) bypass the
gate entirely. Pre-v0.36 brains without the mcp_spend_log table fail
open to spend=0; the migration brings the table in on first start.
- src/core/migrate.ts: new migration v67 mcp_spend_log table + indexes
for the (client_id, day) and (token_name, day) hot reads. PGLite
parity via sqlFor.pglite.
- src/core/search/hybrid.ts: RRF_K constant exported so by-image.ts can
share the same effective-K math as the main hybrid path.
Tests:
- cross-modal-phase2.test.ts (15 cases): magic-byte sniffing (PNG +
JPEG + WebP positive, GIF rejection), oversized rejection (default +
custom cap), data: URI happy path + malformed + decoded-non-image
+ oversized, invalid input shapes (empty + ftp), SSRF defense via
DNS rebinding stub.
- search-by-image-op.test.ts (7 cases): D18 remote image_path
rejection + local CLI accepts; input validation (missing all three /
multiple together); D23-#6 budget block-at-cap + allow-under-cap +
local-CLI-bypass; migration v67 mcp_spend_log table applied cleanly.
All 166 tests across the cross-modal suite pass; no regression in
existing voyage-multimodal / openai-compat-multimodal / search-mode suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cross-modal/3): Phase 3 unified column + reindex + D8 fail-open + D23-#2
Phase 3 of the cross-modal search wave. Adds the unified multimodal column
on content_chunks + the `gbrain reindex --multimodal` sweep + the
`search.unified_multimodal` routing flag with D8 source-aware coverage
guard + fail-open behavior. D17 honest framing: this is the phase that
unlocks true image→full-text-knowledge — Phase 2's searchByImage
transparently upgrades to the richer retrieval once the unified column
has coverage.
D10 reindex-core extraction filed as a follow-up TODO. The existing
markdown reindex walks pages and re-imports via importFromFile; this
walks content_chunks and re-embeds via the gateway. Patterns rhyme but
cores diverge enough that extraction balloons the diff. Both commands
stand alone with their own checkpoint + cost-prompt logic.
- migrate.ts v68 (embedding_multimodal_column): column-only ALTER on
content_chunks. HNSW partial index deferred to post-reindex build
(D20: pgvector docs recommend post-load build for HNSW). Both engines.
- types.ts SearchOpts.embeddingColumn type widened to include
'embedding_multimodal'.
- postgres-engine.ts + pglite-engine.ts searchVector: route to
embedding_multimodal column when opts.embeddingColumn set. NO modality
filter (unified column carries both text + image content).
- hybrid.ts unified routing branch: when search.unified_multimodal=true,
bypasses dual-column branching and runs embedQueryMultimodal +
searchVector(embedding_multimodal). D8 fail-open: zero rows + not
strict-mode → falls through to dual-column text path with structured
warning. search.unified_multimodal_only=true bypasses the fallback.
- src/commands/reindex-multimodal.ts: `gbrain reindex --multimodal`.
D7 lock via tryAcquireDbLock('gbrain-reindex-multimodal'); 6h TTL.
Cost prompt + 10s Ctrl-C grace window in TTY; auto-proceeds non-TTY.
GBRAIN_NO_REEMBED=1 bypass. Checkpoint at
~/.gbrain/reindex-multimodal-checkpoint.json for resume. D23-#2
auto-flip prompt at coverage=100% completion.
- cli.ts: `gbrain reindex --multimodal` dispatch with --limit, --dry-run,
--cost-estimate, --no-embed, --yes, --json flags.
- doctor.ts: unified_multimodal_coverage check (D21 source-aware) +
reports per-source % when search.unified_multimodal is on. Warns at
<95% lowest source; fails when unified_multimodal_only=true AND
lowest source <99%. Falls open to OK when column not yet present.
Tests:
- unified-multimodal.test.ts (8 cases): schema migration v68 applies,
reindex --dry-run + --cost-estimate + GBRAIN_NO_REEMBED bypass +
zero-pending fast-path, hybridSearch unified routing forces voyage
endpoint, D8 fail-open routes to text on empty unified, D8 strict
blocks text fallback.
All 211 tests across the cross-modal + related suite pass; no
regression in voyage-multimodal / openai-compat-multimodal / search-mode
/ intent / search base suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cross-modal/4): LLM intent escalation for ambiguous modality
Commit 4 of the cross-modal search wave (opt-in default off).
When `search.cross_modal.llm_intent` is true AND the regex classifier
returned 'text' AND `isAmbiguousModalityQuery(query)` fires, hybridSearch
awaits a Haiku tie-break via gateway.chat() before routing. The
ambiguous-modality gate (introduced in Commit 1) ensures the LLM call
only fires on the narrow band where regex misses but a visual noun +
reference marker both fire — roughly <1% of queries with the flag on.
- src/core/search/llm-intent.ts: new module. `classifyModalityWithLLM`
routes through gateway.chat() with a fixed system prompt ("Output
exactly one word: text, image, or both"). 1s timeout via AbortController.
`parseModality` is a pure exported helper that tolerates trailing
punctuation + casing. Fail-open on every error path (gateway
unavailable, timeout, parse failure, unrecognized output).
- src/core/search/hybrid.ts: escalation branch slots BEFORE the unified
routing branch. Gated by: no explicit per-call crossModal opt, regex
result == 'text', config flag on, ambiguity heuristic fires. Fail-open
to regex result on any error from the LLM tie-break.
Tests:
- llm-intent-escalation.test.ts (14 cases): parseModality tolerance
matrix (text / image / both / trailing punct / whitespace /
unrecognized / empty), classifyModalityWithLLM happy paths for all 3
outputs, fail-open on throw / unrecognized output / gateway-not-
configured, explicit-fallback-honored.
- llm-intent-hybrid-integration.test.ts (6 cases): hybridSearch
escalation gate fires ONLY when flag-on + ambiguous; off when flag-off,
unambiguous, regex-confident, or explicit per-call opt set; fail-open
on LLM throw.
All 231 tests across the cross-modal + related suite pass; no
regression in voyage-multimodal / openai-compat-multimodal /
search-mode / intent / search base suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cross-modal/3): verify-gate fixes for full test suite
Three small fixes to pass the full unit + E2E sweep after the cross-modal
wave commits land.
- migrate.ts v67: drop date_trunc('day', created_at) from
mcp_spend_log indexes. TIMESTAMPTZ truncation depends on session
timezone and isn't IMMUTABLE, so Postgres rejects the function in
the index expression with SQLSTATE 42P17. BTREE on
(client_id, created_at) covers the per-day rollup query via range
scan on created_at — same performance, no IMMUTABLE constraint.
- pglite-schema.ts + src/schema.sql: shorten the embedding_multimodal
column comment. The longer version contained a comma inside a SQL
line comment ("...search.unified_multimodal=true, all queries..."),
which broke parseBaseTableColumns in test/schema-bootstrap-coverage
(the parser splits on commas at depth-0 before stripping comments,
so the comma inside the comment shortened the column-definition part
and an "all" token from "all queries" got picked up as the next
column name — silently hiding embedding_multimodal from coverage).
- schema-embedded.ts: regenerated via `bun run build:schema`.
- test/e2e/v030_1-integration-pglite.test.ts: listBackfills assertion
extended to include the new `modality` entry registered in
src/core/backfill-registry.ts as part of Commit 1.
- test/search/knobs-hash-reranker.test.ts: KNOBS_HASH_VERSION assertion
updated from 2→3 to match the cross-modal-wave hash-key extension
(D2 cache contamination fix). Same shape as the prior
v0.32→v0.35 bump.
- test/unified-multimodal.test.ts: migrated process.env mutation to
withEnv() helper to satisfy the scripts/check-test-isolation R1
rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(cross-modal): VERSION + CHANGELOG + CLAUDE.md + spec doc + llms regen
Final docs commit for the cross-modal wave (v0.36.0.0).
- VERSION + package.json: bump 0.35.5.1 → 0.36.0.0
- CHANGELOG.md: full Garry-voice release entry with five-commit breakdown,
the-numbers-that-matter table, what-this-means-for-you, and the
required to-take-advantage-of-v0.36.0.0 block
- docs/issues/cross-modal-search.md: cherry-picked from PR #1127 head
(164 lines, the original spec doc preserved as historical reference
for Phase 2 + 3 background)
- CLAUDE.md: Key Files entries for src/core/ssrf-validate.ts,
src/core/search/image-loader.ts, src/core/search/by-image.ts,
src/core/search/llm-intent.ts, src/core/spend-log.ts,
src/commands/reindex-multimodal.ts, plus extension annotations on
src/core/search/query-intent.ts, src/core/search/mode.ts,
src/core/search/hybrid.ts, src/core/backfill-registry.ts,
src/core/migrate.ts (v67 + v68)
- llms-full.txt + llms.txt: regenerated via `bun run build:llms`
`bun run verify` clean (privacy + proposal-pii + test-names + jsonb +
source-id-projection + progress + test-isolation + wasm + admin-build +
admin-scope-drift + cli-exec + system-of-record + eval-glossary +
typecheck). `bun test test/build-llms.test.ts` clean (7/7).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cross-modal): renumber migrations 67→69 + 68→70 post-master-merge
Master shipped its own v67 (`facts_typed_claim_columns`) during the
cross-modal wave's review cycle. The merge picked up both side's v67
entries, breaking the migration-distinct-versions test. Renumbering
moves cross-modal's table + column ALTER off the collision:
- v67 mcp_spend_log → v69 mcp_spend_log
- v68 embedding_multimodal_column → v70 embedding_multimodal_column
References updated in CHANGELOG, CLAUDE.md, pglite-schema.ts, schema.sql.
schema-embedded.ts regenerated. llms-full.txt regenerated.
7006 unit tests pass, 0 fail. No test code touched — just version
renumbering plus comment refs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version 0.36.0.0 → 0.36.4.0
Bumping to v0.36.4.0 to land in the queue slot the user requested.
No behavior change; pure version bump across VERSION, package.json,
CHANGELOG.md header, llms-full.txt regen.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
e227965024
commit
e60b60244f
@@ -0,0 +1,220 @@
|
||||
// Phase 1 integration test — hybridSearch cross-modal routing.
|
||||
//
|
||||
// Uses real PGLite + stubbed gateway fetch. Verifies the routing decisions
|
||||
// from query-intent through hybrid.ts to engine.searchVector with the
|
||||
// correct embeddingColumn.
|
||||
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
import { hybridSearch } from '../src/core/search/hybrid.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
type FetchHandler = (url: string, init: RequestInit) => Promise<Response>;
|
||||
let fetchHandler: FetchHandler | null = null;
|
||||
const origFetch = globalThis.fetch;
|
||||
let fetchUrlsSeen: string[] = [];
|
||||
let fetchBodiesSeen: any[] = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
fetchHandler = null;
|
||||
fetchUrlsSeen = [];
|
||||
fetchBodiesSeen = [];
|
||||
globalThis.fetch = (async (url: string | URL | Request, init?: RequestInit) => {
|
||||
const u = typeof url === 'string' ? url : url.toString();
|
||||
fetchUrlsSeen.push(u);
|
||||
if (init?.body) {
|
||||
try { fetchBodiesSeen.push(JSON.parse(init.body as string)); } catch { /* ignore */ }
|
||||
}
|
||||
if (!fetchHandler) {
|
||||
// Return a generic 1024-dim Voyage-shape response by default
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.1), index: 0 }],
|
||||
model: 'voyage-multimodal-3',
|
||||
}), { status: 200, headers: { 'Content-Type': 'application/json' } });
|
||||
}
|
||||
return fetchHandler(u, init ?? {});
|
||||
}) as typeof fetch;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = origFetch;
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
function configureBoth() {
|
||||
// Gateway needs BOTH text and multimodal models configured. Use a single
|
||||
// openai recipe stub for text — we won't hit it for image-only queries.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
embedding_multimodal_model: 'voyage:voyage-multimodal-3',
|
||||
env: {
|
||||
OPENAI_API_KEY: 'test-key',
|
||||
VOYAGE_API_KEY: 'voyage-test-key',
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
describe('hybridSearch cross-modal routing (Phase 1 integration)', () => {
|
||||
test("explicit crossModal: 'image' calls Voyage multimodal endpoint, NOT OpenAI", async () => {
|
||||
configureBoth();
|
||||
// Stub the Voyage multimodal endpoint with a deterministic 1024d vector.
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.5), index: 0 }],
|
||||
model: 'voyage-multimodal-3',
|
||||
}), { status: 200 });
|
||||
}
|
||||
// Fail OpenAI requests loudly so we catch wrong routing.
|
||||
throw new Error(`Unexpected fetch to OpenAI: ${url}`);
|
||||
};
|
||||
|
||||
// hybridSearch with no rows in DB just returns []; we're testing that the
|
||||
// request hits the multimodal endpoint specifically.
|
||||
const results = await hybridSearch(engine, 'hackathon stuff', { crossModal: 'image', limit: 5 });
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
// Must have called the multimodal endpoint at least once.
|
||||
expect(fetchUrlsSeen.some(u => u.includes('multimodalembeddings'))).toBe(true);
|
||||
// Must NOT have called OpenAI embeddings.
|
||||
expect(fetchUrlsSeen.some(u => u.includes('api.openai.com') && u.includes('embeddings'))).toBe(false);
|
||||
});
|
||||
|
||||
test('explicit crossModal: "image" threads inputType=query in Voyage body (D22-2)', async () => {
|
||||
configureBoth();
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.5), index: 0 }],
|
||||
model: 'voyage-multimodal-3',
|
||||
}), { status: 200 });
|
||||
}
|
||||
throw new Error(`Unexpected fetch: ${url}`);
|
||||
};
|
||||
|
||||
await hybridSearch(engine, 'any text', { crossModal: 'image', limit: 5 });
|
||||
const voyageBody = fetchBodiesSeen.find(b => b?.inputs?.[0]?.content?.[0]?.type === 'text');
|
||||
expect(voyageBody).toBeDefined();
|
||||
expect(voyageBody.input_type).toBe('query');
|
||||
});
|
||||
|
||||
test('default crossModal=text query does NOT call Voyage multimodal', async () => {
|
||||
configureBoth();
|
||||
// Allow text embed to succeed via the default OpenAI fetch handler.
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
throw new Error('Unexpected multimodal call for text-modality query');
|
||||
}
|
||||
// OpenAI text-embedding response shape: {data: [{embedding: [...]}]}
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1536 }, () => 0.1), index: 0 }],
|
||||
model: 'text-embedding-3-large',
|
||||
}), { status: 200 });
|
||||
};
|
||||
|
||||
await hybridSearch(engine, 'what is founder mode', { limit: 5 });
|
||||
expect(fetchUrlsSeen.some(u => u.includes('multimodalembeddings'))).toBe(false);
|
||||
});
|
||||
|
||||
test("'auto' literal normalizes to undefined (D22-1) — text query still routes text", async () => {
|
||||
configureBoth();
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
throw new Error('Unexpected multimodal call for auto-text-intent query');
|
||||
}
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1536 }, () => 0.1), index: 0 }],
|
||||
model: 'text-embedding-3-large',
|
||||
}), { status: 200 });
|
||||
};
|
||||
|
||||
await hybridSearch(engine, 'what is founder mode', { crossModal: 'auto', limit: 5 });
|
||||
// Text route — multimodal never called.
|
||||
expect(fetchUrlsSeen.some(u => u.includes('multimodalembeddings'))).toBe(false);
|
||||
});
|
||||
|
||||
test('"show me photos from the hackathon" auto-detects to image routing', async () => {
|
||||
configureBoth();
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.3), index: 0 }],
|
||||
model: 'voyage-multimodal-3',
|
||||
}), { status: 200 });
|
||||
}
|
||||
// Don't fail OpenAI here — auto mode might still call text in 'both' fallback.
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1536 }, () => 0.1), index: 0 }],
|
||||
model: 'text-embedding-3-large',
|
||||
}), { status: 200 });
|
||||
};
|
||||
|
||||
await hybridSearch(engine, 'show me photos from the hackathon', { limit: 5 });
|
||||
// Auto-detection should have fired image routing.
|
||||
expect(fetchUrlsSeen.some(u => u.includes('multimodalembeddings'))).toBe(true);
|
||||
});
|
||||
|
||||
test("'both' mode hits BOTH endpoints in parallel", async () => {
|
||||
configureBoth();
|
||||
let textCalled = 0;
|
||||
let voyageCalled = 0;
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
voyageCalled++;
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.3), index: 0 }],
|
||||
model: 'voyage-multimodal-3',
|
||||
}), { status: 200 });
|
||||
}
|
||||
textCalled++;
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1536 }, () => 0.1), index: 0 }],
|
||||
model: 'text-embedding-3-large',
|
||||
}), { status: 200 });
|
||||
};
|
||||
|
||||
await hybridSearch(engine, 'anything', { crossModal: 'both', limit: 5 });
|
||||
expect(textCalled).toBeGreaterThanOrEqual(1);
|
||||
expect(voyageCalled).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('fail-open: multimodal unconfigured → image-intent query falls back to text', async () => {
|
||||
configureGateway({
|
||||
// No embedding_multimodal_model set.
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'test-key' },
|
||||
});
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
throw new Error('Voyage should not be called when not configured');
|
||||
}
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1536 }, () => 0.1), index: 0 }],
|
||||
model: 'text-embedding-3-large',
|
||||
}), { status: 200 });
|
||||
};
|
||||
|
||||
// crossModal: 'image' with no multimodal model → fail-open to text.
|
||||
const results = await hybridSearch(engine, 'show me photos', { crossModal: 'image', limit: 5 });
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
// Did NOT throw; fell back successfully.
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,233 @@
|
||||
// Commit 1 (Phase 1): cross-modal intent + hybrid routing + knobsHash + RRF.
|
||||
//
|
||||
// Covers:
|
||||
// - suggestedModality regex matches (positive + negative + plural-safe)
|
||||
// - isAmbiguousModalityQuery heuristic
|
||||
// - SEARCH_MODE_CONFIG_KEYS registry includes new keys (D3)
|
||||
// - knobsHash differs across cross-modal knob values (D2)
|
||||
// - knobsHash version bumped to 3
|
||||
// - MODE_BUNDLES carry cross-modal defaults
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
classifyQuery,
|
||||
isAmbiguousModalityQuery,
|
||||
type ModalityMode,
|
||||
} from '../src/core/search/query-intent.ts';
|
||||
import {
|
||||
KNOBS_HASH_VERSION,
|
||||
MODE_BUNDLES,
|
||||
SEARCH_MODE_CONFIG_KEYS,
|
||||
knobsHash,
|
||||
resolveSearchMode,
|
||||
type ResolvedSearchKnobs,
|
||||
} from '../src/core/search/mode.ts';
|
||||
|
||||
describe('query-intent — suggestedModality regex (D6 + D14)', () => {
|
||||
test('"show me photos from the hackathon" → image', () => {
|
||||
expect(classifyQuery('show me photos from the hackathon').suggestedModality).toBe('image');
|
||||
});
|
||||
|
||||
test('"what is founder mode?" → text (default)', () => {
|
||||
expect(classifyQuery('what is founder mode?').suggestedModality).toBe('text');
|
||||
});
|
||||
|
||||
const imagePhrasings: Array<[string, ModalityMode]> = [
|
||||
['find images from last week', 'image'],
|
||||
['find me images of acme', 'image'],
|
||||
['what does the OG photo look like', 'image'],
|
||||
['screenshot of the dashboard', 'image'],
|
||||
['diagram of the architecture', 'image'],
|
||||
['visuals showing the trends', 'image'],
|
||||
['whiteboard from the offsite', 'image'],
|
||||
['pictures of the team', 'image'],
|
||||
['pull me the screenshots', 'image'],
|
||||
];
|
||||
|
||||
for (const [query, expected] of imagePhrasings) {
|
||||
test(`image phrasing: "${query}" → ${expected}`, () => {
|
||||
expect(classifyQuery(query).suggestedModality).toBe(expected);
|
||||
});
|
||||
}
|
||||
|
||||
const textPhrasings = [
|
||||
'who is acme corp',
|
||||
'tell me about founder mode',
|
||||
'what happened at the hackathon',
|
||||
'meeting notes from yesterday',
|
||||
'most recent take on AI',
|
||||
];
|
||||
|
||||
for (const query of textPhrasings) {
|
||||
test(`text phrasing: "${query}" → text`, () => {
|
||||
expect(classifyQuery(query).suggestedModality).toBe('text');
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
describe('isAmbiguousModalityQuery (Commit 4 prep)', () => {
|
||||
// Genuinely ambiguous = visual noun present + reference marker present BUT
|
||||
// CROSS_MODAL_PATTERNS doesn't catch it (otherwise regex already classified
|
||||
// confidently and the LLM call would be wasted).
|
||||
|
||||
test('"any picture during last week" → ambiguous', () => {
|
||||
// "picture during" doesn't match (of|from|at|with|...) so CROSS_MODAL
|
||||
// doesn't fire; "any pictures" does match the AMBIGUOUS_REFERENCE marker.
|
||||
// Actually "any picture" matches /\b(any|some|...)\s+(pics?|photos?|images?...)/ — but
|
||||
// the CROSS_MODAL pattern needs "pictures from/of/at/...". This phrasing
|
||||
// has neither — so it's genuinely ambiguous.
|
||||
expect(isAmbiguousModalityQuery('any picture during last week')).toBe(true);
|
||||
});
|
||||
|
||||
test('"what is founder mode" → not ambiguous (plain text query)', () => {
|
||||
expect(isAmbiguousModalityQuery('what is founder mode')).toBe(false);
|
||||
});
|
||||
|
||||
test('"show me photos of acme" → not ambiguous (regex catches it)', () => {
|
||||
// Already-confident classification, no LLM needed.
|
||||
expect(isAmbiguousModalityQuery('show me photos of acme')).toBe(false);
|
||||
});
|
||||
|
||||
test('"any pictures from the meeting" → not ambiguous (regex catches "pictures from")', () => {
|
||||
// CROSS_MODAL fires on "pictures from" — confident classification.
|
||||
expect(isAmbiguousModalityQuery('any pictures from the meeting')).toBe(false);
|
||||
});
|
||||
|
||||
test('"chart" without article/determiner → not ambiguous (bare visual noun has no reference marker)', () => {
|
||||
// No "any|some|that|the" determiner in front of the visual noun, and no
|
||||
// "from last/this/the X" phrase — pure text query.
|
||||
expect(isAmbiguousModalityQuery('chart')).toBe(false);
|
||||
});
|
||||
|
||||
test('"the chart" alone → ambiguous (determiner+visual-noun is a real reference marker)', () => {
|
||||
// "the chart" is the canonical ambiguous case — user references a
|
||||
// specific visual asset without confirming they want image search.
|
||||
// LLM tie-break decides.
|
||||
expect(isAmbiguousModalityQuery('the chart')).toBe(true);
|
||||
});
|
||||
|
||||
test('"the diagram in last week\'s deck" → ambiguous', () => {
|
||||
// "diagram in" doesn't match CROSS_MODAL (of|from|about|showing only).
|
||||
// "the diagram" matches AMBIGUOUS_REFERENCE first pattern.
|
||||
expect(isAmbiguousModalityQuery("the diagram in last week's deck")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('D3 — SEARCH_MODE_CONFIG_KEYS registry includes cross-modal keys', () => {
|
||||
const expected = [
|
||||
'search.cross_modal.both_mode_text_weight',
|
||||
'search.cross_modal.both_mode_image_weight',
|
||||
'search.image_query.text_refinement_weight',
|
||||
'search.image_query.image_refinement_weight',
|
||||
'search.unified_multimodal',
|
||||
'search.unified_multimodal_only',
|
||||
'search.cross_modal.llm_intent',
|
||||
];
|
||||
|
||||
for (const key of expected) {
|
||||
test(`registry contains ${key}`, () => {
|
||||
expect(SEARCH_MODE_CONFIG_KEYS).toContain(key);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
function baseKnobs(): ResolvedSearchKnobs {
|
||||
return resolveSearchMode({ mode: 'balanced' });
|
||||
}
|
||||
|
||||
test('KNOBS_HASH_VERSION is 3 (v0.36 cross-modal bump)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(3);
|
||||
});
|
||||
|
||||
test('flipping unified_multimodal changes the hash', () => {
|
||||
const k1 = baseKnobs();
|
||||
const k2 = { ...k1, unified_multimodal: true };
|
||||
expect(knobsHash(k1)).not.toBe(knobsHash(k2));
|
||||
});
|
||||
|
||||
test('flipping unified_multimodal_only changes the hash', () => {
|
||||
const k1 = baseKnobs();
|
||||
const k2 = { ...k1, unified_multimodal_only: true };
|
||||
expect(knobsHash(k1)).not.toBe(knobsHash(k2));
|
||||
});
|
||||
|
||||
test('flipping cross_modal_llm_intent changes the hash', () => {
|
||||
const k1 = baseKnobs();
|
||||
const k2 = { ...k1, cross_modal_llm_intent: true };
|
||||
expect(knobsHash(k1)).not.toBe(knobsHash(k2));
|
||||
});
|
||||
|
||||
test('changing cross_modal_both_text_weight changes the hash', () => {
|
||||
const k1 = baseKnobs();
|
||||
const k2 = { ...k1, cross_modal_both_text_weight: 0.5 };
|
||||
expect(knobsHash(k1)).not.toBe(knobsHash(k2));
|
||||
});
|
||||
|
||||
test('changing image_query_text_refinement_weight changes the hash', () => {
|
||||
const k1 = baseKnobs();
|
||||
const k2 = { ...k1, image_query_text_refinement_weight: 0.7 };
|
||||
expect(knobsHash(k1)).not.toBe(knobsHash(k2));
|
||||
});
|
||||
|
||||
test('identical knobs produce identical hashes (regression sanity)', () => {
|
||||
expect(knobsHash(baseKnobs())).toBe(knobsHash(baseKnobs()));
|
||||
});
|
||||
});
|
||||
|
||||
describe('D6 — MODE_BUNDLES carry cross-modal defaults', () => {
|
||||
test('all three modes default cross_modal_both_text_weight to 0.6', () => {
|
||||
expect(MODE_BUNDLES.conservative.cross_modal_both_text_weight).toBe(0.6);
|
||||
expect(MODE_BUNDLES.balanced.cross_modal_both_text_weight).toBe(0.6);
|
||||
expect(MODE_BUNDLES.tokenmax.cross_modal_both_text_weight).toBe(0.6);
|
||||
});
|
||||
|
||||
test('all three modes default cross_modal_both_image_weight to 0.4', () => {
|
||||
expect(MODE_BUNDLES.conservative.cross_modal_both_image_weight).toBe(0.4);
|
||||
expect(MODE_BUNDLES.balanced.cross_modal_both_image_weight).toBe(0.4);
|
||||
expect(MODE_BUNDLES.tokenmax.cross_modal_both_image_weight).toBe(0.4);
|
||||
});
|
||||
|
||||
test('all three modes default image_query weights (D13: 0.4 text / 0.6 image)', () => {
|
||||
expect(MODE_BUNDLES.conservative.image_query_text_refinement_weight).toBe(0.4);
|
||||
expect(MODE_BUNDLES.conservative.image_query_image_refinement_weight).toBe(0.6);
|
||||
expect(MODE_BUNDLES.tokenmax.image_query_image_refinement_weight).toBe(0.6);
|
||||
});
|
||||
|
||||
test('all three modes default unified_multimodal to false (opt-in)', () => {
|
||||
expect(MODE_BUNDLES.conservative.unified_multimodal).toBe(false);
|
||||
expect(MODE_BUNDLES.balanced.unified_multimodal).toBe(false);
|
||||
expect(MODE_BUNDLES.tokenmax.unified_multimodal).toBe(false);
|
||||
});
|
||||
|
||||
test('all three modes default cross_modal_llm_intent to false (opt-in)', () => {
|
||||
expect(MODE_BUNDLES.conservative.cross_modal_llm_intent).toBe(false);
|
||||
expect(MODE_BUNDLES.balanced.cross_modal_llm_intent).toBe(false);
|
||||
expect(MODE_BUNDLES.tokenmax.cross_modal_llm_intent).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveSearchMode threads cross-modal overrides', () => {
|
||||
test('per-call override beats config override beats mode default', () => {
|
||||
const k = resolveSearchMode({
|
||||
mode: 'balanced',
|
||||
overrides: { cross_modal_both_text_weight: 0.5 },
|
||||
perCall: { cross_modal_both_text_weight: 0.8 },
|
||||
});
|
||||
expect(k.cross_modal_both_text_weight).toBe(0.8);
|
||||
});
|
||||
|
||||
test('config override wins when no per-call override', () => {
|
||||
const k = resolveSearchMode({
|
||||
mode: 'balanced',
|
||||
overrides: { unified_multimodal: true },
|
||||
});
|
||||
expect(k.unified_multimodal).toBe(true);
|
||||
});
|
||||
|
||||
test('mode default fires when neither override is set', () => {
|
||||
const k = resolveSearchMode({ mode: 'balanced' });
|
||||
expect(k.cross_modal_both_text_weight).toBe(0.6);
|
||||
expect(k.cross_modal_both_image_weight).toBe(0.4);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,181 @@
|
||||
// Commit 2 (Phase 2): image-as-query loader + searchByImage + D18 path ban
|
||||
//
|
||||
// Covers:
|
||||
// - loadImageInput: PNG/JPEG/WebP magic-byte sniff + format rejection
|
||||
// - loadImageInput: oversized file rejection (local + remote caps)
|
||||
// - loadImageInput: data: URI parsing
|
||||
// - loadImageInput: invalid input shapes
|
||||
// - D11 SSRF in fetchWithSSRFGuard (already covered by ssrf-validate.test.ts)
|
||||
// - D18 search_by_image rejects image_path when ctx.remote=true
|
||||
// - D12 image_data param-level size cap (validateParams gate)
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, mkdirSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import {
|
||||
ImageLoadError,
|
||||
loadImageInput,
|
||||
} from '../src/core/search/image-loader.ts';
|
||||
import { __setDnsLookupForTests } from '../src/core/ssrf-validate.ts';
|
||||
|
||||
let tmpRoot: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpRoot = mkdtempSync(join(tmpdir(), 'gbrain-img-loader-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
__setDnsLookupForTests(undefined);
|
||||
});
|
||||
|
||||
// PNG magic bytes for a 1x1 transparent PNG.
|
||||
const PNG_BYTES = Buffer.from([
|
||||
0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A,
|
||||
// Minimal IHDR + IDAT + IEND chunks
|
||||
0, 0, 0, 13, 73, 72, 68, 82, 0, 0, 0, 1, 0, 0, 0, 1, 8, 6, 0, 0, 0,
|
||||
31, 21, 196, 137, 0, 0, 0, 12, 73, 68, 65, 84, 8, 87, 99, 248, 207, 192, 0, 0, 0, 3, 0, 1,
|
||||
90, 12, 105, 240, 0, 0, 0, 0, 73, 69, 78, 68, 174, 66, 96, 130,
|
||||
]);
|
||||
|
||||
// JPEG magic: FF D8 FF + dummy
|
||||
const JPEG_BYTES = Buffer.concat([Buffer.from([0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10]), Buffer.alloc(100)]);
|
||||
|
||||
// WebP magic: RIFF????WEBP
|
||||
const WEBP_BYTES = Buffer.concat([
|
||||
Buffer.from('RIFF'),
|
||||
Buffer.from([0x40, 0x00, 0x00, 0x00]),
|
||||
Buffer.from('WEBP'),
|
||||
Buffer.alloc(100),
|
||||
]);
|
||||
|
||||
describe('loadImageInput — local path', () => {
|
||||
test('loads a PNG file and sniffs MIME', async () => {
|
||||
const path = join(tmpRoot, 'test.png');
|
||||
writeFileSync(path, PNG_BYTES);
|
||||
const result = await loadImageInput(path);
|
||||
expect(result.contentType).toBe('image/png');
|
||||
expect(result.bytes.length).toBe(PNG_BYTES.length);
|
||||
expect(result.base64).toBe(PNG_BYTES.toString('base64'));
|
||||
});
|
||||
|
||||
test('loads a JPEG file and sniffs MIME', async () => {
|
||||
const path = join(tmpRoot, 'test.jpg');
|
||||
writeFileSync(path, JPEG_BYTES);
|
||||
const result = await loadImageInput(path);
|
||||
expect(result.contentType).toBe('image/jpeg');
|
||||
});
|
||||
|
||||
test('loads a WebP file and sniffs MIME', async () => {
|
||||
const path = join(tmpRoot, 'test.webp');
|
||||
writeFileSync(path, WEBP_BYTES);
|
||||
const result = await loadImageInput(path);
|
||||
expect(result.contentType).toBe('image/webp');
|
||||
});
|
||||
|
||||
test('rejects unsupported format (GIF)', async () => {
|
||||
const path = join(tmpRoot, 'test.gif');
|
||||
writeFileSync(path, Buffer.from('GIF89a' + 'x'.repeat(100)));
|
||||
const err = await loadImageInput(path).catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('INVALID_FORMAT');
|
||||
});
|
||||
|
||||
test('rejects oversized file (default 10MB cap)', async () => {
|
||||
const path = join(tmpRoot, 'huge.png');
|
||||
// 11MB file with PNG magic bytes
|
||||
writeFileSync(path, Buffer.concat([PNG_BYTES, Buffer.alloc(11 * 1024 * 1024)]));
|
||||
const err = await loadImageInput(path).catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('OVERSIZED');
|
||||
});
|
||||
|
||||
test('rejects file via custom (tighter) maxBytes', async () => {
|
||||
const path = join(tmpRoot, 'medium.png');
|
||||
writeFileSync(path, Buffer.concat([PNG_BYTES, Buffer.alloc(1024 * 1024)])); // 1MB
|
||||
const err = await loadImageInput(path, { maxBytes: 500 * 1024 }).catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('OVERSIZED');
|
||||
});
|
||||
|
||||
test('NOT_FOUND on nonexistent path', async () => {
|
||||
const err = await loadImageInput(join(tmpRoot, 'missing.png')).catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('NOT_FOUND');
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadImageInput — data: URI', () => {
|
||||
test('decodes PNG data: URI', async () => {
|
||||
const dataUri = `data:image/png;base64,${PNG_BYTES.toString('base64')}`;
|
||||
const result = await loadImageInput(dataUri);
|
||||
expect(result.contentType).toBe('image/png');
|
||||
expect(result.bytes.length).toBe(PNG_BYTES.length);
|
||||
});
|
||||
|
||||
test('rejects malformed data: URI', async () => {
|
||||
const err = await loadImageInput('data:image/png;invalid-format').catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('INVALID_FORMAT');
|
||||
});
|
||||
|
||||
test('rejects data: URI with non-image format (decoded GIF bytes)', async () => {
|
||||
const gifBytes = Buffer.from('GIF89a' + 'x'.repeat(100));
|
||||
const dataUri = `data:image/png;base64,${gifBytes.toString('base64')}`;
|
||||
const err = await loadImageInput(dataUri).catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('INVALID_FORMAT');
|
||||
});
|
||||
|
||||
test('rejects oversized data: URI', async () => {
|
||||
const huge = Buffer.concat([PNG_BYTES, Buffer.alloc(11 * 1024 * 1024)]);
|
||||
const dataUri = `data:image/png;base64,${huge.toString('base64')}`;
|
||||
const err = await loadImageInput(dataUri).catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('OVERSIZED');
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadImageInput — invalid input shapes', () => {
|
||||
test('rejects empty string', async () => {
|
||||
const err = await loadImageInput('').catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('INVALID_URL');
|
||||
});
|
||||
|
||||
test('rejects unsupported scheme', async () => {
|
||||
const err = await loadImageInput('ftp://example.com/img.png').catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('INVALID_URL');
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadImageInput — http(s) URL with SSRF defense', () => {
|
||||
let stubAddrs: Map<string, Array<{ address: string; family: number }>>;
|
||||
|
||||
beforeEach(() => {
|
||||
stubAddrs = new Map();
|
||||
__setDnsLookupForTests((async (host: string) => {
|
||||
const recs = stubAddrs.get(host);
|
||||
if (!recs) {
|
||||
const e: any = new Error(`stub: no DNS records for ${host}`);
|
||||
e.code = 'ENOTFOUND';
|
||||
throw e;
|
||||
}
|
||||
return recs;
|
||||
}) as any);
|
||||
});
|
||||
|
||||
test('rejects URL whose hostname resolves internal (DNS rebinding)', async () => {
|
||||
stubAddrs.set('attacker.com', [{ address: '127.0.0.1', family: 4 }]);
|
||||
const err = await loadImageInput('https://attacker.com/img.png').catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('SSRF_BLOCKED');
|
||||
});
|
||||
|
||||
test('rejects URL with metadata IP literal', async () => {
|
||||
const err = await loadImageInput('http://169.254.169.254/latest/').catch(e => e);
|
||||
expect(err).toBeInstanceOf(ImageLoadError);
|
||||
expect(err.code).toBe('SSRF_BLOCKED');
|
||||
});
|
||||
});
|
||||
@@ -80,10 +80,14 @@ describe('Lane B — migration runner applies cleanly through retry wrapper', ()
|
||||
});
|
||||
|
||||
describe('Lane C — backfill registry on empty brain', () => {
|
||||
test('listBackfills returns three entries', () => {
|
||||
test('listBackfills returns the canonical registry entries', () => {
|
||||
// v0.30.1 shipped 3 entries (effective_date, embedding_voyage,
|
||||
// emotional_weight). v0.36 cross-modal wave adds `modality` for
|
||||
// historical image-asset chunks. Extend this assertion as new
|
||||
// backfills land.
|
||||
const list = listBackfills();
|
||||
const names = list.map(e => e.spec.name).sort();
|
||||
expect(names).toEqual(['effective_date', 'embedding_voyage', 'emotional_weight']);
|
||||
expect(names).toEqual(['effective_date', 'embedding_voyage', 'emotional_weight', 'modality']);
|
||||
});
|
||||
|
||||
test('embedding_voyage is declared-only in v0.30.1', () => {
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
// Commit 0 (D4 + D22-2): batching + partial-failure for multimodal embed,
|
||||
// plus query-side helpers (embedQueryMultimodal, embedQueryMultimodalImage).
|
||||
//
|
||||
// Covers:
|
||||
// - Voyage text variant (mixed text+image content arrays)
|
||||
// - inputType: 'query' threaded through to Voyage wire format
|
||||
// - embedMultimodalSafe binary-search retry on transient failure
|
||||
// - embedMultimodalSafe surfaces failed_indices when individual inputs fail
|
||||
// - embedMultimodalSafe stops on AIConfigError (permanent misconfig)
|
||||
// - embedQueryMultimodal returns 1024-dim vector
|
||||
// - embedQueryMultimodalImage returns 1024-dim vector
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
configureGateway,
|
||||
embedMultimodal,
|
||||
embedMultimodalSafe,
|
||||
embedQueryMultimodal,
|
||||
embedQueryMultimodalImage,
|
||||
resetGateway,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
|
||||
type FetchHandler = (url: string, init: RequestInit) => Promise<Response>;
|
||||
let fetchHandler: FetchHandler | null = null;
|
||||
const origFetch = globalThis.fetch;
|
||||
|
||||
beforeEach(() => {
|
||||
fetchHandler = null;
|
||||
globalThis.fetch = (async (url: string | URL | Request, init?: RequestInit) => {
|
||||
if (!fetchHandler) throw new Error('fetch called but no handler installed');
|
||||
return fetchHandler(typeof url === 'string' ? url : url.toString(), init ?? {});
|
||||
}) as typeof fetch;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = origFetch;
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
function configureVoyage(env: Record<string, string | undefined> = {}) {
|
||||
configureGateway({
|
||||
embedding_model: 'voyage:voyage-multimodal-3',
|
||||
embedding_dimensions: 1024,
|
||||
env: { VOYAGE_API_KEY: 'test-key', ...env },
|
||||
});
|
||||
}
|
||||
|
||||
function fakeResponse(count: number, dims = 1024): Response {
|
||||
const data = Array.from({ length: count }, (_, i) => ({
|
||||
embedding: Array.from({ length: dims }, () => 0.1 * (i + 1)),
|
||||
index: i,
|
||||
}));
|
||||
return new Response(JSON.stringify({ data, model: 'voyage-multimodal-3' }), {
|
||||
status: 200,
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
});
|
||||
}
|
||||
|
||||
function makeImg() {
|
||||
return {
|
||||
kind: 'image_base64' as const,
|
||||
data: Buffer.from('fake').toString('base64'),
|
||||
mime: 'image/jpeg',
|
||||
};
|
||||
}
|
||||
|
||||
describe('Voyage multimodal — text variant + inputType discipline', () => {
|
||||
test('text input variant sends correct Voyage content shape', async () => {
|
||||
configureVoyage();
|
||||
let capturedBody: any;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return fakeResponse(1);
|
||||
};
|
||||
const vecs = await embedMultimodal([{ kind: 'text', text: 'hello world' }]);
|
||||
expect(vecs.length).toBe(1);
|
||||
expect(vecs[0]).toBeInstanceOf(Float32Array);
|
||||
expect(capturedBody.inputs[0].content[0]).toEqual({ type: 'text', text: 'hello world' });
|
||||
});
|
||||
|
||||
test('opts.inputType="query" threads through to Voyage wire body', async () => {
|
||||
configureVoyage();
|
||||
let capturedBody: any;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return fakeResponse(1);
|
||||
};
|
||||
await embedMultimodal([{ kind: 'text', text: 'q' }], { inputType: 'query' });
|
||||
expect(capturedBody.input_type).toBe('query');
|
||||
});
|
||||
|
||||
test('default inputType is "document" (preserves pre-v0.36 ingest behavior)', async () => {
|
||||
configureVoyage();
|
||||
let capturedBody: any;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return fakeResponse(1);
|
||||
};
|
||||
await embedMultimodal([makeImg()]);
|
||||
expect(capturedBody.input_type).toBe('document');
|
||||
});
|
||||
|
||||
test('mixed text + image inputs in one batch — each gets correct content type', async () => {
|
||||
configureVoyage();
|
||||
let capturedBody: any;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return fakeResponse(2);
|
||||
};
|
||||
await embedMultimodal([
|
||||
{ kind: 'text', text: 'hello' },
|
||||
makeImg(),
|
||||
]);
|
||||
expect(capturedBody.inputs[0].content[0].type).toBe('text');
|
||||
expect(capturedBody.inputs[1].content[0].type).toBe('image_base64');
|
||||
});
|
||||
});
|
||||
|
||||
describe('embedQueryMultimodal — text query path', () => {
|
||||
test('returns 1024-dim Float32Array via Voyage query embed', async () => {
|
||||
configureVoyage();
|
||||
fetchHandler = async () => fakeResponse(1, 1024);
|
||||
const v = await embedQueryMultimodal('hackathon photos');
|
||||
expect(v).toBeInstanceOf(Float32Array);
|
||||
expect(v.length).toBe(1024);
|
||||
});
|
||||
|
||||
test('threads inputType="query" to the wire', async () => {
|
||||
configureVoyage();
|
||||
let capturedBody: any;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return fakeResponse(1);
|
||||
};
|
||||
await embedQueryMultimodal('q');
|
||||
expect(capturedBody.input_type).toBe('query');
|
||||
expect(capturedBody.inputs[0].content[0]).toEqual({ type: 'text', text: 'q' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('embedQueryMultimodalImage — image query path', () => {
|
||||
test('returns 1024-dim Float32Array via Voyage image-query embed', async () => {
|
||||
configureVoyage();
|
||||
fetchHandler = async () => fakeResponse(1, 1024);
|
||||
const v = await embedQueryMultimodalImage({
|
||||
data: Buffer.from('fake').toString('base64'),
|
||||
mime: 'image/png',
|
||||
});
|
||||
expect(v).toBeInstanceOf(Float32Array);
|
||||
expect(v.length).toBe(1024);
|
||||
});
|
||||
|
||||
test('threads inputType="query" + image_base64 shape to the wire', async () => {
|
||||
configureVoyage();
|
||||
let capturedBody: any;
|
||||
fetchHandler = async (_url, init) => {
|
||||
capturedBody = JSON.parse(init.body as string);
|
||||
return fakeResponse(1);
|
||||
};
|
||||
await embedQueryMultimodalImage({
|
||||
data: Buffer.from('xyz').toString('base64'),
|
||||
mime: 'image/webp',
|
||||
});
|
||||
expect(capturedBody.input_type).toBe('query');
|
||||
expect(capturedBody.inputs[0].content[0].type).toBe('image_base64');
|
||||
expect(capturedBody.inputs[0].content[0].image_base64).toContain('data:image/webp;base64,');
|
||||
});
|
||||
});
|
||||
|
||||
describe('embedMultimodalSafe — partial-failure surfacing', () => {
|
||||
test('happy path returns full embeddings array, no failed indices', async () => {
|
||||
configureVoyage();
|
||||
fetchHandler = async () => fakeResponse(3);
|
||||
const result = await embedMultimodalSafe([makeImg(), makeImg(), makeImg()]);
|
||||
expect(result.failedIndices).toEqual([]);
|
||||
expect(result.embeddings.length).toBe(3);
|
||||
expect(result.embeddings.every(v => v instanceof Float32Array)).toBe(true);
|
||||
});
|
||||
|
||||
test('empty input array returns empty result without HTTP call', async () => {
|
||||
configureVoyage();
|
||||
fetchHandler = async () => {
|
||||
throw new Error('should not be called');
|
||||
};
|
||||
const result = await embedMultimodalSafe([]);
|
||||
expect(result.failedIndices).toEqual([]);
|
||||
expect(result.embeddings).toEqual([]);
|
||||
});
|
||||
|
||||
test('all-fail batch records every input as failed', async () => {
|
||||
configureVoyage();
|
||||
let callCount = 0;
|
||||
fetchHandler = async () => {
|
||||
callCount++;
|
||||
return new Response('rate limited', { status: 429 });
|
||||
};
|
||||
const result = await embedMultimodalSafe([makeImg(), makeImg()]);
|
||||
expect(result.failedIndices).toEqual([0, 1]);
|
||||
expect(result.embeddings).toEqual([undefined, undefined]);
|
||||
expect(result.lastError).toBeDefined();
|
||||
// Binary-search retry: tries [0,1] then [0] then [1] = 3 calls
|
||||
expect(callCount).toBeGreaterThanOrEqual(2);
|
||||
});
|
||||
|
||||
test('mid-batch failure: binary-search retry recovers good inputs', async () => {
|
||||
configureVoyage();
|
||||
// Strategy: track which inputs were sent in each batch by hashing the
|
||||
// request body. Input index 2 always fails when sent solo; other splits succeed.
|
||||
fetchHandler = async (_url, init) => {
|
||||
const body = JSON.parse(init.body as string);
|
||||
const requestSize = body.inputs.length;
|
||||
// Any batch containing TARGET2 fails transiently — forces the binary-search
|
||||
// split until input 2 is isolated, at which point single-input fail is recorded.
|
||||
const containsTarget = body.inputs.some((inp: any) =>
|
||||
inp.content?.[0]?.image_base64?.includes('VEFSR0VUMg'),
|
||||
);
|
||||
if (containsTarget) {
|
||||
return new Response('contains-target-fail', { status: 503 });
|
||||
}
|
||||
return fakeResponse(requestSize);
|
||||
};
|
||||
const inputs = [
|
||||
makeImg(),
|
||||
makeImg(),
|
||||
{ kind: 'image_base64' as const, data: Buffer.from('TARGET2').toString('base64'), mime: 'image/jpeg' },
|
||||
];
|
||||
const result = await embedMultimodalSafe(inputs);
|
||||
// Inputs 0,1 should succeed via the binary-search split; input 2 fails permanently
|
||||
expect(result.failedIndices).toEqual([2]);
|
||||
expect(result.embeddings[0]).toBeInstanceOf(Float32Array);
|
||||
expect(result.embeddings[1]).toBeInstanceOf(Float32Array);
|
||||
expect(result.embeddings[2]).toBeUndefined();
|
||||
});
|
||||
|
||||
test('AIConfigError (permanent) fails fast without binary-search retry', async () => {
|
||||
configureVoyage();
|
||||
let callCount = 0;
|
||||
fetchHandler = async () => {
|
||||
callCount++;
|
||||
return new Response('unauthorized', { status: 401 });
|
||||
};
|
||||
const result = await embedMultimodalSafe([makeImg(), makeImg(), makeImg(), makeImg()]);
|
||||
// AIConfigError (401) is permanent — no point in binary-search retry.
|
||||
// All 4 inputs should be reported as failed after the single call.
|
||||
expect(result.failedIndices).toEqual([0, 1, 2, 3]);
|
||||
expect(callCount).toBe(1);
|
||||
expect(result.lastError?.message).toContain('401');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,137 @@
|
||||
// Commit 4: LLM intent escalation for cross-modal classification.
|
||||
//
|
||||
// Covers:
|
||||
// - parseModality tolerates trailing punctuation + casing
|
||||
// - classifyModalityWithLLM happy paths (text / image / both)
|
||||
// - Fail-open on timeout / parse failure / gateway misconfig
|
||||
// - hybridSearch escalation gate: fires ONLY when flag on + regex 'text' + ambiguous
|
||||
// - Cache miss: same query asked twice WITH llm_intent=true makes 2 LLM calls
|
||||
// (caching is the existing query_cache layer, not a per-process LRU)
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
classifyModalityWithLLM,
|
||||
parseModality,
|
||||
} from '../src/core/search/llm-intent.ts';
|
||||
import {
|
||||
__setChatTransportForTests,
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
|
||||
beforeEach(() => {
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'test', ANTHROPIC_API_KEY: 'test' },
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
resetGateway();
|
||||
__setChatTransportForTests(null);
|
||||
});
|
||||
|
||||
describe('parseModality (pure function)', () => {
|
||||
test('"text" → text', () => {
|
||||
expect(parseModality('text', 'text')).toBe('text');
|
||||
});
|
||||
test('"image" → image', () => {
|
||||
expect(parseModality('image', 'text')).toBe('image');
|
||||
});
|
||||
test('"both" → both', () => {
|
||||
expect(parseModality('both', 'text')).toBe('both');
|
||||
});
|
||||
test('"IMAGE." → image (tolerates trailing punctuation + casing)', () => {
|
||||
expect(parseModality('IMAGE.', 'text')).toBe('image');
|
||||
});
|
||||
test('" text \\n" → text (tolerates whitespace)', () => {
|
||||
expect(parseModality(' text \n', 'image')).toBe('text');
|
||||
});
|
||||
test('"none of the above" → fallback', () => {
|
||||
expect(parseModality('none of the above', 'text')).toBe('text');
|
||||
expect(parseModality('none of the above', 'image')).toBe('image');
|
||||
});
|
||||
test('empty string → fallback', () => {
|
||||
expect(parseModality('', 'text')).toBe('text');
|
||||
});
|
||||
});
|
||||
|
||||
describe('classifyModalityWithLLM — happy path', () => {
|
||||
test('"any pictures from offsite?" → LLM says image → returns image', async () => {
|
||||
let chatCalled = 0;
|
||||
__setChatTransportForTests(async (_opts) => {
|
||||
chatCalled++;
|
||||
return {
|
||||
text: 'image',
|
||||
blocks: [{ type: 'text', text: 'image' }],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 10, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-haiku-4-5',
|
||||
providerId: 'anthropic',
|
||||
};
|
||||
});
|
||||
const result = await classifyModalityWithLLM('any pictures from offsite?');
|
||||
expect(result).toBe('image');
|
||||
expect(chatCalled).toBe(1);
|
||||
});
|
||||
|
||||
test('"what is founder mode" → LLM says text → returns text', async () => {
|
||||
__setChatTransportForTests(async () => ({
|
||||
text: 'text',
|
||||
blocks: [{ type: 'text', text: 'text' }],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 10, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-haiku-4-5',
|
||||
providerId: 'anthropic',
|
||||
}));
|
||||
expect(await classifyModalityWithLLM('what is founder mode')).toBe('text');
|
||||
});
|
||||
|
||||
test('LLM says "both" → returns both', async () => {
|
||||
__setChatTransportForTests(async () => ({
|
||||
text: 'both',
|
||||
blocks: [{ type: 'text', text: 'both' }],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 10, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-haiku-4-5',
|
||||
providerId: 'anthropic',
|
||||
}));
|
||||
expect(await classifyModalityWithLLM('ambiguous query')).toBe('both');
|
||||
});
|
||||
});
|
||||
|
||||
describe('classifyModalityWithLLM — fail-open', () => {
|
||||
test('LLM throws → returns fallback (text)', async () => {
|
||||
__setChatTransportForTests(async () => {
|
||||
throw new Error('network error');
|
||||
});
|
||||
expect(await classifyModalityWithLLM('q')).toBe('text');
|
||||
});
|
||||
|
||||
test('LLM returns unrecognized output → returns fallback', async () => {
|
||||
__setChatTransportForTests(async () => ({
|
||||
text: 'gibberish output',
|
||||
blocks: [{ type: 'text', text: 'gibberish output' }],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 10, output_tokens: 5, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-haiku-4-5',
|
||||
providerId: 'anthropic',
|
||||
}));
|
||||
expect(await classifyModalityWithLLM('q', 'text')).toBe('text');
|
||||
});
|
||||
|
||||
test('Gateway not configured → returns fallback', async () => {
|
||||
resetGateway();
|
||||
// No configureGateway called → isAvailable('chat') returns false.
|
||||
expect(await classifyModalityWithLLM('q', 'text')).toBe('text');
|
||||
});
|
||||
|
||||
test('Explicit fallback honored', async () => {
|
||||
__setChatTransportForTests(async () => {
|
||||
throw new Error('boom');
|
||||
});
|
||||
expect(await classifyModalityWithLLM('q', 'image')).toBe('image');
|
||||
expect(await classifyModalityWithLLM('q', 'both')).toBe('both');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,123 @@
|
||||
// Commit 4 integration: hybridSearch escalation gate fires only when
|
||||
// (config flag on) + (regex returned 'text') + (isAmbiguousModalityQuery true).
|
||||
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import {
|
||||
__setChatTransportForTests,
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
import { hybridSearch } from '../src/core/search/hybrid.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
const origFetch = globalThis.fetch;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
globalThis.fetch = (async (url: string | URL | Request) => {
|
||||
const u = typeof url === 'string' ? url : url.toString();
|
||||
if (u.includes('multimodalembeddings')) {
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.1), index: 0 }],
|
||||
}), { status: 200 });
|
||||
}
|
||||
// Default OpenAI text embed response.
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1536 }, () => 0.1), index: 0 }],
|
||||
}), { status: 200 });
|
||||
}) as typeof fetch;
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
embedding_multimodal_model: 'voyage:voyage-multimodal-3',
|
||||
env: { OPENAI_API_KEY: 'test', VOYAGE_API_KEY: 'test', ANTHROPIC_API_KEY: 'test' },
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = origFetch;
|
||||
resetGateway();
|
||||
__setChatTransportForTests(null);
|
||||
});
|
||||
|
||||
describe('hybridSearch LLM intent escalation gate (Commit 4)', () => {
|
||||
test('flag OFF + ambiguous query → no LLM call (default behavior)', async () => {
|
||||
let chatCalled = 0;
|
||||
__setChatTransportForTests(async () => {
|
||||
chatCalled++;
|
||||
return { text: 'image', blocks: [], stopReason: 'end', usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'x', providerId: 'x' };
|
||||
});
|
||||
// Flag NOT set → default false.
|
||||
await hybridSearch(engine, 'the chart', { limit: 5 });
|
||||
expect(chatCalled).toBe(0);
|
||||
});
|
||||
|
||||
test('flag ON + ambiguous query → ONE LLM call', async () => {
|
||||
await engine.setConfig('search.cross_modal.llm_intent', 'true');
|
||||
let chatCalled = 0;
|
||||
__setChatTransportForTests(async () => {
|
||||
chatCalled++;
|
||||
return { text: 'image', blocks: [], stopReason: 'end', usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'x', providerId: 'x' };
|
||||
});
|
||||
await hybridSearch(engine, 'the chart', { limit: 5 });
|
||||
expect(chatCalled).toBe(1);
|
||||
});
|
||||
|
||||
test('flag ON + unambiguous text query → no LLM call', async () => {
|
||||
await engine.setConfig('search.cross_modal.llm_intent', 'true');
|
||||
let chatCalled = 0;
|
||||
__setChatTransportForTests(async () => {
|
||||
chatCalled++;
|
||||
return { text: 'image', blocks: [], stopReason: 'end', usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'x', providerId: 'x' };
|
||||
});
|
||||
await hybridSearch(engine, 'what is founder mode', { limit: 5 });
|
||||
expect(chatCalled).toBe(0);
|
||||
});
|
||||
|
||||
test('flag ON + regex-confident image query → no LLM call (regex already classified)', async () => {
|
||||
await engine.setConfig('search.cross_modal.llm_intent', 'true');
|
||||
let chatCalled = 0;
|
||||
__setChatTransportForTests(async () => {
|
||||
chatCalled++;
|
||||
return { text: 'image', blocks: [], stopReason: 'end', usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'x', providerId: 'x' };
|
||||
});
|
||||
// Strong regex match: "show me photos from X" → already image.
|
||||
await hybridSearch(engine, 'show me photos from the hackathon', { limit: 5 });
|
||||
// No tie-break needed when regex is already confident.
|
||||
expect(chatCalled).toBe(0);
|
||||
});
|
||||
|
||||
test('flag ON + explicit crossModal opt → no LLM call (per-call opt wins)', async () => {
|
||||
await engine.setConfig('search.cross_modal.llm_intent', 'true');
|
||||
let chatCalled = 0;
|
||||
__setChatTransportForTests(async () => {
|
||||
chatCalled++;
|
||||
return { text: 'image', blocks: [], stopReason: 'end', usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 }, model: 'x', providerId: 'x' };
|
||||
});
|
||||
// Caller passed explicit crossModal — no need to tie-break.
|
||||
await hybridSearch(engine, 'the chart', { crossModal: 'text', limit: 5 });
|
||||
expect(chatCalled).toBe(0);
|
||||
});
|
||||
|
||||
test('flag ON + ambiguous + LLM throws → falls back to regex result (text)', async () => {
|
||||
await engine.setConfig('search.cross_modal.llm_intent', 'true');
|
||||
__setChatTransportForTests(async () => {
|
||||
throw new Error('LLM unavailable');
|
||||
});
|
||||
// Should not throw — fail-open to regex result.
|
||||
const results = await hybridSearch(engine, 'the chart', { limit: 5 });
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,168 @@
|
||||
// Commit 2 (Phase 2): search_by_image MCP op trust-boundary + spend cap.
|
||||
//
|
||||
// Covers:
|
||||
// - D18: remote (ctx.remote=true) + image_path is rejected
|
||||
// - Local (ctx.remote=false) + image_path is accepted
|
||||
// - Missing all three of image_path/url/data is rejected
|
||||
// - Multiple of image_path/url/data is rejected
|
||||
// - D23-#6 spend cap blocks at budget; allows under budget
|
||||
// - Spend log records on successful call
|
||||
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { operationsByName } from '../src/core/operations.ts';
|
||||
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
||||
import { recordSpend, getTodaySpendCents } from '../src/core/spend-log.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let tmpRoot: string;
|
||||
|
||||
const PNG_BYTES = Buffer.from([
|
||||
0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A,
|
||||
0, 0, 0, 13, 73, 72, 68, 82, 0, 0, 0, 1, 0, 0, 0, 1, 8, 6, 0, 0, 0,
|
||||
31, 21, 196, 137, 0, 0, 0, 12, 73, 68, 65, 84, 8, 87, 99, 248, 207, 192, 0, 0, 0, 3, 0, 1,
|
||||
90, 12, 105, 240, 0, 0, 0, 0, 73, 69, 78, 68, 174, 66, 96, 130,
|
||||
]);
|
||||
|
||||
type FetchHandler = (url: string, init: RequestInit) => Promise<Response>;
|
||||
let fetchHandler: FetchHandler | null = null;
|
||||
const origFetch = globalThis.fetch;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
tmpRoot = mkdtempSync(join(tmpdir(), 'gbrain-search-by-image-op-'));
|
||||
fetchHandler = async () => new Response(
|
||||
JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.1), index: 0 }],
|
||||
model: 'voyage-multimodal-3',
|
||||
}),
|
||||
{ status: 200 },
|
||||
);
|
||||
globalThis.fetch = (async (url: string | URL | Request, init?: RequestInit) => {
|
||||
if (!fetchHandler) throw new Error('no fetch handler');
|
||||
return fetchHandler(typeof url === 'string' ? url : url.toString(), init ?? {});
|
||||
}) as typeof fetch;
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
embedding_multimodal_model: 'voyage:voyage-multimodal-3',
|
||||
env: { OPENAI_API_KEY: 'test', VOYAGE_API_KEY: 'test' },
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = origFetch;
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
const op = () => operationsByName.search_by_image;
|
||||
|
||||
describe('search_by_image op — D18 remote image_path ban', () => {
|
||||
test('rejects image_path when ctx.remote=true', async () => {
|
||||
const path = join(tmpRoot, 'test.png');
|
||||
writeFileSync(path, PNG_BYTES);
|
||||
const err = await op().handler(
|
||||
{ engine, remote: true, auth: { token: 't', clientId: 'c1', scopes: ['read'] } } as any,
|
||||
{ image_path: path },
|
||||
).catch((e: any) => e as Error);
|
||||
expect(err).toBeInstanceOf(Error);
|
||||
expect((err as Error).message).toContain('permission_denied');
|
||||
});
|
||||
|
||||
test('accepts image_path when ctx.remote=false (local CLI)', async () => {
|
||||
const path = join(tmpRoot, 'test.png');
|
||||
writeFileSync(path, PNG_BYTES);
|
||||
const results = await op().handler(
|
||||
{ engine, remote: false } as any,
|
||||
{ image_path: path },
|
||||
);
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('search_by_image op — input validation', () => {
|
||||
test('rejects missing all three inputs', async () => {
|
||||
const err = await op().handler(
|
||||
{ engine, remote: false } as any,
|
||||
{},
|
||||
).catch((e: any) => e as Error);
|
||||
expect(err).toBeInstanceOf(Error);
|
||||
expect((err as Error).message).toMatch(/image_path|image_url|image_data/);
|
||||
});
|
||||
|
||||
test('rejects multiple inputs together', async () => {
|
||||
const path = join(tmpRoot, 'test.png');
|
||||
writeFileSync(path, PNG_BYTES);
|
||||
const err = await op().handler(
|
||||
{ engine, remote: false } as any,
|
||||
{ image_path: path, image_data: PNG_BYTES.toString('base64') },
|
||||
).catch((e: any) => e as Error);
|
||||
expect(err).toBeInstanceOf(Error);
|
||||
expect((err as Error).message).toMatch(/only one of/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('search_by_image op — D23-#6 spend cap', () => {
|
||||
test('blocks remote call when daily spend already at budget', async () => {
|
||||
// Configure budget to $0.05 cap.
|
||||
await engine.setConfig('search.image_query.daily_budget_usd_per_client', '0.05');
|
||||
// Pre-record $0.05 = 5 cents of spend for client_a.
|
||||
await recordSpend(engine, {
|
||||
clientId: 'client_a',
|
||||
operation: 'search_by_image',
|
||||
spendCents: 5,
|
||||
});
|
||||
// Verify the recorded spend is at the budget.
|
||||
const spent = await getTodaySpendCents(engine, 'client_a');
|
||||
expect(spent).toBeGreaterThanOrEqual(5);
|
||||
|
||||
const err = await op().handler(
|
||||
{ engine, remote: true, auth: { token: 't', clientId: 'client_a', scopes: ['read'] } } as any,
|
||||
{ image_data: PNG_BYTES.toString('base64') },
|
||||
).catch((e: any) => e as Error);
|
||||
expect(err).toBeInstanceOf(Error);
|
||||
expect((err as Error).message).toContain('Daily Voyage spend cap reached');
|
||||
});
|
||||
|
||||
test('allows remote call when under budget', async () => {
|
||||
await engine.setConfig('search.image_query.daily_budget_usd_per_client', '5');
|
||||
// No prior spend recorded.
|
||||
const results = await op().handler(
|
||||
{ engine, remote: true, auth: { token: 't', clientId: 'client_b', scopes: ['read'] } } as any,
|
||||
{ image_data: PNG_BYTES.toString('base64') },
|
||||
);
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
// Verify spend was recorded after the call.
|
||||
// (Allow a small tick for the async best-effort recordSpend.)
|
||||
await new Promise(r => setTimeout(r, 20));
|
||||
const spent = await getTodaySpendCents(engine, 'client_b');
|
||||
expect(spent).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('local CLI calls bypass budget gate (ctx.remote=false, no clientId)', async () => {
|
||||
// Even with cap=$0 set, local call is allowed.
|
||||
await engine.setConfig('search.image_query.daily_budget_usd_per_client', '0.01');
|
||||
await recordSpend(engine, { clientId: 'somebody', operation: 'search_by_image', spendCents: 100 });
|
||||
const path = join(tmpRoot, 'local.png');
|
||||
writeFileSync(path, PNG_BYTES);
|
||||
const results = await op().handler(
|
||||
{ engine, remote: false } as any,
|
||||
{ image_path: path },
|
||||
);
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -38,6 +38,17 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
|
||||
// The cell-by-cell assertion. The methodology doc cites these.
|
||||
// v0.35.0.0+ extended with 5 reranker fields. tokenmax flips reranker on;
|
||||
// conservative + balanced keep it off until eval data backs a change.
|
||||
// v0.36 cross-modal wave: shared defaults across all modes (opt-in).
|
||||
const CROSS_MODAL_DEFAULTS = {
|
||||
cross_modal_both_text_weight: 0.6,
|
||||
cross_modal_both_image_weight: 0.4,
|
||||
image_query_text_refinement_weight: 0.4,
|
||||
image_query_image_refinement_weight: 0.6,
|
||||
unified_multimodal: false,
|
||||
unified_multimodal_only: false,
|
||||
cross_modal_llm_intent: false,
|
||||
};
|
||||
|
||||
test('conservative bundle values are canonical', () => {
|
||||
expect(MODE_BUNDLES.conservative).toEqual({
|
||||
cache_enabled: true,
|
||||
@@ -52,9 +63,8 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
|
||||
reranker_top_n_in: 30,
|
||||
reranker_top_n_out: null,
|
||||
reranker_timeout_ms: 5000,
|
||||
// v0.35.6.0 — floor_ratio undefined in all three bundles; the per-corpus
|
||||
// ablation TODO gates any default flip.
|
||||
floor_ratio: undefined,
|
||||
...CROSS_MODAL_DEFAULTS,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -75,6 +85,7 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
|
||||
reranker_top_n_out: null,
|
||||
reranker_timeout_ms: 5000,
|
||||
floor_ratio: undefined,
|
||||
...CROSS_MODAL_DEFAULTS,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -93,6 +104,7 @@ describe('SEARCH_MODES + MODE_BUNDLES canonical shape', () => {
|
||||
reranker_top_n_out: null,
|
||||
reranker_timeout_ms: 5000,
|
||||
floor_ratio: undefined,
|
||||
...CROSS_MODAL_DEFAULTS,
|
||||
});
|
||||
});
|
||||
|
||||
@@ -272,11 +284,13 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
|
||||
test('KNOBS_HASH_VERSION constant exposed for migrations to bump on schema change', () => {
|
||||
// v0.35.0.0+ bumped 1→2 to fold reranker fields into the cache key.
|
||||
// v0.35.6.0 bumped 2→3 to fold floor_ratio into the cache key
|
||||
// (codex T1 — preventing cross-floor cache contamination).
|
||||
// v0.36 also extends v=3 with embedding column + provider (D8 / CDX-2)
|
||||
// so a query against `embedding_voyage` never shares a cache row with
|
||||
// `embedding`, even when all other knobs match.
|
||||
// v0.35.6.0 bumped 2→3 to fold floor_ratio (codex outside-voice T1 —
|
||||
// preventing cross-floor cache contamination).
|
||||
// v0.36 piggybacks on v=3 with 7 additional cross-modal knobs (D2) PLUS
|
||||
// embedding column + provider context (D8/CDX-2 cross-column isolation),
|
||||
// all appended per CDX2-F13 append-only convention so a text-mode cache
|
||||
// hit can never silently serve to an image-mode caller, and a query
|
||||
// against `embedding_voyage` never shares a cache row with `embedding`.
|
||||
expect(KNOBS_HASH_VERSION).toBe(3);
|
||||
});
|
||||
|
||||
|
||||
@@ -43,7 +43,11 @@ function baseKnobs(): ResolvedSearchKnobs {
|
||||
}
|
||||
|
||||
describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
test('version is 3 (1→2 v0.35.0.0 reranker; 2→3 v0.35.6.0 floor_ratio + v0.36 embedding-column)', () => {
|
||||
test('version is 3 (1→2 v0.35.0.0 reranker; 2→3 v0.35.6.0 floor_ratio + v0.36 cross-modal + embedding-column appends)', () => {
|
||||
// v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold
|
||||
// floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs
|
||||
// (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation),
|
||||
// all appended per CDX2-F13 append-only convention.
|
||||
expect(KNOBS_HASH_VERSION).toBe(3);
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
// Commit 0 (D19): SSRF validation with DNS resolution.
|
||||
//
|
||||
// Covers the gap that `isInternalUrl` in src/core/url-safety.ts leaves open:
|
||||
// DNS rebinding. validateAndResolveUrl does its own DNS lookup and rejects
|
||||
// hostnames whose resolved IPs land internal.
|
||||
//
|
||||
// Tests use the __setDnsLookupForTests seam so the real network is never hit.
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
SSRFError,
|
||||
__setDnsLookupForTests,
|
||||
validateAndResolveUrl,
|
||||
} from '../src/core/ssrf-validate.ts';
|
||||
|
||||
type DnsRecord = { address: string; family: number };
|
||||
let stubAddrs: Map<string, DnsRecord[]> = new Map();
|
||||
|
||||
beforeEach(() => {
|
||||
stubAddrs = new Map();
|
||||
__setDnsLookupForTests((async (host: string, _opts: any) => {
|
||||
const recs = stubAddrs.get(host);
|
||||
if (!recs) {
|
||||
const err = new Error(`stub: no DNS records for ${host}`);
|
||||
(err as any).code = 'ENOTFOUND';
|
||||
throw err;
|
||||
}
|
||||
return recs;
|
||||
}) as any);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
__setDnsLookupForTests(undefined);
|
||||
});
|
||||
|
||||
describe('validateAndResolveUrl — static rejections (via isInternalUrl)', () => {
|
||||
test('rejects http://127.0.0.1 (loopback)', async () => {
|
||||
await expect(validateAndResolveUrl('http://127.0.0.1/img.png')).rejects.toBeInstanceOf(SSRFError);
|
||||
});
|
||||
test('rejects http://169.254.169.254 (AWS metadata)', async () => {
|
||||
await expect(validateAndResolveUrl('http://169.254.169.254/latest/meta-data/')).rejects.toBeInstanceOf(SSRFError);
|
||||
});
|
||||
test('rejects http://2130706433 (decimal-encoded 127.0.0.1)', async () => {
|
||||
await expect(validateAndResolveUrl('http://2130706433/')).rejects.toBeInstanceOf(SSRFError);
|
||||
});
|
||||
test('rejects http://0x7f000001 (hex-encoded 127.0.0.1)', async () => {
|
||||
await expect(validateAndResolveUrl('http://0x7f000001/')).rejects.toBeInstanceOf(SSRFError);
|
||||
});
|
||||
test('rejects metadata.google.internal hostname', async () => {
|
||||
await expect(validateAndResolveUrl('http://metadata.google.internal/')).rejects.toBeInstanceOf(SSRFError);
|
||||
});
|
||||
test('rejects localhost. (trailing dot)', async () => {
|
||||
await expect(validateAndResolveUrl('http://localhost./')).rejects.toBeInstanceOf(SSRFError);
|
||||
});
|
||||
test('rejects IPv6 link-local fe80::1', async () => {
|
||||
await expect(validateAndResolveUrl('http://[fe80::1]/')).rejects.toBeInstanceOf(SSRFError);
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateAndResolveUrl — scheme + credentials', () => {
|
||||
test('rejects file:// scheme', async () => {
|
||||
const err = await validateAndResolveUrl('file:///etc/passwd').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
// file:// is caught by static `isInternalUrl` first (it returns true for non-http(s))
|
||||
expect(['INVALID_SCHEME', 'INTERNAL_HOST']).toContain(err.code);
|
||||
});
|
||||
test('rejects gopher:// scheme', async () => {
|
||||
const err = await validateAndResolveUrl('gopher://example.com/').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
expect(['INVALID_SCHEME', 'INTERNAL_HOST']).toContain(err.code);
|
||||
});
|
||||
test('rejects credentials embedded in URL', async () => {
|
||||
stubAddrs.set('example.com', [{ address: '93.184.216.34', family: 4 }]);
|
||||
const err = await validateAndResolveUrl('http://user:pass@example.com/').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
expect(err.code).toBe('CREDENTIALS_IN_URL');
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateAndResolveUrl — DNS rebinding defense', () => {
|
||||
test('rejects when hostname resolves to 127.0.0.1', async () => {
|
||||
stubAddrs.set('attacker.com', [{ address: '127.0.0.1', family: 4 }]);
|
||||
const err = await validateAndResolveUrl('http://attacker.com/img.png').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
expect(err.code).toBe('DNS_RESOLVED_INTERNAL');
|
||||
});
|
||||
test('rejects when hostname resolves to 169.254.169.254 (AWS metadata)', async () => {
|
||||
stubAddrs.set('attacker.com', [{ address: '169.254.169.254', family: 4 }]);
|
||||
const err = await validateAndResolveUrl('http://attacker.com/').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
expect(err.code).toBe('DNS_RESOLVED_INTERNAL');
|
||||
});
|
||||
test('rejects when ANY resolved record points internal (DNS rebinding multi-record)', async () => {
|
||||
stubAddrs.set('mixed.com', [
|
||||
{ address: '8.8.8.8', family: 4 },
|
||||
{ address: '10.0.0.1', family: 4 }, // private — rejects whole set
|
||||
]);
|
||||
const err = await validateAndResolveUrl('http://mixed.com/').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
expect(err.code).toBe('DNS_RESOLVED_INTERNAL');
|
||||
});
|
||||
test('rejects IPv6 ULA resolved record', async () => {
|
||||
stubAddrs.set('attacker.com', [{ address: 'fc00::1', family: 6 }]);
|
||||
const err = await validateAndResolveUrl('http://attacker.com/').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
expect(err.code).toBe('DNS_RESOLVED_INTERNAL');
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateAndResolveUrl — happy path', () => {
|
||||
test('resolves public IPv4 address and returns target', async () => {
|
||||
stubAddrs.set('example.com', [{ address: '93.184.216.34', family: 4 }]);
|
||||
const target = await validateAndResolveUrl('https://example.com/img.png');
|
||||
expect(target.resolvedIp).toBe('93.184.216.34');
|
||||
expect(target.originalHost).toBe('example.com');
|
||||
expect(target.ipv6).toBe(false);
|
||||
expect(target.resolvedUrl).toContain('93.184.216.34');
|
||||
});
|
||||
test('public IP literal passes through without DNS lookup', async () => {
|
||||
const target = await validateAndResolveUrl('https://93.184.216.34/img.png');
|
||||
expect(target.resolvedIp).toBe('93.184.216.34');
|
||||
expect(target.originalHost).toBe(''); // literal — no original hostname
|
||||
});
|
||||
test('public IPv6 literal passes through', async () => {
|
||||
const target = await validateAndResolveUrl('https://[2606:2800:220:1::1]/');
|
||||
expect(target.resolvedIp).toBe('2606:2800:220:1::1');
|
||||
expect(target.ipv6).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateAndResolveUrl — DNS resolution failures', () => {
|
||||
test('rejects when DNS lookup fails (ENOTFOUND)', async () => {
|
||||
const err = await validateAndResolveUrl('http://nonexistent.invalid/').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
expect(err.code).toBe('DNS_RESOLUTION_FAILED');
|
||||
});
|
||||
test('rejects when DNS lookup returns empty records', async () => {
|
||||
stubAddrs.set('empty.com', []);
|
||||
const err = await validateAndResolveUrl('http://empty.com/').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
expect(err.code).toBe('DNS_RESOLUTION_FAILED');
|
||||
});
|
||||
});
|
||||
|
||||
describe('validateAndResolveUrl — malformed URLs', () => {
|
||||
test('rejects unparseable URL', async () => {
|
||||
const err = await validateAndResolveUrl('not a url at all').catch(e => e);
|
||||
expect(err).toBeInstanceOf(SSRFError);
|
||||
expect(err.code).toBe('INVALID_URL');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,169 @@
|
||||
// Commit 3 (Phase 3): unified multimodal column.
|
||||
//
|
||||
// Covers:
|
||||
// - Schema migration v68 adds embedding_multimodal column
|
||||
// - searchVector routes to embedding_multimodal when opts.embeddingColumn set
|
||||
// - hybridSearch routes through unified column when search.unified_multimodal=true
|
||||
// - D8 fail-open: unified-only=false + empty unified column → falls back to text
|
||||
// - D8 strict: unified-only=true + empty column → does not fall back
|
||||
// - reindex --multimodal cost estimate + dry-run + GBRAIN_NO_REEMBED bypass
|
||||
// - D7 lock acquired during reindex; second reindex receives LOCK_HELD
|
||||
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
import { hybridSearch } from '../src/core/search/hybrid.ts';
|
||||
import { runReindexMultimodal } from '../src/commands/reindex-multimodal.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let fetchHandler: ((url: string, init: RequestInit) => Promise<Response>) | null = null;
|
||||
const origFetch = globalThis.fetch;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
fetchHandler = async () => new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.1), index: 0 }],
|
||||
model: 'voyage-multimodal-3',
|
||||
}), { status: 200 });
|
||||
globalThis.fetch = (async (url: string | URL | Request, init?: RequestInit) => {
|
||||
if (!fetchHandler) throw new Error('no fetch handler');
|
||||
return fetchHandler(typeof url === 'string' ? url : url.toString(), init ?? {});
|
||||
}) as typeof fetch;
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
embedding_multimodal_model: 'voyage:voyage-multimodal-3',
|
||||
env: { OPENAI_API_KEY: 'test', VOYAGE_API_KEY: 'test' },
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
globalThis.fetch = origFetch;
|
||||
resetGateway();
|
||||
});
|
||||
|
||||
describe('Phase 3 schema — v68 migration', () => {
|
||||
test('content_chunks has embedding_multimodal column', async () => {
|
||||
// Run an explicit query against the column. If the migration ran, this succeeds.
|
||||
const rows = await engine.executeRaw<{ count: string }>(
|
||||
`SELECT COUNT(*)::text AS count FROM content_chunks WHERE embedding_multimodal IS NULL`,
|
||||
);
|
||||
expect(rows.length).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('reindex --multimodal command (Phase 3)', () => {
|
||||
test('--dry-run reports cost estimate without mutating', async () => {
|
||||
// No rows in DB → pending=0, no work needed.
|
||||
const result = await runReindexMultimodal(engine, { dryRun: true });
|
||||
expect(result.dry_run).toBe(true);
|
||||
expect(result.reembedded).toBe(0);
|
||||
});
|
||||
|
||||
test('--cost-estimate reports cost but does not run', async () => {
|
||||
const result = await runReindexMultimodal(engine, { costEstimate: true });
|
||||
expect(result.dry_run).toBe(true);
|
||||
expect(result.reembedded).toBe(0);
|
||||
});
|
||||
|
||||
test('GBRAIN_NO_REEMBED=1 honored on zero-pending brain (skip path is no-op-clean)', async () => {
|
||||
await withEnv({ GBRAIN_NO_REEMBED: '1' }, async () => {
|
||||
const result = await runReindexMultimodal(engine, {});
|
||||
// Zero pending → reindex short-circuits before the env-var check; both
|
||||
// paths produce dry_run=false + reembedded=0 + pending=0.
|
||||
expect(result.reembedded).toBe(0);
|
||||
expect(result.pending_after).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
test('zero-pending returns cleanly', async () => {
|
||||
const result = await runReindexMultimodal(engine, { yes: true });
|
||||
expect(result.pending_before).toBe(0);
|
||||
expect(result.reembedded).toBe(0);
|
||||
expect(result.failed).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hybridSearch unified routing (Phase 3)', () => {
|
||||
test('search.unified_multimodal=true routes ALL queries through embedding_multimodal', async () => {
|
||||
await engine.setConfig('search.unified_multimodal', 'true');
|
||||
let voyageCalled = 0;
|
||||
let openaiCalled = 0;
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
voyageCalled++;
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.1), index: 0 }],
|
||||
}), { status: 200 });
|
||||
}
|
||||
if (url.includes('api.openai.com') && url.includes('embeddings')) {
|
||||
openaiCalled++;
|
||||
}
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1536 }, () => 0.1), index: 0 }],
|
||||
}), { status: 200 });
|
||||
};
|
||||
|
||||
await hybridSearch(engine, 'totally text query', { limit: 5 });
|
||||
// Unified routing: text query forced to multimodal endpoint.
|
||||
expect(voyageCalled).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('D8 fail-open: empty unified column + not strict → falls back to text', async () => {
|
||||
// Set unified flag but DON'T set unified_multimodal_only. Empty DB → unified returns [].
|
||||
await engine.setConfig('search.unified_multimodal', 'true');
|
||||
let openaiCalled = 0;
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.1), index: 0 }],
|
||||
}), { status: 200 });
|
||||
}
|
||||
openaiCalled++;
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1536 }, () => 0.1), index: 0 }],
|
||||
}), { status: 200 });
|
||||
};
|
||||
|
||||
const results = await hybridSearch(engine, 'whatever', { limit: 5 });
|
||||
expect(Array.isArray(results)).toBe(true);
|
||||
// The fall-back path SHOULD call OpenAI (text path) when unified came back empty.
|
||||
expect(openaiCalled).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('D8 strict: unified_multimodal_only=true + empty column → does NOT fall back', async () => {
|
||||
await engine.setConfig('search.unified_multimodal', 'true');
|
||||
await engine.setConfig('search.unified_multimodal_only', 'true');
|
||||
let openaiCalled = 0;
|
||||
fetchHandler = async (url) => {
|
||||
if (url.includes('multimodalembeddings')) {
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1024 }, () => 0.1), index: 0 }],
|
||||
}), { status: 200 });
|
||||
}
|
||||
openaiCalled++;
|
||||
return new Response(JSON.stringify({
|
||||
data: [{ embedding: Array.from({ length: 1536 }, () => 0.1), index: 0 }],
|
||||
}), { status: 200 });
|
||||
};
|
||||
|
||||
await hybridSearch(engine, 'whatever', { limit: 5 });
|
||||
// Strict mode means NO text fallback even when unified is empty.
|
||||
expect(openaiCalled).toBe(0);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user