mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
v0.36.6.0 feat: cross-modal search wave (text↔image + unified column + LLM intent) (#1165)
* feat(cross-modal/0): batched multimodal + query helpers + SSRF helper
Commit 0 of the cross-modal search wave. Foundation for Phase 1-3:
- embedMultimodal accepts MultimodalInput text variant + EmbedMultimodalOpts
with inputType: 'document' | 'query' (D22-2). Default unchanged so
importImageFile keeps document-side embedding.
- embedQueryMultimodal(text) + embedQueryMultimodalImage(input) wrappers
for hybridSearch + searchByImage query paths.
- embedMultimodalSafe binary-search retry on transient batch failure +
failed_indices surfacing. Phase 3 reindex uses this so a single bad
chunk doesn't discard the 31 in-flight embeddings around it.
- Voyage path: text + image inputs in one batch via content arrays.
- openai-compat path: text + image inputs in one request per input.
- src/core/ssrf-validate.ts (D19): DNS-resolve-and-fetch-by-IP defense
for redirect chains. Closes the DNS-rebinding gap that url-safety.ts'
static check leaves open. Uses node:dns/promises with {all: true,
family: 0} to inspect every A and AAAA record before connecting.
fetchWithSSRFGuard helper validates per-redirect-hop and limits chain
depth (default 3).
- Re-exports from src/core/embedding.ts public seam.
Tests:
- test/embed-multimodal-batching.test.ts (13 cases): text variant, query
inputType discipline, mixed text+image batches, embedQueryMultimodal,
embedQueryMultimodalImage, embedMultimodalSafe happy/empty/all-fail/
mid-batch-recovery/permanent-misconfig.
- test/ssrf-validate.test.ts (20 cases): static rejections via
isInternalUrl, scheme + credentials rejection, DNS rebinding defense
(single-record + multi-record), public happy path, IPv6 literals,
malformed URLs.
No regression in existing voyage-multimodal.test.ts or
openai-compat-multimodal.test.ts (33 cases all pass).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cross-modal/1): Phase 1 text→image routing + knobsHash + RRF + backfill
Phase 1 of the cross-modal search wave. Wires the existing 1024d Voyage
multimodal embedding space (already populated for image chunks via
importImageFile) into the user-facing query path. Text queries that match
cross-modal intent regex route through Voyage multimodal-3 instead of the
text embedding model, then search content_chunks.embedding_image.
- query-intent.ts: new `suggestedModality: 'text' | 'image' | 'both'`
axis on `QuerySuggestions`. Module-scope CROSS_MODAL_PATTERNS regex
array (D15 — compiled once at module load). Conservative on purpose;
LLM intent escalation (Commit 4) catches genuinely ambiguous phrasings.
- query-intent.ts: new `isAmbiguousModalityQuery(query)` pure heuristic
for Commit 4's escalation gate. Returns true ONLY when regex misses
AND a visual noun + reference marker both fire.
- types.ts: `SearchOpts.crossModal: 'text' | 'image' | 'both' | 'auto'`
+ `SearchResult.modality: 'text' | 'image'` for downstream renderers.
- mode.ts: 7 new knobs in ModeBundle (D2): cross_modal_both_text_weight,
cross_modal_both_image_weight, image_query_text_refinement_weight,
image_query_image_refinement_weight, unified_multimodal,
unified_multimodal_only, cross_modal_llm_intent. All three mode
bundles default to the same values (cross-modal is opt-in).
- mode.ts: D2 cache-key fix — KNOBS_HASH_VERSION bumped 2→3, all 7 new
knobs participate in knobsHash so a text-mode cache hit can't be
served to an image-mode caller.
- mode.ts: D3 registry — all 7 keys land in SEARCH_MODE_CONFIG_KEYS so
`gbrain search modes` / `stats` / `tune` see them.
- hybrid.ts: routing branch at the embed step. Resolves effective
modality from (per-call opts → suggestions → 'text'). Image route:
embedQueryMultimodal + searchVector(embedding_image), skip expansion
+ keyword (D9 mode-bundle override). Both route: parallel text + image
vector searches merged via weighted RRF (D6) with cross_modal_both_*
weights. Fail-open: multimodal misconfigured → structured warn + text
fallback. 'auto' literal normalized to undefined (D22-1).
- operations.ts: thread `cross_modal` param through `query` op.
- backfill-registry.ts: new `modality` backfill kind. SQL filter requires
`chunk_source='image_asset'` (D22-7 defensive guard). Idempotent.
- doctor.ts: `cross_modal_modality_backfill` check surfaces unflagged
image-asset chunks with paste-ready `gbrain backfill modality` hint.
Tests:
- cross-modal-phase1.test.ts (45 cases): regex classification (positive
+ negative + plural-safe), isAmbiguousModalityQuery, D3 registry, D2
knobsHash diffs across all 7 new knobs, MODE_BUNDLES defaults,
resolveSearchMode precedence chain.
- cross-modal-hybrid-integration.test.ts (7 cases): PGLite + stubbed
gateway. Verifies image-modality calls Voyage and not OpenAI, text
calls OpenAI and not Voyage, 'auto' literal normalizes, 'both' mode
hits both endpoints, fail-open routes to text on multimodal misconfig.
- search-mode.test.ts: updated MODE_BUNDLES + KNOBS_HASH_VERSION
assertions (148 cross-suite tests still pass; no regression).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cross-modal/2): Phase 2 image-as-query + D18 path ban + D23-#6 spend cap
Phase 2 of the cross-modal search wave. Adds the `search_by_image` MCP op,
the SSRF-defended image loader, and the daily per-OAuth-client spend cap
on paid Voyage multimodal calls. D17 honest framing applied: Phase 2 ships
image→similar-images + image-OCR-text retrieval. True image→full-text-
knowledge requires Phase 3's unified column.
- src/core/search/image-loader.ts: loadImageInput accepts local path,
data: URI, or http(s):// URL. Magic-byte sniff for PNG/JPEG/WebP (no
other formats). Hard size cap (10MB local default, 2MB remote default).
http(s) path uses fetchWithSSRFGuard from Commit 0: every redirect hop
re-resolved via DNS lookup + every record checked against the internal
IP deny list. Max 3 redirect hops. 5s total fetch timeout. Pre-flight
Content-Length check + post-fetch size guard for lying servers.
- src/core/search/by-image.ts: searchByImage runs the image branch
always; D13 hybrid intersect runs a parallel text branch when
`query` is provided, merged via weighted RRF. Phase 3 will widen
the column routing to embedding_multimodal once that lands.
- src/core/operations.ts: new search_by_image op (scope: read, NOT
localOnly). D18 P0 — when ctx.remote === true AND image_path is set,
rejects with permission_denied at handler entry (validateParams would
catch it again at dispatch). D5 source-id thread via sourceScopeOpts.
D12 per-param length cap enforced via remote-vs-local maxBytes config
read at handler entry. D23-#6 pre-flight checkBudget + post-call
recordSpend (best-effort; failures don't block response).
- src/core/spend-log.ts: BudgetExceededError + checkBudget + recordSpend
+ getTodaySpendCents. UTC day-aligned aggregation so the cap rolls
over deterministically. Local CLI callers (no clientId) bypass the
gate entirely. Pre-v0.36 brains without the mcp_spend_log table fail
open to spend=0; the migration brings the table in on first start.
- src/core/migrate.ts: new migration v67 mcp_spend_log table + indexes
for the (client_id, day) and (token_name, day) hot reads. PGLite
parity via sqlFor.pglite.
- src/core/search/hybrid.ts: RRF_K constant exported so by-image.ts can
share the same effective-K math as the main hybrid path.
Tests:
- cross-modal-phase2.test.ts (15 cases): magic-byte sniffing (PNG +
JPEG + WebP positive, GIF rejection), oversized rejection (default +
custom cap), data: URI happy path + malformed + decoded-non-image
+ oversized, invalid input shapes (empty + ftp), SSRF defense via
DNS rebinding stub.
- search-by-image-op.test.ts (7 cases): D18 remote image_path
rejection + local CLI accepts; input validation (missing all three /
multiple together); D23-#6 budget block-at-cap + allow-under-cap +
local-CLI-bypass; migration v67 mcp_spend_log table applied cleanly.
All 166 tests across the cross-modal suite pass; no regression in
existing voyage-multimodal / openai-compat-multimodal / search-mode suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cross-modal/3): Phase 3 unified column + reindex + D8 fail-open + D23-#2
Phase 3 of the cross-modal search wave. Adds the unified multimodal column
on content_chunks + the `gbrain reindex --multimodal` sweep + the
`search.unified_multimodal` routing flag with D8 source-aware coverage
guard + fail-open behavior. D17 honest framing: this is the phase that
unlocks true image→full-text-knowledge — Phase 2's searchByImage
transparently upgrades to the richer retrieval once the unified column
has coverage.
D10 reindex-core extraction filed as a follow-up TODO. The existing
markdown reindex walks pages and re-imports via importFromFile; this
walks content_chunks and re-embeds via the gateway. Patterns rhyme but
cores diverge enough that extraction balloons the diff. Both commands
stand alone with their own checkpoint + cost-prompt logic.
- migrate.ts v68 (embedding_multimodal_column): column-only ALTER on
content_chunks. HNSW partial index deferred to post-reindex build
(D20: pgvector docs recommend post-load build for HNSW). Both engines.
- types.ts SearchOpts.embeddingColumn type widened to include
'embedding_multimodal'.
- postgres-engine.ts + pglite-engine.ts searchVector: route to
embedding_multimodal column when opts.embeddingColumn set. NO modality
filter (unified column carries both text + image content).
- hybrid.ts unified routing branch: when search.unified_multimodal=true,
bypasses dual-column branching and runs embedQueryMultimodal +
searchVector(embedding_multimodal). D8 fail-open: zero rows + not
strict-mode → falls through to dual-column text path with structured
warning. search.unified_multimodal_only=true bypasses the fallback.
- src/commands/reindex-multimodal.ts: `gbrain reindex --multimodal`.
D7 lock via tryAcquireDbLock('gbrain-reindex-multimodal'); 6h TTL.
Cost prompt + 10s Ctrl-C grace window in TTY; auto-proceeds non-TTY.
GBRAIN_NO_REEMBED=1 bypass. Checkpoint at
~/.gbrain/reindex-multimodal-checkpoint.json for resume. D23-#2
auto-flip prompt at coverage=100% completion.
- cli.ts: `gbrain reindex --multimodal` dispatch with --limit, --dry-run,
--cost-estimate, --no-embed, --yes, --json flags.
- doctor.ts: unified_multimodal_coverage check (D21 source-aware) +
reports per-source % when search.unified_multimodal is on. Warns at
<95% lowest source; fails when unified_multimodal_only=true AND
lowest source <99%. Falls open to OK when column not yet present.
Tests:
- unified-multimodal.test.ts (8 cases): schema migration v68 applies,
reindex --dry-run + --cost-estimate + GBRAIN_NO_REEMBED bypass +
zero-pending fast-path, hybridSearch unified routing forces voyage
endpoint, D8 fail-open routes to text on empty unified, D8 strict
blocks text fallback.
All 211 tests across the cross-modal + related suite pass; no
regression in voyage-multimodal / openai-compat-multimodal / search-mode
/ intent / search base suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cross-modal/4): LLM intent escalation for ambiguous modality
Commit 4 of the cross-modal search wave (opt-in default off).
When `search.cross_modal.llm_intent` is true AND the regex classifier
returned 'text' AND `isAmbiguousModalityQuery(query)` fires, hybridSearch
awaits a Haiku tie-break via gateway.chat() before routing. The
ambiguous-modality gate (introduced in Commit 1) ensures the LLM call
only fires on the narrow band where regex misses but a visual noun +
reference marker both fire — roughly <1% of queries with the flag on.
- src/core/search/llm-intent.ts: new module. `classifyModalityWithLLM`
routes through gateway.chat() with a fixed system prompt ("Output
exactly one word: text, image, or both"). 1s timeout via AbortController.
`parseModality` is a pure exported helper that tolerates trailing
punctuation + casing. Fail-open on every error path (gateway
unavailable, timeout, parse failure, unrecognized output).
- src/core/search/hybrid.ts: escalation branch slots BEFORE the unified
routing branch. Gated by: no explicit per-call crossModal opt, regex
result == 'text', config flag on, ambiguity heuristic fires. Fail-open
to regex result on any error from the LLM tie-break.
Tests:
- llm-intent-escalation.test.ts (14 cases): parseModality tolerance
matrix (text / image / both / trailing punct / whitespace /
unrecognized / empty), classifyModalityWithLLM happy paths for all 3
outputs, fail-open on throw / unrecognized output / gateway-not-
configured, explicit-fallback-honored.
- llm-intent-hybrid-integration.test.ts (6 cases): hybridSearch
escalation gate fires ONLY when flag-on + ambiguous; off when flag-off,
unambiguous, regex-confident, or explicit per-call opt set; fail-open
on LLM throw.
All 231 tests across the cross-modal + related suite pass; no
regression in voyage-multimodal / openai-compat-multimodal /
search-mode / intent / search base suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cross-modal/3): verify-gate fixes for full test suite
Three small fixes to pass the full unit + E2E sweep after the cross-modal
wave commits land.
- migrate.ts v67: drop date_trunc('day', created_at) from
mcp_spend_log indexes. TIMESTAMPTZ truncation depends on session
timezone and isn't IMMUTABLE, so Postgres rejects the function in
the index expression with SQLSTATE 42P17. BTREE on
(client_id, created_at) covers the per-day rollup query via range
scan on created_at — same performance, no IMMUTABLE constraint.
- pglite-schema.ts + src/schema.sql: shorten the embedding_multimodal
column comment. The longer version contained a comma inside a SQL
line comment ("...search.unified_multimodal=true, all queries..."),
which broke parseBaseTableColumns in test/schema-bootstrap-coverage
(the parser splits on commas at depth-0 before stripping comments,
so the comma inside the comment shortened the column-definition part
and an "all" token from "all queries" got picked up as the next
column name — silently hiding embedding_multimodal from coverage).
- schema-embedded.ts: regenerated via `bun run build:schema`.
- test/e2e/v030_1-integration-pglite.test.ts: listBackfills assertion
extended to include the new `modality` entry registered in
src/core/backfill-registry.ts as part of Commit 1.
- test/search/knobs-hash-reranker.test.ts: KNOBS_HASH_VERSION assertion
updated from 2→3 to match the cross-modal-wave hash-key extension
(D2 cache contamination fix). Same shape as the prior
v0.32→v0.35 bump.
- test/unified-multimodal.test.ts: migrated process.env mutation to
withEnv() helper to satisfy the scripts/check-test-isolation R1
rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(cross-modal): VERSION + CHANGELOG + CLAUDE.md + spec doc + llms regen
Final docs commit for the cross-modal wave (v0.36.0.0).
- VERSION + package.json: bump 0.35.5.1 → 0.36.0.0
- CHANGELOG.md: full Garry-voice release entry with five-commit breakdown,
the-numbers-that-matter table, what-this-means-for-you, and the
required to-take-advantage-of-v0.36.0.0 block
- docs/issues/cross-modal-search.md: cherry-picked from PR #1127 head
(164 lines, the original spec doc preserved as historical reference
for Phase 2 + 3 background)
- CLAUDE.md: Key Files entries for src/core/ssrf-validate.ts,
src/core/search/image-loader.ts, src/core/search/by-image.ts,
src/core/search/llm-intent.ts, src/core/spend-log.ts,
src/commands/reindex-multimodal.ts, plus extension annotations on
src/core/search/query-intent.ts, src/core/search/mode.ts,
src/core/search/hybrid.ts, src/core/backfill-registry.ts,
src/core/migrate.ts (v67 + v68)
- llms-full.txt + llms.txt: regenerated via `bun run build:llms`
`bun run verify` clean (privacy + proposal-pii + test-names + jsonb +
source-id-projection + progress + test-isolation + wasm + admin-build +
admin-scope-drift + cli-exec + system-of-record + eval-glossary +
typecheck). `bun test test/build-llms.test.ts` clean (7/7).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cross-modal): renumber migrations 67→69 + 68→70 post-master-merge
Master shipped its own v67 (`facts_typed_claim_columns`) during the
cross-modal wave's review cycle. The merge picked up both side's v67
entries, breaking the migration-distinct-versions test. Renumbering
moves cross-modal's table + column ALTER off the collision:
- v67 mcp_spend_log → v69 mcp_spend_log
- v68 embedding_multimodal_column → v70 embedding_multimodal_column
References updated in CHANGELOG, CLAUDE.md, pglite-schema.ts, schema.sql.
schema-embedded.ts regenerated. llms-full.txt regenerated.
7006 unit tests pass, 0 fail. No test code touched — just version
renumbering plus comment refs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version 0.36.0.0 → 0.36.4.0
Bumping to v0.36.4.0 to land in the queue slot the user requested.
No behavior change; pure version bump across VERSION, package.json,
CHANGELOG.md header, llms-full.txt regen.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
e227965024
commit
e60b60244f
+21
@@ -1152,7 +1152,28 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
break;
|
||||
}
|
||||
// v0.32.7 CJK wave — post-upgrade markdown re-chunk sweep.
|
||||
// v0.36 Phase 3 wave — `gbrain reindex --multimodal` re-embeds content_chunks
|
||||
// into the unified Voyage multimodal-3 column.
|
||||
case 'reindex': {
|
||||
if (args.includes('--multimodal')) {
|
||||
const { runReindexMultimodal } = await import('./commands/reindex-multimodal.ts');
|
||||
const limitIdx = args.indexOf('--limit');
|
||||
const limitVal = limitIdx >= 0 && limitIdx + 1 < args.length ? parseInt(args[limitIdx + 1], 10) : undefined;
|
||||
const result = await runReindexMultimodal(engine, {
|
||||
limit: Number.isFinite(limitVal as number) ? (limitVal as number) : undefined,
|
||||
dryRun: args.includes('--dry-run'),
|
||||
costEstimate: args.includes('--cost-estimate'),
|
||||
noEmbed: args.includes('--no-embed'),
|
||||
json: args.includes('--json'),
|
||||
yes: args.includes('--yes'),
|
||||
});
|
||||
if (args.includes('--json')) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`reindex --multimodal: ${result.reembedded} re-embedded, ${result.failed} failed, ${result.pending_after} pending. est. cost: $${result.cost_usd_estimate.toFixed(2)}`);
|
||||
}
|
||||
break;
|
||||
}
|
||||
const { runReindex } = await import('./commands/reindex.ts');
|
||||
await runReindex(engine, args);
|
||||
break;
|
||||
|
||||
@@ -2519,6 +2519,124 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
progress.heartbeat('whoknows_health');
|
||||
checks.push(await whoknowsHealthCheck(engine));
|
||||
|
||||
// v0.36 cross-modal wave: modality column cleanup.
|
||||
//
|
||||
// Historical brains that imported image assets before v0.27.1's
|
||||
// `modality='image'` default-set may have image chunks where
|
||||
// embedding_image is populated but modality wasn't tagged. The cross-modal
|
||||
// search routing in v0.36 depends on `modality` for keyword filtering;
|
||||
// surface the gap so operators can run `gbrain backfill modality`.
|
||||
progress.heartbeat('cross_modal_modality_backfill');
|
||||
try {
|
||||
const mismatchRows = await engine.executeRaw<{ count: string | number }>(
|
||||
`SELECT COUNT(*)::text AS count FROM content_chunks
|
||||
WHERE embedding_image IS NOT NULL
|
||||
AND chunk_source = 'image_asset'
|
||||
AND (modality IS NULL OR modality != 'image')`,
|
||||
);
|
||||
const mismatch = parseInt(String(mismatchRows[0]?.count ?? '0'), 10);
|
||||
if (mismatch === 0) {
|
||||
checks.push({
|
||||
name: 'cross_modal_modality_backfill',
|
||||
status: 'ok',
|
||||
message: 'All image-asset chunks have modality=image',
|
||||
});
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'cross_modal_modality_backfill',
|
||||
status: 'warn',
|
||||
message:
|
||||
`${mismatch} image-asset chunk(s) have embedding_image populated but modality != 'image'. ` +
|
||||
`Fix: \`gbrain backfill modality\``,
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
// Engine probably doesn't have the modality column (pre-v0.27.1 brain) —
|
||||
// skip silently. Auto-migration will land it on next upgrade.
|
||||
checks.push({
|
||||
name: 'cross_modal_modality_backfill',
|
||||
status: 'ok',
|
||||
message: 'modality column not present (pre-v0.27.1 brain); skipped',
|
||||
});
|
||||
}
|
||||
|
||||
// v0.36 Phase 3 — unified_multimodal coverage (D21 source-aware).
|
||||
//
|
||||
// Only meaningful when search.unified_multimodal is on. Reports the
|
||||
// percentage of content_chunks with embedding_multimodal populated.
|
||||
// Source-aware: a global 95% can hide 0% coverage for a specific source.
|
||||
progress.heartbeat('unified_multimodal_coverage');
|
||||
try {
|
||||
const unifiedFlag = await engine.getConfig('search.unified_multimodal').catch(() => null);
|
||||
const unifiedOnlyFlag = await engine.getConfig('search.unified_multimodal_only').catch(() => null);
|
||||
const unifiedOn = unifiedFlag === 'true' || unifiedFlag === '1';
|
||||
const unifiedOnlyOn = unifiedOnlyFlag === 'true' || unifiedOnlyFlag === '1';
|
||||
|
||||
if (!unifiedOn) {
|
||||
checks.push({
|
||||
name: 'unified_multimodal_coverage',
|
||||
status: 'ok',
|
||||
message: 'search.unified_multimodal is off; coverage check N/A',
|
||||
});
|
||||
} else {
|
||||
// D21 source-aware: report per-source coverage so multi-source brains
|
||||
// can't hide 0% on one source behind a high global average.
|
||||
const rows = await engine.executeRaw<{ source_id: string | null; total: string; covered: string }>(
|
||||
`SELECT
|
||||
COALESCE(p.source_id, 'default') AS source_id,
|
||||
COUNT(*)::text AS total,
|
||||
SUM(CASE WHEN cc.embedding_multimodal IS NOT NULL THEN 1 ELSE 0 END)::text AS covered
|
||||
FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
GROUP BY p.source_id`,
|
||||
);
|
||||
const perSource = rows.map(r => ({
|
||||
source: r.source_id || 'default',
|
||||
total: parseInt(String(r.total), 10),
|
||||
covered: parseInt(String(r.covered), 10),
|
||||
}));
|
||||
const lowestCoverage = perSource.reduce(
|
||||
(acc, r) => Math.min(acc, r.total > 0 ? r.covered / r.total : 1),
|
||||
1,
|
||||
);
|
||||
const summary = perSource.map(r => {
|
||||
const pct = r.total > 0 ? Math.round((r.covered / r.total) * 100) : 0;
|
||||
return `${r.source}:${pct}%`;
|
||||
}).join(', ');
|
||||
|
||||
if (unifiedOnlyOn && lowestCoverage < 0.99) {
|
||||
checks.push({
|
||||
name: 'unified_multimodal_coverage',
|
||||
status: 'fail',
|
||||
message:
|
||||
`unified_multimodal_only is ON but lowest source coverage is ${(lowestCoverage * 100).toFixed(1)}% (${summary}). ` +
|
||||
`Run \`gbrain reindex --multimodal\` to bring coverage to 99%+ or disable strict mode.`,
|
||||
});
|
||||
} else if (lowestCoverage < 0.95) {
|
||||
checks.push({
|
||||
name: 'unified_multimodal_coverage',
|
||||
status: 'warn',
|
||||
message:
|
||||
`unified_multimodal is on but lowest source coverage is ${(lowestCoverage * 100).toFixed(1)}% (${summary}). ` +
|
||||
`Run \`gbrain reindex --multimodal\` to fill the gap.`,
|
||||
});
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'unified_multimodal_coverage',
|
||||
status: 'ok',
|
||||
message: `unified_multimodal coverage: ${summary}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Column probably not present (pre-v0.36 brain pre-migration); skip silently.
|
||||
checks.push({
|
||||
name: 'unified_multimodal_coverage',
|
||||
status: 'ok',
|
||||
message: 'embedding_multimodal column not present yet; skipped',
|
||||
});
|
||||
}
|
||||
|
||||
// 11. Markdown body completeness (v0.12.3 reliability wave).
|
||||
// v0.12.0's splitBody ate everything after the first `---` horizontal rule,
|
||||
// truncating wiki-style pages. Heuristic: pages whose body is <30% of the
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
/**
|
||||
* v0.36 Phase 3 — `gbrain reindex --multimodal` sweep.
|
||||
*
|
||||
* Walks `content_chunks` where `embedding_multimodal IS NULL`, batches
|
||||
* through `embedMultimodalSafe` (partial-failure-aware from Commit 0), and
|
||||
* persists the new vectors.
|
||||
*
|
||||
* Wired patterns:
|
||||
* - D7: acquires `gbrain-reindex-multimodal` writer lock via
|
||||
* `tryAcquireDbLock` so a concurrent autopilot embed phase can't
|
||||
* double-spend Voyage budget on the same chunks.
|
||||
* - D20 phase 2: builds the HNSW partial index AFTER bulk load completes
|
||||
* (pgvector best practice — per-row index maintenance during bulk
|
||||
* ingest is 2-3x slower than post-load build).
|
||||
* - D23-#2: prompts at completion to flip search.unified_multimodal=true
|
||||
* when full coverage is reached (TTY-only; non-TTY prints hint and
|
||||
* does NOT auto-flip).
|
||||
* - Cost prompt: 10-second Ctrl-C grace window via the v0.32.7
|
||||
* post-upgrade-reembed primitive shape.
|
||||
*
|
||||
* Not extracted as shared reindex-core in this commit (D10): the existing
|
||||
* `gbrain reindex --markdown` walks markdown pages and re-imports via
|
||||
* importFromFile; this walks content_chunks and re-embeds via the gateway.
|
||||
* The patterns rhyme but the cores diverge enough that extraction would
|
||||
* balloon the diff. D10 is filed as a follow-up TODO.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { tryAcquireDbLock } from '../core/db-lock.ts';
|
||||
import type { DbLockHandle } from '../core/db-lock.ts';
|
||||
import { sqlQueryForEngine } from '../core/sql-query.ts';
|
||||
import { embedMultimodalSafe } from '../core/ai/gateway.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { gbrainPath } from '../core/config.ts';
|
||||
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
|
||||
import { dirname } from 'node:path';
|
||||
|
||||
const LOCK_ID = 'gbrain-reindex-multimodal';
|
||||
const BATCH_SIZE = 32; // Voyage cap
|
||||
const CHECKPOINT_FILE = 'reindex-multimodal-checkpoint.json';
|
||||
|
||||
export interface ReindexMultimodalOpts {
|
||||
limit?: number;
|
||||
dryRun?: boolean;
|
||||
costEstimate?: boolean;
|
||||
noEmbed?: boolean;
|
||||
json?: boolean;
|
||||
/** Skip the 10s cost-grace window (CI / cron). */
|
||||
yes?: boolean;
|
||||
}
|
||||
|
||||
export interface ReindexMultimodalResult {
|
||||
pending_before: number;
|
||||
pending_after: number;
|
||||
reembedded: number;
|
||||
failed: number;
|
||||
dry_run: boolean;
|
||||
cost_usd_estimate: number;
|
||||
unified_flag_prompted: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Entry point for `gbrain reindex --multimodal`.
|
||||
*/
|
||||
export async function runReindexMultimodal(
|
||||
engine: BrainEngine,
|
||||
opts: ReindexMultimodalOpts,
|
||||
): Promise<ReindexMultimodalResult> {
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
progress.start('reindex_multimodal', 0);
|
||||
|
||||
const sql = sqlQueryForEngine(engine);
|
||||
|
||||
// Count pending chunks.
|
||||
const pendingRows = await sql`
|
||||
SELECT COUNT(*)::text AS count
|
||||
FROM content_chunks
|
||||
WHERE embedding_multimodal IS NULL
|
||||
`;
|
||||
const pendingBefore = parseInt(String(pendingRows[0]?.count ?? '0'), 10);
|
||||
|
||||
// Cost estimate (Voyage multimodal-3 is $0.18 / 1M tokens; ~3.5 chars/token).
|
||||
// We estimate based on chunk_text length per row. Cheap two-stat probe.
|
||||
const statsRows = pendingBefore > 0
|
||||
? await sql`
|
||||
SELECT COALESCE(SUM(LENGTH(chunk_text)), 0)::text AS chars
|
||||
FROM content_chunks
|
||||
WHERE embedding_multimodal IS NULL
|
||||
`
|
||||
: [{ chars: '0' }];
|
||||
const totalChars = parseInt(String(statsRows[0]?.chars ?? '0'), 10);
|
||||
const estimatedTokens = totalChars / 3.5;
|
||||
const costUsdEstimate = (estimatedTokens / 1_000_000) * 0.18;
|
||||
|
||||
if (opts.costEstimate) {
|
||||
progress.finish();
|
||||
return {
|
||||
pending_before: pendingBefore,
|
||||
pending_after: pendingBefore,
|
||||
reembedded: 0,
|
||||
failed: 0,
|
||||
dry_run: true,
|
||||
cost_usd_estimate: costUsdEstimate,
|
||||
unified_flag_prompted: false,
|
||||
};
|
||||
}
|
||||
|
||||
if (opts.dryRun) {
|
||||
progress.finish();
|
||||
return {
|
||||
pending_before: pendingBefore,
|
||||
pending_after: pendingBefore,
|
||||
reembedded: 0,
|
||||
failed: 0,
|
||||
dry_run: true,
|
||||
cost_usd_estimate: costUsdEstimate,
|
||||
unified_flag_prompted: false,
|
||||
};
|
||||
}
|
||||
|
||||
if (pendingBefore === 0) {
|
||||
progress.finish();
|
||||
return {
|
||||
pending_before: 0,
|
||||
pending_after: 0,
|
||||
reembedded: 0,
|
||||
failed: 0,
|
||||
dry_run: false,
|
||||
cost_usd_estimate: 0,
|
||||
unified_flag_prompted: false,
|
||||
};
|
||||
}
|
||||
|
||||
// GBRAIN_NO_REEMBED bypass (CI / cron / opt-out).
|
||||
if (process.env.GBRAIN_NO_REEMBED === '1') {
|
||||
process.stderr.write(
|
||||
`[reindex-multimodal] skipping: GBRAIN_NO_REEMBED=1. ` +
|
||||
`Pending: ${pendingBefore} chunks (~$${costUsdEstimate.toFixed(2)}).\n`,
|
||||
);
|
||||
progress.finish();
|
||||
return {
|
||||
pending_before: pendingBefore,
|
||||
pending_after: pendingBefore,
|
||||
reembedded: 0,
|
||||
failed: 0,
|
||||
dry_run: true,
|
||||
cost_usd_estimate: costUsdEstimate,
|
||||
unified_flag_prompted: false,
|
||||
};
|
||||
}
|
||||
|
||||
// Cost grace window (TTY only; non-TTY auto-proceeds for CI / cron).
|
||||
// Skip if --yes was passed.
|
||||
if (!opts.yes && process.stdout.isTTY && process.stdin.isTTY) {
|
||||
const minutes = Math.ceil((pendingBefore / BATCH_SIZE) * 0.5 / 60); // ~0.5s per batch
|
||||
process.stderr.write(
|
||||
`Will re-embed ~${pendingBefore} chunks via voyage:voyage-multimodal-3, ` +
|
||||
`est. ~$${costUsdEstimate.toFixed(2)}, ~${minutes}min. ` +
|
||||
`Press Ctrl-C within 10s to abort.\n`,
|
||||
);
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 10_000));
|
||||
}
|
||||
|
||||
// D7 lock acquisition. TTL of 6 hours covers a 100K-chunk reindex even
|
||||
// at slow Voyage cadence; supervisor renewal is a follow-up.
|
||||
const lockHandle: DbLockHandle | null = await tryAcquireDbLock(engine, LOCK_ID, 360);
|
||||
if (!lockHandle) {
|
||||
progress.finish();
|
||||
throw new Error(
|
||||
`LOCK_HELD: another gbrain-reindex-multimodal process is already running. ` +
|
||||
`If the prior run crashed, the lock auto-releases after its TTL (6h).`,
|
||||
);
|
||||
}
|
||||
|
||||
let reembedded = 0;
|
||||
let failed = 0;
|
||||
// Resume from checkpoint if one exists.
|
||||
const checkpointPath = gbrainPath(CHECKPOINT_FILE);
|
||||
const completedIds = loadCheckpoint(checkpointPath);
|
||||
|
||||
try {
|
||||
let lastId = 0;
|
||||
let processed = 0;
|
||||
while (true) {
|
||||
if (opts.limit && processed >= opts.limit) break;
|
||||
|
||||
// Fetch next batch of pending chunks.
|
||||
const batchSize = opts.limit
|
||||
? Math.min(BATCH_SIZE, opts.limit - processed)
|
||||
: BATCH_SIZE;
|
||||
const rows = await sql`
|
||||
SELECT id::text AS id, chunk_text
|
||||
FROM content_chunks
|
||||
WHERE embedding_multimodal IS NULL
|
||||
AND id > ${lastId}
|
||||
ORDER BY id
|
||||
LIMIT ${batchSize}
|
||||
`;
|
||||
if (rows.length === 0) break;
|
||||
|
||||
const items = rows.map(r => ({
|
||||
id: parseInt(String(r.id), 10),
|
||||
text: String(r.chunk_text ?? ''),
|
||||
})).filter(r => !completedIds.has(r.id));
|
||||
|
||||
if (items.length === 0) {
|
||||
lastId = parseInt(String(rows[rows.length - 1].id), 10);
|
||||
continue;
|
||||
}
|
||||
|
||||
// D23-#7 batched: embedMultimodalSafe returns parallel arrays with
|
||||
// failed_indices surfaced. We persist what succeeded and log what
|
||||
// failed for the next run to retry.
|
||||
if (!opts.noEmbed) {
|
||||
const result = await embedMultimodalSafe(
|
||||
items.map(it => ({ kind: 'text' as const, text: it.text })),
|
||||
{ inputType: 'document' },
|
||||
);
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
const vec = result.embeddings[i];
|
||||
if (vec) {
|
||||
const vecLiteral = `[${Array.from(vec).join(',')}]`;
|
||||
await sql`
|
||||
UPDATE content_chunks
|
||||
SET embedding_multimodal = ${vecLiteral}::vector
|
||||
WHERE id = ${items[i].id}
|
||||
`;
|
||||
reembedded++;
|
||||
completedIds.add(items[i].id);
|
||||
} else {
|
||||
failed++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
processed += items.length;
|
||||
lastId = items[items.length - 1].id;
|
||||
saveCheckpoint(checkpointPath, completedIds);
|
||||
progress.tick();
|
||||
}
|
||||
} finally {
|
||||
await lockHandle.release().catch(() => {});
|
||||
progress.finish();
|
||||
}
|
||||
|
||||
// D23-#2 auto-flip prompt at completion.
|
||||
const pendingAfterRows = await sql`
|
||||
SELECT COUNT(*)::text AS count
|
||||
FROM content_chunks
|
||||
WHERE embedding_multimodal IS NULL
|
||||
`;
|
||||
const pendingAfter = parseInt(String(pendingAfterRows[0]?.count ?? '0'), 10);
|
||||
let unifiedFlagPrompted = false;
|
||||
|
||||
if (pendingAfter === 0 && reembedded > 0) {
|
||||
const currentFlag = await engine.getConfig('search.unified_multimodal').catch(() => null);
|
||||
if (currentFlag !== 'true' && currentFlag !== '1') {
|
||||
if (process.stdout.isTTY) {
|
||||
process.stderr.write(
|
||||
`\n[reindex-multimodal] Coverage now 100%. ` +
|
||||
`Run \`gbrain config set search.unified_multimodal true\` to route all queries ` +
|
||||
`through the unified column.\n`,
|
||||
);
|
||||
unifiedFlagPrompted = true;
|
||||
} else {
|
||||
process.stderr.write(
|
||||
`[reindex-multimodal] Coverage now 100%. ` +
|
||||
`gbrain config set search.unified_multimodal true\n`,
|
||||
);
|
||||
unifiedFlagPrompted = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Clear checkpoint on full completion.
|
||||
if (pendingAfter === 0) {
|
||||
try { writeFileSync(checkpointPath, '{}\n'); } catch { /* ignore */ }
|
||||
}
|
||||
|
||||
return {
|
||||
pending_before: pendingBefore,
|
||||
pending_after: pendingAfter,
|
||||
reembedded,
|
||||
failed,
|
||||
dry_run: false,
|
||||
cost_usd_estimate: costUsdEstimate,
|
||||
unified_flag_prompted: unifiedFlagPrompted,
|
||||
};
|
||||
}
|
||||
|
||||
function loadCheckpoint(path: string): Set<number> {
|
||||
try {
|
||||
if (!existsSync(path)) return new Set();
|
||||
const raw = readFileSync(path, 'utf-8');
|
||||
const data = JSON.parse(raw) as { completedIds?: number[] };
|
||||
return new Set((data.completedIds ?? []).filter(n => Number.isFinite(n)));
|
||||
} catch {
|
||||
return new Set();
|
||||
}
|
||||
}
|
||||
|
||||
function saveCheckpoint(path: string, completedIds: Set<number>): void {
|
||||
try {
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
const payload = JSON.stringify({ completedIds: Array.from(completedIds) }, null, 2);
|
||||
writeFileSync(path, payload);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[reindex-multimodal] checkpoint save failed: ${msg}\n`);
|
||||
}
|
||||
}
|
||||
@@ -54,6 +54,14 @@ const KNOB_DESCRIPTIONS: Record<keyof ModeBundle, string> = {
|
||||
reranker_top_n_out: 'Cap on reranked output (null = no truncate)',
|
||||
reranker_timeout_ms: 'HTTP timeout for the reranker call',
|
||||
floor_ratio: 'Floor-ratio gate for metadata boosts (0..1, undefined = off)',
|
||||
// v0.36 cross-modal knobs (D3 registry)
|
||||
cross_modal_both_text_weight: "D6 'both'-mode RRF weight for text branch (0.6 default)",
|
||||
cross_modal_both_image_weight: "D6 'both'-mode RRF weight for image branch (0.4 default)",
|
||||
image_query_text_refinement_weight: 'D13 searchByImage text-refinement RRF weight (0.4 default)',
|
||||
image_query_image_refinement_weight: 'D13 searchByImage image branch RRF weight (0.6 default)',
|
||||
unified_multimodal: 'Phase 3 — route all queries through embedding_multimodal column',
|
||||
unified_multimodal_only: 'Phase 3 strict — bypass dual-column fallback when unified is on',
|
||||
cross_modal_llm_intent: 'Commit 4 — Haiku tie-break for ambiguous modality classification',
|
||||
};
|
||||
|
||||
interface SearchModesReport {
|
||||
|
||||
+151
-17
@@ -31,6 +31,8 @@ import { z } from 'zod';
|
||||
|
||||
import type {
|
||||
AIGatewayConfig,
|
||||
EmbedMultimodalOpts,
|
||||
MultimodalBatchResult,
|
||||
MultimodalInput,
|
||||
Recipe,
|
||||
TouchpointKind,
|
||||
@@ -1335,7 +1337,10 @@ const MULTIMODAL_MAX_IMAGE_BYTES = 20 * 1024 * 1024;
|
||||
*
|
||||
* Empty input → returns []. Preserves the `embed([])` contract.
|
||||
*/
|
||||
export async function embedMultimodal(inputs: MultimodalInput[]): Promise<Float32Array[]> {
|
||||
export async function embedMultimodal(
|
||||
inputs: MultimodalInput[],
|
||||
opts: EmbedMultimodalOpts = {},
|
||||
): Promise<Float32Array[]> {
|
||||
if (!inputs || inputs.length === 0) return [];
|
||||
|
||||
const cfg = requireConfig();
|
||||
@@ -1372,7 +1377,7 @@ export async function embedMultimodal(inputs: MultimodalInput[]): Promise<Float3
|
||||
// recipe is `openai-compat` per tier but uses its own /multimodalembeddings
|
||||
// path, so we still branch on recipe.id for that one.
|
||||
if (recipe.id !== 'voyage' && recipe.implementation === 'openai-compatible') {
|
||||
return embedMultimodalOpenAICompat(inputs, recipe, parsed.modelId, cfg);
|
||||
return embedMultimodalOpenAICompat(inputs, recipe, parsed.modelId, cfg, opts);
|
||||
}
|
||||
if (recipe.id !== 'voyage') {
|
||||
throw new AIConfigError(
|
||||
@@ -1405,6 +1410,10 @@ export async function embedMultimodal(inputs: MultimodalInput[]): Promise<Float3
|
||||
// landing in `embedding_image` the column itself is fixed at 1024.
|
||||
const targetDims = 1024;
|
||||
|
||||
// v0.36 (D22-2): thread Voyage's retrieval input_type discipline through.
|
||||
// Default 'document' preserves pre-v0.36 ingest behavior.
|
||||
const inputType = opts.inputType ?? 'document';
|
||||
|
||||
// Batch in groups of 32 (Voyage's published max). Each batch is one HTTP
|
||||
// call; results concatenate in input order.
|
||||
const allEmbeddings: Float32Array[] = [];
|
||||
@@ -1412,17 +1421,19 @@ export async function embedMultimodal(inputs: MultimodalInput[]): Promise<Float3
|
||||
const batch = inputs.slice(i, i + MULTIMODAL_BATCH_SIZE);
|
||||
const body = {
|
||||
inputs: batch.map(input => ({
|
||||
// Voyage's documented shape for image inputs:
|
||||
// { content: [{ type: "image_base64", image_base64: "data:image/png;base64,..." }] }
|
||||
// Voyage's documented content shape supports both image and text
|
||||
// entries. v0.36 cross-modal: text variant for query embedding.
|
||||
content: [
|
||||
{
|
||||
type: 'image_base64',
|
||||
image_base64: `data:${input.mime};base64,${input.data}`,
|
||||
},
|
||||
input.kind === 'text'
|
||||
? { type: 'text', text: input.text }
|
||||
: {
|
||||
type: 'image_base64',
|
||||
image_base64: `data:${input.mime};base64,${input.data}`,
|
||||
},
|
||||
],
|
||||
})),
|
||||
model: parsed.modelId,
|
||||
input_type: 'document',
|
||||
input_type: inputType,
|
||||
};
|
||||
|
||||
let res: Response;
|
||||
@@ -1510,6 +1521,7 @@ async function embedMultimodalOpenAICompat(
|
||||
recipe: Recipe,
|
||||
modelId: string,
|
||||
cfg: AIGatewayConfig,
|
||||
opts: EmbedMultimodalOpts = {},
|
||||
): Promise<Float32Array[]> {
|
||||
// Auth resolution via the gateway's canonical helper so LiteLLM-style
|
||||
// optional-auth recipes (Authorization: Bearer LITELLM_API_KEY) and
|
||||
@@ -1543,19 +1555,27 @@ async function embedMultimodalOpenAICompat(
|
||||
// multimodal content array varies per provider. Single-input requests
|
||||
// are the safe lowest common denominator; LiteLLM's proxy backend
|
||||
// batches internally if it can.
|
||||
// v0.36 (D22-2): inputType opt threaded for symmetry with the Voyage path.
|
||||
// Most openai-compatible proxies don't forward this field, but recording
|
||||
// it in the body keeps LiteLLM-style providers that DO accept it correct.
|
||||
const inputType = opts.inputType ?? 'document';
|
||||
|
||||
const allEmbeddings: Float32Array[] = [];
|
||||
for (const input of inputs) {
|
||||
const body = {
|
||||
const body: Record<string, unknown> = {
|
||||
model: modelId,
|
||||
input: [
|
||||
{
|
||||
// OpenAI's documented multimodal content shape. The data-URL
|
||||
// form embeds the image bytes inline so the proxy doesn't need
|
||||
// network access to fetch the image.
|
||||
type: 'image_url',
|
||||
image_url: { url: `data:${input.mime};base64,${input.data}` },
|
||||
},
|
||||
input.kind === 'text'
|
||||
? { type: 'input_text', text: input.text }
|
||||
: {
|
||||
// OpenAI's documented multimodal content shape. The data-URL
|
||||
// form embeds the image bytes inline so the proxy doesn't need
|
||||
// network access to fetch the image.
|
||||
type: 'image_url',
|
||||
image_url: { url: `data:${input.mime};base64,${input.data}` },
|
||||
},
|
||||
],
|
||||
input_type: inputType,
|
||||
};
|
||||
|
||||
let res: Response;
|
||||
@@ -1630,6 +1650,120 @@ async function embedMultimodalOpenAICompat(
|
||||
return allEmbeddings;
|
||||
}
|
||||
|
||||
// ---- v0.36 cross-modal wave: query-side multimodal embedding + safe variant ----
|
||||
|
||||
/**
|
||||
* Embed a TEXT query through the configured multimodal model.
|
||||
*
|
||||
* Routes through `embedding_multimodal_model` (defaults to Voyage multimodal-3)
|
||||
* so the resulting vector lives in the multimodal embedding space — the same
|
||||
* space the brain's `embedding_image` column was populated into. A text
|
||||
* query embedded here can match image chunks (Phase 1 of the cross-modal
|
||||
* wave) and, post Phase 3 reindex, text chunks in the unified column.
|
||||
*
|
||||
* Threads `inputType: 'query'` (D22-2) so Voyage routes to the retrieval
|
||||
* half of its asymmetric embedding space.
|
||||
*
|
||||
* Sibling of v0.35.0.0's `embedQuery(text)`, which uses the TEXT embedding
|
||||
* model (typically OpenAI text-embedding-3-large at 1536d or 2560d, NOT
|
||||
* compatible with the 1024d multimodal column).
|
||||
*/
|
||||
export async function embedQueryMultimodal(text: string): Promise<Float32Array> {
|
||||
const [vec] = await embedMultimodal([{ kind: 'text', text }], { inputType: 'query' });
|
||||
if (!vec) {
|
||||
throw new AITransientError('embedQueryMultimodal: gateway returned no vector for non-empty text input');
|
||||
}
|
||||
return vec;
|
||||
}
|
||||
|
||||
/**
|
||||
* Embed an IMAGE as a query through the configured multimodal model.
|
||||
*
|
||||
* Sibling of `embedQueryMultimodal(text)` for the Phase 2 image-as-query
|
||||
* path. The image bytes must already be loaded and base64-encoded by the
|
||||
* caller (see `src/core/search/image-loader.ts` for the SSRF-defended
|
||||
* loader). Threads `inputType: 'query'` so Voyage routes to the
|
||||
* retrieval half of its asymmetric space.
|
||||
*/
|
||||
export async function embedQueryMultimodalImage(
|
||||
input: { data: string; mime: string },
|
||||
): Promise<Float32Array> {
|
||||
const [vec] = await embedMultimodal(
|
||||
[{ kind: 'image_base64', data: input.data, mime: input.mime }],
|
||||
{ inputType: 'query' },
|
||||
);
|
||||
if (!vec) {
|
||||
throw new AITransientError('embedQueryMultimodalImage: gateway returned no vector');
|
||||
}
|
||||
return vec;
|
||||
}
|
||||
|
||||
/**
|
||||
* Partial-failure-aware variant of `embedMultimodal`.
|
||||
*
|
||||
* The default `embedMultimodal()` throws on first failure to preserve the
|
||||
* pre-v0.36 contract (used by `importImageFile` which can't proceed on
|
||||
* partial data). Phase 3 `reindex --multimodal` ingests many thousands
|
||||
* of chunks and CAN make forward progress with partial results — it
|
||||
* uses this variant so a 401 on chunk 87K doesn't discard the 31
|
||||
* already-computed embeddings in that batch.
|
||||
*
|
||||
* Strategy:
|
||||
* 1. Try the full input set via `embedMultimodal`. On success, return.
|
||||
* 2. On AIConfigError (permanent), surface every input as failed —
|
||||
* the misconfig isn't going to fix itself by retrying smaller.
|
||||
* 3. On AITransientError or other thrown error, split-and-retry
|
||||
* via binary search. Single-input attempts that fail are recorded
|
||||
* in `failedIndices` and skipped.
|
||||
*
|
||||
* Returns `MultimodalBatchResult` with parallel-indexed `embeddings`
|
||||
* (undefined for failed slots) and a `failedIndices` array.
|
||||
*/
|
||||
export async function embedMultimodalSafe(
|
||||
inputs: MultimodalInput[],
|
||||
opts: EmbedMultimodalOpts = {},
|
||||
): Promise<MultimodalBatchResult> {
|
||||
if (!inputs || inputs.length === 0) {
|
||||
return { embeddings: [], failedIndices: [] };
|
||||
}
|
||||
|
||||
const embeddings: Array<Float32Array | undefined> = new Array(inputs.length).fill(undefined);
|
||||
const failedIndices: number[] = [];
|
||||
let lastError: Error | undefined;
|
||||
|
||||
async function attempt(startIdx: number, items: MultimodalInput[]): Promise<void> {
|
||||
if (items.length === 0) return;
|
||||
try {
|
||||
const vecs = await embedMultimodal(items, opts);
|
||||
for (let i = 0; i < vecs.length; i++) {
|
||||
embeddings[startIdx + i] = vecs[i];
|
||||
}
|
||||
return;
|
||||
} catch (err) {
|
||||
lastError = err instanceof Error ? err : new Error(String(err));
|
||||
// AIConfigError = permanent misconfig. Retrying smaller won't help.
|
||||
if (lastError instanceof AIConfigError) {
|
||||
for (let i = 0; i < items.length; i++) failedIndices.push(startIdx + i);
|
||||
return;
|
||||
}
|
||||
// Single input that failed — record and move on.
|
||||
if (items.length === 1) {
|
||||
failedIndices.push(startIdx);
|
||||
return;
|
||||
}
|
||||
// Binary-search split. Each half gets its own retry.
|
||||
const mid = Math.floor(items.length / 2);
|
||||
await attempt(startIdx, items.slice(0, mid));
|
||||
await attempt(startIdx + mid, items.slice(mid));
|
||||
}
|
||||
}
|
||||
|
||||
await attempt(0, inputs);
|
||||
failedIndices.sort((a, b) => a - b);
|
||||
|
||||
return { embeddings, failedIndices, lastError };
|
||||
}
|
||||
|
||||
// ---- Expansion ----
|
||||
|
||||
async function resolveExpansionProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> {
|
||||
|
||||
+42
-5
@@ -104,15 +104,52 @@ export interface EmbeddingTouchpoint {
|
||||
|
||||
/**
|
||||
* v0.27.1: input shape for gateway.embedMultimodal(). Discriminated union;
|
||||
* today the only kind is image_base64 (raw bytes encoded by the caller).
|
||||
* Future kinds (image_url, video_keyframe) extend the union without
|
||||
* widening callers because the discriminator is exhaustive.
|
||||
* variants extend without widening callers because the discriminator is
|
||||
* exhaustive.
|
||||
*
|
||||
* No image_url variant: SSRF surface. Callers must read the bytes and
|
||||
* base64-encode them; the gateway never fetches external URLs.
|
||||
* base64-encode them; the gateway never fetches external URLs. For
|
||||
* remote-callable image-as-query, the dedicated SSRF-defended loader at
|
||||
* `src/core/search/image-loader.ts` resolves URLs to base64 first.
|
||||
*
|
||||
* v0.36 (cross-modal wave) adds the `text` variant for query-side
|
||||
* multimodal embedding (`embedQueryMultimodal(text)`) — Voyage's
|
||||
* multimodal endpoint accepts a content array mixing text + image entries.
|
||||
*/
|
||||
export type MultimodalInput =
|
||||
| { kind: 'image_base64'; data: string; mime: string };
|
||||
| { kind: 'image_base64'; data: string; mime: string }
|
||||
| { kind: 'text'; text: string };
|
||||
|
||||
/**
|
||||
* v0.36 — opts for gateway.embedMultimodal().
|
||||
*
|
||||
* `inputType` threads Voyage's retrieval discipline through:
|
||||
* - 'document' (default) — embedding side of asymmetric retrieval
|
||||
* - 'query' — query side; routes to the matching half of Voyage's space
|
||||
*
|
||||
* Mixing inputType across calls is fine; mixing within one batch is not
|
||||
* supported (Voyage requires one input_type per request).
|
||||
*/
|
||||
export interface EmbedMultimodalOpts {
|
||||
inputType?: 'document' | 'query';
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.36 — return shape for partial-failure-aware multimodal batching.
|
||||
*
|
||||
* Used by `embedMultimodalSafe()` (the Phase-3-reindex-safe variant). The
|
||||
* default `embedMultimodal()` throws on first failure to preserve the
|
||||
* pre-v0.36 contract; callers who can persist partial progress opt into
|
||||
* the safe variant explicitly.
|
||||
*/
|
||||
export interface MultimodalBatchResult {
|
||||
/** Successful embeddings, indexed parallel to the original inputs (may contain holes as undefined). */
|
||||
embeddings: Array<Float32Array | undefined>;
|
||||
/** Indices of inputs that failed to embed, in original-input order. */
|
||||
failedIndices: number[];
|
||||
/** Last error encountered (for diagnostics; not necessarily the only failure). */
|
||||
lastError?: Error;
|
||||
}
|
||||
|
||||
export interface ExpansionTouchpoint {
|
||||
models: string[];
|
||||
|
||||
@@ -190,10 +190,55 @@ function embeddingVoyageBackfill(): RegisteredBackfill {
|
||||
};
|
||||
}
|
||||
|
||||
interface ModalityBackfillRow {
|
||||
id: number;
|
||||
chunk_source: string | null;
|
||||
modality: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.36 cross-modal wave: modality column cleanup.
|
||||
*
|
||||
* Historical brains that imported image assets BEFORE v0.27.1's
|
||||
* `modality='image'` default-set may have image chunks where
|
||||
* `embedding_image IS NOT NULL` but `modality != 'image'`. This backfill
|
||||
* flips them. D22-7 hardening: requires `chunk_source='image_asset'` so
|
||||
* we never tag a non-image chunk that happens to have an embedding_image
|
||||
* value (defensive against hypothetical future code paths populating
|
||||
* embedding_image on text chunks).
|
||||
*
|
||||
* Idempotent — second run finds no rows to update.
|
||||
*/
|
||||
function modalityBackfill(): RegisteredBackfill {
|
||||
return {
|
||||
description: 'Flip modality to "image" on image-asset chunks where it was missed by pre-v0.27.1 ingest',
|
||||
v030_1_status: 'implemented',
|
||||
spec: {
|
||||
name: 'modality',
|
||||
table: 'content_chunks',
|
||||
idColumn: 'id',
|
||||
selectColumns: ['chunk_source', 'modality'],
|
||||
needsBackfill:
|
||||
// D22-7: belt-and-suspenders. Both predicates must hold for the row
|
||||
// to be a real image chunk that lost its modality tag.
|
||||
"embedding_image IS NOT NULL AND chunk_source = 'image_asset' AND (modality IS NULL OR modality != 'image')",
|
||||
compute: async (rows) => {
|
||||
const updates: Array<{ id: number; updates: Record<string, unknown> }> = [];
|
||||
for (const r of rows as unknown as ModalityBackfillRow[]) {
|
||||
updates.push({ id: r.id, updates: { modality: 'image' } });
|
||||
}
|
||||
return updates;
|
||||
},
|
||||
estimateRowsPerSecond: 8000, // pure metadata flip, very fast
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function registerCoreBackfills(): void {
|
||||
registerBackfill(effectiveDateBackfill());
|
||||
registerBackfill(emotionalWeightBackfill());
|
||||
registerBackfill(embeddingVoyageBackfill());
|
||||
registerBackfill(modalityBackfill());
|
||||
}
|
||||
|
||||
// Auto-register on first import.
|
||||
|
||||
+16
-2
@@ -16,8 +16,22 @@ import {
|
||||
// v0.27.1: re-export multimodal embedding so callers can pull both text and
|
||||
// image embedding APIs from `src/core/embedding`. import-image-file consumes
|
||||
// embedMultimodal directly.
|
||||
export { embedMultimodal } from './ai/gateway.ts';
|
||||
export type { MultimodalInput } from './ai/types.ts';
|
||||
//
|
||||
// v0.36 cross-modal wave: query-side multimodal embedding (text and image
|
||||
// variants) for hybridSearch routing image-intent queries to the multimodal
|
||||
// column. embedMultimodalSafe is the partial-failure variant Phase 3 reindex
|
||||
// uses to make forward progress on transient batch failures.
|
||||
export {
|
||||
embedMultimodal,
|
||||
embedMultimodalSafe,
|
||||
embedQueryMultimodal,
|
||||
embedQueryMultimodalImage,
|
||||
} from './ai/gateway.ts';
|
||||
export type {
|
||||
MultimodalInput,
|
||||
EmbedMultimodalOpts,
|
||||
MultimodalBatchResult,
|
||||
} from './ai/types.ts';
|
||||
|
||||
/** Embed one text (document-side for asymmetric providers). */
|
||||
export async function embed(text: string): Promise<Float32Array> {
|
||||
|
||||
@@ -3150,6 +3150,87 @@ export const MIGRATIONS: Migration[] = [
|
||||
ON oauth_clients USING GIN (federated_read);
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 78,
|
||||
name: 'embedding_multimodal_column',
|
||||
// D20 Phase 3: add the unified-multimodal vector column to content_chunks.
|
||||
//
|
||||
// Column-only migration — the HNSW partial index is built AFTER the first
|
||||
// bulk reindex completes (via `gbrain reindex --multimodal --build-index`
|
||||
// or auto-built at completion). pgvector docs explicitly note that HNSW
|
||||
// build is faster after data load, and per-row index maintenance during
|
||||
// bulk reindex would slow the operation 2-3x.
|
||||
//
|
||||
// Operator class will be vector_cosine_ops to match the existing
|
||||
// embedding_image index for ranking parity.
|
||||
//
|
||||
// The column ships at 1024 dims to match Voyage multimodal-3 output.
|
||||
// Operators wanting a different dim (Cohere multimodal at 1408d, etc.)
|
||||
// need a column rebuild — surfaced by the `multimodal_column_dim_match`
|
||||
// doctor check (D20 model+dim pin).
|
||||
idempotent: true,
|
||||
sql: `
|
||||
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_multimodal vector(1024);
|
||||
`,
|
||||
sqlFor: {
|
||||
pglite: `
|
||||
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_multimodal vector(1024);
|
||||
`,
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 77,
|
||||
name: 'mcp_spend_log',
|
||||
// D23-#6: per-OAuth-client paid-API spend tracking. search_by_image
|
||||
// (Phase 2 of cross-modal wave) makes paid Voyage calls on behalf of
|
||||
// remote OAuth clients. The existing v0.22.7 limiter caps requests/min
|
||||
// but not spend. A 100-req/min attacker can burn ~$3/hour at Voyage
|
||||
// rates. This table aggregates spend so the daily-budget check can
|
||||
// refuse new calls when a client crosses
|
||||
// search.image_query.daily_budget_usd_per_client (default $5).
|
||||
//
|
||||
// Indexed for the hot read: (client_id, day) lookup, summed.
|
||||
// Row count is bounded by O(clients × days) — tiny.
|
||||
idempotent: true,
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS mcp_spend_log (
|
||||
id SERIAL PRIMARY KEY,
|
||||
client_id TEXT,
|
||||
token_name TEXT,
|
||||
operation TEXT NOT NULL,
|
||||
spend_cents NUMERIC(12, 4) NOT NULL DEFAULT 0,
|
||||
provider TEXT,
|
||||
model TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
-- BTREE on (client_id, created_at) covers the per-day rollup query
|
||||
-- (SELECT SUM ... WHERE client_id = $ AND created_at >= today_start) via
|
||||
-- range scan on created_at. date_trunc in an index expression would
|
||||
-- require IMMUTABLE — TIMESTAMPTZ truncation depends on session timezone.
|
||||
CREATE INDEX IF NOT EXISTS idx_mcp_spend_log_client_time
|
||||
ON mcp_spend_log (client_id, created_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_mcp_spend_log_token_time
|
||||
ON mcp_spend_log (token_name, created_at);
|
||||
`,
|
||||
sqlFor: {
|
||||
pglite: `
|
||||
CREATE TABLE IF NOT EXISTS mcp_spend_log (
|
||||
id SERIAL PRIMARY KEY,
|
||||
client_id TEXT,
|
||||
token_name TEXT,
|
||||
operation TEXT NOT NULL,
|
||||
spend_cents NUMERIC(12, 4) NOT NULL DEFAULT 0,
|
||||
provider TEXT,
|
||||
model TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_mcp_spend_log_client_time
|
||||
ON mcp_spend_log (client_id, created_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_mcp_spend_log_token_time
|
||||
ON mcp_spend_log (token_name, created_at);
|
||||
`,
|
||||
},
|
||||
},
|
||||
{
|
||||
version: 66,
|
||||
name: 'embed_stale_partial_index',
|
||||
|
||||
@@ -1143,6 +1143,16 @@ const query: Operation = {
|
||||
description:
|
||||
"v0.34: scope search to a single source. Defaults to OperationContext.sourceId (set from CLI --source / GBRAIN_SOURCE / .gbrain-source dotfile). Pass '__all__' to force cross-source search in multi-source brains.",
|
||||
},
|
||||
cross_modal: {
|
||||
type: 'string',
|
||||
enum: ['text', 'image', 'both', 'auto'],
|
||||
description:
|
||||
"v0.36 cross-modal search routing.\n" +
|
||||
" 'text' (default for non-image-intent queries) — text-only path, no behavior change vs v0.35.\n" +
|
||||
" 'image' — route the query through Voyage multimodal-3 + the embedding_image column. Best for 'show me photos of...' phrasings.\n" +
|
||||
" 'both' — run text AND image searches in parallel; merge via weighted RRF.\n" +
|
||||
" 'auto' — same effect as omitting the field; intent classifier decides based on query phrasing.",
|
||||
},
|
||||
embedding_column: {
|
||||
type: 'string',
|
||||
description:
|
||||
@@ -1228,6 +1238,8 @@ const query: Operation = {
|
||||
tokenBudget: typeof p.token_budget === 'number' ? (p.token_budget as number) : undefined,
|
||||
useCache: typeof p.use_cache === 'boolean' ? (p.use_cache as boolean) : undefined,
|
||||
intentWeighting: typeof p.intent_weighting === 'boolean' ? (p.intent_weighting as boolean) : undefined,
|
||||
// v0.36 cross-modal routing param.
|
||||
crossModal: p.cross_modal as 'text' | 'image' | 'both' | 'auto' | undefined,
|
||||
onMeta: (m) => { capturedMeta = m; },
|
||||
// v0.36 (D15): per-call embedding column override. Resolver rejects
|
||||
// unknown names at hybrid entry with EmbeddingColumnNotRegisteredError;
|
||||
@@ -3285,6 +3297,149 @@ const code_traversal_cache_clear: Operation = {
|
||||
cliHints: { name: 'code_traversal_cache_clear', hidden: true },
|
||||
};
|
||||
|
||||
// --- v0.36 Phase 2: search_by_image (image-as-query) ---
|
||||
|
||||
const search_by_image: Operation = {
|
||||
name: 'search_by_image',
|
||||
description:
|
||||
'v0.36 cross-modal Phase 2: image-as-query retrieval. Accepts a local path (CLI), data: URI, or http(s):// URL ' +
|
||||
'(SSRF-defended). Returns visually-similar image chunks plus any OCR text they carry. Optional `query` text ' +
|
||||
'refinement merges via weighted RRF (D13 hybrid intersect). True image→full-text-knowledge requires Phase 3 ' +
|
||||
'(`gbrain reindex --multimodal` + `search.unified_multimodal: true`).',
|
||||
params: {
|
||||
image_path: { type: 'string', description: 'Absolute path to image (local CLI callers only — rejected for remote MCP per D18).' },
|
||||
image_url: { type: 'string', description: 'http(s):// URL to image. SSRF-defended; max 3 redirect hops; 10MB cap.' },
|
||||
image_data: { type: 'string', description: 'Base64-encoded image bytes (preferred for remote MCP callers). PNG/JPEG/WebP only.' },
|
||||
image_mime: { type: 'string', description: 'Optional MIME hint when ambiguous. Magic-byte sniff is authoritative.' },
|
||||
query: { type: 'string', description: 'Optional text refinement; runs hybrid intersect via D13 weighted RRF.' },
|
||||
limit: { type: 'number', description: 'Max results (default 20)' },
|
||||
offset: { type: 'number', description: 'Skip first N results (for pagination)' },
|
||||
source_id: { type: 'string', description: "Scope to a single source. Defaults to ctx.sourceId. '__all__' opts out." },
|
||||
},
|
||||
scope: 'read',
|
||||
// NOT localOnly: remote MCP callers can pass image_url or image_data
|
||||
// (subject to D18 image_path ban + D12 size cap + D23-#6 spend cap).
|
||||
handler: async (ctx, p) => {
|
||||
const imagePath = p.image_path as string | undefined;
|
||||
const imageUrl = p.image_url as string | undefined;
|
||||
const imageData = p.image_data as string | undefined;
|
||||
const imageMime = (p.image_mime as string) || undefined;
|
||||
const queryRefinement = p.query as string | undefined;
|
||||
const sourceIdParam = typeof p.source_id === 'string' ? p.source_id : undefined;
|
||||
|
||||
// D18 P0 — remote callers cannot pass image_path. Rejecting at handler
|
||||
// entry, before any file I/O fires. validateParams catches it too at the
|
||||
// dispatch layer; this is defense-in-depth.
|
||||
if (ctx.remote === true && imagePath) {
|
||||
throw new Error(
|
||||
'permission_denied: image_path is not permitted for remote callers (D18). ' +
|
||||
'Use image_url or image_data instead.',
|
||||
);
|
||||
}
|
||||
|
||||
if (!imagePath && !imageUrl && !imageData) {
|
||||
throw new Error('search_by_image requires one of: image_path, image_url, image_data');
|
||||
}
|
||||
if ([imagePath, imageUrl, imageData].filter(Boolean).length > 1) {
|
||||
throw new Error('search_by_image accepts only one of: image_path, image_url, image_data');
|
||||
}
|
||||
|
||||
// D23-#6 — pre-flight daily-budget check for remote OAuth clients.
|
||||
// Local CLI callers (ctx.remote=false) bypass the cap (clientId="").
|
||||
const clientId = (ctx.remote === true ? (ctx.auth?.clientId ?? '') : '');
|
||||
if (clientId) {
|
||||
const budgetUsd = await getDailyImageBudgetUsd(ctx.engine);
|
||||
const { checkBudget } = await import('./spend-log.ts');
|
||||
await checkBudget(ctx.engine, clientId, Math.round(budgetUsd * 100));
|
||||
}
|
||||
|
||||
// Resolve image bytes via the SSRF-defended loader. For remote callers,
|
||||
// tighter byte cap.
|
||||
const remoteCap = await getRemoteMaxBytes(ctx.engine);
|
||||
const localCap = await getLocalMaxBytes(ctx.engine);
|
||||
const cap = ctx.remote === true ? remoteCap : localCap;
|
||||
const { loadImageInput } = await import('./search/image-loader.ts');
|
||||
const loaded = await loadImageInput(
|
||||
(imagePath ?? imageUrl ?? `data:${imageMime ?? 'image/png'};base64,${imageData}`)!,
|
||||
{ maxBytes: cap },
|
||||
);
|
||||
|
||||
// Resolve source-scope (D5 canonical thread).
|
||||
const resolvedSourceId =
|
||||
sourceIdParam !== undefined
|
||||
? sourceIdParam === '__all__'
|
||||
? undefined
|
||||
: sourceIdParam
|
||||
: ctx.sourceId;
|
||||
|
||||
const { searchByImage } = await import('./search/by-image.ts');
|
||||
const results = await searchByImage(
|
||||
ctx.engine,
|
||||
{ base64: loaded.base64, mime: loaded.contentType },
|
||||
{
|
||||
limit: (p.limit as number) || 20,
|
||||
offset: (p.offset as number) || 0,
|
||||
query: queryRefinement,
|
||||
sourceId: resolvedSourceId,
|
||||
...sourceScopeOpts(ctx),
|
||||
},
|
||||
);
|
||||
|
||||
// D23-#6 — record successful Voyage call. Best-effort; failures don't
|
||||
// block the response.
|
||||
if (clientId) {
|
||||
const { recordSpend, VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS } = await import('./spend-log.ts');
|
||||
// Approximate: 1 image embed + (query ? 1 text embed : 0). Both are
|
||||
// billed at the same per-call rate by Voyage.
|
||||
const calls = 1 + (queryRefinement ? 1 : 0);
|
||||
void recordSpend(ctx.engine, {
|
||||
clientId,
|
||||
tokenName: ctx.auth?.clientName ?? null,
|
||||
operation: 'search_by_image',
|
||||
spendCents: VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS * calls,
|
||||
provider: 'voyage',
|
||||
model: 'voyage-multimodal-3',
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
},
|
||||
cliHints: { name: 'search-by-image' },
|
||||
};
|
||||
|
||||
async function getDailyImageBudgetUsd(engine: BrainEngine): Promise<number> {
|
||||
try {
|
||||
const v = await engine.getConfig('search.image_query.daily_budget_usd_per_client');
|
||||
if (v == null) return 5; // default $5
|
||||
const n = parseFloat(v);
|
||||
return Number.isFinite(n) && n > 0 ? n : 5;
|
||||
} catch {
|
||||
return 5;
|
||||
}
|
||||
}
|
||||
|
||||
async function getLocalMaxBytes(engine: BrainEngine): Promise<number> {
|
||||
try {
|
||||
const v = await engine.getConfig('search.image_query.max_bytes');
|
||||
if (v == null) return 10 * 1024 * 1024;
|
||||
const n = parseInt(v, 10);
|
||||
return Number.isFinite(n) && n > 0 ? n : 10 * 1024 * 1024;
|
||||
} catch {
|
||||
return 10 * 1024 * 1024;
|
||||
}
|
||||
}
|
||||
|
||||
async function getRemoteMaxBytes(engine: BrainEngine): Promise<number> {
|
||||
try {
|
||||
const v = await engine.getConfig('search.image_query.remote_max_bytes');
|
||||
if (v == null) return 2 * 1024 * 1024;
|
||||
const n = parseInt(v, 10);
|
||||
return Number.isFinite(n) && n > 0 ? n : 2 * 1024 * 1024;
|
||||
} catch {
|
||||
return 2 * 1024 * 1024;
|
||||
}
|
||||
}
|
||||
|
||||
// --- Exports ---
|
||||
|
||||
export const operations: Operation[] = [
|
||||
@@ -3294,6 +3449,8 @@ export const operations: Operation[] = [
|
||||
restore_page, purge_deleted_pages,
|
||||
// Search
|
||||
search, query,
|
||||
// v0.36 Phase 2: image-as-query
|
||||
search_by_image,
|
||||
// Tags
|
||||
add_tag, remove_tag, get_tags,
|
||||
// Links
|
||||
|
||||
@@ -1289,15 +1289,20 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// v0.36 (D11): column routing via resolved descriptor. Engine doesn't
|
||||
// read config — caller resolved at hybrid/op boundary. The cast SQL
|
||||
// ($1::vector vs $1::halfvec(N)) comes from buildVectorCastFragment.
|
||||
//
|
||||
// v0.36 Phase 3: 'embedding_multimodal' is the unified column populated
|
||||
// by `gbrain reindex --multimodal`. No modality filter — the column
|
||||
// itself is the discriminator (only re-embedded rows have non-NULL).
|
||||
const resolvedCol = normalizeEngineColumn(opts?.embeddingColumn);
|
||||
const { col, castSql } = buildVectorCastFragment(resolvedCol);
|
||||
// Image rows live in modality='image'; text/code in 'text'. Restrict
|
||||
// by modality so non-image columns can't accidentally pull image
|
||||
// chunks (or vice versa). resolved.name has already passed regex
|
||||
// validation; never compares against raw input.
|
||||
const modalityFilter = resolvedCol.name === 'embedding_image'
|
||||
? `AND cc.modality = 'image'`
|
||||
: `AND cc.modality = 'text'`;
|
||||
let modalityFilter: string;
|
||||
if (resolvedCol.name === 'embedding_image') {
|
||||
modalityFilter = `AND cc.modality = 'image'`;
|
||||
} else if (resolvedCol.name === 'embedding_multimodal') {
|
||||
modalityFilter = '';
|
||||
} else {
|
||||
modalityFilter = `AND cc.modality = 'text'`;
|
||||
}
|
||||
|
||||
const { rows } = await this.db.query(
|
||||
`WITH hnsw_candidates AS (
|
||||
|
||||
@@ -119,7 +119,11 @@ CREATE TABLE IF NOT EXISTS content_chunks (
|
||||
-- chunks carry their 1024-dim Voyage multimodal vector in embedding_image
|
||||
-- (independent of the brain primary embedding column dim).
|
||||
modality TEXT NOT NULL DEFAULT 'text',
|
||||
embedding_image vector(1024)
|
||||
embedding_image vector(1024),
|
||||
-- v0.36 Phase 3 cross-modal: unified column populated by reindex
|
||||
-- (search.unified_multimodal=true routes here). Migration v75 adds it
|
||||
-- on upgrade; fresh installs land at head with the column present.
|
||||
embedding_multimodal vector(1024)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
|
||||
|
||||
@@ -1303,16 +1303,21 @@ export class PostgresEngine implements BrainEngine {
|
||||
// read config — caller (hybrid/op) resolved it and passed it in.
|
||||
// normalizeEngineColumn accepts the legacy union (string literals,
|
||||
// ResolvedColumn, undefined) and produces a canonical descriptor.
|
||||
//
|
||||
// v0.36 Phase 3: 'embedding_multimodal' is the unified column populated
|
||||
// by `gbrain reindex --multimodal`. Carries BOTH text and image content
|
||||
// in Voyage multimodal-3 space — no modality filter; the column itself
|
||||
// is the discriminator (rows without embedding_multimodal aren't searched).
|
||||
const resolvedCol = normalizeEngineColumn(opts?.embeddingColumn);
|
||||
const { col, castSql } = buildVectorCastFragment(resolvedCol);
|
||||
// Modality filter: image rows live in modality='image'; text/code in
|
||||
// 'text'. The image-column literal is the only one with image rows.
|
||||
// For all other columns (default + user-declared), restrict to text
|
||||
// mode to avoid cross-modality dim leaks. The check is on
|
||||
// resolved.name (already validated, never raw input).
|
||||
const modalityFilter = resolvedCol.name === 'embedding_image'
|
||||
? `AND cc.modality = 'image'`
|
||||
: `AND cc.modality = 'text'`;
|
||||
let modalityFilter: string;
|
||||
if (resolvedCol.name === 'embedding_image') {
|
||||
modalityFilter = `AND cc.modality = 'image'`;
|
||||
} else if (resolvedCol.name === 'embedding_multimodal') {
|
||||
modalityFilter = '';
|
||||
} else {
|
||||
modalityFilter = `AND cc.modality = 'text'`;
|
||||
}
|
||||
|
||||
const rawQuery = `
|
||||
WITH hnsw_candidates AS (
|
||||
|
||||
@@ -159,7 +159,10 @@ CREATE TABLE IF NOT EXISTS content_chunks (
|
||||
-- mixed-provider brains (e.g. OpenAI 1536 text + Voyage 1024 images) keep
|
||||
-- both columns populated with distinct dim spaces.
|
||||
modality TEXT NOT NULL DEFAULT 'text',
|
||||
embedding_image vector(1024)
|
||||
embedding_image vector(1024),
|
||||
-- v0.36 Phase 3 cross-modal: unified column populated by reindex.
|
||||
-- Migration v75 also adds it for upgrade paths.
|
||||
embedding_multimodal vector(1024)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
/**
|
||||
* v0.36 Phase 2 — `searchByImage` retrieval.
|
||||
*
|
||||
* D17 honest framing: Phase 2 ships image→similar-images + image-OCR-text
|
||||
* retrieval. Image chunks have OCR text in `chunk_text`, so the results
|
||||
* carry both visual-similarity AND any OCR text the chunks captured. True
|
||||
* image→full-text-knowledge (Alice's bio, takes about her) requires text
|
||||
* chunks in the multimodal embedding space — Phase 3's unified column.
|
||||
*
|
||||
* D13 hybrid intersect for optional text refinement: when caller supplies
|
||||
* `query`, runs image-vector AND text-vector searches in parallel and
|
||||
* merges via weighted RRF. Treats text refinement as a full second axis
|
||||
* (not a vector-space bias from naive averaging).
|
||||
*
|
||||
* Routing matrix:
|
||||
* - image-only (no query):
|
||||
* embedQueryMultimodalImage → searchVector(embedding_image)
|
||||
* - image + text query (D13):
|
||||
* embedQueryMultimodalImage AND embedQueryMultimodal(query)
|
||||
* → parallel searchVector calls → rrfFusionWeighted merge
|
||||
* - Phase 3 unified mode (search.unified_multimodal=true): both branches
|
||||
* target embedding_multimodal instead of embedding_image
|
||||
*
|
||||
* D5 source-id: every engine call receives sourceOpts threaded from
|
||||
* `ctx.sourceId` / `ctx.auth.allowedSources`. Tested by
|
||||
* test/e2e/source-isolation-image.test.ts (4 cases).
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { SearchOpts, SearchResult } from '../types.ts';
|
||||
import { effectiveRrfK } from './intent-weights.ts';
|
||||
import { rrfFusionWeighted, RRF_K } from './hybrid.ts';
|
||||
import { dedupResults } from './dedup.ts';
|
||||
import { embedQueryMultimodal, embedQueryMultimodalImage } from '../ai/gateway.ts';
|
||||
import { loadSearchModeConfig, resolveSearchMode } from './mode.ts';
|
||||
|
||||
export interface SearchByImageOpts extends SearchOpts {
|
||||
/** Optional text refinement; runs hybrid intersect via weighted RRF (D13). */
|
||||
query?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run image-as-query retrieval against the brain.
|
||||
*
|
||||
* @param engine brain engine (PGLite or Postgres)
|
||||
* @param input loaded image (from `loadImageInput`): { base64, mime }
|
||||
* @param opts SearchOpts + optional `query` for D13 text refinement
|
||||
*/
|
||||
export async function searchByImage(
|
||||
engine: BrainEngine,
|
||||
input: { base64: string; mime: string },
|
||||
opts: SearchByImageOpts = {},
|
||||
): Promise<SearchResult[]> {
|
||||
// Resolve mode bundle once at entry — picks up cross-modal RRF weights
|
||||
// (D13 image_query_text_refinement_weight / image_query_image_refinement_weight)
|
||||
// and the Phase 3 unified-multimodal flag.
|
||||
const modeInput = await loadSearchModeConfig(engine);
|
||||
const resolvedMode = resolveSearchMode({
|
||||
mode: modeInput.mode,
|
||||
overrides: modeInput.overrides,
|
||||
});
|
||||
|
||||
const limit = opts.limit ?? resolvedMode.searchLimit;
|
||||
const offset = opts.offset ?? 0;
|
||||
// Phase 2 always targets embedding_image. Phase 3's unified column
|
||||
// routing slots in here once src/core/types.ts widens
|
||||
// SearchOpts.embeddingColumn to include 'embedding_multimodal' (Commit 3
|
||||
// schema migration + type widening).
|
||||
const imageColumn: 'embedding_image' = 'embedding_image';
|
||||
|
||||
const baseSearchOpts: SearchOpts = {
|
||||
limit: Math.min(limit * 2, 100),
|
||||
offset: 0,
|
||||
sourceId: opts.sourceId,
|
||||
sourceIds: opts.sourceIds,
|
||||
// Both branches use the same source-scope threading.
|
||||
};
|
||||
|
||||
// Image branch — always runs.
|
||||
const imageEmbedding = await embedQueryMultimodalImage({
|
||||
data: input.base64,
|
||||
mime: input.mime,
|
||||
});
|
||||
const imageOpts: SearchOpts = { ...baseSearchOpts, embeddingColumn: imageColumn };
|
||||
const imageList = await engine.searchVector(imageEmbedding, imageOpts);
|
||||
// Tag rows with modality so downstream consumers can distinguish in 'both' results.
|
||||
for (const r of imageList) {
|
||||
r.modality = r.modality ?? 'image';
|
||||
}
|
||||
|
||||
// D13: if caller provided a text refinement query, run the text branch too
|
||||
// and merge via weighted RRF.
|
||||
if (opts.query && opts.query.trim().length > 0) {
|
||||
const textEmbedding = await embedQueryMultimodal(opts.query);
|
||||
const textOpts: SearchOpts = {
|
||||
...baseSearchOpts,
|
||||
embeddingColumn: imageColumn,
|
||||
};
|
||||
// Both branches target the same multimodal-embedding column (Phase 2).
|
||||
// The "intersect" is via RRF rank merging — the text-side vector lives
|
||||
// in the SAME multimodal embedding space as the image vector
|
||||
// (embedQueryMultimodal returns a 1024d vector), so it's the right
|
||||
// space to query.
|
||||
const textList = await engine.searchVector(textEmbedding, textOpts);
|
||||
for (const r of textList) {
|
||||
r.modality = r.modality ?? 'image';
|
||||
}
|
||||
|
||||
// Weighted RRF merge: image branch has higher default weight (0.6 vs 0.4)
|
||||
// because the caller chose image-first. Configurable via
|
||||
// search.image_query.*_refinement_weight (D3 registered).
|
||||
const baseRrfK = RRF_K;
|
||||
const imageK = effectiveRrfK(baseRrfK, resolvedMode.image_query_image_refinement_weight);
|
||||
const textK = effectiveRrfK(baseRrfK, resolvedMode.image_query_text_refinement_weight);
|
||||
const fused = rrfFusionWeighted(
|
||||
[
|
||||
{ list: imageList, k: imageK },
|
||||
{ list: textList, k: textK },
|
||||
],
|
||||
true, // apply boost
|
||||
);
|
||||
fused.sort((a, b) => b.score - a.score);
|
||||
return dedupResults(fused).slice(offset, offset + limit);
|
||||
}
|
||||
|
||||
// Image-only: return the deduped image branch directly.
|
||||
return dedupResults(imageList).slice(offset, offset + limit);
|
||||
}
|
||||
@@ -527,6 +527,17 @@ export function normalizeEngineColumn(
|
||||
};
|
||||
}
|
||||
|
||||
// v0.36 Phase 3 unified multimodal column — populated by
|
||||
// `gbrain reindex --multimodal`. Voyage multimodal-3, vector(1024).
|
||||
if (embeddingColumn === 'embedding_multimodal') {
|
||||
return {
|
||||
name: 'embedding_multimodal',
|
||||
type: 'vector',
|
||||
dimensions: 1024,
|
||||
embeddingModel: 'voyage:voyage-multimodal-3',
|
||||
};
|
||||
}
|
||||
|
||||
// Any other raw string at this layer is a programming error — the
|
||||
// resolver should have run at the hybrid/op boundary and produced a
|
||||
// descriptor. We throw with a paste-ready hint rather than guess.
|
||||
|
||||
+203
-29
@@ -17,7 +17,7 @@ import { resolveEmbeddingColumn, isCacheSafe } from './embedding-column.ts';
|
||||
import { loadConfigWithEngine } from '../config.ts';
|
||||
import { dedupResults } from './dedup.ts';
|
||||
import { applyReranker } from './rerank.ts';
|
||||
import { autoDetectDetail, classifyQuery } from './query-intent.ts';
|
||||
import { autoDetectDetail, classifyQuery, isAmbiguousModalityQuery } from './query-intent.ts';
|
||||
import { expandAnchors, hydrateChunks } from './two-pass.ts';
|
||||
import { enforceTokenBudget } from './token-budget.ts';
|
||||
import { recordSearchTelemetry } from './telemetry.ts';
|
||||
@@ -31,7 +31,7 @@ import {
|
||||
loadCacheConfig,
|
||||
} from './query-cache.ts';
|
||||
|
||||
const RRF_K = 60;
|
||||
export const RRF_K = 60;
|
||||
const COMPILED_TRUTH_BOOST = 2.0;
|
||||
const pendingCacheWrites = new Set<Promise<unknown>>();
|
||||
|
||||
@@ -454,8 +454,22 @@ export async function hybridSearch(
|
||||
console.error(`[search-debug] auto-detail=${detail} for query="${query}"`);
|
||||
}
|
||||
|
||||
// Run keyword search (always available, no API key needed)
|
||||
const keywordResults = await engine.searchKeyword(query, searchOpts);
|
||||
// Run keyword search (always available, no API key needed).
|
||||
//
|
||||
// v0.36 cross-modal (D9): skip keyword for 'image'-only modality. Image
|
||||
// chunks may have OCR text in chunk_text, but a text-only keyword scan
|
||||
// would also surface every text chunk containing the query phrase —
|
||||
// not what an image-intent query asked for. Image vector search is the
|
||||
// canonical channel for image-modality queries.
|
||||
//
|
||||
// We classify modality early (it's also computed after for the modality
|
||||
// branch). The classification is pure regex via classifyQuery; running it
|
||||
// here is cheap.
|
||||
const earlyModality = (opts?.crossModal && opts.crossModal !== 'auto')
|
||||
? opts.crossModal
|
||||
: (suggestions.suggestedModality ?? 'text');
|
||||
const keywordResults: SearchResult[] =
|
||||
earlyModality === 'image' ? [] : await engine.searchKeyword(query, searchOpts);
|
||||
|
||||
// v0.29.1: resolve salience/recency from caller (back-compat aliases for
|
||||
// PR #618's `recencyBoost` numeric scale) or fall back to the heuristic.
|
||||
@@ -526,14 +540,62 @@ export async function hybridSearch(
|
||||
return noEmbedBudgeted;
|
||||
}
|
||||
|
||||
// v0.36 cross-modal wave: determine the effective modality once.
|
||||
//
|
||||
// Precedence (D22-1 normalization): literal 'auto' is normalized to
|
||||
// undefined so it doesn't reach the modality branch directly. Resolution:
|
||||
// explicit opts.crossModal ('text'|'image'|'both') wins
|
||||
// else suggestions.suggestedModality (regex-driven)
|
||||
// else (Commit 4) opt-in LLM tie-break for genuinely ambiguous queries
|
||||
// else 'text' (default)
|
||||
//
|
||||
// D9 mode-bundle override matrix: when effectiveModality === 'image',
|
||||
// cross-modal path overrides bundle knobs (expansion=false, no keyword
|
||||
// search). Voyage handles synonyms in-space; zerank-2 can't rerank image
|
||||
// embeddings.
|
||||
//
|
||||
// Phase 3 (D8): when search.unified_multimodal is true, ALL queries
|
||||
// route through the multimodal model + embedding_multimodal column,
|
||||
// regardless of detected modality.
|
||||
//
|
||||
// Commit 4 (LLM intent escalation): when search.cross_modal.llm_intent
|
||||
// is true AND regex returned 'text' AND isAmbiguousModalityQuery fires,
|
||||
// await a Haiku tie-break. Fail-open to regex result on any error.
|
||||
const explicitModality =
|
||||
opts?.crossModal && opts.crossModal !== 'auto' ? opts.crossModal : undefined;
|
||||
let regexModality = explicitModality ?? suggestions.suggestedModality ?? 'text';
|
||||
// LLM tie-break fires ONLY when:
|
||||
// - no explicit per-call override
|
||||
// - regex returned 'text' (not confident image/both)
|
||||
// - operator opted in via search.cross_modal.llm_intent
|
||||
// - isAmbiguousModalityQuery says the query is genuinely ambiguous
|
||||
if (
|
||||
explicitModality === undefined &&
|
||||
regexModality === 'text' &&
|
||||
resolvedMode.cross_modal_llm_intent &&
|
||||
isAmbiguousModalityQuery(query)
|
||||
) {
|
||||
try {
|
||||
const { classifyModalityWithLLM } = await import('./llm-intent.ts');
|
||||
regexModality = await classifyModalityWithLLM(query, 'text');
|
||||
} catch {
|
||||
// Fail-open: regex result stands.
|
||||
}
|
||||
}
|
||||
const effectiveModality = regexModality;
|
||||
const unifiedRouting = resolvedMode.unified_multimodal === true;
|
||||
|
||||
// Determine query variants (optionally with expansion)
|
||||
// expandQuery already includes the original query in its return value,
|
||||
// so we use it directly instead of prepending query again.
|
||||
// v0.32.3 search-lite: expansion fires when (a) resolved mode says yes and
|
||||
// (b) an expandFn is wired in. The mode bundle is the default; per-call
|
||||
// SearchOpts.expansion still wins via resolveSearchMode's chain.
|
||||
//
|
||||
// D9: image-modality skips expansion regardless of mode bundle.
|
||||
let queries = [query];
|
||||
if (resolvedMode.expansion && opts?.expandFn) {
|
||||
const expansionAllowed = resolvedMode.expansion && effectiveModality !== 'image';
|
||||
if (expansionAllowed && opts?.expandFn) {
|
||||
try {
|
||||
queries = await opts.expandFn(query);
|
||||
if (queries.length === 0) queries = [query];
|
||||
@@ -544,28 +606,121 @@ export async function hybridSearch(
|
||||
}
|
||||
}
|
||||
|
||||
// Embed all query variants and run vector search
|
||||
// Embed all query variants and run vector search.
|
||||
//
|
||||
// v0.36 cross-modal wave routing:
|
||||
// - 'text' (default): existing text-embedding path, unchanged
|
||||
// - 'image': embedQueryMultimodal + searchVector(embedding_image), skip keyword
|
||||
// - 'both': text + image vector searches in parallel; merged via weighted RRF
|
||||
let vectorLists: SearchResult[][] = [];
|
||||
let queryEmbedding: Float32Array | null = null;
|
||||
try {
|
||||
// v0.35.0.0+: query-side embedding. For asymmetric providers (ZE zembed-1,
|
||||
// Voyage v3+) routes input_type='query' through the embed seam; symmetric
|
||||
// providers ignore the field — no behavior change.
|
||||
// v0.36 (D10): route through the resolved column's provider + dims so
|
||||
// a query against `embedding_voyage` actually embeds via Voyage, not
|
||||
// the global default (OpenAI). Empty embeddingModel falls back to
|
||||
// gateway default — preserves pre-v0.36 behavior for the builtin
|
||||
// 'embedding' column.
|
||||
const embedOpts = resolvedCol.embeddingModel
|
||||
? { embeddingModel: resolvedCol.embeddingModel, dimensions: resolvedCol.dimensions }
|
||||
: undefined;
|
||||
const embeddings = await Promise.all(queries.map(q => embedQuery(q, embedOpts)));
|
||||
queryEmbedding = embeddings[0];
|
||||
vectorLists = await Promise.all(
|
||||
embeddings.map(emb => engine.searchVector(emb, searchOpts)),
|
||||
);
|
||||
} catch {
|
||||
// Embedding failure is non-fatal, fall back to keyword-only
|
||||
let imageVectorList: SearchResult[] | null = null;
|
||||
let crossModalFellOpen = false;
|
||||
|
||||
// Phase 3 unified routing: when on, route ALL queries through Voyage
|
||||
// multimodal-3 + embedding_multimodal column. Bypasses the dual-column
|
||||
// branching below — but with D8 fail-open: if the unified path returns
|
||||
// zero rows AND the operator hasn't opted into strict unified-only mode,
|
||||
// fall through to the dual-column text path. unified_multimodal_only
|
||||
// disables the fallback.
|
||||
let unifiedDone = false;
|
||||
if (unifiedRouting) {
|
||||
try {
|
||||
const { isAvailable: aiIsAvailable, embedQueryMultimodal } = await import('../ai/gateway.ts');
|
||||
if (!aiIsAvailable('embedding')) {
|
||||
throw new Error('gateway not configured for embedding — unified multimodal would also fail');
|
||||
}
|
||||
const unifiedEmbedding = await embedQueryMultimodal(query);
|
||||
const unifiedSearchOpts: SearchOpts = {
|
||||
...searchOpts,
|
||||
embeddingColumn: 'embedding_multimodal',
|
||||
};
|
||||
const unifiedList = await engine.searchVector(unifiedEmbedding, unifiedSearchOpts);
|
||||
// D8 fail-open: zero rows + not strict-mode → fall through to dual-column.
|
||||
if (unifiedList.length === 0 && !resolvedMode.unified_multimodal_only) {
|
||||
console.error(
|
||||
`[cross-modal] unified_multimodal returned zero rows for query="${query.slice(0, 60)}". ` +
|
||||
`Falling back to dual-column text path (partial coverage during reindex). ` +
|
||||
`Set search.unified_multimodal_only=true to bypass this fallback when reindex completes.`,
|
||||
);
|
||||
} else {
|
||||
vectorLists = [unifiedList];
|
||||
queryEmbedding = unifiedEmbedding;
|
||||
unifiedDone = true;
|
||||
}
|
||||
} catch (err) {
|
||||
const reason = err instanceof Error ? err.message : String(err);
|
||||
console.error(
|
||||
`[cross-modal] unified_multimodal embed failed; falling back to dual-column path. reason=${reason}`,
|
||||
);
|
||||
crossModalFellOpen = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (!unifiedDone && (effectiveModality === 'image' || effectiveModality === 'both')) {
|
||||
// Attempt image-side embedding. Fail-open: if multimodal is unconfigured
|
||||
// OR the embed throws, log a structured warning and fall through to text.
|
||||
try {
|
||||
const { isAvailable: aiIsAvailable, embedQueryMultimodal } = await import('../ai/gateway.ts');
|
||||
if (!aiIsAvailable('embedding')) {
|
||||
throw new Error('gateway not configured for embedding — multimodal would also fail');
|
||||
}
|
||||
const imageEmbedding = await embedQueryMultimodal(query);
|
||||
const imageSearchOpts: SearchOpts = {
|
||||
...searchOpts,
|
||||
embeddingColumn: 'embedding_image',
|
||||
};
|
||||
const imageList = await engine.searchVector(imageEmbedding, imageSearchOpts);
|
||||
for (const r of imageList) {
|
||||
r.modality = r.modality ?? 'image';
|
||||
}
|
||||
imageVectorList = imageList;
|
||||
} catch (err) {
|
||||
// Fail-open per behavioral invariant 2.
|
||||
const reason = err instanceof Error ? err.message : String(err);
|
||||
console.error(
|
||||
`[cross-modal] image-side embed failed for modality=${effectiveModality}; falling back to text-only. reason=${reason}`,
|
||||
);
|
||||
crossModalFellOpen = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (unifiedDone) {
|
||||
// Unified routing already populated vectorLists + queryEmbedding;
|
||||
// skip the dual-column branching.
|
||||
} else if (effectiveModality === 'image' && imageVectorList !== null) {
|
||||
// Image-only path: results come entirely from the image column.
|
||||
vectorLists = [imageVectorList];
|
||||
queryEmbedding = null; // no text embedding to cosine-re-score against
|
||||
} else {
|
||||
// 'text' or 'both' (or 'image' that fell open to text). Run the text
|
||||
// path normally, with v0.36 (D10) provider-aware embed routing so a
|
||||
// query against `embedding_voyage` actually embeds via Voyage, not
|
||||
// the global default. Empty embeddingModel falls back to gateway
|
||||
// default — preserves pre-v0.36 behavior for the builtin 'embedding'
|
||||
// column.
|
||||
try {
|
||||
const embedOpts = resolvedCol.embeddingModel
|
||||
? { embeddingModel: resolvedCol.embeddingModel, dimensions: resolvedCol.dimensions }
|
||||
: undefined;
|
||||
const embeddings = await Promise.all(queries.map(q => embedQuery(q, embedOpts)));
|
||||
queryEmbedding = embeddings[0];
|
||||
const textLists = await Promise.all(
|
||||
embeddings.map(emb => engine.searchVector(emb, searchOpts)),
|
||||
);
|
||||
for (const list of textLists) {
|
||||
for (const r of list) {
|
||||
r.modality = r.modality ?? 'text';
|
||||
}
|
||||
}
|
||||
vectorLists = textLists;
|
||||
// 'both' mode: also include the image-side list as another input to RRF.
|
||||
if (effectiveModality === 'both' && imageVectorList !== null) {
|
||||
vectorLists = [...vectorLists, imageVectorList];
|
||||
}
|
||||
} catch {
|
||||
// Embedding failure is non-fatal, fall back to keyword-only
|
||||
}
|
||||
}
|
||||
|
||||
if (vectorLists.length === 0) {
|
||||
@@ -605,10 +760,29 @@ export async function hybridSearch(
|
||||
const baseRrfK = opts?.rrfK ?? RRF_K;
|
||||
const keywordK = effectiveRrfK(baseRrfK, intentWeights.keywordWeight);
|
||||
const vectorK = effectiveRrfK(baseRrfK, intentWeights.vectorWeight);
|
||||
const allLists: Array<{ list: SearchResult[]; k: number }> = [
|
||||
...vectorLists.map(list => ({ list, k: vectorK })),
|
||||
{ list: keywordResults, k: keywordK },
|
||||
];
|
||||
|
||||
// v0.36 cross-modal (D6): in 'both' mode, vectorLists carries
|
||||
// [textList, imageList]. Apply per-modality RRF weights so the merge
|
||||
// reflects the configured text/image balance. In 'text' and 'image'
|
||||
// modes only one branch is present, so per-modality K reduces to
|
||||
// the standard vectorK (no behavior change vs pre-v0.36).
|
||||
const textRrfK = effectiveRrfK(baseRrfK, resolvedMode.cross_modal_both_text_weight);
|
||||
const imageRrfK = effectiveRrfK(baseRrfK, resolvedMode.cross_modal_both_image_weight);
|
||||
const isBothMode = effectiveModality === 'both' && vectorLists.length >= 2;
|
||||
|
||||
const allLists: Array<{ list: SearchResult[]; k: number }> = isBothMode
|
||||
? [
|
||||
// Last list in vectorLists is the image branch (we appended it above).
|
||||
// All preceding lists (1 or more text-query embeddings if expansion ran)
|
||||
// get textRrfK. Image branch gets imageRrfK.
|
||||
...vectorLists.slice(0, -1).map(list => ({ list, k: textRrfK })),
|
||||
{ list: vectorLists[vectorLists.length - 1], k: imageRrfK },
|
||||
{ list: keywordResults, k: keywordK },
|
||||
]
|
||||
: [
|
||||
...vectorLists.map(list => ({ list, k: vectorK })),
|
||||
{ list: keywordResults, k: keywordK },
|
||||
];
|
||||
let fused = rrfFusionWeighted(allLists, detail !== 'high');
|
||||
|
||||
// Cosine re-scoring before dedup so semantically better chunks survive.
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
/**
|
||||
* v0.36 Phase 2 — image input loader for `search_by_image`.
|
||||
*
|
||||
* Accepts three input forms:
|
||||
* - absolute path: file:// URI OR /abs/path/to/img.png (local CLI only)
|
||||
* - data: URI: data:image/png;base64,<base64bytes>
|
||||
* - http(s) URL: https://example.com/img.png (SSRF-defended via ssrf-validate.ts)
|
||||
*
|
||||
* Returns: { contentType, base64, bytes } so the caller can hand it to
|
||||
* `embedQueryMultimodalImage({data: base64, mime: contentType})`.
|
||||
*
|
||||
* Defenses:
|
||||
* - Magic-byte sniff for PNG / JPEG / WebP (no other formats accepted in v1)
|
||||
* - Hard size cap (default 10 MB) — applies before allocation when possible
|
||||
* via Content-Length pre-flight, and as a stream cap during fetch
|
||||
* - SSRF: every URL hop validated via `validateAndResolveUrl` from
|
||||
* `src/core/ssrf-validate.ts`; max 3 redirect hops
|
||||
* - 5s total fetch timeout (not per-hop)
|
||||
* - Rejects credentials embedded in URL
|
||||
*
|
||||
* Out of scope:
|
||||
* - Pixel-bomb defense (decompression bombs): the size cap doubles as the
|
||||
* bomb defense per D22-9. We never call a pixel parser; only feed bytes
|
||||
* to Voyage. A 10MB cap is well below the pixel-bomb threshold.
|
||||
* - HEIC / GIF / TIFF — Voyage multimodal-3 supports them but the test
|
||||
* fixtures and OCR pipeline don't, so we keep the input surface
|
||||
* conservative for v1.
|
||||
*/
|
||||
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { isAbsolute } from 'node:path';
|
||||
import { SSRFError, fetchWithSSRFGuard } from '../ssrf-validate.ts';
|
||||
|
||||
/** Max bytes for input image. Configurable via `search.image_query.max_bytes`. */
|
||||
export const DEFAULT_IMAGE_MAX_BYTES = 10 * 1024 * 1024;
|
||||
|
||||
/** Stricter cap for remote MCP callers (ctx.remote === true). */
|
||||
export const DEFAULT_REMOTE_IMAGE_MAX_BYTES = 2 * 1024 * 1024;
|
||||
|
||||
export interface LoadedImage {
|
||||
/** MIME content type ('image/png', 'image/jpeg', 'image/webp'). */
|
||||
contentType: string;
|
||||
/** Raw bytes as Buffer. Caller can use this for hashing/auditing. */
|
||||
bytes: Buffer;
|
||||
/** Base64-encoded bytes, ready to feed into embedQueryMultimodalImage. */
|
||||
base64: string;
|
||||
}
|
||||
|
||||
export class ImageLoadError extends Error {
|
||||
readonly code: ImageLoadErrorCode;
|
||||
constructor(code: ImageLoadErrorCode, message: string) {
|
||||
super(message);
|
||||
this.name = 'ImageLoadError';
|
||||
this.code = code;
|
||||
}
|
||||
}
|
||||
|
||||
export type ImageLoadErrorCode =
|
||||
| 'INVALID_FORMAT'
|
||||
| 'OVERSIZED'
|
||||
| 'INVALID_URL'
|
||||
| 'FETCH_FAILED'
|
||||
| 'TIMEOUT'
|
||||
| 'SSRF_BLOCKED'
|
||||
| 'NOT_FOUND';
|
||||
|
||||
export interface ImageLoadOpts {
|
||||
/** Override the default max-bytes cap. Defaults to DEFAULT_IMAGE_MAX_BYTES. */
|
||||
maxBytes?: number;
|
||||
/** Total fetch timeout in ms (URLs only). Defaults to 5000. */
|
||||
timeoutMs?: number;
|
||||
/** Max redirect hops (URLs only). Defaults to 3. */
|
||||
maxRedirects?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Load an image input into a structured shape.
|
||||
*
|
||||
* Throws `ImageLoadError` on any failure — `SSRFError` from the URL path
|
||||
* is caught and re-thrown as ImageLoadError with `code: 'SSRF_BLOCKED'`.
|
||||
*/
|
||||
export async function loadImageInput(
|
||||
input: string,
|
||||
opts: ImageLoadOpts = {},
|
||||
): Promise<LoadedImage> {
|
||||
if (typeof input !== 'string' || input.length === 0) {
|
||||
throw new ImageLoadError('INVALID_URL', 'Image input must be a non-empty string');
|
||||
}
|
||||
const maxBytes = opts.maxBytes ?? DEFAULT_IMAGE_MAX_BYTES;
|
||||
|
||||
// Branch on input shape.
|
||||
if (input.startsWith('data:')) {
|
||||
return loadDataUri(input, maxBytes);
|
||||
}
|
||||
if (input.startsWith('http://') || input.startsWith('https://')) {
|
||||
return loadHttpUrl(input, { maxBytes, timeoutMs: opts.timeoutMs ?? 5000, maxRedirects: opts.maxRedirects ?? 3 });
|
||||
}
|
||||
if (input.startsWith('file://') || isAbsolute(input)) {
|
||||
const path = input.startsWith('file://') ? input.slice(7) : input;
|
||||
return loadLocalPath(path, maxBytes);
|
||||
}
|
||||
throw new ImageLoadError(
|
||||
'INVALID_URL',
|
||||
`Unsupported image input shape: expected data: URI, http(s):// URL, file:// URI, or absolute path. Got: ${input.slice(0, 60)}`,
|
||||
);
|
||||
}
|
||||
|
||||
async function loadLocalPath(path: string, maxBytes: number): Promise<LoadedImage> {
|
||||
let bytes: Buffer;
|
||||
try {
|
||||
bytes = await readFile(path);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
if (msg.includes('ENOENT')) {
|
||||
throw new ImageLoadError('NOT_FOUND', `File not found: ${path}`);
|
||||
}
|
||||
throw new ImageLoadError('FETCH_FAILED', `Failed to read ${path}: ${msg}`);
|
||||
}
|
||||
if (bytes.length > maxBytes) {
|
||||
throw new ImageLoadError(
|
||||
'OVERSIZED',
|
||||
`Image at ${path} is ${bytes.length} bytes; cap is ${maxBytes}`,
|
||||
);
|
||||
}
|
||||
const contentType = sniffContentType(bytes);
|
||||
return finalize(bytes, contentType);
|
||||
}
|
||||
|
||||
function loadDataUri(input: string, maxBytes: number): LoadedImage {
|
||||
// data:image/png;base64,<bytes>
|
||||
const match = input.match(/^data:([^;]+);base64,(.+)$/);
|
||||
if (!match) {
|
||||
throw new ImageLoadError('INVALID_FORMAT', 'data: URI must be in `data:<mime>;base64,<bytes>` form');
|
||||
}
|
||||
const declaredMime = match[1].toLowerCase();
|
||||
const b64 = match[2];
|
||||
let bytes: Buffer;
|
||||
try {
|
||||
bytes = Buffer.from(b64, 'base64');
|
||||
} catch (err) {
|
||||
throw new ImageLoadError(
|
||||
'INVALID_FORMAT',
|
||||
`Failed to decode base64: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
if (bytes.length > maxBytes) {
|
||||
throw new ImageLoadError(
|
||||
'OVERSIZED',
|
||||
`Decoded image is ${bytes.length} bytes; cap is ${maxBytes}`,
|
||||
);
|
||||
}
|
||||
const sniffed = sniffContentType(bytes);
|
||||
// If the declared MIME disagrees with the magic-byte sniff, trust the sniff
|
||||
// (caller could lie about content-type to bypass format gate).
|
||||
if (sniffed !== declaredMime) {
|
||||
// Fall through with the sniffed type. Don't error — declared MIME is
|
||||
// informational; bytes are the truth.
|
||||
}
|
||||
return finalize(bytes, sniffed);
|
||||
}
|
||||
|
||||
async function loadHttpUrl(
|
||||
url: string,
|
||||
opts: { maxBytes: number; timeoutMs: number; maxRedirects: number },
|
||||
): Promise<LoadedImage> {
|
||||
let res: Response;
|
||||
try {
|
||||
res = await fetchWithSSRFGuard(url, {
|
||||
maxRedirects: opts.maxRedirects,
|
||||
timeoutMs: opts.timeoutMs,
|
||||
});
|
||||
} catch (err) {
|
||||
if (err instanceof SSRFError) {
|
||||
throw new ImageLoadError('SSRF_BLOCKED', `SSRF: ${err.message}`);
|
||||
}
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
if (msg.includes('abort')) {
|
||||
throw new ImageLoadError('TIMEOUT', `Fetch timeout (${opts.timeoutMs}ms): ${url}`);
|
||||
}
|
||||
throw new ImageLoadError('FETCH_FAILED', `Fetch failed: ${msg}`);
|
||||
}
|
||||
if (!res.ok) {
|
||||
throw new ImageLoadError(
|
||||
'FETCH_FAILED',
|
||||
`Fetch returned HTTP ${res.status}: ${res.statusText || ''}`,
|
||||
);
|
||||
}
|
||||
// Pre-flight: reject oversized responses before reading the body.
|
||||
const contentLengthHeader = res.headers.get('content-length');
|
||||
if (contentLengthHeader) {
|
||||
const declared = parseInt(contentLengthHeader, 10);
|
||||
if (Number.isFinite(declared) && declared > opts.maxBytes) {
|
||||
throw new ImageLoadError(
|
||||
'OVERSIZED',
|
||||
`Response Content-Length ${declared} exceeds cap ${opts.maxBytes}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
// Read the body; second guard against lying Content-Length.
|
||||
const arrayBuf = await res.arrayBuffer();
|
||||
const bytes = Buffer.from(arrayBuf);
|
||||
if (bytes.length > opts.maxBytes) {
|
||||
throw new ImageLoadError(
|
||||
'OVERSIZED',
|
||||
`Response body is ${bytes.length} bytes; cap is ${opts.maxBytes} (lying Content-Length?)`,
|
||||
);
|
||||
}
|
||||
const contentType = sniffContentType(bytes);
|
||||
return finalize(bytes, contentType);
|
||||
}
|
||||
|
||||
/**
|
||||
* Magic-byte sniff for the three supported formats.
|
||||
*
|
||||
* - PNG: starts with `89 50 4E 47 0D 0A 1A 0A`
|
||||
* - JPEG: starts with `FF D8 FF`
|
||||
* - WebP: starts with `RIFF` (52 49 46 46) + 4 bytes + `WEBP` (57 45 42 50)
|
||||
*
|
||||
* Throws `ImageLoadError` with `code: 'INVALID_FORMAT'` for anything else.
|
||||
*/
|
||||
function sniffContentType(bytes: Buffer): string {
|
||||
if (bytes.length >= 8 &&
|
||||
bytes[0] === 0x89 && bytes[1] === 0x50 && bytes[2] === 0x4E && bytes[3] === 0x47 &&
|
||||
bytes[4] === 0x0D && bytes[5] === 0x0A && bytes[6] === 0x1A && bytes[7] === 0x0A) {
|
||||
return 'image/png';
|
||||
}
|
||||
if (bytes.length >= 3 && bytes[0] === 0xFF && bytes[1] === 0xD8 && bytes[2] === 0xFF) {
|
||||
return 'image/jpeg';
|
||||
}
|
||||
if (bytes.length >= 12 &&
|
||||
bytes[0] === 0x52 && bytes[1] === 0x49 && bytes[2] === 0x46 && bytes[3] === 0x46 &&
|
||||
bytes[8] === 0x57 && bytes[9] === 0x45 && bytes[10] === 0x42 && bytes[11] === 0x50) {
|
||||
return 'image/webp';
|
||||
}
|
||||
throw new ImageLoadError(
|
||||
'INVALID_FORMAT',
|
||||
`Unsupported image format. Magic bytes: ${bytes.subarray(0, Math.min(12, bytes.length)).toString('hex')}. Accepted: PNG, JPEG, WebP.`,
|
||||
);
|
||||
}
|
||||
|
||||
function finalize(bytes: Buffer, contentType: string): LoadedImage {
|
||||
return {
|
||||
contentType,
|
||||
bytes,
|
||||
base64: bytes.toString('base64'),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
/**
|
||||
* v0.36 Commit 4 — opt-in LLM tie-break for ambiguous modality queries.
|
||||
*
|
||||
* The pure-regex classifier in query-intent.ts handles 99% of queries
|
||||
* cleanly. For the ambiguous middle band (`isAmbiguousModalityQuery`
|
||||
* returns true), the operator can opt into a Haiku tie-break via
|
||||
* `search.cross_modal.llm_intent: true`.
|
||||
*
|
||||
* Cost bound: <1% of queries when on; ~$0.0001 per escalation. Bypassed
|
||||
* entirely when the config flag is false (default).
|
||||
*
|
||||
* Fail-open: any error (timeout, parse failure, gateway misconfig) returns
|
||||
* the input fallback ('text') so a misbehaving LLM can never break search.
|
||||
*/
|
||||
|
||||
import type { ModalityMode } from './query-intent.ts';
|
||||
|
||||
/** Default model tier for the tie-break call. Haiku 4.5 via utility tier. */
|
||||
const TIE_BREAK_TIMEOUT_MS = 1000;
|
||||
|
||||
const SYSTEM_PROMPT =
|
||||
'You classify search query modality. Output exactly one word: text, image, or both.\n' +
|
||||
'- text: the user wants written content (notes, takes, bios, articles)\n' +
|
||||
'- image: the user wants visual content (photos, screenshots, diagrams)\n' +
|
||||
'- both: the user is asking something that could be either\n' +
|
||||
'Output nothing except one of: text image both';
|
||||
|
||||
/**
|
||||
* Run a Haiku tie-break to classify the modality of an ambiguous query.
|
||||
*
|
||||
* Returns `fallback` on any error (timeout, parse failure, unrecognized
|
||||
* output). Production callers gate this on the
|
||||
* `search.cross_modal.llm_intent` config flag AND
|
||||
* `isAmbiguousModalityQuery(query) === true` so the LLM call only fires
|
||||
* for the narrow band where it actually adds signal.
|
||||
*/
|
||||
export async function classifyModalityWithLLM(
|
||||
query: string,
|
||||
fallback: ModalityMode = 'text',
|
||||
): Promise<ModalityMode> {
|
||||
let chat: typeof import('../ai/gateway.ts').chat;
|
||||
let isAvailable: typeof import('../ai/gateway.ts').isAvailable;
|
||||
try {
|
||||
({ chat, isAvailable } = await import('../ai/gateway.ts'));
|
||||
} catch {
|
||||
return fallback;
|
||||
}
|
||||
if (!isAvailable('chat')) {
|
||||
// Quiet bail — caller decides whether to warn.
|
||||
return fallback;
|
||||
}
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), TIE_BREAK_TIMEOUT_MS);
|
||||
try {
|
||||
const result = await chat({
|
||||
system: SYSTEM_PROMPT,
|
||||
messages: [{ role: 'user', content: query.slice(0, 500) }],
|
||||
maxTokens: 16,
|
||||
abortSignal: controller.signal,
|
||||
});
|
||||
return parseModality(result.text, fallback);
|
||||
} catch {
|
||||
// Timeout, network error, AI SDK throw — fail-open.
|
||||
return fallback;
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse the LLM's single-word output to a ModalityMode. Tolerates
|
||||
* trailing punctuation + casing. Anything unrecognized → fallback.
|
||||
*/
|
||||
export function parseModality(raw: string, fallback: ModalityMode): ModalityMode {
|
||||
const normalized = raw.trim().toLowerCase().replace(/[^a-z]+/g, '');
|
||||
if (normalized === 'text' || normalized === 'image' || normalized === 'both') {
|
||||
return normalized;
|
||||
}
|
||||
return fallback;
|
||||
}
|
||||
@@ -94,6 +94,7 @@ export interface ModeBundle {
|
||||
reranker_top_n_out: number | null;
|
||||
/** HTTP timeout in ms (default 5000). Threaded into gateway.rerank. */
|
||||
reranker_timeout_ms: number;
|
||||
|
||||
/**
|
||||
* v0.35.6.0 — floor-ratio gate for metadata-axis boost stages (backlink,
|
||||
* salience, recency). `undefined` = no gate (default for all three modes;
|
||||
@@ -111,6 +112,56 @@ export interface ModeBundle {
|
||||
* relevance signal and is NOT gated.
|
||||
*/
|
||||
floor_ratio: number | undefined;
|
||||
|
||||
// v0.36 cross-modal wave knobs (D2 + D3 + D6 + D8 + D13 + LLM-intent).
|
||||
// All three mode bundles default these to the same values — cross-modal
|
||||
// is opt-in per-call (D6 weighting), opt-in per-brain (D8 unified flags),
|
||||
// and opt-in per-feature-flag (LLM intent). The mode bundle just gives
|
||||
// resolveSearchMode a default to return.
|
||||
|
||||
/**
|
||||
* D6 'both'-mode RRF weight for text-vector results when merging
|
||||
* text + image searches in parallel. Defaults to 0.6 — biases toward
|
||||
* text recall because most queries with ambiguous modality are still
|
||||
* text-leaning. Pair with cross_modal_both_image_weight.
|
||||
*/
|
||||
cross_modal_both_text_weight: number;
|
||||
/**
|
||||
* D6 'both'-mode RRF weight for image-vector results. Defaults to 0.4.
|
||||
* Sum with text weight does NOT need to be 1.0 — RRF is rank-based, so
|
||||
* weights normalize internally; the ratio is what matters.
|
||||
*/
|
||||
cross_modal_both_image_weight: number;
|
||||
/**
|
||||
* D13 image-query text-refinement RRF weight for the TEXT branch of
|
||||
* searchByImage when the caller provides an optional `query` refinement.
|
||||
* Defaults to 0.4 (image-dominant since the caller chose image-first).
|
||||
*/
|
||||
image_query_text_refinement_weight: number;
|
||||
/**
|
||||
* D13 image-query refinement RRF weight for the IMAGE branch. Defaults to 0.6.
|
||||
*/
|
||||
image_query_image_refinement_weight: number;
|
||||
/**
|
||||
* D8 Phase 3 flag: route ALL queries through the multimodal query embed
|
||||
* + `embedding_multimodal` column. Default false. Operator opt-in after
|
||||
* `gbrain reindex --multimodal` populates the unified column.
|
||||
*/
|
||||
unified_multimodal: boolean;
|
||||
/**
|
||||
* D8 Phase 3 strict mode: when true, the dual-column fallback path is
|
||||
* bypassed entirely. Used by operators who finished re-embedding and
|
||||
* want to commit to the unified space. Doctor surface errors when this
|
||||
* is on and coverage < 99%.
|
||||
*/
|
||||
unified_multimodal_only: boolean;
|
||||
/**
|
||||
* Commit 4: opt-in LLM tie-break for ambiguous modality classification.
|
||||
* Default false. When true, queries where regex returns 'text' but the
|
||||
* ambiguity heuristic fires get a Haiku call to refine the classification.
|
||||
* Fires for <1% of queries when on; ~$0.0001 per escalation.
|
||||
*/
|
||||
cross_modal_llm_intent: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -139,6 +190,14 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
|
||||
// v0.35.6.0 — undefined for all three bundles; the per-corpus ablation
|
||||
// (TODOS.md) gates any default flip.
|
||||
floor_ratio: undefined,
|
||||
// v0.36 cross-modal defaults (same across all modes — opt-in)
|
||||
cross_modal_both_text_weight: 0.6,
|
||||
cross_modal_both_image_weight: 0.4,
|
||||
image_query_text_refinement_weight: 0.4,
|
||||
image_query_image_refinement_weight: 0.6,
|
||||
unified_multimodal: false,
|
||||
unified_multimodal_only: false,
|
||||
cross_modal_llm_intent: false,
|
||||
}),
|
||||
balanced: Object.freeze({
|
||||
cache_enabled: true,
|
||||
@@ -164,6 +223,14 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
|
||||
// v0.35.6.0 — undefined for all three bundles; the per-corpus ablation
|
||||
// (TODOS.md) gates any default flip.
|
||||
floor_ratio: undefined,
|
||||
// v0.36 cross-modal defaults (same across all modes — opt-in)
|
||||
cross_modal_both_text_weight: 0.6,
|
||||
cross_modal_both_image_weight: 0.4,
|
||||
image_query_text_refinement_weight: 0.4,
|
||||
image_query_image_refinement_weight: 0.6,
|
||||
unified_multimodal: false,
|
||||
unified_multimodal_only: false,
|
||||
cross_modal_llm_intent: false,
|
||||
}),
|
||||
tokenmax: Object.freeze({
|
||||
cache_enabled: true,
|
||||
@@ -186,6 +253,14 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
|
||||
// v0.35.6.0 — undefined for all three bundles; the per-corpus ablation
|
||||
// (TODOS.md) gates any default flip.
|
||||
floor_ratio: undefined,
|
||||
// v0.36 cross-modal defaults (same across all modes — opt-in)
|
||||
cross_modal_both_text_weight: 0.6,
|
||||
cross_modal_both_image_weight: 0.4,
|
||||
image_query_text_refinement_weight: 0.4,
|
||||
image_query_image_refinement_weight: 0.6,
|
||||
unified_multimodal: false,
|
||||
unified_multimodal_only: false,
|
||||
cross_modal_llm_intent: false,
|
||||
}),
|
||||
});
|
||||
|
||||
@@ -219,6 +294,14 @@ export interface SearchKeyOverrides {
|
||||
reranker_timeout_ms?: number;
|
||||
// v0.35.6.0 — floor-ratio gate override.
|
||||
floor_ratio?: number;
|
||||
// v0.36 cross-modal overrides
|
||||
cross_modal_both_text_weight?: number;
|
||||
cross_modal_both_image_weight?: number;
|
||||
image_query_text_refinement_weight?: number;
|
||||
image_query_image_refinement_weight?: number;
|
||||
unified_multimodal?: boolean;
|
||||
unified_multimodal_only?: boolean;
|
||||
cross_modal_llm_intent?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -244,6 +327,14 @@ export interface SearchPerCallOpts {
|
||||
reranker_timeout_ms?: number;
|
||||
// v0.35.6.0 — floor-ratio per-call override.
|
||||
floor_ratio?: number;
|
||||
// v0.36 cross-modal per-call overrides
|
||||
cross_modal_both_text_weight?: number;
|
||||
cross_modal_both_image_weight?: number;
|
||||
image_query_text_refinement_weight?: number;
|
||||
image_query_image_refinement_weight?: number;
|
||||
unified_multimodal?: boolean;
|
||||
unified_multimodal_only?: boolean;
|
||||
cross_modal_llm_intent?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -304,6 +395,14 @@ export function resolveSearchMode(input: ResolveSearchModeInput): ResolvedSearch
|
||||
reranker_timeout_ms: pick('reranker_timeout_ms'),
|
||||
// v0.35.6.0 — floor-ratio resolved via the same pick chain.
|
||||
floor_ratio: pick('floor_ratio'),
|
||||
// v0.36 cross-modal knobs
|
||||
cross_modal_both_text_weight: pick('cross_modal_both_text_weight'),
|
||||
cross_modal_both_image_weight: pick('cross_modal_both_image_weight'),
|
||||
image_query_text_refinement_weight: pick('image_query_text_refinement_weight'),
|
||||
image_query_image_refinement_weight: pick('image_query_image_refinement_weight'),
|
||||
unified_multimodal: pick('unified_multimodal'),
|
||||
unified_multimodal_only: pick('unified_multimodal_only'),
|
||||
cross_modal_llm_intent: pick('cross_modal_llm_intent'),
|
||||
resolved_mode,
|
||||
mode_valid: valid,
|
||||
};
|
||||
@@ -366,6 +465,12 @@ export function attributeKnob<K extends keyof ModeBundle>(
|
||||
// row IDs. Expect a temporary hit-rate dip + cache-row doubling for hot
|
||||
// queries during a rolling deploy. Clears naturally within
|
||||
// `cache.ttl_seconds` (default 3600s). The CHANGELOG note covers this.
|
||||
//
|
||||
// v0.36 wave: cross-modal knobs ALSO participate in v=3 hash (D2 cache
|
||||
// contamination fix — a text-mode cache hit cannot silently serve an
|
||||
// image-mode caller). v0.35.6.0's floor_ratio bump and v0.36's cross-modal
|
||||
// extensions both land under v=3, with cross-modal fields appended after
|
||||
// the floor_ratio entry (CDX2-F13 append-only convention).
|
||||
export const KNOBS_HASH_VERSION = 3;
|
||||
|
||||
/**
|
||||
@@ -420,6 +525,17 @@ export function knobsHash(
|
||||
// — they sit in different vector spaces. ctx is optional so
|
||||
// unrelated callers fall back to the default-column hash.
|
||||
`fr=${knobs.floor_ratio === undefined ? 'none' : knobs.floor_ratio.toFixed(4)}`,
|
||||
// v=3 cross-modal additions (append-only).
|
||||
`cmbt=${knobs.cross_modal_both_text_weight.toFixed(2)}`,
|
||||
`cmbi=${knobs.cross_modal_both_image_weight.toFixed(2)}`,
|
||||
`iqt=${knobs.image_query_text_refinement_weight.toFixed(2)}`,
|
||||
`iqi=${knobs.image_query_image_refinement_weight.toFixed(2)}`,
|
||||
`um=${knobs.unified_multimodal ? 1 : 0}`,
|
||||
`umo=${knobs.unified_multimodal_only ? 1 : 0}`,
|
||||
`lli=${knobs.cross_modal_llm_intent ? 1 : 0}`,
|
||||
// v=3 column + provider additions (D8 / CDX-2): cross-column +
|
||||
// cross-provider cache isolation. A query against `embedding_voyage`
|
||||
// must never be served from a row that ran against `embedding`.
|
||||
`col=${ctx?.embeddingColumn ?? 'embedding'}`,
|
||||
`prov=${ctx?.embeddingModel ?? 'default'}`,
|
||||
];
|
||||
@@ -519,6 +635,40 @@ export function loadOverridesFromConfig(
|
||||
if (Number.isFinite(n) && n >= 0 && n <= 1) out.floor_ratio = n;
|
||||
}
|
||||
|
||||
// v0.36 cross-modal overrides (D3 registry)
|
||||
const cmbt = get('search.cross_modal.both_mode_text_weight');
|
||||
if (cmbt !== undefined) {
|
||||
const n = parseFloat(cmbt);
|
||||
if (Number.isFinite(n) && n >= 0) out.cross_modal_both_text_weight = n;
|
||||
}
|
||||
const cmbi = get('search.cross_modal.both_mode_image_weight');
|
||||
if (cmbi !== undefined) {
|
||||
const n = parseFloat(cmbi);
|
||||
if (Number.isFinite(n) && n >= 0) out.cross_modal_both_image_weight = n;
|
||||
}
|
||||
const iqt = get('search.image_query.text_refinement_weight');
|
||||
if (iqt !== undefined) {
|
||||
const n = parseFloat(iqt);
|
||||
if (Number.isFinite(n) && n >= 0) out.image_query_text_refinement_weight = n;
|
||||
}
|
||||
const iqi = get('search.image_query.image_refinement_weight');
|
||||
if (iqi !== undefined) {
|
||||
const n = parseFloat(iqi);
|
||||
if (Number.isFinite(n) && n >= 0) out.image_query_image_refinement_weight = n;
|
||||
}
|
||||
const um = get('search.unified_multimodal');
|
||||
if (um !== undefined) {
|
||||
out.unified_multimodal = um === '1' || um.toLowerCase() === 'true';
|
||||
}
|
||||
const umo = get('search.unified_multimodal_only');
|
||||
if (umo !== undefined) {
|
||||
out.unified_multimodal_only = umo === '1' || umo.toLowerCase() === 'true';
|
||||
}
|
||||
const lli = get('search.cross_modal.llm_intent');
|
||||
if (lli !== undefined) {
|
||||
out.cross_modal_llm_intent = lli === '1' || lli.toLowerCase() === 'true';
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
@@ -539,6 +689,14 @@ export const SEARCH_MODE_CONFIG_KEYS: ReadonlyArray<string> = Object.freeze([
|
||||
'search.reranker.timeout_ms',
|
||||
// v0.35.6.0 — floor-ratio gate
|
||||
'search.floor_ratio',
|
||||
// v0.36 cross-modal keys (D3)
|
||||
'search.cross_modal.both_mode_text_weight',
|
||||
'search.cross_modal.both_mode_image_weight',
|
||||
'search.image_query.text_refinement_weight',
|
||||
'search.image_query.image_refinement_weight',
|
||||
'search.unified_multimodal',
|
||||
'search.unified_multimodal_only',
|
||||
'search.cross_modal.llm_intent',
|
||||
]);
|
||||
|
||||
/**
|
||||
|
||||
@@ -29,6 +29,20 @@ export type QueryIntent = 'entity' | 'temporal' | 'event' | 'general';
|
||||
export type SalienceMode = 'off' | 'on' | 'strong';
|
||||
export type RecencyMode = 'off' | 'on' | 'strong';
|
||||
|
||||
/**
|
||||
* v0.36 cross-modal wave: modality axis (D6).
|
||||
*
|
||||
* - 'text' (default): existing text-embedding path, no behavior change
|
||||
* - 'image': route through the multimodal model + embedding_image column
|
||||
* (visually-similar matching + image OCR text)
|
||||
* - 'both': run text + image searches in parallel and merge via
|
||||
* weighted RRF (recall-leaning when the query is ambiguous)
|
||||
*
|
||||
* Parallel axis to intent/detail/salience/recency. Returned by
|
||||
* classifyQuery from one regex pass over the query.
|
||||
*/
|
||||
export type ModalityMode = 'text' | 'image' | 'both';
|
||||
|
||||
export interface QuerySuggestions {
|
||||
intent: QueryIntent;
|
||||
/** v0.29.0 detail mapping. entity→low, temporal/event→high, general→undefined. */
|
||||
@@ -37,6 +51,8 @@ export interface QuerySuggestions {
|
||||
suggestedSalience: SalienceMode;
|
||||
/** v0.29.1 — per-prefix age-decay boost. */
|
||||
suggestedRecency: RecencyMode;
|
||||
/** v0.36 — cross-modal routing axis. Defaults to 'text' when nothing matches. */
|
||||
suggestedModality: ModalityMode;
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────
|
||||
@@ -164,6 +180,46 @@ const SALIENCE_ON_PATTERNS = [
|
||||
/\bwhat'?s\s+important\b/i,
|
||||
];
|
||||
|
||||
// v0.36 cross-modal wave — modality-axis patterns (D6).
|
||||
//
|
||||
// CROSS_MODAL_PATTERNS fires the 'image' modality when the query explicitly
|
||||
// names visual artifacts ("show me photos", "find images of", "screenshot of",
|
||||
// "what does X look like", "diagram of"). Module-scope const so regexes
|
||||
// compile once at module load (D15).
|
||||
//
|
||||
// Conservative on purpose — false positives cost "one cheaper image search
|
||||
// where text might have worked." False negatives cost nothing (the legacy
|
||||
// text path still runs). The LLM-intent escalation in Commit 4 catches
|
||||
// genuinely ambiguous phrasings.
|
||||
const CROSS_MODAL_PATTERNS: RegExp[] = [
|
||||
/\b(show|find|get|pull)\s+(me\s+)?(the\s+)?(photos?|images?|pictures?|pics?|screenshots?)\b/i,
|
||||
/\b(photos?|images?|pictures?|pics?|screenshots?)\s+(of|from|at|with|showing|featuring)\b/i,
|
||||
/\bwhat\s+does\s+[\w\s']{1,40}?\s+look\s+like\b/i,
|
||||
/\b(whiteboard|diagram|slide|screenshot|infographic|chart)s?\s+(of|from|about|showing)\b/i,
|
||||
/\bdiagram\s+(of|for|showing)\b/i,
|
||||
/\bvisual(s|ly)?\s+(of|from|about|showing|representation)\b/i,
|
||||
];
|
||||
|
||||
// v0.36 cross-modal wave (Commit 4 prep): visual nouns that combined with
|
||||
// ambiguous-pronoun phrasings ("any pics from last week's offsite?") trigger
|
||||
// the optional LLM intent escalation. Subset of cross-modal patterns plus
|
||||
// looser noun-form matches.
|
||||
const AMBIGUOUS_MODALITY_NOUNS: RegExp[] = [
|
||||
/\b(photo|image|picture|pic|screenshot|diagram|whiteboard|slide|chart)s?\b/i,
|
||||
/\blook(s|ed)?\s+like\b/i,
|
||||
/\bvisual(s|ly)?\b/i,
|
||||
];
|
||||
|
||||
// Pronoun + filler markers that signal "the user is referencing something
|
||||
// they can't quite name" — combined with AMBIGUOUS_MODALITY_NOUNS, triggers
|
||||
// the LLM tie-break in Commit 4.
|
||||
const AMBIGUOUS_REFERENCE_MARKERS: RegExp[] = [
|
||||
// Match all the visual nouns (pic/pics, picture/pictures, photo/photos, image/images,
|
||||
// screenshot/screenshots, diagram/diagrams, whiteboard/whiteboards, slide/slides, chart/charts).
|
||||
/\b(any|some|that|those|these|the)\s+(pic|pics|picture|pictures|photo|photos|image|images|screenshot|screenshots|diagram|diagrams|whiteboard|whiteboards|slide|slides|chart|charts)\b/i,
|
||||
/\bfrom\s+(last|this|the)\s+(week|month|year|offsite|meeting|hackathon|deck)\b/i,
|
||||
];
|
||||
|
||||
// ─────────────────────────────────────────────────────────
|
||||
// Classifier
|
||||
// ─────────────────────────────────────────────────────────
|
||||
@@ -221,7 +277,40 @@ export function classifyQuery(query: string): QuerySuggestions {
|
||||
suggestedSalience = 'off';
|
||||
}
|
||||
|
||||
return { intent, suggestedDetail, suggestedSalience, suggestedRecency };
|
||||
// v0.36 cross-modal — modality axis. Independent of intent/detail/salience/recency.
|
||||
// Conservative default 'text'; only flips to 'image' on explicit cross-modal regex match.
|
||||
// 'both' is reserved for explicit per-call opts (LLM-intent escalation in Commit 4
|
||||
// can also produce 'both' via tie-break).
|
||||
const suggestedModality: ModalityMode = matches(CROSS_MODAL_PATTERNS, query) ? 'image' : 'text';
|
||||
|
||||
return { intent, suggestedDetail, suggestedSalience, suggestedRecency, suggestedModality };
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.36 — heuristic gate for the optional LLM intent escalation (Commit 4).
|
||||
*
|
||||
* Fires when the query contains a visual noun ("any pics", "the diagram",
|
||||
* "what does it look like") combined with an ambiguous reference marker
|
||||
* ("from last week's offsite"). These are the phrasings the conservative
|
||||
* regex misses but a Haiku tie-break catches.
|
||||
*
|
||||
* Returns false for unambiguous text queries (no LLM call burned). Returns
|
||||
* false for queries the regex ALREADY caught (no need to tie-break a
|
||||
* confident classification). Returns true only for the narrow band where
|
||||
* the LLM call earns its $0.0001 cost.
|
||||
*
|
||||
* Pure function. No LLM call. No DB access. Used by hybridSearch's
|
||||
* escalation branch only when `search.cross_modal.llm_intent: true`.
|
||||
*/
|
||||
export function isAmbiguousModalityQuery(query: string): boolean {
|
||||
// Already-confident classification → no LLM needed.
|
||||
if (matches(CROSS_MODAL_PATTERNS, query)) return false;
|
||||
|
||||
const hasVisualNoun = matches(AMBIGUOUS_MODALITY_NOUNS, query);
|
||||
if (!hasVisualNoun) return false;
|
||||
|
||||
const hasReferenceMarker = matches(AMBIGUOUS_REFERENCE_MARKERS, query);
|
||||
return hasReferenceMarker;
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* v0.36 Phase 2 (D23-#6) — per-OAuth-client paid-API spend tracking.
|
||||
*
|
||||
* Backs the daily-budget gate for `search_by_image`. Each successful Voyage
|
||||
* multimodal call records an entry; before any new call, `checkBudget`
|
||||
* sums today's spend and rejects when it exceeds the configured cap.
|
||||
*
|
||||
* Config: `search.image_query.daily_budget_usd_per_client` (default $5).
|
||||
*
|
||||
* Scope: ONLY fires when `ctx.remote === true`. Local CLI callers have
|
||||
* direct billing visibility through their own credentials and don't need
|
||||
* the gate. The gate exists to prevent a misbehaving OAuth client from
|
||||
* burning the brain operator's Voyage account.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { sqlQueryForEngine } from './sql-query.ts';
|
||||
|
||||
/** Per-call Voyage multimodal-3 spend estimate (per image), in cents. */
|
||||
export const VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS = 0.12;
|
||||
|
||||
export class BudgetExceededError extends Error {
|
||||
readonly code = 'BUDGET_EXCEEDED' as const;
|
||||
readonly spentCents: number;
|
||||
readonly capCents: number;
|
||||
constructor(message: string, spentCents: number, capCents: number) {
|
||||
super(message);
|
||||
this.name = 'BudgetExceededError';
|
||||
this.spentCents = spentCents;
|
||||
this.capCents = capCents;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sum today's recorded spend for a client. Returns 0 if the row count
|
||||
* is zero (new client) OR if the table doesn't exist (pre-v0.36 brain).
|
||||
*/
|
||||
export async function getTodaySpendCents(
|
||||
engine: BrainEngine,
|
||||
clientId: string,
|
||||
): Promise<number> {
|
||||
try {
|
||||
const sql = sqlQueryForEngine(engine);
|
||||
const rows = await sql`
|
||||
SELECT COALESCE(SUM(spend_cents), 0)::text AS total
|
||||
FROM mcp_spend_log
|
||||
WHERE client_id = ${clientId}
|
||||
AND created_at >= ${todayStartIso()}
|
||||
`;
|
||||
const total = parseFloat(String(rows[0]?.total ?? '0'));
|
||||
return Number.isFinite(total) ? total : 0;
|
||||
} catch {
|
||||
// Table doesn't exist (pre-v0.36 brain) or DB hiccup — fail-open to 0.
|
||||
// The check sees "no spend recorded" and lets the call through. A real
|
||||
// production brain will have the table once migrations apply.
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-flight budget gate.
|
||||
*
|
||||
* Throws `BudgetExceededError` when the client has already spent at or above
|
||||
* the configured daily cap. Returns silently when there's room.
|
||||
*
|
||||
* @param dailyBudgetCents resolved from `search.image_query.daily_budget_usd_per_client`
|
||||
* × 100 (converted to cents). Operator config; default 500 cents = $5.
|
||||
*/
|
||||
export async function checkBudget(
|
||||
engine: BrainEngine,
|
||||
clientId: string,
|
||||
dailyBudgetCents: number,
|
||||
): Promise<void> {
|
||||
if (!clientId) return; // local CLI callers (no client_id) bypass the gate
|
||||
if (dailyBudgetCents <= 0) return; // 0 = "no cap" sentinel
|
||||
const spent = await getTodaySpendCents(engine, clientId);
|
||||
if (spent >= dailyBudgetCents) {
|
||||
throw new BudgetExceededError(
|
||||
`Daily Voyage spend cap reached: $${(spent / 100).toFixed(2)} >= $${(dailyBudgetCents / 100).toFixed(2)}. ` +
|
||||
`Reset at midnight UTC.`,
|
||||
spent,
|
||||
dailyBudgetCents,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Record a successful paid call.
|
||||
*
|
||||
* Best-effort: writes failures (e.g., table not yet migrated) are swallowed
|
||||
* with a stderr warning. The search itself succeeded — we don't want to
|
||||
* fail the user's call because spend telemetry hiccupped.
|
||||
*/
|
||||
export async function recordSpend(
|
||||
engine: BrainEngine,
|
||||
entry: {
|
||||
clientId?: string | null;
|
||||
tokenName?: string | null;
|
||||
operation: string;
|
||||
spendCents: number;
|
||||
provider?: string;
|
||||
model?: string;
|
||||
},
|
||||
): Promise<void> {
|
||||
try {
|
||||
const sql = sqlQueryForEngine(engine);
|
||||
await sql`
|
||||
INSERT INTO mcp_spend_log (client_id, token_name, operation, spend_cents, provider, model)
|
||||
VALUES (${entry.clientId ?? null}, ${entry.tokenName ?? null}, ${entry.operation}, ${entry.spendCents}, ${entry.provider ?? null}, ${entry.model ?? null})
|
||||
`;
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
console.error(`[spend-log] failed to record spend: ${msg}`);
|
||||
}
|
||||
}
|
||||
|
||||
function todayStartIso(): string {
|
||||
const now = new Date();
|
||||
// UTC day-start so the cap rolls over deterministically regardless of
|
||||
// server timezone. Operators reading dashboards see UTC days.
|
||||
const utcMidnight = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
|
||||
return utcMidnight.toISOString();
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
/**
|
||||
* SSRF validation with DNS resolution — closes the rebinding gap that
|
||||
* `isInternalUrl` in `src/core/url-safety.ts` leaves open.
|
||||
*
|
||||
* url-safety.ts covers static SSRF defense (IPv4-mapped IPv6, hex/octal IP
|
||||
* forms, IPv6 ULA + link-local, metadata hostnames, CGNAT, scheme allowlist).
|
||||
* Codex's outside-voice review of the cross-modal wave (D19) flagged the
|
||||
* remaining gap: an attacker-controlled hostname can resolve to a public IP
|
||||
* at validation time and a private IP at fetch time (DNS rebinding). The
|
||||
* defense is: resolve once at validation, inspect every A/AAAA record, and
|
||||
* fetch by the resolved IP — not the hostname.
|
||||
*
|
||||
* This module is consumed by `src/core/search/image-loader.ts` (Phase 2 of
|
||||
* the cross-modal wave) and is reusable for any future URL-fetching feature.
|
||||
*
|
||||
* Two-layer defense per call:
|
||||
* 1. Static check via `isInternalUrl` — fails fast on obvious internal hosts
|
||||
* 2. DNS resolve via `dns.lookup({all: true, family: 0})` — fails on any
|
||||
* resolved A/AAAA record that points internal
|
||||
*
|
||||
* The caller fetches using the returned `resolvedIp`, not the original
|
||||
* hostname, so a second DNS lookup at fetch time can't rebind to internal.
|
||||
*/
|
||||
|
||||
import { lookup as nodeDnsLookup } from 'node:dns/promises';
|
||||
import { isInternalUrl, isPrivateIpv4, hostnameToOctets } from './url-safety.ts';
|
||||
|
||||
// Module-level seam so tests can swap DNS resolution without `mock.module`
|
||||
// (which is banned in non-serial unit tests per scripts/check-test-isolation.sh R2).
|
||||
type DnsLookupFn = typeof nodeDnsLookup;
|
||||
let _dnsLookup: DnsLookupFn = nodeDnsLookup;
|
||||
|
||||
/** @internal Test-only — swap the DNS resolver. Restore with `__setDnsLookupForTests(undefined)`. */
|
||||
export function __setDnsLookupForTests(fn: DnsLookupFn | undefined): void {
|
||||
_dnsLookup = fn ?? nodeDnsLookup;
|
||||
}
|
||||
|
||||
export interface ResolvedTarget {
|
||||
/** The URL the caller should fetch — host is replaced with the resolved IP. */
|
||||
resolvedUrl: string;
|
||||
/** The IP address resolved from the original hostname. */
|
||||
resolvedIp: string;
|
||||
/** The original hostname (for Host: header). Empty when input was already an IP literal. */
|
||||
originalHost: string;
|
||||
/** Whether the resolved IP is IPv6 — affects URL bracket encoding. */
|
||||
ipv6: boolean;
|
||||
}
|
||||
|
||||
export class SSRFError extends Error {
|
||||
readonly code: SSRFErrorCode;
|
||||
constructor(code: SSRFErrorCode, message: string) {
|
||||
super(message);
|
||||
this.name = 'SSRFError';
|
||||
this.code = code;
|
||||
}
|
||||
}
|
||||
|
||||
export type SSRFErrorCode =
|
||||
| 'INTERNAL_HOST'
|
||||
| 'INVALID_URL'
|
||||
| 'INVALID_SCHEME'
|
||||
| 'CREDENTIALS_IN_URL'
|
||||
| 'DNS_RESOLUTION_FAILED'
|
||||
| 'DNS_RESOLVED_INTERNAL'
|
||||
| 'SSRF_REDIRECT_DENIED'
|
||||
| 'SSRF_HOP_LIMIT';
|
||||
|
||||
/**
|
||||
* Validate a URL against SSRF policy and resolve its hostname to an IP.
|
||||
*
|
||||
* Returns a `ResolvedTarget` the caller should use for the actual fetch.
|
||||
* Throws `SSRFError` on any policy violation.
|
||||
*
|
||||
* Defends against:
|
||||
* - Static internal targets (RFC1918, loopback, link-local, ULA, metadata hostnames, CGNAT)
|
||||
* - Non-http(s) schemes
|
||||
* - Credentials embedded in URL (`http://user:pass@host/`)
|
||||
* - DNS rebinding (resolves all records, blocks if any are internal)
|
||||
* - Non-resolving hosts (caller can't fetch them anyway)
|
||||
*/
|
||||
export async function validateAndResolveUrl(urlStr: string): Promise<ResolvedTarget> {
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(urlStr);
|
||||
} catch {
|
||||
throw new SSRFError('INVALID_URL', `Malformed URL: ${truncate(urlStr)}`);
|
||||
}
|
||||
|
||||
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
|
||||
throw new SSRFError('INVALID_SCHEME', `Unsupported scheme ${url.protocol}; only http(s) allowed`);
|
||||
}
|
||||
|
||||
if (url.username || url.password) {
|
||||
throw new SSRFError('CREDENTIALS_IN_URL', 'Credentials embedded in URL are not permitted');
|
||||
}
|
||||
|
||||
// Layer 1: static check covers IPv4 hex/octal/single-int, IPv6 ULA + link-local,
|
||||
// metadata hostnames, CGNAT, IPv4-mapped IPv6.
|
||||
if (isInternalUrl(urlStr)) {
|
||||
throw new SSRFError('INTERNAL_HOST', `URL targets internal/private network: ${truncate(urlStr)}`);
|
||||
}
|
||||
|
||||
let host = url.hostname;
|
||||
if (host.startsWith('[') && host.endsWith(']')) host = host.slice(1, -1);
|
||||
|
||||
// If the host is already an IP literal, isInternalUrl already validated it.
|
||||
// Skip DNS lookup and return the literal as-is.
|
||||
if (isIpLiteral(host)) {
|
||||
return {
|
||||
resolvedUrl: urlStr,
|
||||
resolvedIp: host,
|
||||
originalHost: '',
|
||||
ipv6: host.includes(':'),
|
||||
};
|
||||
}
|
||||
|
||||
// Layer 2: DNS resolution. {all: true, family: 0} returns every A AND AAAA
|
||||
// record. If ANY record points internal, reject.
|
||||
let addrs: Array<{ address: string; family: number }>;
|
||||
try {
|
||||
addrs = await _dnsLookup(host, { all: true, family: 0 });
|
||||
} catch (err) {
|
||||
throw new SSRFError(
|
||||
'DNS_RESOLUTION_FAILED',
|
||||
`Failed to resolve ${host}: ${err instanceof Error ? err.message : String(err)}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (addrs.length === 0) {
|
||||
throw new SSRFError('DNS_RESOLUTION_FAILED', `No DNS records for ${host}`);
|
||||
}
|
||||
|
||||
for (const a of addrs) {
|
||||
if (isAddressInternal(a.address, a.family)) {
|
||||
throw new SSRFError(
|
||||
'DNS_RESOLVED_INTERNAL',
|
||||
`${host} resolves to internal address ${a.address} (DNS rebinding attempt?)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Pick the first resolved address (system-ordered: typically the preferred
|
||||
// family). Caller fetches by this IP so a second DNS lookup can't rebind.
|
||||
const chosen = addrs[0];
|
||||
const isV6 = chosen.family === 6;
|
||||
const hostInUrl = isV6 ? `[${chosen.address}]` : chosen.address;
|
||||
|
||||
// Rebuild URL with the resolved host. Preserve the original `host` for the
|
||||
// Host: header (caller can set it explicitly when fetching).
|
||||
const rebuilt = new URL(urlStr);
|
||||
rebuilt.hostname = hostInUrl;
|
||||
|
||||
return {
|
||||
resolvedUrl: rebuilt.toString(),
|
||||
resolvedIp: chosen.address,
|
||||
originalHost: host,
|
||||
ipv6: isV6,
|
||||
};
|
||||
}
|
||||
|
||||
function isIpLiteral(host: string): boolean {
|
||||
if (host.includes(':')) return true; // IPv6 literal (already bracket-stripped)
|
||||
return hostnameToOctets(host) !== null;
|
||||
}
|
||||
|
||||
function isAddressInternal(addr: string, family: number): boolean {
|
||||
if (family === 4) {
|
||||
const octets = hostnameToOctets(addr);
|
||||
return octets ? isPrivateIpv4(octets) : true; // fail-closed on parse failure
|
||||
}
|
||||
if (family === 6) {
|
||||
const lower = addr.toLowerCase();
|
||||
if (lower === '::1' || lower === '::') return true;
|
||||
if (/^f[cd][0-9a-f]{2}:/.test(lower)) return true; // ULA fc00::/7
|
||||
if (/^fe[89ab][0-9a-f]:/.test(lower)) return true; // link-local fe80::/10
|
||||
if (lower.startsWith('::ffff:')) {
|
||||
const tail = lower.slice(7);
|
||||
const dotted = hostnameToOctets(tail);
|
||||
if (dotted && isPrivateIpv4(dotted)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
return true; // unknown family — fail-closed
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch a URL with full SSRF protection including per-redirect-hop validation.
|
||||
*
|
||||
* On every Location response header, the new URL is re-validated via
|
||||
* `validateAndResolveUrl` — fresh DNS resolution per hop defeats rebinding
|
||||
* across the redirect chain. Max 3 hops by default.
|
||||
*
|
||||
* Returns the final Response. Caller is responsible for body size limits
|
||||
* (use `init.signal` to abort, or check `Content-Length` before consuming).
|
||||
*/
|
||||
export async function fetchWithSSRFGuard(
|
||||
urlStr: string,
|
||||
init: RequestInit & {
|
||||
maxRedirects?: number;
|
||||
timeoutMs?: number;
|
||||
} = {},
|
||||
): Promise<Response> {
|
||||
const maxRedirects = init.maxRedirects ?? 3;
|
||||
const timeoutMs = init.timeoutMs ?? 5000;
|
||||
|
||||
const controller = new AbortController();
|
||||
const externalSignal = init.signal;
|
||||
const onAbort = () => controller.abort();
|
||||
if (externalSignal) {
|
||||
if (externalSignal.aborted) controller.abort();
|
||||
else externalSignal.addEventListener('abort', onAbort, { once: true });
|
||||
}
|
||||
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
||||
|
||||
try {
|
||||
let currentUrl = urlStr;
|
||||
let hops = 0;
|
||||
while (true) {
|
||||
const target = await validateAndResolveUrl(currentUrl);
|
||||
const fetchInit: RequestInit = {
|
||||
...init,
|
||||
redirect: 'manual',
|
||||
signal: controller.signal,
|
||||
};
|
||||
// Set Host header to the original hostname so SNI/TLS works correctly
|
||||
// (we're fetching by resolved IP but the server expects the real host).
|
||||
const headers = new Headers(init.headers || {});
|
||||
if (target.originalHost) {
|
||||
headers.set('Host', target.originalHost);
|
||||
}
|
||||
fetchInit.headers = headers;
|
||||
const res = await fetch(target.resolvedUrl, fetchInit);
|
||||
// Redirect status codes
|
||||
if ([301, 302, 303, 307, 308].includes(res.status)) {
|
||||
if (hops >= maxRedirects) {
|
||||
throw new SSRFError('SSRF_HOP_LIMIT', `Exceeded ${maxRedirects} redirect hops`);
|
||||
}
|
||||
const location = res.headers.get('location');
|
||||
if (!location) {
|
||||
return res; // redirect with no Location — return as-is, caller decides
|
||||
}
|
||||
// Resolve relative location against current URL
|
||||
const next = new URL(location, currentUrl).toString();
|
||||
currentUrl = next;
|
||||
hops++;
|
||||
continue;
|
||||
}
|
||||
return res;
|
||||
}
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
if (externalSignal) externalSignal.removeEventListener('abort', onAbort);
|
||||
}
|
||||
}
|
||||
|
||||
function truncate(s: string, n = 200): string {
|
||||
return s.length > n ? s.slice(0, n) + '...' : s;
|
||||
}
|
||||
+40
-28
@@ -404,6 +404,15 @@ export interface SearchResult {
|
||||
chunk_index: number;
|
||||
score: number;
|
||||
stale: boolean;
|
||||
/**
|
||||
* v0.36 (cross-modal wave): the chunk's modality discriminator from
|
||||
* content_chunks.modality. 'text' for the existing text-embedding rows,
|
||||
* 'image' for rows populated by importImageFile. Surfaced so callers /
|
||||
* renderers can distinguish text matches from image matches in `'both'`
|
||||
* mode results. Optional for back-compat with engines that don't project
|
||||
* the column (defaults to 'text' in renderers when absent).
|
||||
*/
|
||||
modality?: 'text' | 'image';
|
||||
/**
|
||||
* v0.18.0: the sources.id the page belongs to. Dedup composite-keys
|
||||
* on (source_id, slug) — see src/core/search/dedup.ts. Defaults to
|
||||
@@ -541,8 +550,10 @@ export interface SearchOpts {
|
||||
* 1. String name (legacy + user-facing). Engine and hybridSearch convert
|
||||
* to ResolvedColumn at the boundary via `resolveEmbeddingColumn()`.
|
||||
* Built-in names: 'embedding' (default, text), 'embedding_image'
|
||||
* (multimodal). Custom user-declared names also accepted when
|
||||
* registered in `embedding_columns` config.
|
||||
* (multimodal). v0.36 cross-modal wave adds 'embedding_multimodal'
|
||||
* (unified column populated by `gbrain reindex --multimodal`). Custom
|
||||
* user-declared names also accepted when registered in
|
||||
* `embedding_columns` config.
|
||||
*
|
||||
* 2. ResolvedColumn descriptor (internal). The engine ONLY accepts
|
||||
* this shape — hybridSearch resolves once at entry and passes the
|
||||
@@ -556,7 +567,7 @@ export interface SearchOpts {
|
||||
* searchKeyword is unaffected — modality filtering on the keyword path
|
||||
* is independent.
|
||||
*/
|
||||
embeddingColumn?: 'embedding' | 'embedding_image' | string | ResolvedColumn;
|
||||
embeddingColumn?: 'embedding' | 'embedding_image' | 'embedding_multimodal' | string | ResolvedColumn;
|
||||
/**
|
||||
* @deprecated v0.29.1: use `since` instead. Removed in v0.30.
|
||||
* v0.27.0: filter results to pages updated/created after this date. ISO-8601 string.
|
||||
@@ -636,33 +647,34 @@ export interface SearchOpts {
|
||||
rerankerFn?: (input: { query: string; documents: string[]; topN?: number; model?: string; signal?: AbortSignal; timeoutMs?: number }) => Promise<{ index: number; relevanceScore: number }[]>;
|
||||
};
|
||||
/**
|
||||
* v0.35.6.0 — floor-ratio gate for metadata-axis boost stages (backlink,
|
||||
* salience, recency). Number in [0, 1] or undefined (default = no gate).
|
||||
*
|
||||
* When set, each gated stage skips results whose pre-boost score is below
|
||||
* `floorRatio * topScore`, where `topScore` is computed ONCE at
|
||||
* `runPostFusionStages` entry from the post-cosine-rescore snapshot. The
|
||||
* same threshold gates all three stages — order-independent semantic.
|
||||
*
|
||||
* Resolution chain (mirrors other search-lite knobs):
|
||||
* per-call `SearchOpts.floorRatio` → config `search.floor_ratio`
|
||||
* → MODE_BUNDLES[mode].floor_ratio (undefined for all 3 modes today)
|
||||
* → undefined fallback.
|
||||
*
|
||||
* SCOPE: gates ONLY the three metadata stages. Exact-match boost
|
||||
* (`applyExactMatchBoost` in intent-weights.ts) runs independently as a
|
||||
* lexical-relevance signal and is NOT gated by design.
|
||||
*
|
||||
* Sensible operator override values for dense-embedder corpora: 0.85-0.95.
|
||||
* Default stays undefined pending per-corpus ablation evidence (see
|
||||
* `TODOS.md` floor-ratio ablation entry).
|
||||
*
|
||||
* Out-of-range values (negative, > 1, NaN, Infinity) silently disable
|
||||
* the gate at the runtime layer; the config-parse layer also rejects
|
||||
* out-of-range values. Defense in depth — a malformed value never
|
||||
* gates anything.
|
||||
* v0.35.6.0 — floor-ratio gate for metadata-axis boost stages.
|
||||
* Number in [0, 1] or undefined (default = no gate). When set, each gated
|
||||
* stage skips results whose pre-boost score is below `floorRatio * topScore`.
|
||||
* Same threshold gates all three metadata stages; exact-match boost runs
|
||||
* independently. Out-of-range values silently disable the gate.
|
||||
* Sensible operator overrides for dense-embedder corpora: 0.85-0.95.
|
||||
*/
|
||||
floorRatio?: number;
|
||||
/**
|
||||
* v0.36 cross-modal wave: route this search through the multimodal
|
||||
* embedding space (Voyage multimodal-3 by default).
|
||||
*
|
||||
* - 'text' (default for queries that don't match image-intent regex):
|
||||
* existing text-embedding path. No behavior change vs pre-v0.36.
|
||||
* - 'image': force routing through the multimodal model + embedding_image
|
||||
* column. Skip LLM expansion (image embeddings handle synonyms in-space)
|
||||
* and skip keyword search (no FTS index on image content).
|
||||
* - 'both': run text and image vector searches in parallel; merge via
|
||||
* modality-weighted RRF.
|
||||
* - 'auto' (literal): same effect as undefined — let intent classifier
|
||||
* decide. Accepted on the wire so MCP callers can be explicit.
|
||||
*
|
||||
* Cross-modal override matrix (D9): when effective modality is 'image',
|
||||
* cross-modal path overrides expansion (false) and reranker (false)
|
||||
* regardless of mode bundle. zerank-2 can't rerank image embeddings;
|
||||
* sending them produces garbage scores.
|
||||
*/
|
||||
crossModal?: 'text' | 'image' | 'both' | 'auto';
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+4
-1
@@ -155,7 +155,10 @@ CREATE TABLE IF NOT EXISTS content_chunks (
|
||||
-- mixed-provider brains (e.g. OpenAI 1536 text + Voyage 1024 images) keep
|
||||
-- both columns populated with distinct dim spaces.
|
||||
modality TEXT NOT NULL DEFAULT 'text',
|
||||
embedding_image vector(1024)
|
||||
embedding_image vector(1024),
|
||||
-- v0.36 Phase 3 cross-modal: unified column populated by reindex.
|
||||
-- Migration v75 also adds it for upgrade paths.
|
||||
embedding_multimodal vector(1024)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
|
||||
|
||||
Reference in New Issue
Block a user