admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-31 04:07:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

v0.36.6.0 feat: cross-modal search wave (text↔image + unified column + LLM intent) (#1165)

Browse Source 
* feat(cross-modal/0): batched multimodal + query helpers + SSRF helper

Commit 0 of the cross-modal search wave. Foundation for Phase 1-3:

- embedMultimodal accepts MultimodalInput text variant + EmbedMultimodalOpts
  with inputType: 'document' | 'query' (D22-2). Default unchanged so
  importImageFile keeps document-side embedding.
- embedQueryMultimodal(text) + embedQueryMultimodalImage(input) wrappers
  for hybridSearch + searchByImage query paths.
- embedMultimodalSafe binary-search retry on transient batch failure +
  failed_indices surfacing. Phase 3 reindex uses this so a single bad
  chunk doesn't discard the 31 in-flight embeddings around it.
- Voyage path: text + image inputs in one batch via content arrays.
- openai-compat path: text + image inputs in one request per input.
- src/core/ssrf-validate.ts (D19): DNS-resolve-and-fetch-by-IP defense
  for redirect chains. Closes the DNS-rebinding gap that url-safety.ts'
  static check leaves open. Uses node:dns/promises with {all: true,
  family: 0} to inspect every A and AAAA record before connecting.
  fetchWithSSRFGuard helper validates per-redirect-hop and limits chain
  depth (default 3).
- Re-exports from src/core/embedding.ts public seam.

Tests:
- test/embed-multimodal-batching.test.ts (13 cases): text variant, query
  inputType discipline, mixed text+image batches, embedQueryMultimodal,
  embedQueryMultimodalImage, embedMultimodalSafe happy/empty/all-fail/
  mid-batch-recovery/permanent-misconfig.
- test/ssrf-validate.test.ts (20 cases): static rejections via
  isInternalUrl, scheme + credentials rejection, DNS rebinding defense
  (single-record + multi-record), public happy path, IPv6 literals,
  malformed URLs.

No regression in existing voyage-multimodal.test.ts or
openai-compat-multimodal.test.ts (33 cases all pass).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cross-modal/1): Phase 1 text→image routing + knobsHash + RRF + backfill

Phase 1 of the cross-modal search wave. Wires the existing 1024d Voyage
multimodal embedding space (already populated for image chunks via
importImageFile) into the user-facing query path. Text queries that match
cross-modal intent regex route through Voyage multimodal-3 instead of the
text embedding model, then search content_chunks.embedding_image.

- query-intent.ts: new `suggestedModality: 'text' | 'image' | 'both'`
  axis on `QuerySuggestions`. Module-scope CROSS_MODAL_PATTERNS regex
  array (D15 — compiled once at module load). Conservative on purpose;
  LLM intent escalation (Commit 4) catches genuinely ambiguous phrasings.
- query-intent.ts: new `isAmbiguousModalityQuery(query)` pure heuristic
  for Commit 4's escalation gate. Returns true ONLY when regex misses
  AND a visual noun + reference marker both fire.
- types.ts: `SearchOpts.crossModal: 'text' | 'image' | 'both' | 'auto'`
  + `SearchResult.modality: 'text' | 'image'` for downstream renderers.
- mode.ts: 7 new knobs in ModeBundle (D2): cross_modal_both_text_weight,
  cross_modal_both_image_weight, image_query_text_refinement_weight,
  image_query_image_refinement_weight, unified_multimodal,
  unified_multimodal_only, cross_modal_llm_intent. All three mode
  bundles default to the same values (cross-modal is opt-in).
- mode.ts: D2 cache-key fix — KNOBS_HASH_VERSION bumped 2→3, all 7 new
  knobs participate in knobsHash so a text-mode cache hit can't be
  served to an image-mode caller.
- mode.ts: D3 registry — all 7 keys land in SEARCH_MODE_CONFIG_KEYS so
  `gbrain search modes` / `stats` / `tune` see them.
- hybrid.ts: routing branch at the embed step. Resolves effective
  modality from (per-call opts → suggestions → 'text'). Image route:
  embedQueryMultimodal + searchVector(embedding_image), skip expansion
  + keyword (D9 mode-bundle override). Both route: parallel text + image
  vector searches merged via weighted RRF (D6) with cross_modal_both_*
  weights. Fail-open: multimodal misconfigured → structured warn + text
  fallback. 'auto' literal normalized to undefined (D22-1).
- operations.ts: thread `cross_modal` param through `query` op.
- backfill-registry.ts: new `modality` backfill kind. SQL filter requires
  `chunk_source='image_asset'` (D22-7 defensive guard). Idempotent.
- doctor.ts: `cross_modal_modality_backfill` check surfaces unflagged
  image-asset chunks with paste-ready `gbrain backfill modality` hint.

Tests:
- cross-modal-phase1.test.ts (45 cases): regex classification (positive
  + negative + plural-safe), isAmbiguousModalityQuery, D3 registry, D2
  knobsHash diffs across all 7 new knobs, MODE_BUNDLES defaults,
  resolveSearchMode precedence chain.
- cross-modal-hybrid-integration.test.ts (7 cases): PGLite + stubbed
  gateway. Verifies image-modality calls Voyage and not OpenAI, text
  calls OpenAI and not Voyage, 'auto' literal normalizes, 'both' mode
  hits both endpoints, fail-open routes to text on multimodal misconfig.
- search-mode.test.ts: updated MODE_BUNDLES + KNOBS_HASH_VERSION
  assertions (148 cross-suite tests still pass; no regression).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cross-modal/2): Phase 2 image-as-query + D18 path ban + D23-#6 spend cap

Phase 2 of the cross-modal search wave. Adds the `search_by_image` MCP op,
the SSRF-defended image loader, and the daily per-OAuth-client spend cap
on paid Voyage multimodal calls. D17 honest framing applied: Phase 2 ships
image→similar-images + image-OCR-text retrieval. True image→full-text-
knowledge requires Phase 3's unified column.

- src/core/search/image-loader.ts: loadImageInput accepts local path,
  data: URI, or http(s):// URL. Magic-byte sniff for PNG/JPEG/WebP (no
  other formats). Hard size cap (10MB local default, 2MB remote default).
  http(s) path uses fetchWithSSRFGuard from Commit 0: every redirect hop
  re-resolved via DNS lookup + every record checked against the internal
  IP deny list. Max 3 redirect hops. 5s total fetch timeout. Pre-flight
  Content-Length check + post-fetch size guard for lying servers.
- src/core/search/by-image.ts: searchByImage runs the image branch
  always; D13 hybrid intersect runs a parallel text branch when
  `query` is provided, merged via weighted RRF. Phase 3 will widen
  the column routing to embedding_multimodal once that lands.
- src/core/operations.ts: new search_by_image op (scope: read, NOT
  localOnly). D18 P0 — when ctx.remote === true AND image_path is set,
  rejects with permission_denied at handler entry (validateParams would
  catch it again at dispatch). D5 source-id thread via sourceScopeOpts.
  D12 per-param length cap enforced via remote-vs-local maxBytes config
  read at handler entry. D23-#6 pre-flight checkBudget + post-call
  recordSpend (best-effort; failures don't block response).
- src/core/spend-log.ts: BudgetExceededError + checkBudget + recordSpend
  + getTodaySpendCents. UTC day-aligned aggregation so the cap rolls
  over deterministically. Local CLI callers (no clientId) bypass the
  gate entirely. Pre-v0.36 brains without the mcp_spend_log table fail
  open to spend=0; the migration brings the table in on first start.
- src/core/migrate.ts: new migration v67 mcp_spend_log table + indexes
  for the (client_id, day) and (token_name, day) hot reads. PGLite
  parity via sqlFor.pglite.
- src/core/search/hybrid.ts: RRF_K constant exported so by-image.ts can
  share the same effective-K math as the main hybrid path.

Tests:
- cross-modal-phase2.test.ts (15 cases): magic-byte sniffing (PNG +
  JPEG + WebP positive, GIF rejection), oversized rejection (default +
  custom cap), data: URI happy path + malformed + decoded-non-image
  + oversized, invalid input shapes (empty + ftp), SSRF defense via
  DNS rebinding stub.
- search-by-image-op.test.ts (7 cases): D18 remote image_path
  rejection + local CLI accepts; input validation (missing all three /
  multiple together); D23-#6 budget block-at-cap + allow-under-cap +
  local-CLI-bypass; migration v67 mcp_spend_log table applied cleanly.

All 166 tests across the cross-modal suite pass; no regression in
existing voyage-multimodal / openai-compat-multimodal / search-mode suites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cross-modal/3): Phase 3 unified column + reindex + D8 fail-open + D23-#2

Phase 3 of the cross-modal search wave. Adds the unified multimodal column
on content_chunks + the `gbrain reindex --multimodal` sweep + the
`search.unified_multimodal` routing flag with D8 source-aware coverage
guard + fail-open behavior. D17 honest framing: this is the phase that
unlocks true image→full-text-knowledge — Phase 2's searchByImage
transparently upgrades to the richer retrieval once the unified column
has coverage.

D10 reindex-core extraction filed as a follow-up TODO. The existing
markdown reindex walks pages and re-imports via importFromFile; this
walks content_chunks and re-embeds via the gateway. Patterns rhyme but
cores diverge enough that extraction balloons the diff. Both commands
stand alone with their own checkpoint + cost-prompt logic.

- migrate.ts v68 (embedding_multimodal_column): column-only ALTER on
  content_chunks. HNSW partial index deferred to post-reindex build
  (D20: pgvector docs recommend post-load build for HNSW). Both engines.
- types.ts SearchOpts.embeddingColumn type widened to include
  'embedding_multimodal'.
- postgres-engine.ts + pglite-engine.ts searchVector: route to
  embedding_multimodal column when opts.embeddingColumn set. NO modality
  filter (unified column carries both text + image content).
- hybrid.ts unified routing branch: when search.unified_multimodal=true,
  bypasses dual-column branching and runs embedQueryMultimodal +
  searchVector(embedding_multimodal). D8 fail-open: zero rows + not
  strict-mode → falls through to dual-column text path with structured
  warning. search.unified_multimodal_only=true bypasses the fallback.
- src/commands/reindex-multimodal.ts: `gbrain reindex --multimodal`.
  D7 lock via tryAcquireDbLock('gbrain-reindex-multimodal'); 6h TTL.
  Cost prompt + 10s Ctrl-C grace window in TTY; auto-proceeds non-TTY.
  GBRAIN_NO_REEMBED=1 bypass. Checkpoint at
  ~/.gbrain/reindex-multimodal-checkpoint.json for resume. D23-#2
  auto-flip prompt at coverage=100% completion.
- cli.ts: `gbrain reindex --multimodal` dispatch with --limit, --dry-run,
  --cost-estimate, --no-embed, --yes, --json flags.
- doctor.ts: unified_multimodal_coverage check (D21 source-aware) +
  reports per-source % when search.unified_multimodal is on. Warns at
  <95% lowest source; fails when unified_multimodal_only=true AND
  lowest source <99%. Falls open to OK when column not yet present.

Tests:
- unified-multimodal.test.ts (8 cases): schema migration v68 applies,
  reindex --dry-run + --cost-estimate + GBRAIN_NO_REEMBED bypass +
  zero-pending fast-path, hybridSearch unified routing forces voyage
  endpoint, D8 fail-open routes to text on empty unified, D8 strict
  blocks text fallback.

All 211 tests across the cross-modal + related suite pass; no
regression in voyage-multimodal / openai-compat-multimodal / search-mode
/ intent / search base suites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cross-modal/4): LLM intent escalation for ambiguous modality

Commit 4 of the cross-modal search wave (opt-in default off).

When `search.cross_modal.llm_intent` is true AND the regex classifier
returned 'text' AND `isAmbiguousModalityQuery(query)` fires, hybridSearch
awaits a Haiku tie-break via gateway.chat() before routing. The
ambiguous-modality gate (introduced in Commit 1) ensures the LLM call
only fires on the narrow band where regex misses but a visual noun +
reference marker both fire — roughly <1% of queries with the flag on.

- src/core/search/llm-intent.ts: new module. `classifyModalityWithLLM`
  routes through gateway.chat() with a fixed system prompt ("Output
  exactly one word: text, image, or both"). 1s timeout via AbortController.
  `parseModality` is a pure exported helper that tolerates trailing
  punctuation + casing. Fail-open on every error path (gateway
  unavailable, timeout, parse failure, unrecognized output).
- src/core/search/hybrid.ts: escalation branch slots BEFORE the unified
  routing branch. Gated by: no explicit per-call crossModal opt, regex
  result == 'text', config flag on, ambiguity heuristic fires. Fail-open
  to regex result on any error from the LLM tie-break.

Tests:
- llm-intent-escalation.test.ts (14 cases): parseModality tolerance
  matrix (text / image / both / trailing punct / whitespace /
  unrecognized / empty), classifyModalityWithLLM happy paths for all 3
  outputs, fail-open on throw / unrecognized output / gateway-not-
  configured, explicit-fallback-honored.
- llm-intent-hybrid-integration.test.ts (6 cases): hybridSearch
  escalation gate fires ONLY when flag-on + ambiguous; off when flag-off,
  unambiguous, regex-confident, or explicit per-call opt set; fail-open
  on LLM throw.

All 231 tests across the cross-modal + related suite pass; no
regression in voyage-multimodal / openai-compat-multimodal /
search-mode / intent / search base suites.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cross-modal/3): verify-gate fixes for full test suite

Three small fixes to pass the full unit + E2E sweep after the cross-modal
wave commits land.

- migrate.ts v67: drop date_trunc('day', created_at) from
  mcp_spend_log indexes. TIMESTAMPTZ truncation depends on session
  timezone and isn't IMMUTABLE, so Postgres rejects the function in
  the index expression with SQLSTATE 42P17. BTREE on
  (client_id, created_at) covers the per-day rollup query via range
  scan on created_at — same performance, no IMMUTABLE constraint.
- pglite-schema.ts + src/schema.sql: shorten the embedding_multimodal
  column comment. The longer version contained a comma inside a SQL
  line comment ("...search.unified_multimodal=true, all queries..."),
  which broke parseBaseTableColumns in test/schema-bootstrap-coverage
  (the parser splits on commas at depth-0 before stripping comments,
  so the comma inside the comment shortened the column-definition part
  and an "all" token from "all queries" got picked up as the next
  column name — silently hiding embedding_multimodal from coverage).
- schema-embedded.ts: regenerated via `bun run build:schema`.
- test/e2e/v030_1-integration-pglite.test.ts: listBackfills assertion
  extended to include the new `modality` entry registered in
  src/core/backfill-registry.ts as part of Commit 1.
- test/search/knobs-hash-reranker.test.ts: KNOBS_HASH_VERSION assertion
  updated from 2→3 to match the cross-modal-wave hash-key extension
  (D2 cache contamination fix). Same shape as the prior
  v0.32→v0.35 bump.
- test/unified-multimodal.test.ts: migrated process.env mutation to
  withEnv() helper to satisfy the scripts/check-test-isolation R1
  rule.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(cross-modal): VERSION + CHANGELOG + CLAUDE.md + spec doc + llms regen

Final docs commit for the cross-modal wave (v0.36.0.0).

- VERSION + package.json: bump 0.35.5.1 → 0.36.0.0
- CHANGELOG.md: full Garry-voice release entry with five-commit breakdown,
  the-numbers-that-matter table, what-this-means-for-you, and the
  required to-take-advantage-of-v0.36.0.0 block
- docs/issues/cross-modal-search.md: cherry-picked from PR #1127 head
  (164 lines, the original spec doc preserved as historical reference
  for Phase 2 + 3 background)
- CLAUDE.md: Key Files entries for src/core/ssrf-validate.ts,
  src/core/search/image-loader.ts, src/core/search/by-image.ts,
  src/core/search/llm-intent.ts, src/core/spend-log.ts,
  src/commands/reindex-multimodal.ts, plus extension annotations on
  src/core/search/query-intent.ts, src/core/search/mode.ts,
  src/core/search/hybrid.ts, src/core/backfill-registry.ts,
  src/core/migrate.ts (v67 + v68)
- llms-full.txt + llms.txt: regenerated via `bun run build:llms`

`bun run verify` clean (privacy + proposal-pii + test-names + jsonb +
source-id-projection + progress + test-isolation + wasm + admin-build +
admin-scope-drift + cli-exec + system-of-record + eval-glossary +
typecheck). `bun test test/build-llms.test.ts` clean (7/7).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cross-modal): renumber migrations 67→69 + 68→70 post-master-merge

Master shipped its own v67 (`facts_typed_claim_columns`) during the
cross-modal wave's review cycle. The merge picked up both side's v67
entries, breaking the migration-distinct-versions test. Renumbering
moves cross-modal's table + column ALTER off the collision:

- v67 mcp_spend_log → v69 mcp_spend_log
- v68 embedding_multimodal_column → v70 embedding_multimodal_column

References updated in CHANGELOG, CLAUDE.md, pglite-schema.ts, schema.sql.
schema-embedded.ts regenerated. llms-full.txt regenerated.

7006 unit tests pass, 0 fail. No test code touched — just version
renumbering plus comment refs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version 0.36.0.0 → 0.36.4.0

Bumping to v0.36.4.0 to land in the queue slot the user requested.
No behavior change; pure version bump across VERSION, package.json,
CHANGELOG.md header, llms-full.txt regen.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-19 17:14:53 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent e227965024
commit e60b60244f
43 changed files with 4303 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/core/ai/gateway.ts
+151 -17
View File
@@ -31,6 +31,8 @@ import { z } from 'zod';
import type {
AIGatewayConfig,
EmbedMultimodalOpts,
MultimodalBatchResult,
MultimodalInput,
Recipe,
TouchpointKind,
@@ -1335,7 +1337,10 @@ const MULTIMODAL_MAX_IMAGE_BYTES = 20 * 1024 * 1024;
*
* Empty input → returns []. Preserves the `embed([])` contract.
*/
export async function embedMultimodal(inputs: MultimodalInput[]): Promise<Float32Array[]> {
export async function embedMultimodal(
inputs: MultimodalInput[],
opts: EmbedMultimodalOpts = {},
): Promise<Float32Array[]> {
if (!inputs || inputs.length === 0) return [];
const cfg = requireConfig();
@@ -1372,7 +1377,7 @@ export async function embedMultimodal(inputs: MultimodalInput[]): Promise<Float3
// recipe is `openai-compat` per tier but uses its own /multimodalembeddings
// path, so we still branch on recipe.id for that one.
if (recipe.id !== 'voyage' && recipe.implementation === 'openai-compatible') {
return embedMultimodalOpenAICompat(inputs, recipe, parsed.modelId, cfg);
return embedMultimodalOpenAICompat(inputs, recipe, parsed.modelId, cfg, opts);
}
if (recipe.id !== 'voyage') {
throw new AIConfigError(
@@ -1405,6 +1410,10 @@ export async function embedMultimodal(inputs: MultimodalInput[]): Promise<Float3
// landing in `embedding_image` the column itself is fixed at 1024.
const targetDims = 1024;
// v0.36 (D22-2): thread Voyage's retrieval input_type discipline through.
// Default 'document' preserves pre-v0.36 ingest behavior.
const inputType = opts.inputType ?? 'document';
// Batch in groups of 32 (Voyage's published max). Each batch is one HTTP
// call; results concatenate in input order.
const allEmbeddings: Float32Array[] = [];
@@ -1412,17 +1421,19 @@ export async function embedMultimodal(inputs: MultimodalInput[]): Promise<Float3
const batch = inputs.slice(i, i + MULTIMODAL_BATCH_SIZE);
const body = {
inputs: batch.map(input => ({
// Voyage's documented shape for image inputs:
// { content: [{ type: "image_base64", image_base64: "data:image/png;base64,..." }] }
// Voyage's documented content shape supports both image and text
// entries. v0.36 cross-modal: text variant for query embedding.
content: [
{
type: 'image_base64',
image_base64: `data:${input.mime};base64,${input.data}`,
},
input.kind === 'text'
? { type: 'text', text: input.text }
: {
type: 'image_base64',
image_base64: `data:${input.mime};base64,${input.data}`,
},
],
})),
model: parsed.modelId,
input_type: 'document',
input_type: inputType,
};
let res: Response;
@@ -1510,6 +1521,7 @@ async function embedMultimodalOpenAICompat(
recipe: Recipe,
modelId: string,
cfg: AIGatewayConfig,
opts: EmbedMultimodalOpts = {},
): Promise<Float32Array[]> {
// Auth resolution via the gateway's canonical helper so LiteLLM-style
// optional-auth recipes (Authorization: Bearer LITELLM_API_KEY) and
@@ -1543,19 +1555,27 @@ async function embedMultimodalOpenAICompat(
// multimodal content array varies per provider. Single-input requests
// are the safe lowest common denominator; LiteLLM's proxy backend
// batches internally if it can.
// v0.36 (D22-2): inputType opt threaded for symmetry with the Voyage path.
// Most openai-compatible proxies don't forward this field, but recording
// it in the body keeps LiteLLM-style providers that DO accept it correct.
const inputType = opts.inputType ?? 'document';
const allEmbeddings: Float32Array[] = [];
for (const input of inputs) {
const body = {
const body: Record<string, unknown> = {
model: modelId,
input: [
{
// OpenAI's documented multimodal content shape. The data-URL
// form embeds the image bytes inline so the proxy doesn't need
// network access to fetch the image.
type: 'image_url',
image_url: { url: `data:${input.mime};base64,${input.data}` },
},
input.kind === 'text'
? { type: 'input_text', text: input.text }
: {
// OpenAI's documented multimodal content shape. The data-URL
// form embeds the image bytes inline so the proxy doesn't need
// network access to fetch the image.
type: 'image_url',
image_url: { url: `data:${input.mime};base64,${input.data}` },
},
],
input_type: inputType,
};
let res: Response;
@@ -1630,6 +1650,120 @@ async function embedMultimodalOpenAICompat(
return allEmbeddings;
}
// ---- v0.36 cross-modal wave: query-side multimodal embedding + safe variant ----
/**
* Embed a TEXT query through the configured multimodal model.
*
* Routes through `embedding_multimodal_model` (defaults to Voyage multimodal-3)
* so the resulting vector lives in the multimodal embedding space — the same
* space the brain's `embedding_image` column was populated into. A text
* query embedded here can match image chunks (Phase 1 of the cross-modal
* wave) and, post Phase 3 reindex, text chunks in the unified column.
*
* Threads `inputType: 'query'` (D22-2) so Voyage routes to the retrieval
* half of its asymmetric embedding space.
*
* Sibling of v0.35.0.0's `embedQuery(text)`, which uses the TEXT embedding
* model (typically OpenAI text-embedding-3-large at 1536d or 2560d, NOT
* compatible with the 1024d multimodal column).
*/
export async function embedQueryMultimodal(text: string): Promise<Float32Array> {
const [vec] = await embedMultimodal([{ kind: 'text', text }], { inputType: 'query' });
if (!vec) {
throw new AITransientError('embedQueryMultimodal: gateway returned no vector for non-empty text input');
}
return vec;
}
/**
* Embed an IMAGE as a query through the configured multimodal model.
*
* Sibling of `embedQueryMultimodal(text)` for the Phase 2 image-as-query
* path. The image bytes must already be loaded and base64-encoded by the
* caller (see `src/core/search/image-loader.ts` for the SSRF-defended
* loader). Threads `inputType: 'query'` so Voyage routes to the
* retrieval half of its asymmetric space.
*/
export async function embedQueryMultimodalImage(
input: { data: string; mime: string },
): Promise<Float32Array> {
const [vec] = await embedMultimodal(
[{ kind: 'image_base64', data: input.data, mime: input.mime }],
{ inputType: 'query' },
);
if (!vec) {
throw new AITransientError('embedQueryMultimodalImage: gateway returned no vector');
}
return vec;
}
/**
* Partial-failure-aware variant of `embedMultimodal`.
*
* The default `embedMultimodal()` throws on first failure to preserve the
* pre-v0.36 contract (used by `importImageFile` which can't proceed on
* partial data). Phase 3 `reindex --multimodal` ingests many thousands
* of chunks and CAN make forward progress with partial results — it
* uses this variant so a 401 on chunk 87K doesn't discard the 31
* already-computed embeddings in that batch.
*
* Strategy:
* 1. Try the full input set via `embedMultimodal`. On success, return.
* 2. On AIConfigError (permanent), surface every input as failed —
* the misconfig isn't going to fix itself by retrying smaller.
* 3. On AITransientError or other thrown error, split-and-retry
* via binary search. Single-input attempts that fail are recorded
* in `failedIndices` and skipped.
*
* Returns `MultimodalBatchResult` with parallel-indexed `embeddings`
* (undefined for failed slots) and a `failedIndices` array.
*/
export async function embedMultimodalSafe(
inputs: MultimodalInput[],
opts: EmbedMultimodalOpts = {},
): Promise<MultimodalBatchResult> {
if (!inputs || inputs.length === 0) {
return { embeddings: [], failedIndices: [] };
}
const embeddings: Array<Float32Array | undefined> = new Array(inputs.length).fill(undefined);
const failedIndices: number[] = [];
let lastError: Error | undefined;
async function attempt(startIdx: number, items: MultimodalInput[]): Promise<void> {
if (items.length === 0) return;
try {
const vecs = await embedMultimodal(items, opts);
for (let i = 0; i < vecs.length; i++) {
embeddings[startIdx + i] = vecs[i];
}
return;
} catch (err) {
lastError = err instanceof Error ? err : new Error(String(err));
// AIConfigError = permanent misconfig. Retrying smaller won't help.
if (lastError instanceof AIConfigError) {
for (let i = 0; i < items.length; i++) failedIndices.push(startIdx + i);
return;
}
// Single input that failed — record and move on.
if (items.length === 1) {
failedIndices.push(startIdx);
return;
}
// Binary-search split. Each half gets its own retry.
const mid = Math.floor(items.length / 2);
await attempt(startIdx, items.slice(0, mid));
await attempt(startIdx + mid, items.slice(mid));
}
}
await attempt(0, inputs);
failedIndices.sort((a, b) => a - b);
return { embeddings, failedIndices, lastError };
}
// ---- Expansion ----
async function resolveExpansionProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> {
src/core/ai/types.ts
+42 -5
View File
@@ -104,15 +104,52 @@ export interface EmbeddingTouchpoint {
/**
* v0.27.1: input shape for gateway.embedMultimodal(). Discriminated union;
* today the only kind is image_base64 (raw bytes encoded by the caller).
* Future kinds (image_url, video_keyframe) extend the union without
* widening callers because the discriminator is exhaustive.
* variants extend without widening callers because the discriminator is
* exhaustive.
*
* No image_url variant: SSRF surface. Callers must read the bytes and
* base64-encode them; the gateway never fetches external URLs.
* base64-encode them; the gateway never fetches external URLs. For
* remote-callable image-as-query, the dedicated SSRF-defended loader at
* `src/core/search/image-loader.ts` resolves URLs to base64 first.
*
* v0.36 (cross-modal wave) adds the `text` variant for query-side
* multimodal embedding (`embedQueryMultimodal(text)`) — Voyage's
* multimodal endpoint accepts a content array mixing text + image entries.
*/
export type MultimodalInput =
| { kind: 'image_base64'; data: string; mime: string };
| { kind: 'image_base64'; data: string; mime: string }
| { kind: 'text'; text: string };
/**
* v0.36 — opts for gateway.embedMultimodal().
*
* `inputType` threads Voyage's retrieval discipline through:
* - 'document' (default) — embedding side of asymmetric retrieval
* - 'query' — query side; routes to the matching half of Voyage's space
*
* Mixing inputType across calls is fine; mixing within one batch is not
* supported (Voyage requires one input_type per request).
*/
export interface EmbedMultimodalOpts {
inputType?: 'document' | 'query';
}
/**
* v0.36 — return shape for partial-failure-aware multimodal batching.
*
* Used by `embedMultimodalSafe()` (the Phase-3-reindex-safe variant). The
* default `embedMultimodal()` throws on first failure to preserve the
* pre-v0.36 contract; callers who can persist partial progress opt into
* the safe variant explicitly.
*/
export interface MultimodalBatchResult {
/** Successful embeddings, indexed parallel to the original inputs (may contain holes as undefined). */
embeddings: Array<Float32Array | undefined>;
/** Indices of inputs that failed to embed, in original-input order. */
failedIndices: number[];
/** Last error encountered (for diagnostics; not necessarily the only failure). */
lastError?: Error;
}
export interface ExpansionTouchpoint {
models: string[];
src/core/backfill-registry.ts
+45
View File
@@ -190,10 +190,55 @@ function embeddingVoyageBackfill(): RegisteredBackfill {
};
}
interface ModalityBackfillRow {
id: number;
chunk_source: string | null;
modality: string | null;
}
/**
* v0.36 cross-modal wave: modality column cleanup.
*
* Historical brains that imported image assets BEFORE v0.27.1's
* `modality='image'` default-set may have image chunks where
* `embedding_image IS NOT NULL` but `modality != 'image'`. This backfill
* flips them. D22-7 hardening: requires `chunk_source='image_asset'` so
* we never tag a non-image chunk that happens to have an embedding_image
* value (defensive against hypothetical future code paths populating
* embedding_image on text chunks).
*
* Idempotent — second run finds no rows to update.
*/
function modalityBackfill(): RegisteredBackfill {
return {
description: 'Flip modality to "image" on image-asset chunks where it was missed by pre-v0.27.1 ingest',
v030_1_status: 'implemented',
spec: {
name: 'modality',
table: 'content_chunks',
idColumn: 'id',
selectColumns: ['chunk_source', 'modality'],
needsBackfill:
// D22-7: belt-and-suspenders. Both predicates must hold for the row
// to be a real image chunk that lost its modality tag.
"embedding_image IS NOT NULL AND chunk_source = 'image_asset' AND (modality IS NULL OR modality != 'image')",
compute: async (rows) => {
const updates: Array<{ id: number; updates: Record<string, unknown> }> = [];
for (const r of rows as unknown as ModalityBackfillRow[]) {
updates.push({ id: r.id, updates: { modality: 'image' } });
}
return updates;
},
estimateRowsPerSecond: 8000, // pure metadata flip, very fast
},
};
}
function registerCoreBackfills(): void {
registerBackfill(effectiveDateBackfill());
registerBackfill(emotionalWeightBackfill());
registerBackfill(embeddingVoyageBackfill());
registerBackfill(modalityBackfill());
}
// Auto-register on first import.
src/core/embedding.ts
+16 -2
View File
@@ -16,8 +16,22 @@ import {
// v0.27.1: re-export multimodal embedding so callers can pull both text and
// image embedding APIs from `src/core/embedding`. import-image-file consumes
// embedMultimodal directly.
export { embedMultimodal } from './ai/gateway.ts';
export type { MultimodalInput } from './ai/types.ts';
//
// v0.36 cross-modal wave: query-side multimodal embedding (text and image
// variants) for hybridSearch routing image-intent queries to the multimodal
// column. embedMultimodalSafe is the partial-failure variant Phase 3 reindex
// uses to make forward progress on transient batch failures.
export {
embedMultimodal,
embedMultimodalSafe,
embedQueryMultimodal,
embedQueryMultimodalImage,
} from './ai/gateway.ts';
export type {
MultimodalInput,
EmbedMultimodalOpts,
MultimodalBatchResult,
} from './ai/types.ts';
/** Embed one text (document-side for asymmetric providers). */
export async function embed(text: string): Promise<Float32Array> {
src/core/migrate.ts
+81
View File
@@ -3150,6 +3150,87 @@ export const MIGRATIONS: Migration[] = [
ON oauth_clients USING GIN (federated_read);
`,
},
{
version: 78,
name: 'embedding_multimodal_column',
// D20 Phase 3: add the unified-multimodal vector column to content_chunks.
//
// Column-only migration — the HNSW partial index is built AFTER the first
// bulk reindex completes (via `gbrain reindex --multimodal --build-index`
// or auto-built at completion). pgvector docs explicitly note that HNSW
// build is faster after data load, and per-row index maintenance during
// bulk reindex would slow the operation 2-3x.
//
// Operator class will be vector_cosine_ops to match the existing
// embedding_image index for ranking parity.
//
// The column ships at 1024 dims to match Voyage multimodal-3 output.
// Operators wanting a different dim (Cohere multimodal at 1408d, etc.)
// need a column rebuild — surfaced by the `multimodal_column_dim_match`
// doctor check (D20 model+dim pin).
idempotent: true,
sql: `
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_multimodal vector(1024);
`,
sqlFor: {
pglite: `
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS embedding_multimodal vector(1024);
`,
},
},
{
version: 77,
name: 'mcp_spend_log',
// D23-#6: per-OAuth-client paid-API spend tracking. search_by_image
// (Phase 2 of cross-modal wave) makes paid Voyage calls on behalf of
// remote OAuth clients. The existing v0.22.7 limiter caps requests/min
// but not spend. A 100-req/min attacker can burn ~$3/hour at Voyage
// rates. This table aggregates spend so the daily-budget check can
// refuse new calls when a client crosses
// search.image_query.daily_budget_usd_per_client (default $5).
//
// Indexed for the hot read: (client_id, day) lookup, summed.
// Row count is bounded by O(clients × days) — tiny.
idempotent: true,
sql: `
CREATE TABLE IF NOT EXISTS mcp_spend_log (
id SERIAL PRIMARY KEY,
client_id TEXT,
token_name TEXT,
operation TEXT NOT NULL,
spend_cents NUMERIC(12, 4) NOT NULL DEFAULT 0,
provider TEXT,
model TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- BTREE on (client_id, created_at) covers the per-day rollup query
-- (SELECT SUM ... WHERE client_id = $ AND created_at >= today_start) via
-- range scan on created_at. date_trunc in an index expression would
-- require IMMUTABLE — TIMESTAMPTZ truncation depends on session timezone.
CREATE INDEX IF NOT EXISTS idx_mcp_spend_log_client_time
ON mcp_spend_log (client_id, created_at);
CREATE INDEX IF NOT EXISTS idx_mcp_spend_log_token_time
ON mcp_spend_log (token_name, created_at);
`,
sqlFor: {
pglite: `
CREATE TABLE IF NOT EXISTS mcp_spend_log (
id SERIAL PRIMARY KEY,
client_id TEXT,
token_name TEXT,
operation TEXT NOT NULL,
spend_cents NUMERIC(12, 4) NOT NULL DEFAULT 0,
provider TEXT,
model TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_mcp_spend_log_client_time
ON mcp_spend_log (client_id, created_at);
CREATE INDEX IF NOT EXISTS idx_mcp_spend_log_token_time
ON mcp_spend_log (token_name, created_at);
`,
},
},
{
version: 66,
name: 'embed_stale_partial_index',
src/core/operations.ts
+157
View File
@@ -1143,6 +1143,16 @@ const query: Operation = {
description:
"v0.34: scope search to a single source. Defaults to OperationContext.sourceId (set from CLI --source / GBRAIN_SOURCE / .gbrain-source dotfile). Pass '__all__' to force cross-source search in multi-source brains.",
},
cross_modal: {
type: 'string',
enum: ['text', 'image', 'both', 'auto'],
description:
"v0.36 cross-modal search routing.\n" +
" 'text' (default for non-image-intent queries) — text-only path, no behavior change vs v0.35.\n" +
" 'image' — route the query through Voyage multimodal-3 + the embedding_image column. Best for 'show me photos of...' phrasings.\n" +
" 'both' — run text AND image searches in parallel; merge via weighted RRF.\n" +
" 'auto' — same effect as omitting the field; intent classifier decides based on query phrasing.",
},
embedding_column: {
type: 'string',
description:
@@ -1228,6 +1238,8 @@ const query: Operation = {
tokenBudget: typeof p.token_budget === 'number' ? (p.token_budget as number) : undefined,
useCache: typeof p.use_cache === 'boolean' ? (p.use_cache as boolean) : undefined,
intentWeighting: typeof p.intent_weighting === 'boolean' ? (p.intent_weighting as boolean) : undefined,
// v0.36 cross-modal routing param.
crossModal: p.cross_modal as 'text' | 'image' | 'both' | 'auto' | undefined,
onMeta: (m) => { capturedMeta = m; },
// v0.36 (D15): per-call embedding column override. Resolver rejects
// unknown names at hybrid entry with EmbeddingColumnNotRegisteredError;
@@ -3285,6 +3297,149 @@ const code_traversal_cache_clear: Operation = {
cliHints: { name: 'code_traversal_cache_clear', hidden: true },
};
// --- v0.36 Phase 2: search_by_image (image-as-query) ---
const search_by_image: Operation = {
name: 'search_by_image',
description:
'v0.36 cross-modal Phase 2: image-as-query retrieval. Accepts a local path (CLI), data: URI, or http(s):// URL ' +
'(SSRF-defended). Returns visually-similar image chunks plus any OCR text they carry. Optional `query` text ' +
'refinement merges via weighted RRF (D13 hybrid intersect). True image→full-text-knowledge requires Phase 3 ' +
'(`gbrain reindex --multimodal` + `search.unified_multimodal: true`).',
params: {
image_path: { type: 'string', description: 'Absolute path to image (local CLI callers only — rejected for remote MCP per D18).' },
image_url: { type: 'string', description: 'http(s):// URL to image. SSRF-defended; max 3 redirect hops; 10MB cap.' },
image_data: { type: 'string', description: 'Base64-encoded image bytes (preferred for remote MCP callers). PNG/JPEG/WebP only.' },
image_mime: { type: 'string', description: 'Optional MIME hint when ambiguous. Magic-byte sniff is authoritative.' },
query: { type: 'string', description: 'Optional text refinement; runs hybrid intersect via D13 weighted RRF.' },
limit: { type: 'number', description: 'Max results (default 20)' },
offset: { type: 'number', description: 'Skip first N results (for pagination)' },
source_id: { type: 'string', description: "Scope to a single source. Defaults to ctx.sourceId. '__all__' opts out." },
},
scope: 'read',
// NOT localOnly: remote MCP callers can pass image_url or image_data
// (subject to D18 image_path ban + D12 size cap + D23-#6 spend cap).
handler: async (ctx, p) => {
const imagePath = p.image_path as string | undefined;
const imageUrl = p.image_url as string | undefined;
const imageData = p.image_data as string | undefined;
const imageMime = (p.image_mime as string) || undefined;
const queryRefinement = p.query as string | undefined;
const sourceIdParam = typeof p.source_id === 'string' ? p.source_id : undefined;
// D18 P0 — remote callers cannot pass image_path. Rejecting at handler
// entry, before any file I/O fires. validateParams catches it too at the
// dispatch layer; this is defense-in-depth.
if (ctx.remote === true && imagePath) {
throw new Error(
'permission_denied: image_path is not permitted for remote callers (D18). ' +
'Use image_url or image_data instead.',
);
}
if (!imagePath && !imageUrl && !imageData) {
throw new Error('search_by_image requires one of: image_path, image_url, image_data');
}
if ([imagePath, imageUrl, imageData].filter(Boolean).length > 1) {
throw new Error('search_by_image accepts only one of: image_path, image_url, image_data');
}
// D23-#6 — pre-flight daily-budget check for remote OAuth clients.
// Local CLI callers (ctx.remote=false) bypass the cap (clientId="").
const clientId = (ctx.remote === true ? (ctx.auth?.clientId ?? '') : '');
if (clientId) {
const budgetUsd = await getDailyImageBudgetUsd(ctx.engine);
const { checkBudget } = await import('./spend-log.ts');
await checkBudget(ctx.engine, clientId, Math.round(budgetUsd * 100));
}
// Resolve image bytes via the SSRF-defended loader. For remote callers,
// tighter byte cap.
const remoteCap = await getRemoteMaxBytes(ctx.engine);
const localCap = await getLocalMaxBytes(ctx.engine);
const cap = ctx.remote === true ? remoteCap : localCap;
const { loadImageInput } = await import('./search/image-loader.ts');
const loaded = await loadImageInput(
(imagePath ?? imageUrl ?? `data:${imageMime ?? 'image/png'};base64,${imageData}`)!,
{ maxBytes: cap },
);
// Resolve source-scope (D5 canonical thread).
const resolvedSourceId =
sourceIdParam !== undefined
? sourceIdParam === '__all__'
? undefined
: sourceIdParam
: ctx.sourceId;
const { searchByImage } = await import('./search/by-image.ts');
const results = await searchByImage(
ctx.engine,
{ base64: loaded.base64, mime: loaded.contentType },
{
limit: (p.limit as number) || 20,
offset: (p.offset as number) || 0,
query: queryRefinement,
sourceId: resolvedSourceId,
...sourceScopeOpts(ctx),
},
);
// D23-#6 — record successful Voyage call. Best-effort; failures don't
// block the response.
if (clientId) {
const { recordSpend, VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS } = await import('./spend-log.ts');
// Approximate: 1 image embed + (query ? 1 text embed : 0). Both are
// billed at the same per-call rate by Voyage.
const calls = 1 + (queryRefinement ? 1 : 0);
void recordSpend(ctx.engine, {
clientId,
tokenName: ctx.auth?.clientName ?? null,
operation: 'search_by_image',
spendCents: VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS * calls,
provider: 'voyage',
model: 'voyage-multimodal-3',
});
}
return results;
},
cliHints: { name: 'search-by-image' },
};
async function getDailyImageBudgetUsd(engine: BrainEngine): Promise<number> {
try {
const v = await engine.getConfig('search.image_query.daily_budget_usd_per_client');
if (v == null) return 5; // default $5
const n = parseFloat(v);
return Number.isFinite(n) && n > 0 ? n : 5;
} catch {
return 5;
}
}
async function getLocalMaxBytes(engine: BrainEngine): Promise<number> {
try {
const v = await engine.getConfig('search.image_query.max_bytes');
if (v == null) return 10 * 1024 * 1024;
const n = parseInt(v, 10);
return Number.isFinite(n) && n > 0 ? n : 10 * 1024 * 1024;
} catch {
return 10 * 1024 * 1024;
}
}
async function getRemoteMaxBytes(engine: BrainEngine): Promise<number> {
try {
const v = await engine.getConfig('search.image_query.remote_max_bytes');
if (v == null) return 2 * 1024 * 1024;
const n = parseInt(v, 10);
return Number.isFinite(n) && n > 0 ? n : 2 * 1024 * 1024;
} catch {
return 2 * 1024 * 1024;
}
}
// --- Exports ---
export const operations: Operation[] = [
@@ -3294,6 +3449,8 @@ export const operations: Operation[] = [
restore_page, purge_deleted_pages,
// Search
search, query,
// v0.36 Phase 2: image-as-query
search_by_image,
// Tags
add_tag, remove_tag, get_tags,
// Links
src/core/pglite-engine.ts
+12 -7
View File
@@ -1289,15 +1289,20 @@ export class PGLiteEngine implements BrainEngine {
// v0.36 (D11): column routing via resolved descriptor. Engine doesn't
// read config — caller resolved at hybrid/op boundary. The cast SQL
// ($1::vector vs $1::halfvec(N)) comes from buildVectorCastFragment.
//
// v0.36 Phase 3: 'embedding_multimodal' is the unified column populated
// by `gbrain reindex --multimodal`. No modality filter — the column
// itself is the discriminator (only re-embedded rows have non-NULL).
const resolvedCol = normalizeEngineColumn(opts?.embeddingColumn);
const { col, castSql } = buildVectorCastFragment(resolvedCol);
// Image rows live in modality='image'; text/code in 'text'. Restrict
// by modality so non-image columns can't accidentally pull image
// chunks (or vice versa). resolved.name has already passed regex
// validation; never compares against raw input.
const modalityFilter = resolvedCol.name === 'embedding_image'
? `AND cc.modality = 'image'`
: `AND cc.modality = 'text'`;
let modalityFilter: string;
if (resolvedCol.name === 'embedding_image') {
modalityFilter = `AND cc.modality = 'image'`;
} else if (resolvedCol.name === 'embedding_multimodal') {
modalityFilter = '';
} else {
modalityFilter = `AND cc.modality = 'text'`;
}
const { rows } = await this.db.query(
`WITH hnsw_candidates AS (
src/core/pglite-schema.ts
+5 -1
View File
@@ -119,7 +119,11 @@ CREATE TABLE IF NOT EXISTS content_chunks (
-- chunks carry their 1024-dim Voyage multimodal vector in embedding_image
-- (independent of the brain primary embedding column dim).
modality TEXT NOT NULL DEFAULT 'text',
embedding_image vector(1024)
embedding_image vector(1024),
-- v0.36 Phase 3 cross-modal: unified column populated by reindex
-- (search.unified_multimodal=true routes here). Migration v75 adds it
-- on upgrade; fresh installs land at head with the column present.
embedding_multimodal vector(1024)
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
src/core/postgres-engine.ts
+13 -8
View File
@@ -1303,16 +1303,21 @@ export class PostgresEngine implements BrainEngine {
// read config — caller (hybrid/op) resolved it and passed it in.
// normalizeEngineColumn accepts the legacy union (string literals,
// ResolvedColumn, undefined) and produces a canonical descriptor.
//
// v0.36 Phase 3: 'embedding_multimodal' is the unified column populated
// by `gbrain reindex --multimodal`. Carries BOTH text and image content
// in Voyage multimodal-3 space — no modality filter; the column itself
// is the discriminator (rows without embedding_multimodal aren't searched).
const resolvedCol = normalizeEngineColumn(opts?.embeddingColumn);
const { col, castSql } = buildVectorCastFragment(resolvedCol);
// Modality filter: image rows live in modality='image'; text/code in
// 'text'. The image-column literal is the only one with image rows.
// For all other columns (default + user-declared), restrict to text
// mode to avoid cross-modality dim leaks. The check is on
// resolved.name (already validated, never raw input).
const modalityFilter = resolvedCol.name === 'embedding_image'
? `AND cc.modality = 'image'`
: `AND cc.modality = 'text'`;
let modalityFilter: string;
if (resolvedCol.name === 'embedding_image') {
modalityFilter = `AND cc.modality = 'image'`;
} else if (resolvedCol.name === 'embedding_multimodal') {
modalityFilter = '';
} else {
modalityFilter = `AND cc.modality = 'text'`;
}
const rawQuery = `
WITH hnsw_candidates AS (
src/core/schema-embedded.ts
+4 -1
View File
@@ -159,7 +159,10 @@ CREATE TABLE IF NOT EXISTS content_chunks (
-- mixed-provider brains (e.g. OpenAI 1536 text + Voyage 1024 images) keep
-- both columns populated with distinct dim spaces.
modality TEXT NOT NULL DEFAULT 'text',
embedding_image vector(1024)
embedding_image vector(1024),
-- v0.36 Phase 3 cross-modal: unified column populated by reindex.
-- Migration v75 also adds it for upgrade paths.
embedding_multimodal vector(1024)
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_chunks_page_index ON content_chunks(page_id, chunk_index);
src/core/search/by-image.ts
+128
View File
@@ -0,0 +1,128 @@
/**
* v0.36 Phase 2 — `searchByImage` retrieval.
*
* D17 honest framing: Phase 2 ships image→similar-images + image-OCR-text
* retrieval. Image chunks have OCR text in `chunk_text`, so the results
* carry both visual-similarity AND any OCR text the chunks captured. True
* image→full-text-knowledge (Alice's bio, takes about her) requires text
* chunks in the multimodal embedding space — Phase 3's unified column.
*
* D13 hybrid intersect for optional text refinement: when caller supplies
* `query`, runs image-vector AND text-vector searches in parallel and
* merges via weighted RRF. Treats text refinement as a full second axis
* (not a vector-space bias from naive averaging).
*
* Routing matrix:
* - image-only (no query):
* embedQueryMultimodalImage → searchVector(embedding_image)
* - image + text query (D13):
* embedQueryMultimodalImage AND embedQueryMultimodal(query)
* → parallel searchVector calls → rrfFusionWeighted merge
* - Phase 3 unified mode (search.unified_multimodal=true): both branches
* target embedding_multimodal instead of embedding_image
*
* D5 source-id: every engine call receives sourceOpts threaded from
* `ctx.sourceId` / `ctx.auth.allowedSources`. Tested by
* test/e2e/source-isolation-image.test.ts (4 cases).
*/
import type { BrainEngine } from '../engine.ts';
import type { SearchOpts, SearchResult } from '../types.ts';
import { effectiveRrfK } from './intent-weights.ts';
import { rrfFusionWeighted, RRF_K } from './hybrid.ts';
import { dedupResults } from './dedup.ts';
import { embedQueryMultimodal, embedQueryMultimodalImage } from '../ai/gateway.ts';
import { loadSearchModeConfig, resolveSearchMode } from './mode.ts';
export interface SearchByImageOpts extends SearchOpts {
/** Optional text refinement; runs hybrid intersect via weighted RRF (D13). */
query?: string;
}
/**
* Run image-as-query retrieval against the brain.
*
* @param engine brain engine (PGLite or Postgres)
* @param input loaded image (from `loadImageInput`): { base64, mime }
* @param opts SearchOpts + optional `query` for D13 text refinement
*/
export async function searchByImage(
engine: BrainEngine,
input: { base64: string; mime: string },
opts: SearchByImageOpts = {},
): Promise<SearchResult[]> {
// Resolve mode bundle once at entry — picks up cross-modal RRF weights
// (D13 image_query_text_refinement_weight / image_query_image_refinement_weight)
// and the Phase 3 unified-multimodal flag.
const modeInput = await loadSearchModeConfig(engine);
const resolvedMode = resolveSearchMode({
mode: modeInput.mode,
overrides: modeInput.overrides,
});
const limit = opts.limit ?? resolvedMode.searchLimit;
const offset = opts.offset ?? 0;
// Phase 2 always targets embedding_image. Phase 3's unified column
// routing slots in here once src/core/types.ts widens
// SearchOpts.embeddingColumn to include 'embedding_multimodal' (Commit 3
// schema migration + type widening).
const imageColumn: 'embedding_image' = 'embedding_image';
const baseSearchOpts: SearchOpts = {
limit: Math.min(limit * 2, 100),
offset: 0,
sourceId: opts.sourceId,
sourceIds: opts.sourceIds,
// Both branches use the same source-scope threading.
};
// Image branch — always runs.
const imageEmbedding = await embedQueryMultimodalImage({
data: input.base64,
mime: input.mime,
});
const imageOpts: SearchOpts = { ...baseSearchOpts, embeddingColumn: imageColumn };
const imageList = await engine.searchVector(imageEmbedding, imageOpts);
// Tag rows with modality so downstream consumers can distinguish in 'both' results.
for (const r of imageList) {
r.modality = r.modality ?? 'image';
}
// D13: if caller provided a text refinement query, run the text branch too
// and merge via weighted RRF.
if (opts.query && opts.query.trim().length > 0) {
const textEmbedding = await embedQueryMultimodal(opts.query);
const textOpts: SearchOpts = {
...baseSearchOpts,
embeddingColumn: imageColumn,
};
// Both branches target the same multimodal-embedding column (Phase 2).
// The "intersect" is via RRF rank merging — the text-side vector lives
// in the SAME multimodal embedding space as the image vector
// (embedQueryMultimodal returns a 1024d vector), so it's the right
// space to query.
const textList = await engine.searchVector(textEmbedding, textOpts);
for (const r of textList) {
r.modality = r.modality ?? 'image';
}
// Weighted RRF merge: image branch has higher default weight (0.6 vs 0.4)
// because the caller chose image-first. Configurable via
// search.image_query.*_refinement_weight (D3 registered).
const baseRrfK = RRF_K;
const imageK = effectiveRrfK(baseRrfK, resolvedMode.image_query_image_refinement_weight);
const textK = effectiveRrfK(baseRrfK, resolvedMode.image_query_text_refinement_weight);
const fused = rrfFusionWeighted(
[
{ list: imageList, k: imageK },
{ list: textList, k: textK },
],
true, // apply boost
);
fused.sort((a, b) => b.score - a.score);
return dedupResults(fused).slice(offset, offset + limit);
}
// Image-only: return the deduped image branch directly.
return dedupResults(imageList).slice(offset, offset + limit);
}
src/core/search/embedding-column.ts
+11
View File
@@ -527,6 +527,17 @@ export function normalizeEngineColumn(
};
}
// v0.36 Phase 3 unified multimodal column — populated by
// `gbrain reindex --multimodal`. Voyage multimodal-3, vector(1024).
if (embeddingColumn === 'embedding_multimodal') {
return {
name: 'embedding_multimodal',
type: 'vector',
dimensions: 1024,
embeddingModel: 'voyage:voyage-multimodal-3',
};
}
// Any other raw string at this layer is a programming error — the
// resolver should have run at the hybrid/op boundary and produced a
// descriptor. We throw with a paste-ready hint rather than guess.
src/core/search/hybrid.ts
+203 -29
View File
@@ -17,7 +17,7 @@ import { resolveEmbeddingColumn, isCacheSafe } from './embedding-column.ts';
import { loadConfigWithEngine } from '../config.ts';
import { dedupResults } from './dedup.ts';
import { applyReranker } from './rerank.ts';
import { autoDetectDetail, classifyQuery } from './query-intent.ts';
import { autoDetectDetail, classifyQuery, isAmbiguousModalityQuery } from './query-intent.ts';
import { expandAnchors, hydrateChunks } from './two-pass.ts';
import { enforceTokenBudget } from './token-budget.ts';
import { recordSearchTelemetry } from './telemetry.ts';
@@ -31,7 +31,7 @@ import {
loadCacheConfig,
} from './query-cache.ts';
const RRF_K = 60;
export const RRF_K = 60;
const COMPILED_TRUTH_BOOST = 2.0;
const pendingCacheWrites = new Set<Promise<unknown>>();
@@ -454,8 +454,22 @@ export async function hybridSearch(
console.error(`[search-debug] auto-detail=${detail} for query="${query}"`);
}
// Run keyword search (always available, no API key needed)
const keywordResults = await engine.searchKeyword(query, searchOpts);
// Run keyword search (always available, no API key needed).
//
// v0.36 cross-modal (D9): skip keyword for 'image'-only modality. Image
// chunks may have OCR text in chunk_text, but a text-only keyword scan
// would also surface every text chunk containing the query phrase —
// not what an image-intent query asked for. Image vector search is the
// canonical channel for image-modality queries.
//
// We classify modality early (it's also computed after for the modality
// branch). The classification is pure regex via classifyQuery; running it
// here is cheap.
const earlyModality = (opts?.crossModal && opts.crossModal !== 'auto')
? opts.crossModal
: (suggestions.suggestedModality ?? 'text');
const keywordResults: SearchResult[] =
earlyModality === 'image' ? [] : await engine.searchKeyword(query, searchOpts);
// v0.29.1: resolve salience/recency from caller (back-compat aliases for
// PR #618's `recencyBoost` numeric scale) or fall back to the heuristic.
@@ -526,14 +540,62 @@ export async function hybridSearch(
return noEmbedBudgeted;
}
// v0.36 cross-modal wave: determine the effective modality once.
//
// Precedence (D22-1 normalization): literal 'auto' is normalized to
// undefined so it doesn't reach the modality branch directly. Resolution:
// explicit opts.crossModal ('text'|'image'|'both') wins
// else suggestions.suggestedModality (regex-driven)
// else (Commit 4) opt-in LLM tie-break for genuinely ambiguous queries
// else 'text' (default)
//
// D9 mode-bundle override matrix: when effectiveModality === 'image',
// cross-modal path overrides bundle knobs (expansion=false, no keyword
// search). Voyage handles synonyms in-space; zerank-2 can't rerank image
// embeddings.
//
// Phase 3 (D8): when search.unified_multimodal is true, ALL queries
// route through the multimodal model + embedding_multimodal column,
// regardless of detected modality.
//
// Commit 4 (LLM intent escalation): when search.cross_modal.llm_intent
// is true AND regex returned 'text' AND isAmbiguousModalityQuery fires,
// await a Haiku tie-break. Fail-open to regex result on any error.
const explicitModality =
opts?.crossModal && opts.crossModal !== 'auto' ? opts.crossModal : undefined;
let regexModality = explicitModality ?? suggestions.suggestedModality ?? 'text';
// LLM tie-break fires ONLY when:
// - no explicit per-call override
// - regex returned 'text' (not confident image/both)
// - operator opted in via search.cross_modal.llm_intent
// - isAmbiguousModalityQuery says the query is genuinely ambiguous
if (
explicitModality === undefined &&
regexModality === 'text' &&
resolvedMode.cross_modal_llm_intent &&
isAmbiguousModalityQuery(query)
) {
try {
const { classifyModalityWithLLM } = await import('./llm-intent.ts');
regexModality = await classifyModalityWithLLM(query, 'text');
} catch {
// Fail-open: regex result stands.
}
}
const effectiveModality = regexModality;
const unifiedRouting = resolvedMode.unified_multimodal === true;
// Determine query variants (optionally with expansion)
// expandQuery already includes the original query in its return value,
// so we use it directly instead of prepending query again.
// v0.32.3 search-lite: expansion fires when (a) resolved mode says yes and
// (b) an expandFn is wired in. The mode bundle is the default; per-call
// SearchOpts.expansion still wins via resolveSearchMode's chain.
//
// D9: image-modality skips expansion regardless of mode bundle.
let queries = [query];
if (resolvedMode.expansion && opts?.expandFn) {
const expansionAllowed = resolvedMode.expansion && effectiveModality !== 'image';
if (expansionAllowed && opts?.expandFn) {
try {
queries = await opts.expandFn(query);
if (queries.length === 0) queries = [query];
@@ -544,28 +606,121 @@ export async function hybridSearch(
}
}
// Embed all query variants and run vector search
// Embed all query variants and run vector search.
//
// v0.36 cross-modal wave routing:
// - 'text' (default): existing text-embedding path, unchanged
// - 'image': embedQueryMultimodal + searchVector(embedding_image), skip keyword
// - 'both': text + image vector searches in parallel; merged via weighted RRF
let vectorLists: SearchResult[][] = [];
let queryEmbedding: Float32Array | null = null;
try {
// v0.35.0.0+: query-side embedding. For asymmetric providers (ZE zembed-1,
// Voyage v3+) routes input_type='query' through the embed seam; symmetric
// providers ignore the field — no behavior change.
// v0.36 (D10): route through the resolved column's provider + dims so
// a query against `embedding_voyage` actually embeds via Voyage, not
// the global default (OpenAI). Empty embeddingModel falls back to
// gateway default — preserves pre-v0.36 behavior for the builtin
// 'embedding' column.
const embedOpts = resolvedCol.embeddingModel
? { embeddingModel: resolvedCol.embeddingModel, dimensions: resolvedCol.dimensions }
: undefined;
const embeddings = await Promise.all(queries.map(q => embedQuery(q, embedOpts)));
queryEmbedding = embeddings[0];
vectorLists = await Promise.all(
embeddings.map(emb => engine.searchVector(emb, searchOpts)),
);
} catch {
// Embedding failure is non-fatal, fall back to keyword-only
let imageVectorList: SearchResult[] | null = null;
let crossModalFellOpen = false;
// Phase 3 unified routing: when on, route ALL queries through Voyage
// multimodal-3 + embedding_multimodal column. Bypasses the dual-column
// branching below — but with D8 fail-open: if the unified path returns
// zero rows AND the operator hasn't opted into strict unified-only mode,
// fall through to the dual-column text path. unified_multimodal_only
// disables the fallback.
let unifiedDone = false;
if (unifiedRouting) {
try {
const { isAvailable: aiIsAvailable, embedQueryMultimodal } = await import('../ai/gateway.ts');
if (!aiIsAvailable('embedding')) {
throw new Error('gateway not configured for embedding — unified multimodal would also fail');
}
const unifiedEmbedding = await embedQueryMultimodal(query);
const unifiedSearchOpts: SearchOpts = {
...searchOpts,
embeddingColumn: 'embedding_multimodal',
};
const unifiedList = await engine.searchVector(unifiedEmbedding, unifiedSearchOpts);
// D8 fail-open: zero rows + not strict-mode → fall through to dual-column.
if (unifiedList.length === 0 && !resolvedMode.unified_multimodal_only) {
console.error(
`[cross-modal] unified_multimodal returned zero rows for query="${query.slice(0, 60)}". ` +
`Falling back to dual-column text path (partial coverage during reindex). ` +
`Set search.unified_multimodal_only=true to bypass this fallback when reindex completes.`,
);
} else {
vectorLists = [unifiedList];
queryEmbedding = unifiedEmbedding;
unifiedDone = true;
}
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
console.error(
`[cross-modal] unified_multimodal embed failed; falling back to dual-column path. reason=${reason}`,
);
crossModalFellOpen = true;
}
}
if (!unifiedDone && (effectiveModality === 'image' || effectiveModality === 'both')) {
// Attempt image-side embedding. Fail-open: if multimodal is unconfigured
// OR the embed throws, log a structured warning and fall through to text.
try {
const { isAvailable: aiIsAvailable, embedQueryMultimodal } = await import('../ai/gateway.ts');
if (!aiIsAvailable('embedding')) {
throw new Error('gateway not configured for embedding — multimodal would also fail');
}
const imageEmbedding = await embedQueryMultimodal(query);
const imageSearchOpts: SearchOpts = {
...searchOpts,
embeddingColumn: 'embedding_image',
};
const imageList = await engine.searchVector(imageEmbedding, imageSearchOpts);
for (const r of imageList) {
r.modality = r.modality ?? 'image';
}
imageVectorList = imageList;
} catch (err) {
// Fail-open per behavioral invariant 2.
const reason = err instanceof Error ? err.message : String(err);
console.error(
`[cross-modal] image-side embed failed for modality=${effectiveModality}; falling back to text-only. reason=${reason}`,
);
crossModalFellOpen = true;
}
}
if (unifiedDone) {
// Unified routing already populated vectorLists + queryEmbedding;
// skip the dual-column branching.
} else if (effectiveModality === 'image' && imageVectorList !== null) {
// Image-only path: results come entirely from the image column.
vectorLists = [imageVectorList];
queryEmbedding = null; // no text embedding to cosine-re-score against
} else {
// 'text' or 'both' (or 'image' that fell open to text). Run the text
// path normally, with v0.36 (D10) provider-aware embed routing so a
// query against `embedding_voyage` actually embeds via Voyage, not
// the global default. Empty embeddingModel falls back to gateway
// default — preserves pre-v0.36 behavior for the builtin 'embedding'
// column.
try {
const embedOpts = resolvedCol.embeddingModel
? { embeddingModel: resolvedCol.embeddingModel, dimensions: resolvedCol.dimensions }
: undefined;
const embeddings = await Promise.all(queries.map(q => embedQuery(q, embedOpts)));
queryEmbedding = embeddings[0];
const textLists = await Promise.all(
embeddings.map(emb => engine.searchVector(emb, searchOpts)),
);
for (const list of textLists) {
for (const r of list) {
r.modality = r.modality ?? 'text';
}
}
vectorLists = textLists;
// 'both' mode: also include the image-side list as another input to RRF.
if (effectiveModality === 'both' && imageVectorList !== null) {
vectorLists = [...vectorLists, imageVectorList];
}
} catch {
// Embedding failure is non-fatal, fall back to keyword-only
}
}
if (vectorLists.length === 0) {
@@ -605,10 +760,29 @@ export async function hybridSearch(
const baseRrfK = opts?.rrfK ?? RRF_K;
const keywordK = effectiveRrfK(baseRrfK, intentWeights.keywordWeight);
const vectorK = effectiveRrfK(baseRrfK, intentWeights.vectorWeight);
const allLists: Array<{ list: SearchResult[]; k: number }> = [
...vectorLists.map(list => ({ list, k: vectorK })),
{ list: keywordResults, k: keywordK },
];
// v0.36 cross-modal (D6): in 'both' mode, vectorLists carries
// [textList, imageList]. Apply per-modality RRF weights so the merge
// reflects the configured text/image balance. In 'text' and 'image'
// modes only one branch is present, so per-modality K reduces to
// the standard vectorK (no behavior change vs pre-v0.36).
const textRrfK = effectiveRrfK(baseRrfK, resolvedMode.cross_modal_both_text_weight);
const imageRrfK = effectiveRrfK(baseRrfK, resolvedMode.cross_modal_both_image_weight);
const isBothMode = effectiveModality === 'both' && vectorLists.length >= 2;
const allLists: Array<{ list: SearchResult[]; k: number }> = isBothMode
? [
// Last list in vectorLists is the image branch (we appended it above).
// All preceding lists (1 or more text-query embeddings if expansion ran)
// get textRrfK. Image branch gets imageRrfK.
...vectorLists.slice(0, -1).map(list => ({ list, k: textRrfK })),
{ list: vectorLists[vectorLists.length - 1], k: imageRrfK },
{ list: keywordResults, k: keywordK },
]
: [
...vectorLists.map(list => ({ list, k: vectorK })),
{ list: keywordResults, k: keywordK },
];
let fused = rrfFusionWeighted(allLists, detail !== 'high');
// Cosine re-scoring before dedup so semantically better chunks survive.
src/core/search/image-loader.ts
+247
View File
@@ -0,0 +1,247 @@
/**
* v0.36 Phase 2 — image input loader for `search_by_image`.
*
* Accepts three input forms:
* - absolute path: file:// URI OR /abs/path/to/img.png (local CLI only)
* - data: URI: data:image/png;base64,<base64bytes>
* - http(s) URL: https://example.com/img.png (SSRF-defended via ssrf-validate.ts)
*
* Returns: { contentType, base64, bytes } so the caller can hand it to
* `embedQueryMultimodalImage({data: base64, mime: contentType})`.
*
* Defenses:
* - Magic-byte sniff for PNG / JPEG / WebP (no other formats accepted in v1)
* - Hard size cap (default 10 MB) — applies before allocation when possible
* via Content-Length pre-flight, and as a stream cap during fetch
* - SSRF: every URL hop validated via `validateAndResolveUrl` from
* `src/core/ssrf-validate.ts`; max 3 redirect hops
* - 5s total fetch timeout (not per-hop)
* - Rejects credentials embedded in URL
*
* Out of scope:
* - Pixel-bomb defense (decompression bombs): the size cap doubles as the
* bomb defense per D22-9. We never call a pixel parser; only feed bytes
* to Voyage. A 10MB cap is well below the pixel-bomb threshold.
* - HEIC / GIF / TIFF — Voyage multimodal-3 supports them but the test
* fixtures and OCR pipeline don't, so we keep the input surface
* conservative for v1.
*/
import { readFile } from 'node:fs/promises';
import { isAbsolute } from 'node:path';
import { SSRFError, fetchWithSSRFGuard } from '../ssrf-validate.ts';
/** Max bytes for input image. Configurable via `search.image_query.max_bytes`. */
export const DEFAULT_IMAGE_MAX_BYTES = 10 * 1024 * 1024;
/** Stricter cap for remote MCP callers (ctx.remote === true). */
export const DEFAULT_REMOTE_IMAGE_MAX_BYTES = 2 * 1024 * 1024;
export interface LoadedImage {
/** MIME content type ('image/png', 'image/jpeg', 'image/webp'). */
contentType: string;
/** Raw bytes as Buffer. Caller can use this for hashing/auditing. */
bytes: Buffer;
/** Base64-encoded bytes, ready to feed into embedQueryMultimodalImage. */
base64: string;
}
export class ImageLoadError extends Error {
readonly code: ImageLoadErrorCode;
constructor(code: ImageLoadErrorCode, message: string) {
super(message);
this.name = 'ImageLoadError';
this.code = code;
}
}
export type ImageLoadErrorCode =
| 'INVALID_FORMAT'
| 'OVERSIZED'
| 'INVALID_URL'
| 'FETCH_FAILED'
| 'TIMEOUT'
| 'SSRF_BLOCKED'
| 'NOT_FOUND';
export interface ImageLoadOpts {
/** Override the default max-bytes cap. Defaults to DEFAULT_IMAGE_MAX_BYTES. */
maxBytes?: number;
/** Total fetch timeout in ms (URLs only). Defaults to 5000. */
timeoutMs?: number;
/** Max redirect hops (URLs only). Defaults to 3. */
maxRedirects?: number;
}
/**
* Load an image input into a structured shape.
*
* Throws `ImageLoadError` on any failure — `SSRFError` from the URL path
* is caught and re-thrown as ImageLoadError with `code: 'SSRF_BLOCKED'`.
*/
export async function loadImageInput(
input: string,
opts: ImageLoadOpts = {},
): Promise<LoadedImage> {
if (typeof input !== 'string' || input.length === 0) {
throw new ImageLoadError('INVALID_URL', 'Image input must be a non-empty string');
}
const maxBytes = opts.maxBytes ?? DEFAULT_IMAGE_MAX_BYTES;
// Branch on input shape.
if (input.startsWith('data:')) {
return loadDataUri(input, maxBytes);
}
if (input.startsWith('http://') || input.startsWith('https://')) {
return loadHttpUrl(input, { maxBytes, timeoutMs: opts.timeoutMs ?? 5000, maxRedirects: opts.maxRedirects ?? 3 });
}
if (input.startsWith('file://') || isAbsolute(input)) {
const path = input.startsWith('file://') ? input.slice(7) : input;
return loadLocalPath(path, maxBytes);
}
throw new ImageLoadError(
'INVALID_URL',
`Unsupported image input shape: expected data: URI, http(s):// URL, file:// URI, or absolute path. Got: ${input.slice(0, 60)}`,
);
}
async function loadLocalPath(path: string, maxBytes: number): Promise<LoadedImage> {
let bytes: Buffer;
try {
bytes = await readFile(path);
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
if (msg.includes('ENOENT')) {
throw new ImageLoadError('NOT_FOUND', `File not found: ${path}`);
}
throw new ImageLoadError('FETCH_FAILED', `Failed to read ${path}: ${msg}`);
}
if (bytes.length > maxBytes) {
throw new ImageLoadError(
'OVERSIZED',
`Image at ${path} is ${bytes.length} bytes; cap is ${maxBytes}`,
);
}
const contentType = sniffContentType(bytes);
return finalize(bytes, contentType);
}
function loadDataUri(input: string, maxBytes: number): LoadedImage {
// data:image/png;base64,<bytes>
const match = input.match(/^data:([^;]+);base64,(.+)$/);
if (!match) {
throw new ImageLoadError('INVALID_FORMAT', 'data: URI must be in `data:<mime>;base64,<bytes>` form');
}
const declaredMime = match[1].toLowerCase();
const b64 = match[2];
let bytes: Buffer;
try {
bytes = Buffer.from(b64, 'base64');
} catch (err) {
throw new ImageLoadError(
'INVALID_FORMAT',
`Failed to decode base64: ${err instanceof Error ? err.message : String(err)}`,
);
}
if (bytes.length > maxBytes) {
throw new ImageLoadError(
'OVERSIZED',
`Decoded image is ${bytes.length} bytes; cap is ${maxBytes}`,
);
}
const sniffed = sniffContentType(bytes);
// If the declared MIME disagrees with the magic-byte sniff, trust the sniff
// (caller could lie about content-type to bypass format gate).
if (sniffed !== declaredMime) {
// Fall through with the sniffed type. Don't error — declared MIME is
// informational; bytes are the truth.
}
return finalize(bytes, sniffed);
}
async function loadHttpUrl(
url: string,
opts: { maxBytes: number; timeoutMs: number; maxRedirects: number },
): Promise<LoadedImage> {
let res: Response;
try {
res = await fetchWithSSRFGuard(url, {
maxRedirects: opts.maxRedirects,
timeoutMs: opts.timeoutMs,
});
} catch (err) {
if (err instanceof SSRFError) {
throw new ImageLoadError('SSRF_BLOCKED', `SSRF: ${err.message}`);
}
const msg = err instanceof Error ? err.message : String(err);
if (msg.includes('abort')) {
throw new ImageLoadError('TIMEOUT', `Fetch timeout (${opts.timeoutMs}ms): ${url}`);
}
throw new ImageLoadError('FETCH_FAILED', `Fetch failed: ${msg}`);
}
if (!res.ok) {
throw new ImageLoadError(
'FETCH_FAILED',
`Fetch returned HTTP ${res.status}: ${res.statusText || ''}`,
);
}
// Pre-flight: reject oversized responses before reading the body.
const contentLengthHeader = res.headers.get('content-length');
if (contentLengthHeader) {
const declared = parseInt(contentLengthHeader, 10);
if (Number.isFinite(declared) && declared > opts.maxBytes) {
throw new ImageLoadError(
'OVERSIZED',
`Response Content-Length ${declared} exceeds cap ${opts.maxBytes}`,
);
}
}
// Read the body; second guard against lying Content-Length.
const arrayBuf = await res.arrayBuffer();
const bytes = Buffer.from(arrayBuf);
if (bytes.length > opts.maxBytes) {
throw new ImageLoadError(
'OVERSIZED',
`Response body is ${bytes.length} bytes; cap is ${opts.maxBytes} (lying Content-Length?)`,
);
}
const contentType = sniffContentType(bytes);
return finalize(bytes, contentType);
}
/**
* Magic-byte sniff for the three supported formats.
*
* - PNG: starts with `89 50 4E 47 0D 0A 1A 0A`
* - JPEG: starts with `FF D8 FF`
* - WebP: starts with `RIFF` (52 49 46 46) + 4 bytes + `WEBP` (57 45 42 50)
*
* Throws `ImageLoadError` with `code: 'INVALID_FORMAT'` for anything else.
*/
function sniffContentType(bytes: Buffer): string {
if (bytes.length >= 8 &&
bytes[0] === 0x89 && bytes[1] === 0x50 && bytes[2] === 0x4E && bytes[3] === 0x47 &&
bytes[4] === 0x0D && bytes[5] === 0x0A && bytes[6] === 0x1A && bytes[7] === 0x0A) {
return 'image/png';
}
if (bytes.length >= 3 && bytes[0] === 0xFF && bytes[1] === 0xD8 && bytes[2] === 0xFF) {
return 'image/jpeg';
}
if (bytes.length >= 12 &&
bytes[0] === 0x52 && bytes[1] === 0x49 && bytes[2] === 0x46 && bytes[3] === 0x46 &&
bytes[8] === 0x57 && bytes[9] === 0x45 && bytes[10] === 0x42 && bytes[11] === 0x50) {
return 'image/webp';
}
throw new ImageLoadError(
'INVALID_FORMAT',
`Unsupported image format. Magic bytes: ${bytes.subarray(0, Math.min(12, bytes.length)).toString('hex')}. Accepted: PNG, JPEG, WebP.`,
);
}
function finalize(bytes: Buffer, contentType: string): LoadedImage {
return {
contentType,
bytes,
base64: bytes.toString('base64'),
};
}
src/core/search/llm-intent.ts
+80
View File
@@ -0,0 +1,80 @@
/**
* v0.36 Commit 4 — opt-in LLM tie-break for ambiguous modality queries.
*
* The pure-regex classifier in query-intent.ts handles 99% of queries
* cleanly. For the ambiguous middle band (`isAmbiguousModalityQuery`
* returns true), the operator can opt into a Haiku tie-break via
* `search.cross_modal.llm_intent: true`.
*
* Cost bound: <1% of queries when on; ~$0.0001 per escalation. Bypassed
* entirely when the config flag is false (default).
*
* Fail-open: any error (timeout, parse failure, gateway misconfig) returns
* the input fallback ('text') so a misbehaving LLM can never break search.
*/
import type { ModalityMode } from './query-intent.ts';
/** Default model tier for the tie-break call. Haiku 4.5 via utility tier. */
const TIE_BREAK_TIMEOUT_MS = 1000;
const SYSTEM_PROMPT =
'You classify search query modality. Output exactly one word: text, image, or both.\n' +
'- text: the user wants written content (notes, takes, bios, articles)\n' +
'- image: the user wants visual content (photos, screenshots, diagrams)\n' +
'- both: the user is asking something that could be either\n' +
'Output nothing except one of: text image both';
/**
* Run a Haiku tie-break to classify the modality of an ambiguous query.
*
* Returns `fallback` on any error (timeout, parse failure, unrecognized
* output). Production callers gate this on the
* `search.cross_modal.llm_intent` config flag AND
* `isAmbiguousModalityQuery(query) === true` so the LLM call only fires
* for the narrow band where it actually adds signal.
*/
export async function classifyModalityWithLLM(
query: string,
fallback: ModalityMode = 'text',
): Promise<ModalityMode> {
let chat: typeof import('../ai/gateway.ts').chat;
let isAvailable: typeof import('../ai/gateway.ts').isAvailable;
try {
({ chat, isAvailable } = await import('../ai/gateway.ts'));
} catch {
return fallback;
}
if (!isAvailable('chat')) {
// Quiet bail — caller decides whether to warn.
return fallback;
}
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), TIE_BREAK_TIMEOUT_MS);
try {
const result = await chat({
system: SYSTEM_PROMPT,
messages: [{ role: 'user', content: query.slice(0, 500) }],
maxTokens: 16,
abortSignal: controller.signal,
});
return parseModality(result.text, fallback);
} catch {
// Timeout, network error, AI SDK throw — fail-open.
return fallback;
} finally {
clearTimeout(timer);
}
}
/**
* Parse the LLM's single-word output to a ModalityMode. Tolerates
* trailing punctuation + casing. Anything unrecognized → fallback.
*/
export function parseModality(raw: string, fallback: ModalityMode): ModalityMode {
const normalized = raw.trim().toLowerCase().replace(/[^a-z]+/g, '');
if (normalized === 'text' || normalized === 'image' || normalized === 'both') {
return normalized;
}
return fallback;
}
src/core/search/mode.ts
+158
View File
@@ -94,6 +94,7 @@ export interface ModeBundle {
reranker_top_n_out: number | null;
/** HTTP timeout in ms (default 5000). Threaded into gateway.rerank. */
reranker_timeout_ms: number;
/**
* v0.35.6.0 — floor-ratio gate for metadata-axis boost stages (backlink,
* salience, recency). `undefined` = no gate (default for all three modes;
@@ -111,6 +112,56 @@ export interface ModeBundle {
* relevance signal and is NOT gated.
*/
floor_ratio: number | undefined;
// v0.36 cross-modal wave knobs (D2 + D3 + D6 + D8 + D13 + LLM-intent).
// All three mode bundles default these to the same values — cross-modal
// is opt-in per-call (D6 weighting), opt-in per-brain (D8 unified flags),
// and opt-in per-feature-flag (LLM intent). The mode bundle just gives
// resolveSearchMode a default to return.
/**
* D6 'both'-mode RRF weight for text-vector results when merging
* text + image searches in parallel. Defaults to 0.6 — biases toward
* text recall because most queries with ambiguous modality are still
* text-leaning. Pair with cross_modal_both_image_weight.
*/
cross_modal_both_text_weight: number;
/**
* D6 'both'-mode RRF weight for image-vector results. Defaults to 0.4.
* Sum with text weight does NOT need to be 1.0 — RRF is rank-based, so
* weights normalize internally; the ratio is what matters.
*/
cross_modal_both_image_weight: number;
/**
* D13 image-query text-refinement RRF weight for the TEXT branch of
* searchByImage when the caller provides an optional `query` refinement.
* Defaults to 0.4 (image-dominant since the caller chose image-first).
*/
image_query_text_refinement_weight: number;
/**
* D13 image-query refinement RRF weight for the IMAGE branch. Defaults to 0.6.
*/
image_query_image_refinement_weight: number;
/**
* D8 Phase 3 flag: route ALL queries through the multimodal query embed
* + `embedding_multimodal` column. Default false. Operator opt-in after
* `gbrain reindex --multimodal` populates the unified column.
*/
unified_multimodal: boolean;
/**
* D8 Phase 3 strict mode: when true, the dual-column fallback path is
* bypassed entirely. Used by operators who finished re-embedding and
* want to commit to the unified space. Doctor surface errors when this
* is on and coverage < 99%.
*/
unified_multimodal_only: boolean;
/**
* Commit 4: opt-in LLM tie-break for ambiguous modality classification.
* Default false. When true, queries where regex returns 'text' but the
* ambiguity heuristic fires get a Haiku call to refine the classification.
* Fires for <1% of queries when on; ~$0.0001 per escalation.
*/
cross_modal_llm_intent: boolean;
}
/**
@@ -139,6 +190,14 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
// v0.35.6.0 — undefined for all three bundles; the per-corpus ablation
// (TODOS.md) gates any default flip.
floor_ratio: undefined,
// v0.36 cross-modal defaults (same across all modes — opt-in)
cross_modal_both_text_weight: 0.6,
cross_modal_both_image_weight: 0.4,
image_query_text_refinement_weight: 0.4,
image_query_image_refinement_weight: 0.6,
unified_multimodal: false,
unified_multimodal_only: false,
cross_modal_llm_intent: false,
}),
balanced: Object.freeze({
cache_enabled: true,
@@ -164,6 +223,14 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
// v0.35.6.0 — undefined for all three bundles; the per-corpus ablation
// (TODOS.md) gates any default flip.
floor_ratio: undefined,
// v0.36 cross-modal defaults (same across all modes — opt-in)
cross_modal_both_text_weight: 0.6,
cross_modal_both_image_weight: 0.4,
image_query_text_refinement_weight: 0.4,
image_query_image_refinement_weight: 0.6,
unified_multimodal: false,
unified_multimodal_only: false,
cross_modal_llm_intent: false,
}),
tokenmax: Object.freeze({
cache_enabled: true,
@@ -186,6 +253,14 @@ export const MODE_BUNDLES: Readonly<Record<SearchMode, Readonly<ModeBundle>>> =
// v0.35.6.0 — undefined for all three bundles; the per-corpus ablation
// (TODOS.md) gates any default flip.
floor_ratio: undefined,
// v0.36 cross-modal defaults (same across all modes — opt-in)
cross_modal_both_text_weight: 0.6,
cross_modal_both_image_weight: 0.4,
image_query_text_refinement_weight: 0.4,
image_query_image_refinement_weight: 0.6,
unified_multimodal: false,
unified_multimodal_only: false,
cross_modal_llm_intent: false,
}),
});
@@ -219,6 +294,14 @@ export interface SearchKeyOverrides {
reranker_timeout_ms?: number;
// v0.35.6.0 — floor-ratio gate override.
floor_ratio?: number;
// v0.36 cross-modal overrides
cross_modal_both_text_weight?: number;
cross_modal_both_image_weight?: number;
image_query_text_refinement_weight?: number;
image_query_image_refinement_weight?: number;
unified_multimodal?: boolean;
unified_multimodal_only?: boolean;
cross_modal_llm_intent?: boolean;
}
/**
@@ -244,6 +327,14 @@ export interface SearchPerCallOpts {
reranker_timeout_ms?: number;
// v0.35.6.0 — floor-ratio per-call override.
floor_ratio?: number;
// v0.36 cross-modal per-call overrides
cross_modal_both_text_weight?: number;
cross_modal_both_image_weight?: number;
image_query_text_refinement_weight?: number;
image_query_image_refinement_weight?: number;
unified_multimodal?: boolean;
unified_multimodal_only?: boolean;
cross_modal_llm_intent?: boolean;
}
/**
@@ -304,6 +395,14 @@ export function resolveSearchMode(input: ResolveSearchModeInput): ResolvedSearch
reranker_timeout_ms: pick('reranker_timeout_ms'),
// v0.35.6.0 — floor-ratio resolved via the same pick chain.
floor_ratio: pick('floor_ratio'),
// v0.36 cross-modal knobs
cross_modal_both_text_weight: pick('cross_modal_both_text_weight'),
cross_modal_both_image_weight: pick('cross_modal_both_image_weight'),
image_query_text_refinement_weight: pick('image_query_text_refinement_weight'),
image_query_image_refinement_weight: pick('image_query_image_refinement_weight'),
unified_multimodal: pick('unified_multimodal'),
unified_multimodal_only: pick('unified_multimodal_only'),
cross_modal_llm_intent: pick('cross_modal_llm_intent'),
resolved_mode,
mode_valid: valid,
};
@@ -366,6 +465,12 @@ export function attributeKnob<K extends keyof ModeBundle>(
// row IDs. Expect a temporary hit-rate dip + cache-row doubling for hot
// queries during a rolling deploy. Clears naturally within
// `cache.ttl_seconds` (default 3600s). The CHANGELOG note covers this.
//
// v0.36 wave: cross-modal knobs ALSO participate in v=3 hash (D2 cache
// contamination fix — a text-mode cache hit cannot silently serve an
// image-mode caller). v0.35.6.0's floor_ratio bump and v0.36's cross-modal
// extensions both land under v=3, with cross-modal fields appended after
// the floor_ratio entry (CDX2-F13 append-only convention).
export const KNOBS_HASH_VERSION = 3;
/**
@@ -420,6 +525,17 @@ export function knobsHash(
// — they sit in different vector spaces. ctx is optional so
// unrelated callers fall back to the default-column hash.
`fr=${knobs.floor_ratio === undefined ? 'none' : knobs.floor_ratio.toFixed(4)}`,
// v=3 cross-modal additions (append-only).
`cmbt=${knobs.cross_modal_both_text_weight.toFixed(2)}`,
`cmbi=${knobs.cross_modal_both_image_weight.toFixed(2)}`,
`iqt=${knobs.image_query_text_refinement_weight.toFixed(2)}`,
`iqi=${knobs.image_query_image_refinement_weight.toFixed(2)}`,
`um=${knobs.unified_multimodal ? 1 : 0}`,
`umo=${knobs.unified_multimodal_only ? 1 : 0}`,
`lli=${knobs.cross_modal_llm_intent ? 1 : 0}`,
// v=3 column + provider additions (D8 / CDX-2): cross-column +
// cross-provider cache isolation. A query against `embedding_voyage`
// must never be served from a row that ran against `embedding`.
`col=${ctx?.embeddingColumn ?? 'embedding'}`,
`prov=${ctx?.embeddingModel ?? 'default'}`,
];
@@ -519,6 +635,40 @@ export function loadOverridesFromConfig(
if (Number.isFinite(n) && n >= 0 && n <= 1) out.floor_ratio = n;
}
// v0.36 cross-modal overrides (D3 registry)
const cmbt = get('search.cross_modal.both_mode_text_weight');
if (cmbt !== undefined) {
const n = parseFloat(cmbt);
if (Number.isFinite(n) && n >= 0) out.cross_modal_both_text_weight = n;
}
const cmbi = get('search.cross_modal.both_mode_image_weight');
if (cmbi !== undefined) {
const n = parseFloat(cmbi);
if (Number.isFinite(n) && n >= 0) out.cross_modal_both_image_weight = n;
}
const iqt = get('search.image_query.text_refinement_weight');
if (iqt !== undefined) {
const n = parseFloat(iqt);
if (Number.isFinite(n) && n >= 0) out.image_query_text_refinement_weight = n;
}
const iqi = get('search.image_query.image_refinement_weight');
if (iqi !== undefined) {
const n = parseFloat(iqi);
if (Number.isFinite(n) && n >= 0) out.image_query_image_refinement_weight = n;
}
const um = get('search.unified_multimodal');
if (um !== undefined) {
out.unified_multimodal = um === '1' || um.toLowerCase() === 'true';
}
const umo = get('search.unified_multimodal_only');
if (umo !== undefined) {
out.unified_multimodal_only = umo === '1' || umo.toLowerCase() === 'true';
}
const lli = get('search.cross_modal.llm_intent');
if (lli !== undefined) {
out.cross_modal_llm_intent = lli === '1' || lli.toLowerCase() === 'true';
}
return out;
}
@@ -539,6 +689,14 @@ export const SEARCH_MODE_CONFIG_KEYS: ReadonlyArray<string> = Object.freeze([
'search.reranker.timeout_ms',
// v0.35.6.0 — floor-ratio gate
'search.floor_ratio',
// v0.36 cross-modal keys (D3)
'search.cross_modal.both_mode_text_weight',
'search.cross_modal.both_mode_image_weight',
'search.image_query.text_refinement_weight',
'search.image_query.image_refinement_weight',
'search.unified_multimodal',
'search.unified_multimodal_only',
'search.cross_modal.llm_intent',
]);
/**
src/core/search/query-intent.ts
+90 -1
View File
@@ -29,6 +29,20 @@ export type QueryIntent = 'entity' | 'temporal' | 'event' | 'general';
export type SalienceMode = 'off' | 'on' | 'strong';
export type RecencyMode = 'off' | 'on' | 'strong';
/**
* v0.36 cross-modal wave: modality axis (D6).
*
* - 'text' (default): existing text-embedding path, no behavior change
* - 'image': route through the multimodal model + embedding_image column
* (visually-similar matching + image OCR text)
* - 'both': run text + image searches in parallel and merge via
* weighted RRF (recall-leaning when the query is ambiguous)
*
* Parallel axis to intent/detail/salience/recency. Returned by
* classifyQuery from one regex pass over the query.
*/
export type ModalityMode = 'text' | 'image' | 'both';
export interface QuerySuggestions {
intent: QueryIntent;
/** v0.29.0 detail mapping. entity→low, temporal/event→high, general→undefined. */
@@ -37,6 +51,8 @@ export interface QuerySuggestions {
suggestedSalience: SalienceMode;
/** v0.29.1 — per-prefix age-decay boost. */
suggestedRecency: RecencyMode;
/** v0.36 — cross-modal routing axis. Defaults to 'text' when nothing matches. */
suggestedModality: ModalityMode;
}
// ─────────────────────────────────────────────────────────
@@ -164,6 +180,46 @@ const SALIENCE_ON_PATTERNS = [
/\bwhat'?s\s+important\b/i,
];
// v0.36 cross-modal wave — modality-axis patterns (D6).
//
// CROSS_MODAL_PATTERNS fires the 'image' modality when the query explicitly
// names visual artifacts ("show me photos", "find images of", "screenshot of",
// "what does X look like", "diagram of"). Module-scope const so regexes
// compile once at module load (D15).
//
// Conservative on purpose — false positives cost "one cheaper image search
// where text might have worked." False negatives cost nothing (the legacy
// text path still runs). The LLM-intent escalation in Commit 4 catches
// genuinely ambiguous phrasings.
const CROSS_MODAL_PATTERNS: RegExp[] = [
/\b(show|find|get|pull)\s+(me\s+)?(the\s+)?(photos?|images?|pictures?|pics?|screenshots?)\b/i,
/\b(photos?|images?|pictures?|pics?|screenshots?)\s+(of|from|at|with|showing|featuring)\b/i,
/\bwhat\s+does\s+[\w\s']{1,40}?\s+look\s+like\b/i,
/\b(whiteboard|diagram|slide|screenshot|infographic|chart)s?\s+(of|from|about|showing)\b/i,
/\bdiagram\s+(of|for|showing)\b/i,
/\bvisual(s|ly)?\s+(of|from|about|showing|representation)\b/i,
];
// v0.36 cross-modal wave (Commit 4 prep): visual nouns that combined with
// ambiguous-pronoun phrasings ("any pics from last week's offsite?") trigger
// the optional LLM intent escalation. Subset of cross-modal patterns plus
// looser noun-form matches.
const AMBIGUOUS_MODALITY_NOUNS: RegExp[] = [
/\b(photo|image|picture|pic|screenshot|diagram|whiteboard|slide|chart)s?\b/i,
/\blook(s|ed)?\s+like\b/i,
/\bvisual(s|ly)?\b/i,
];
// Pronoun + filler markers that signal "the user is referencing something
// they can't quite name" — combined with AMBIGUOUS_MODALITY_NOUNS, triggers
// the LLM tie-break in Commit 4.
const AMBIGUOUS_REFERENCE_MARKERS: RegExp[] = [
// Match all the visual nouns (pic/pics, picture/pictures, photo/photos, image/images,
// screenshot/screenshots, diagram/diagrams, whiteboard/whiteboards, slide/slides, chart/charts).
/\b(any|some|that|those|these|the)\s+(pic|pics|picture|pictures|photo|photos|image|images|screenshot|screenshots|diagram|diagrams|whiteboard|whiteboards|slide|slides|chart|charts)\b/i,
/\bfrom\s+(last|this|the)\s+(week|month|year|offsite|meeting|hackathon|deck)\b/i,
];
// ─────────────────────────────────────────────────────────
// Classifier
// ─────────────────────────────────────────────────────────
@@ -221,7 +277,40 @@ export function classifyQuery(query: string): QuerySuggestions {
suggestedSalience = 'off';
}
return { intent, suggestedDetail, suggestedSalience, suggestedRecency };
// v0.36 cross-modal — modality axis. Independent of intent/detail/salience/recency.
// Conservative default 'text'; only flips to 'image' on explicit cross-modal regex match.
// 'both' is reserved for explicit per-call opts (LLM-intent escalation in Commit 4
// can also produce 'both' via tie-break).
const suggestedModality: ModalityMode = matches(CROSS_MODAL_PATTERNS, query) ? 'image' : 'text';
return { intent, suggestedDetail, suggestedSalience, suggestedRecency, suggestedModality };
}
/**
* v0.36 — heuristic gate for the optional LLM intent escalation (Commit 4).
*
* Fires when the query contains a visual noun ("any pics", "the diagram",
* "what does it look like") combined with an ambiguous reference marker
* ("from last week's offsite"). These are the phrasings the conservative
* regex misses but a Haiku tie-break catches.
*
* Returns false for unambiguous text queries (no LLM call burned). Returns
* false for queries the regex ALREADY caught (no need to tie-break a
* confident classification). Returns true only for the narrow band where
* the LLM call earns its $0.0001 cost.
*
* Pure function. No LLM call. No DB access. Used by hybridSearch's
* escalation branch only when `search.cross_modal.llm_intent: true`.
*/
export function isAmbiguousModalityQuery(query: string): boolean {
// Already-confident classification → no LLM needed.
if (matches(CROSS_MODAL_PATTERNS, query)) return false;
const hasVisualNoun = matches(AMBIGUOUS_MODALITY_NOUNS, query);
if (!hasVisualNoun) return false;
const hasReferenceMarker = matches(AMBIGUOUS_REFERENCE_MARKERS, query);
return hasReferenceMarker;
}
// ─────────────────────────────────────────────────────────
src/core/spend-log.ts
+123
View File
@@ -0,0 +1,123 @@
/**
* v0.36 Phase 2 (D23-#6) — per-OAuth-client paid-API spend tracking.
*
* Backs the daily-budget gate for `search_by_image`. Each successful Voyage
* multimodal call records an entry; before any new call, `checkBudget`
* sums today's spend and rejects when it exceeds the configured cap.
*
* Config: `search.image_query.daily_budget_usd_per_client` (default $5).
*
* Scope: ONLY fires when `ctx.remote === true`. Local CLI callers have
* direct billing visibility through their own credentials and don't need
* the gate. The gate exists to prevent a misbehaving OAuth client from
* burning the brain operator's Voyage account.
*/
import type { BrainEngine } from './engine.ts';
import { sqlQueryForEngine } from './sql-query.ts';
/** Per-call Voyage multimodal-3 spend estimate (per image), in cents. */
export const VOYAGE_MULTIMODAL_3_PER_IMAGE_CENTS = 0.12;
export class BudgetExceededError extends Error {
readonly code = 'BUDGET_EXCEEDED' as const;
readonly spentCents: number;
readonly capCents: number;
constructor(message: string, spentCents: number, capCents: number) {
super(message);
this.name = 'BudgetExceededError';
this.spentCents = spentCents;
this.capCents = capCents;
}
}
/**
* Sum today's recorded spend for a client. Returns 0 if the row count
* is zero (new client) OR if the table doesn't exist (pre-v0.36 brain).
*/
export async function getTodaySpendCents(
engine: BrainEngine,
clientId: string,
): Promise<number> {
try {
const sql = sqlQueryForEngine(engine);
const rows = await sql`
SELECT COALESCE(SUM(spend_cents), 0)::text AS total
FROM mcp_spend_log
WHERE client_id = ${clientId}
AND created_at >= ${todayStartIso()}
`;
const total = parseFloat(String(rows[0]?.total ?? '0'));
return Number.isFinite(total) ? total : 0;
} catch {
// Table doesn't exist (pre-v0.36 brain) or DB hiccup — fail-open to 0.
// The check sees "no spend recorded" and lets the call through. A real
// production brain will have the table once migrations apply.
return 0;
}
}
/**
* Pre-flight budget gate.
*
* Throws `BudgetExceededError` when the client has already spent at or above
* the configured daily cap. Returns silently when there's room.
*
* @param dailyBudgetCents resolved from `search.image_query.daily_budget_usd_per_client`
* × 100 (converted to cents). Operator config; default 500 cents = $5.
*/
export async function checkBudget(
engine: BrainEngine,
clientId: string,
dailyBudgetCents: number,
): Promise<void> {
if (!clientId) return; // local CLI callers (no client_id) bypass the gate
if (dailyBudgetCents <= 0) return; // 0 = "no cap" sentinel
const spent = await getTodaySpendCents(engine, clientId);
if (spent >= dailyBudgetCents) {
throw new BudgetExceededError(
`Daily Voyage spend cap reached: $${(spent / 100).toFixed(2)} >= $${(dailyBudgetCents / 100).toFixed(2)}. ` +
`Reset at midnight UTC.`,
spent,
dailyBudgetCents,
);
}
}
/**
* Record a successful paid call.
*
* Best-effort: writes failures (e.g., table not yet migrated) are swallowed
* with a stderr warning. The search itself succeeded — we don't want to
* fail the user's call because spend telemetry hiccupped.
*/
export async function recordSpend(
engine: BrainEngine,
entry: {
clientId?: string | null;
tokenName?: string | null;
operation: string;
spendCents: number;
provider?: string;
model?: string;
},
): Promise<void> {
try {
const sql = sqlQueryForEngine(engine);
await sql`
INSERT INTO mcp_spend_log (client_id, token_name, operation, spend_cents, provider, model)
VALUES (${entry.clientId ?? null}, ${entry.tokenName ?? null}, ${entry.operation}, ${entry.spendCents}, ${entry.provider ?? null}, ${entry.model ?? null})
`;
} catch (err) {
const msg = err instanceof Error ? err.message : String(err);
console.error(`[spend-log] failed to record spend: ${msg}`);
}
}
function todayStartIso(): string {
const now = new Date();
// UTC day-start so the cap rolls over deterministically regardless of
// server timezone. Operators reading dashboards see UTC days.
const utcMidnight = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate()));
return utcMidnight.toISOString();
}
src/core/ssrf-validate.ts
+258
View File
@@ -0,0 +1,258 @@
/**
* SSRF validation with DNS resolution — closes the rebinding gap that
* `isInternalUrl` in `src/core/url-safety.ts` leaves open.
*
* url-safety.ts covers static SSRF defense (IPv4-mapped IPv6, hex/octal IP
* forms, IPv6 ULA + link-local, metadata hostnames, CGNAT, scheme allowlist).
* Codex's outside-voice review of the cross-modal wave (D19) flagged the
* remaining gap: an attacker-controlled hostname can resolve to a public IP
* at validation time and a private IP at fetch time (DNS rebinding). The
* defense is: resolve once at validation, inspect every A/AAAA record, and
* fetch by the resolved IP — not the hostname.
*
* This module is consumed by `src/core/search/image-loader.ts` (Phase 2 of
* the cross-modal wave) and is reusable for any future URL-fetching feature.
*
* Two-layer defense per call:
* 1. Static check via `isInternalUrl` — fails fast on obvious internal hosts
* 2. DNS resolve via `dns.lookup({all: true, family: 0})` — fails on any
* resolved A/AAAA record that points internal
*
* The caller fetches using the returned `resolvedIp`, not the original
* hostname, so a second DNS lookup at fetch time can't rebind to internal.
*/
import { lookup as nodeDnsLookup } from 'node:dns/promises';
import { isInternalUrl, isPrivateIpv4, hostnameToOctets } from './url-safety.ts';
// Module-level seam so tests can swap DNS resolution without `mock.module`
// (which is banned in non-serial unit tests per scripts/check-test-isolation.sh R2).
type DnsLookupFn = typeof nodeDnsLookup;
let _dnsLookup: DnsLookupFn = nodeDnsLookup;
/** @internal Test-only — swap the DNS resolver. Restore with `__setDnsLookupForTests(undefined)`. */
export function __setDnsLookupForTests(fn: DnsLookupFn | undefined): void {
_dnsLookup = fn ?? nodeDnsLookup;
}
export interface ResolvedTarget {
/** The URL the caller should fetch — host is replaced with the resolved IP. */
resolvedUrl: string;
/** The IP address resolved from the original hostname. */
resolvedIp: string;
/** The original hostname (for Host: header). Empty when input was already an IP literal. */
originalHost: string;
/** Whether the resolved IP is IPv6 — affects URL bracket encoding. */
ipv6: boolean;
}
export class SSRFError extends Error {
readonly code: SSRFErrorCode;
constructor(code: SSRFErrorCode, message: string) {
super(message);
this.name = 'SSRFError';
this.code = code;
}
}
export type SSRFErrorCode =
| 'INTERNAL_HOST'
| 'INVALID_URL'
| 'INVALID_SCHEME'
| 'CREDENTIALS_IN_URL'
| 'DNS_RESOLUTION_FAILED'
| 'DNS_RESOLVED_INTERNAL'
| 'SSRF_REDIRECT_DENIED'
| 'SSRF_HOP_LIMIT';
/**
* Validate a URL against SSRF policy and resolve its hostname to an IP.
*
* Returns a `ResolvedTarget` the caller should use for the actual fetch.
* Throws `SSRFError` on any policy violation.
*
* Defends against:
* - Static internal targets (RFC1918, loopback, link-local, ULA, metadata hostnames, CGNAT)
* - Non-http(s) schemes
* - Credentials embedded in URL (`http://user:pass@host/`)
* - DNS rebinding (resolves all records, blocks if any are internal)
* - Non-resolving hosts (caller can't fetch them anyway)
*/
export async function validateAndResolveUrl(urlStr: string): Promise<ResolvedTarget> {
let url: URL;
try {
url = new URL(urlStr);
} catch {
throw new SSRFError('INVALID_URL', `Malformed URL: ${truncate(urlStr)}`);
}
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
throw new SSRFError('INVALID_SCHEME', `Unsupported scheme ${url.protocol}; only http(s) allowed`);
}
if (url.username || url.password) {
throw new SSRFError('CREDENTIALS_IN_URL', 'Credentials embedded in URL are not permitted');
}
// Layer 1: static check covers IPv4 hex/octal/single-int, IPv6 ULA + link-local,
// metadata hostnames, CGNAT, IPv4-mapped IPv6.
if (isInternalUrl(urlStr)) {
throw new SSRFError('INTERNAL_HOST', `URL targets internal/private network: ${truncate(urlStr)}`);
}
let host = url.hostname;
if (host.startsWith('[') && host.endsWith(']')) host = host.slice(1, -1);
// If the host is already an IP literal, isInternalUrl already validated it.
// Skip DNS lookup and return the literal as-is.
if (isIpLiteral(host)) {
return {
resolvedUrl: urlStr,
resolvedIp: host,
originalHost: '',
ipv6: host.includes(':'),
};
}
// Layer 2: DNS resolution. {all: true, family: 0} returns every A AND AAAA
// record. If ANY record points internal, reject.
let addrs: Array<{ address: string; family: number }>;
try {
addrs = await _dnsLookup(host, { all: true, family: 0 });
} catch (err) {
throw new SSRFError(
'DNS_RESOLUTION_FAILED',
`Failed to resolve ${host}: ${err instanceof Error ? err.message : String(err)}`,
);
}
if (addrs.length === 0) {
throw new SSRFError('DNS_RESOLUTION_FAILED', `No DNS records for ${host}`);
}
for (const a of addrs) {
if (isAddressInternal(a.address, a.family)) {
throw new SSRFError(
'DNS_RESOLVED_INTERNAL',
`${host} resolves to internal address ${a.address} (DNS rebinding attempt?)`,
);
}
}
// Pick the first resolved address (system-ordered: typically the preferred
// family). Caller fetches by this IP so a second DNS lookup can't rebind.
const chosen = addrs[0];
const isV6 = chosen.family === 6;
const hostInUrl = isV6 ? `[${chosen.address}]` : chosen.address;
// Rebuild URL with the resolved host. Preserve the original `host` for the
// Host: header (caller can set it explicitly when fetching).
const rebuilt = new URL(urlStr);
rebuilt.hostname = hostInUrl;
return {
resolvedUrl: rebuilt.toString(),
resolvedIp: chosen.address,
originalHost: host,
ipv6: isV6,
};
}
function isIpLiteral(host: string): boolean {
if (host.includes(':')) return true; // IPv6 literal (already bracket-stripped)
return hostnameToOctets(host) !== null;
}
function isAddressInternal(addr: string, family: number): boolean {
if (family === 4) {
const octets = hostnameToOctets(addr);
return octets ? isPrivateIpv4(octets) : true; // fail-closed on parse failure
}
if (family === 6) {
const lower = addr.toLowerCase();
if (lower === '::1' || lower === '::') return true;
if (/^f[cd][0-9a-f]{2}:/.test(lower)) return true; // ULA fc00::/7
if (/^fe[89ab][0-9a-f]:/.test(lower)) return true; // link-local fe80::/10
if (lower.startsWith('::ffff:')) {
const tail = lower.slice(7);
const dotted = hostnameToOctets(tail);
if (dotted && isPrivateIpv4(dotted)) return true;
}
return false;
}
return true; // unknown family — fail-closed
}
/**
* Fetch a URL with full SSRF protection including per-redirect-hop validation.
*
* On every Location response header, the new URL is re-validated via
* `validateAndResolveUrl` — fresh DNS resolution per hop defeats rebinding
* across the redirect chain. Max 3 hops by default.
*
* Returns the final Response. Caller is responsible for body size limits
* (use `init.signal` to abort, or check `Content-Length` before consuming).
*/
export async function fetchWithSSRFGuard(
urlStr: string,
init: RequestInit & {
maxRedirects?: number;
timeoutMs?: number;
} = {},
): Promise<Response> {
const maxRedirects = init.maxRedirects ?? 3;
const timeoutMs = init.timeoutMs ?? 5000;
const controller = new AbortController();
const externalSignal = init.signal;
const onAbort = () => controller.abort();
if (externalSignal) {
if (externalSignal.aborted) controller.abort();
else externalSignal.addEventListener('abort', onAbort, { once: true });
}
const timer = setTimeout(() => controller.abort(), timeoutMs);
try {
let currentUrl = urlStr;
let hops = 0;
while (true) {
const target = await validateAndResolveUrl(currentUrl);
const fetchInit: RequestInit = {
...init,
redirect: 'manual',
signal: controller.signal,
};
// Set Host header to the original hostname so SNI/TLS works correctly
// (we're fetching by resolved IP but the server expects the real host).
const headers = new Headers(init.headers || {});
if (target.originalHost) {
headers.set('Host', target.originalHost);
}
fetchInit.headers = headers;
const res = await fetch(target.resolvedUrl, fetchInit);
// Redirect status codes
if ([301, 302, 303, 307, 308].includes(res.status)) {
if (hops >= maxRedirects) {
throw new SSRFError('SSRF_HOP_LIMIT', `Exceeded ${maxRedirects} redirect hops`);
}
const location = res.headers.get('location');
if (!location) {
return res; // redirect with no Location — return as-is, caller decides
}
// Resolve relative location against current URL
const next = new URL(location, currentUrl).toString();
currentUrl = next;
hops++;
continue;
}
return res;
}
} finally {
clearTimeout(timer);
if (externalSignal) externalSignal.removeEventListener('abort', onAbort);
}
}
function truncate(s: string, n = 200): string {
return s.length > n ? s.slice(0, n) + '...' : s;
}
src/core/types.ts
+40 -28
View File
@@ -404,6 +404,15 @@ export interface SearchResult {
chunk_index: number;
score: number;
stale: boolean;
/**
* v0.36 (cross-modal wave): the chunk's modality discriminator from
* content_chunks.modality. 'text' for the existing text-embedding rows,
* 'image' for rows populated by importImageFile. Surfaced so callers /
* renderers can distinguish text matches from image matches in `'both'`
* mode results. Optional for back-compat with engines that don't project
* the column (defaults to 'text' in renderers when absent).
*/
modality?: 'text' | 'image';
/**
* v0.18.0: the sources.id the page belongs to. Dedup composite-keys
* on (source_id, slug) — see src/core/search/dedup.ts. Defaults to
@@ -541,8 +550,10 @@ export interface SearchOpts {
* 1. String name (legacy + user-facing). Engine and hybridSearch convert
* to ResolvedColumn at the boundary via `resolveEmbeddingColumn()`.
* Built-in names: 'embedding' (default, text), 'embedding_image'
* (multimodal). Custom user-declared names also accepted when
* registered in `embedding_columns` config.
* (multimodal). v0.36 cross-modal wave adds 'embedding_multimodal'
* (unified column populated by `gbrain reindex --multimodal`). Custom
* user-declared names also accepted when registered in
* `embedding_columns` config.
*
* 2. ResolvedColumn descriptor (internal). The engine ONLY accepts
* this shape — hybridSearch resolves once at entry and passes the
@@ -556,7 +567,7 @@ export interface SearchOpts {
* searchKeyword is unaffected — modality filtering on the keyword path
* is independent.
*/
embeddingColumn?: 'embedding' | 'embedding_image' | string | ResolvedColumn;
embeddingColumn?: 'embedding' | 'embedding_image' | 'embedding_multimodal' | string | ResolvedColumn;
/**
* @deprecated v0.29.1: use `since` instead. Removed in v0.30.
* v0.27.0: filter results to pages updated/created after this date. ISO-8601 string.
@@ -636,33 +647,34 @@ export interface SearchOpts {
rerankerFn?: (input: { query: string; documents: string[]; topN?: number; model?: string; signal?: AbortSignal; timeoutMs?: number }) => Promise<{ index: number; relevanceScore: number }[]>;
};
/**
* v0.35.6.0 — floor-ratio gate for metadata-axis boost stages (backlink,
* salience, recency). Number in [0, 1] or undefined (default = no gate).
*
* When set, each gated stage skips results whose pre-boost score is below
* `floorRatio * topScore`, where `topScore` is computed ONCE at
* `runPostFusionStages` entry from the post-cosine-rescore snapshot. The
* same threshold gates all three stages — order-independent semantic.
*
* Resolution chain (mirrors other search-lite knobs):
* per-call `SearchOpts.floorRatio` → config `search.floor_ratio`
* → MODE_BUNDLES[mode].floor_ratio (undefined for all 3 modes today)
* → undefined fallback.
*
* SCOPE: gates ONLY the three metadata stages. Exact-match boost
* (`applyExactMatchBoost` in intent-weights.ts) runs independently as a
* lexical-relevance signal and is NOT gated by design.
*
* Sensible operator override values for dense-embedder corpora: 0.85-0.95.
* Default stays undefined pending per-corpus ablation evidence (see
* `TODOS.md` floor-ratio ablation entry).
*
* Out-of-range values (negative, > 1, NaN, Infinity) silently disable
* the gate at the runtime layer; the config-parse layer also rejects
* out-of-range values. Defense in depth — a malformed value never
* gates anything.
* v0.35.6.0 — floor-ratio gate for metadata-axis boost stages.
* Number in [0, 1] or undefined (default = no gate). When set, each gated
* stage skips results whose pre-boost score is below `floorRatio * topScore`.
* Same threshold gates all three metadata stages; exact-match boost runs
* independently. Out-of-range values silently disable the gate.
* Sensible operator overrides for dense-embedder corpora: 0.85-0.95.
*/
floorRatio?: number;
/**
* v0.36 cross-modal wave: route this search through the multimodal
* embedding space (Voyage multimodal-3 by default).
*
* - 'text' (default for queries that don't match image-intent regex):
* existing text-embedding path. No behavior change vs pre-v0.36.
* - 'image': force routing through the multimodal model + embedding_image
* column. Skip LLM expansion (image embeddings handle synonyms in-space)
* and skip keyword search (no FTS index on image content).
* - 'both': run text and image vector searches in parallel; merge via
* modality-weighted RRF.
* - 'auto' (literal): same effect as undefined — let intent classifier
* decide. Accepted on the wire so MCP callers can be explicit.
*
* Cross-modal override matrix (D9): when effective modality is 'image',
* cross-modal path overrides expansion (false) and reranker (false)
* regardless of mode bundle. zerank-2 can't rerank image embeddings;
* sending them produces garbage scores.
*/
crossModal?: 'text' | 'image' | 'both' | 'auto';
}
/**