diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index ac0b675cf..c40e666c9 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -191,7 +191,10 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling. - `src/commands/enrich.ts` + `src/core/enrich/thin.ts` + `src/core/cycle/enrich-thin.ts` — `gbrain enrich --thin`: batch-develops stub (thin) pages via **brain-internal grounded synthesis**. gbrain's model tooling sees only brain-internal context (search / get_page / facts / backlinks), not the web, so enrich consolidates what the brain ALREADY knows about an entity (scattered across meetings, other pages, deals, facts) into one cited page via ONE `gateway.chat` call per page; web research stays the agent-driven `enrich` SKILL's job. `runEnrichCore(engine, opts, signal)` (strict per-source; multi-source iteration is the caller's job) drives `enrichOne` per candidate: `withRefreshingLock('enrich::')` → `getPage` → deterministic retrieve (hybridSearch + getBacklinks + facts + raw_data, source-scoped, sanitized via `INJECTION_PATTERNS`) → `assessGrounding` gate (skip < `MIN_CONTEXT_CHARS`, no LLM) → `buildEnrichPrompt` (grounded dossier, `[Source: slug]` citations, SKIP sentinel) → synth → `put_page` handler (`remote:false`, auto-link + write-through) stamping `enriched_at` + `enriched_by:'cli:enrich'`. Candidate selection is the SQL-native `engine.listEnrichCandidates(opts)` (`src/core/engine.ts` interface + `EnrichCandidate`/`EnrichCandidatesOpts`/`ENRICH_ORDER_SQL` in `src/core/types.ts` + pg/pglite impls): thin-filter + per-page source-correct inbound count (`to_page_id = p.id`, `mentions` excluded) + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, lightweight projection (NO bodies). Resume via `src/core/op-checkpoint.ts` (local `enrichFingerprint`); budget via `BudgetTracker` + `withBudgetTracker` (best-effort under `--workers > 1` — `runSlidingPool` aborts new claims on `BUDGET_EXHAUSTED` but does NOT cancel in-flight `gateway.chat`; pin `--workers 1` for a hard ceiling). `sanitizeContext` (thin.ts) neutralizes the `…` data-envelope delimiters (injection escape, mirrors the `` convention); the `--background` multi-source fan-out idempotency key carries the run fingerprint via exported `backgroundIdempotencyKey(sid, args)` (a bare `enrich:${sid}` would return stale completed jobs); `runEnrichCore` flags `budget_exhausted` post-hoc when `tracker.totalSpent > tracker.cap` even when the gateway swallowed the final-call throw (via read-only `BudgetTracker.cap` getter); `body()` flushes the checkpoint on `BudgetExhausted` before it propagates so resume doesn't re-charge. The opt-in `enrich_thin` cycle phase (default OFF via `cycle.enrich_thin.enabled`) trickles `max_pages_per_tick` (default 3) per source with per-source cost cap enforced as `min(per_source_cap, brain_wide_remaining)` + brain-wide total + walltime caps. Wired into `cycle.ts` (`CyclePhase`/`ALL_PHASES` between `conversation_facts_backfill` and `skillopt`/`embed`; `PHASE_SCOPE='source'`; `NEEDS_LOCK`; dispatch), `cli.ts` (`CLI_ONLY` + `CLI_ONLY_SELF_HELP` + `THIN_CLIENT_REFUSED_COMMANDS` + dispatch), `jobs.ts` (Minion `enrich` handler, strict per-source, NOT in `PROTECTED_JOB_NAMES`). DI seam `opts.synthesizeFn` keeps tests hermetic (no API key, no mock.module). Pinned by `test/enrich/thin.test.ts`, `test/enrich/idempotency.test.ts`, `test/enrich-cycle-phase.test.ts`, `test/e2e/enrich-pglite.test.ts` (grew-cited, skip, ordering, multi-source, recency, resume, budget abort + checkpoint flush, final-call overage, lock-skip, provenance), `test/e2e/engine-parity.test.ts` (`listEnrichCandidates` pg↔pglite parity). - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping. -- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. +- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed ` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. `--include-null-signature` (#3391) lifts the NULL-signature grandfather clause: threads `includeNullSignature: true` into the invalidation + counts so pages that predate the v108 stamp re-embed too after a model swap (both engines' `countStaleChunks`/`sumStaleChunkChars`/`invalidateStaleSignatureEmbeddings` accept the flag; predicate becomes `sig IS NULL OR sig <> current`). Without the flag, a live stale run that just invalidated drifted rows probes for left-behind NULL-signature chunks and emits a loud stderr warning naming the count + the fix — mixed embedding spaces in one index are never silent. Pinned by `test/embedding-migration.test.ts` + `test/e2e/migrate-embeddings-postgres.test.ts`. +- `src/core/retrieval-upgrade-planner.ts` — `runSchemaTransition(engine, targetDim)` (exported) is the ONE atomic dimension-transition path, shared by `ze-switch` and `gbrain migrate embeddings`. In a single transaction it rebuilds ALL THREE dim-pinned text-embedding-space columns at `targetDim` — `content_chunks.embedding`, `query_cache.embedding`, `facts.embedding` — preserving each column's declared type (`vector` vs `halfvec`, probed from `information_schema`) and recreating its HNSW index with the matching opclass, gated on `hnswIndexExpected` (above the per-type dim ceiling pgvector refuses the index and exact scans remain the path). query_cache + facts are created at brain-birth width by `migrate.ts` and NO migration ever ALTERs them, so omitting either leaves it silently broken: a narrow `query_cache.embedding` makes every `store()`/`lookup()` fail inside the cache's own error-swallowing (permanent 0% hit rate), and a narrow `facts.embedding` fails every per-fact embed write (the doctor check that would warn is skipped on PGLite, the default engine). `content_chunks.embedding_image` / `embedding_multimodal` are the deliberate exception — separate multimodal models, dimensions independent of the text model. Pinned by `test/embedding-migration.test.ts` (all three widths + a real INSERT at the new width into each) and `test/e2e/migrate-embeddings-postgres.test.ts`. +- `src/core/embedding-migration.ts` — provider-agnostic embedding migration core (#3390): `planEmbeddingMigration` (workload counts via the widened stale predicates with the TARGET signature + `includeNullSignature`, so a mid-migration re-plan counts only what remains; cost via `embedding-pricing.ts`; `null_signature_chunks` split out for #3391 visibility; reranker-on-outgoing-provider warning), `applyEmbeddingMigration` (env-override gate BEFORE any mutation → in-flight state marker `embedding_migration.state` → `runSchemaTransition` when the ACTUAL column width differs from target → DB-plane `embedding_model`/`embedding_dimensions` → `persistConfig` callback for the file plane → `invalidateStaleSignatureEmbeddings({includeNullSignature: true})` → `SemanticQueryCache.clear()`), `completeEmbeddingMigration` (clears the marker + stamps `embedding_migration.completed`; call only at zero backlog), `resolveMigrationTarget` (validates `provider:model` via `resolveRecipe`, dims via `embeddingDimsForModel` or explicit `--dim`), `migrationSignature` (matches `currentEmbeddingSignature()` shape). Engine-pure; every step idempotent under crash + re-run — the NULL-embedding column is the checkpoint. Reuses `runSchemaTransition` (now exported from `retrieval-upgrade-planner.ts`) so ze-switch and the migration share ONE dimension-transition path. `reconcilePageSignatures(engine, plan)` runs after the re-embed drain and BEFORE the completion probe: it stamps the target signature on every page that has zero NULL-embedding chunks, covering pages whose chunks straddle a `listStaleChunks` batch boundary (the embed loop only stamps when `stale.length === existing.length`, so a split page is embedded correctly but never stamped — without the reconcile a >1-batch brain reports "incomplete" and the re-run re-invalidates and re-pays for those pages). Sound only because apply() invalidated everything not already in the target space; pages with a remaining NULL chunk stay unstamped so a real embed failure still surfaces. Invalidation is ordered BEFORE the config writes so a crash on a same-dim swap leaves rows merely stale (empty results) rather than new-space queries scored against old-space vectors (silently wrong). Pinned by `test/embedding-migration.test.ts` (PGLite) + `test/e2e/migrate-embeddings-postgres.test.ts` (real pgvector). +- `src/commands/migrate-embeddings.ts` — `gbrain migrate embeddings --to [--dim N] [--dry-run] [--yes] [--json] [--no-embed] [--pace[=mode]] [--ignore-env-override]` (alias: `gbrain retrieval-upgrade`, the command README/doctor promised since v0.36). Flow: plan → render (stderr when `--json` so stdout stays JSON-clean) → consent gate (TTY y/N prompt or `--yes`; non-TTY without `--yes` refuses exit 2, mirroring the reindex-code cost gate) → live probe (one embed against the TARGET model/dims BEFORE any mutation — bad key/model/dim fails with nothing changed) → `applyEmbeddingMigration` with `persistEmbeddingFileConfig` (writes `~/.gbrain/config.json` + reconfigures the in-process gateway — the gateway reads file/env, NOT the DB plane) → `runEmbedCore({stale, catchUp, singleFlight, includeNullSignature, pace})` → drain check → `completeEmbeddingMigration` or exit 1 with the resume hint (re-run the same command). Also surfaced as the `migrate_embeddings` op (scope admin, localOnly, hidden cliHints; handler hard-refuses `ctx.remote !== false` and returns `needs_confirmation` + plan without `yes: true`). Pinned by `test/migrate-embeddings-flow.serial.test.ts` (full lifecycle incl. interrupted-run resume on PGLite). - `src/core/conversation-parser/` — 17-pattern built-in chat-format registry + opt-in LLM polish/fallback. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (17 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, bold-time-dash, bold-name-no-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as a module-level default), `parse.ts` (orchestrator with pattern-priority scoring across the first 10 lines + date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + multi-line continuation + timezone warning), `llm-base.ts` (shared `runLlmCall` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; polish and fallback are thin wrappers), `llm-polish.ts` (opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (opt-IN; NO regex inference + NO persistence), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). Pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, ordered after the time-bearing bold patterns) parses `**Speaker:** text` with NO per-line timestamp (Circleback/Granola/Zoom), anchoring every message at `T00:00:00Z` of the frontmatter date (line order preserves sequence, same no-time convention as `irc-classic`); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**`; non-shadow is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break). Because `**Label:** text` is a common prose idiom, the pattern sets optional `PatternEntry.score_full_body: true` so `parse.ts` recomputes the winner's acceptance score over the FULL body before the `SCORING_MIN_ACCEPTANCE` floor, keeping a bold-label notes page at `no_match`. Pattern `bold-paren-time` parses `**Speaker** (HH:MM): text` and `(HH:MM:SS)` (date_source: frontmatter). Fallback gates: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that; `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives. Exported `scorePatternFull(body, entry)`; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` DRY the quick_reject+regex loop. CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser ` exit 0/1/2, wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan ` debug, `list-builtins`, `validate `). Doctor checks: `conversation_format_coverage`, `progressive_batch_audit_health`, `conversation_parser_probe_health`. Pinned by `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,imessage-time-only-12h,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial,bold-name-no-time,bold-time-dash}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. Maintainer guidance: [conversation parser patterns](conversation-parser-patterns.md). - `src/core/progressive-batch/` — shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII formatter for the default `Policy.onStageReport`). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1`, `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by `test/progressive-batch/orchestrator.test.ts` (35 cases, every verdict path). - `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into time-windowed segments (30-min gap or 30-msg cap), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES=25MB`); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd`; **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); and **`--background` via `maybeBackground`**. The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source, and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`. diff --git a/docs/guides/embedding-migration.md b/docs/guides/embedding-migration.md new file mode 100644 index 000000000..ac748609e --- /dev/null +++ b/docs/guides/embedding-migration.md @@ -0,0 +1,138 @@ +# Embedding migration — moving a brain to another embedding provider + +`gbrain migrate embeddings` re-embeds an entire brain onto a different +embedding provider/model, safely and resumably. It is the forward path off a +sunsetting provider (for example ZeroEntropy's hosted API, which shuts down +2026-09-04 and is the shipped default for brains that never picked a model) — +but it is provider-agnostic: any configured `provider:model` works as a +target. + +Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the +README reference). + +## Quick start + +```bash +# Preview the work + cost. Changes nothing. +gbrain migrate embeddings --to openai:text-embedding-3-small --dry-run + +# Run it (interactive confirm shows chunk count + $ estimate first). +gbrain migrate embeddings --to openai:text-embedding-3-small + +# Non-interactive (cron / scripts): --yes is required, else exit 2. +gbrain migrate embeddings --to voyage:voyage-3-large --yes +``` + +`--dim ` overrides the target width; it defaults to the provider recipe's +declared width and is required for recipes that don't declare one (litellm, +llama-server, and other bring-your-own-model providers). + +## What it does, in order + +1. **Plan.** Counts every chunk not already in the target embedding space — + including chunks on pages with **no recorded embedding signature** + (pages embedded before the v108 provenance stamp). Prices the re-embed + from the pricing table; unknown providers print "estimate unavailable" + instead of a fabricated number. +2. **Consent gate.** Prints the plan; requires an interactive `y` or `--yes`. + Non-TTY without `--yes` refuses with exit 2 (mirrors the `reindex-code` + gate in [spend-controls](../operations/spend-controls.md)). Unlike the pure + cost gates there, `spend.posture=tokenmax` does **not** bypass this one: + posture waives the spend *ceiling*, and this gate also guards a + destructive schema rebuild. Under `tokenmax` the dollar figure is marked + informational and the confirmation is still asked. `--yes` is the single + scripted bypass. +3. **Live probe.** One tiny embed against the TARGET provider before any + mutation — validates the API key, model id, and dimension support in a + single call. A bad key fails here, with nothing changed. +4. **Env-override gate.** Refuses when `GBRAIN_EMBEDDING_MODEL` / + `GBRAIN_EMBEDDING_DIMENSIONS` would silently defeat the switch at + runtime (the same guard `ze-switch` uses). `--ignore-env-override` for + people running deliberate experiments. +5. **Apply.** When the target width differs from the actual column width, + runs the same atomic schema transition `ze-switch` uses, in one + transaction. It rebuilds **all three dim-pinned text-embedding-space + columns** — `content_chunks.embedding`, `query_cache.embedding`, and + `facts.embedding` — at the new width, preserving each column's type + (`vector` vs `halfvec`) and recreating its HNSW index. Missing any of the + three leaves it silently broken: a narrow `query_cache.embedding` makes + every cache write and read fail *by design* (the cache swallows errors so + it can never break search) for a permanent 0% hit rate, and a narrow + `facts.embedding` fails every per-fact embed write. The image/multimodal + columns ARE deliberately untouched — they use separate models whose + dimensions are independent of the text embedding model. + Writes `embedding_model` + `embedding_dimensions` to BOTH config planes + (file plane for the runtime gateway, DB plane for doctor), invalidates + every chunk still in the old space — **including NULL-signature pages** — + and purges the semantic query cache so stale cached results can't be + served across the swap. +6. **Re-embed.** The standard embed pipeline (`embed --stale --catch-up`) + with per-source single-flight locks, rate-limit backoff, stderr progress, + and optional DB-contention pacing (`--pace[=mode]`). + +## What the rebuild deletes + +The dimension change **deletes every stored embedding vector** in the brain — +they are in the old model's space and unusable. They are not recoverable: +going back to the previous provider means paying for a second full re-embed. +`content_chunks` vectors are rebuilt by the re-embed pass, the query cache +refills on the next query, and fact embeddings are rewritten on their next +write (or a `gbrain extract` pass). + +## Resume after a kill + +The NULL-embedding column is the checkpoint. If the run is killed (or some +pages fail to embed), re-run the **same command**: chunks already embedded on +the target are never re-embedded, the schema/config steps no-op, and the run +continues where it stopped. An in-flight marker (`embedding_migration.state` +in DB config) records the target; it is cleared only when the backlog drains +to zero. + +A page whose chunks straddle two stale batches is embedded correctly but not +stamped by the embed loop (which only stamps all-or-nothing per batch), so the +migration runs one reconcile pass after the drain that stamps every +fully-embedded page. Without it a large brain would report "incomplete" and the +re-run would pay again for those pages. `--batch-size N` tunes the batch +(default 2000). + +`--no-embed` applies schema + config + invalidation and stops, so you can run +the (potentially long) re-embed later or in the background: + +```bash +gbrain migrate embeddings --to openai:text-embedding-3-small --yes --no-embed +gbrain embed --stale --catch-up --include-null-signature --background +``` + +## During the migration + +While the re-embed runs, semantic search returns degraded (lexical-arm-only) +results for not-yet-re-embedded content. Pick a quiet window for large +brains, or use `--pace` to keep the DB responsive. + +## Pages without an embedding signature (#3391) + +Pages embedded before provenance stamping have `embedding_signature IS NULL` +and are grandfathered by the routine stale sweep (so an upgrade never +surprise-re-embeds a whole corpus). After a provider swap that grandfather +clause would silently leave those pages in the OLD embedding space — mixed +vector spaces in one index, degrading retrieval with nothing in the logs. + +- `gbrain migrate embeddings` always includes them. +- Plain `gbrain embed --stale` warns when a model swap leaves NULL-signature + pages behind, and `gbrain embed --stale --include-null-signature` re-embeds + them. + +## Reranker + +Migrating embeddings does not touch the reranker. If +`search.reranker.model` points at the outgoing provider, the plan prints a +warning; disable it (`gbrain config set search.reranker.enabled false`) or +point it at another provider. + +## Self-hosting instead of migrating + +If the outgoing model's weights are available (zembed-1's are Apache-2.0), +serving them locally via `llama-server` / `ollama` / a LiteLLM proxy +preserves your existing vectors — no re-embed at all. Point +`embedding_model` at the local recipe and keep the same dimensions. The +migration command is for when you'd rather move to a hosted provider. diff --git a/docs/operations/spend-controls.md b/docs/operations/spend-controls.md index 1e2fca9ee..0af1f2c92 100644 --- a/docs/operations/spend-controls.md +++ b/docs/operations/spend-controls.md @@ -49,6 +49,7 @@ The USD-limit knobs accept `off`, `unlimited`, or `none` (case-insensitive) to m | Backfill per-job budget | `embed.backfill_max_usd` | `10` | caps the job's tracker | `off` (`0` → default) | uncapped (still ledgered) | | Backfill cooldown | `embed.backfill_cooldown_min` | `10` | skips re-submission inside window | — (latency knob, not spend) | **not** bypassed | | `reindex-code` cost gate | — (preview before re-embed) | — | TTY prompt / non-TTY refuse + exit 2 | `--max-cost off` | informational | +| `migrate embeddings` consent gate | — (plan + estimate before provider migration) | — | TTY y/N prompt / non-TTY refuse + exit 2 | `--yes` | estimate marked informational, but **still prompts** (guards a destructive schema rebuild, not just spend) | | `enrich` / `onboard --auto` | `--max-usd` (per-call) | — | refuse without a cap (non-TTY) | `--max-usd off` | runs uncapped (still ledgered) | ### Sync inline-embed cost gate diff --git a/docs/progress-events.md b/docs/progress-events.md index 0b0a2a0e7..d6da9a467 100644 --- a/docs/progress-events.md +++ b/docs/progress-events.md @@ -140,6 +140,9 @@ Stable phase names shipped in v0.15.2: - `import.files` - `sync.deletes`, `sync.renames`, `sync.imports` - `migrate.copy_pages`, `migrate.copy_links` +- `migrate.reembed` (the re-embed pass of `gbrain migrate embeddings`; total is the + stale-chunk backlog at the start of the pass, so it can grow slightly if a + writer adds chunks mid-run) - `repair_jsonb.run`, `repair_jsonb..` - `backlinks.scan` - `lint.pages` diff --git a/scripts/e2e-test-map.ts b/scripts/e2e-test-map.ts index c32a34142..1ff067ccb 100644 --- a/scripts/e2e-test-map.ts +++ b/scripts/e2e-test-map.ts @@ -46,7 +46,15 @@ export const E2E_TEST_MAP: Record = { "test/e2e/multi-source-bug-class.test.ts", "test/e2e/synthesize-bigint-job-id-postgres.test.ts", ], - "src/commands/embed.ts": ["test/e2e/multi-source-bug-class.test.ts"], + "src/commands/embed.ts": [ + "test/e2e/multi-source-bug-class.test.ts", + // #3391: the NULL-signature stale predicates differ per engine. + "test/e2e/migrate-embeddings-postgres.test.ts", + ], + // #3390: runSchemaTransition's DDL path + the stale predicates behave + // differently on real pgvector than on PGLite. + "src/core/embedding-migration.ts": ["test/e2e/migrate-embeddings-postgres.test.ts"], + "src/core/retrieval-upgrade-planner.ts": ["test/e2e/migrate-embeddings-postgres.test.ts"], "src/commands/extract.ts": ["test/e2e/multi-source-bug-class.test.ts"], "src/commands/migrate-engine.ts": ["test/e2e/multi-source-bug-class.test.ts"], // Any minions queue/worker/handler change exercises all minion E2E. @@ -64,6 +72,8 @@ export const E2E_TEST_MAP: Record = { "test/e2e/jsonb-roundtrip.test.ts", "test/e2e/engine-parity.test.ts", "test/e2e/schema-drift.test.ts", + // #3391: includeNullSignature stale predicates (engine parity). + "test/e2e/migrate-embeddings-postgres.test.ts", ], // PGLite bootstrap path + parity guard. "src/core/pglite-engine.ts": [ diff --git a/src/cli.ts b/src/cli.ts index 50705174b..779f0c2c8 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -55,7 +55,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown { } // CLI-only commands that bypass the operation layer -export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']); +export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']); // CLI-only commands whose handlers print their own --help text. These are // excluded from the generic short-circuit so detailed per-command and // per-subcommand usage stays reachable. @@ -107,6 +107,10 @@ const CLI_ONLY_SELF_HELP = new Set([ // `gbrain connect --help` prints its own usage (flags + examples) from // runConnect; route around the generic one-line short-circuit. 'connect', + // #3390 — `gbrain migrate embeddings --help` / `gbrain retrieval-upgrade + // --help` print the migration flags from runMigrateEmbeddings. `migrate` + // (engine transfer) keeps its own dispatch too. + 'migrate', 'retrieval-upgrade', ]); // v114 (#1941): alias -> operation lookup, kept separate from `cliOps` so @@ -1055,7 +1059,7 @@ export function formatResult(opName: string, result: unknown): string { * `runRemoteDoctor` for thin-client installs. */ const THIN_CLIENT_REFUSED_COMMANDS = new Set([ - 'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'apply-migrations', + 'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'retrieval-upgrade', 'apply-migrations', 'repair-jsonb', 'orphans', 'integrity', 'serve', // v0.43 (#2095): watch streams against a LOCAL engine; thin clients get // the volunteer_context MCP op instead. @@ -1102,6 +1106,7 @@ const THIN_CLIENT_REFUSE_HINTS: Record = { 'extract-conversation-facts': 'extract-conversation-facts runs on the host (requires local engine + chat gateway). Run on the host machine.', enrich: 'enrich runs on the host (requires local engine + chat gateway for grounded synthesis). Run on the host machine.', migrate: "migrate runs on the host's local engine. Run on the host machine.", + 'retrieval-upgrade': "retrieval-upgrade (embedding migration) rebuilds the host brain's schema + re-embeds. Run on the host machine.", 'apply-migrations': 'schema migrations run on the host. SSH and run there.', 'repair-jsonb': 'repair-jsonb operates on the local DB only.', integrity: 'integrity scans local files. Run on the host machine.', @@ -1752,10 +1757,33 @@ async function handleCliOnly(command: string, args: string[]) { } // doctor is handled before connectEngine() above case 'migrate': { + // #3390: `gbrain migrate embeddings --to ` — the + // provider-agnostic embedding migration. Everything else stays the + // engine-transfer path (`migrate --to `). + if (args[0] === 'embeddings') { + const { runMigrateEmbeddings } = await import('./commands/migrate-embeddings.ts'); + await runMigrateEmbeddings(engine, args.slice(1)); + break; + } + if (args.includes('--help') || args.includes('-h')) { + console.log('Usage: gbrain migrate --to [--url ] [--path ] [--force]'); + console.log(' gbrain migrate embeddings --to [--dim N] [--dry-run] [--yes]'); + console.log(''); + console.log('The first form transfers the brain between engines; the second re-embeds'); + console.log('onto a different embedding provider (run `gbrain migrate embeddings --help`).'); + break; + } const { runMigrateEngine } = await import('./commands/migrate-engine.ts'); await runMigrateEngine(engine, args); break; } + case 'retrieval-upgrade': { + // The command README.md + doctor.ts promised since v0.36 but never + // dispatched. Alias for `migrate embeddings` (#3390). + const { runMigrateEmbeddings } = await import('./commands/migrate-embeddings.ts'); + await runMigrateEmbeddings(engine, args); + break; + } case 'eval': { // v0.32 EXP-5: `eval takes-quality {run,trend,regress}` requires a // brain (samples takes from DB / reads runs table). `replay` was @@ -2352,6 +2380,7 @@ USAGE SETUP init [--pglite|--supabase|--url] Create brain (PGLite default, no server) migrate --to Transfer brain between engines + migrate embeddings --to Re-embed onto another embedding provider upgrade Self-update check-update [--json] Check for new versions doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings) diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index a695a80f8..e029087c4 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -5923,7 +5923,7 @@ export async function buildChecks( // that doesn't match the gateway's resolved default. Empty-brain vs // non-empty-brain branching determines the repair hint: // - empty brain (no embedded chunks) → `gbrain init --force --embedding-model …` - // - non-empty brain → `gbrain retrieval-upgrade --to … --reindex` + // - non-empty brain → `gbrain migrate embeddings --to … --dim …` (#3390) // The bug-reporter's `rm -rf ~/.gbrain` recovery is never the right answer. let surfacedUnconfiguredDrift = false; try { @@ -5954,7 +5954,7 @@ export async function buildChecks( if (totalChunks > 0) { const fix = embeddedCount === 0 ? `No embeddings yet — drop the empty schema and re-init at the right dim:\n gbrain init --force --pglite --embedding-model ${configuredModel} --embedding-dimensions ${configuredDims}` - : `Non-empty brain (${embeddedCount} embedded chunks). Migrate cleanly:\n gbrain retrieval-upgrade --to ${configuredModel} --reindex`; + : `Non-empty brain (${embeddedCount} embedded chunks). Migrate cleanly:\n gbrain migrate embeddings --to ${configuredModel} --dim ${configuredDims}`; checks.push({ name: 'embedding_provider', diff --git a/src/commands/embed.ts b/src/commands/embed.ts index 85ae2c282..7b3c227aa 100644 --- a/src/commands/embed.ts +++ b/src/commands/embed.ts @@ -115,6 +115,16 @@ export interface EmbedOpts { * Errors/warnings still go to stderr regardless. */ quiet?: boolean; + /** + * #3391: widen signature-drift invalidation to pages with NO recorded + * embedding_signature (pre-v108). By default those are grandfathered + * (never invalidated) so a routine upgrade doesn't surprise-re-embed a + * whole corpus — but after a provider/model swap the grandfather clause + * silently leaves them in the OLD embedding space, mixing two vector + * spaces in one index. `gbrain migrate embeddings` and + * `gbrain embed --stale --include-null-signature` set this. + */ + includeNullSignature?: boolean; } /** @@ -356,6 +366,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis pacer, paceMaxConcurrency, quiet: opts.quiet, + includeNullSignature: opts.includeNullSignature, }, opts.signal); } finally { // E1: surface pacing telemetry (human + structured) when pacing was on. @@ -469,6 +480,8 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise= 0 ? args[priorityIdx + 1] : undefined; const priority = priorityRaw === 'recent' ? 'recent' as const : undefined; const catchUp = args.includes('--catch-up'); + // #3391: re-embed pages that predate the embedding_signature stamp too. + const includeNullSignature = args.includes('--include-null-signature'); const pace = parsePaceArgs(args); let opts: EmbedOpts; @@ -476,11 +489,11 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise !a.startsWith('--')), dryRun, sourceId, batchSize, priority, catchUp }; } else if (all || stale) { // E-2: CLI-only single-flight for stale runs (the minion path locks itself). - opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }) }; + opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }), ...(includeNullSignature && { includeNullSignature: true }) }; } else { const slug = args.find(a => !a.startsWith('--')); if (!slug) { - serr('Usage: gbrain embed [|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up]'); + serr('Usage: gbrain embed [|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up] [--include-null-signature]'); process.exit(1); } opts = { slug, dryRun, sourceId, batchSize, priority, catchUp }; @@ -657,6 +670,8 @@ async function embedAll( paceMaxConcurrency?: number; /** #394: suppress human stdout summaries (structured-output callers). */ quiet?: boolean; + /** #3391: lift the NULL-signature grandfather clause (see EmbedOpts). */ + includeNullSignature?: boolean; }, signal?: AbortSignal, ) { @@ -845,6 +860,8 @@ async function embedAllStale( paceMaxConcurrency?: number; /** #394: suppress human stdout summaries (structured-output callers). */ quiet?: boolean; + /** #3391: lift the NULL-signature grandfather clause (see EmbedOpts). */ + includeNullSignature?: boolean; }, signature?: string, externalSignal?: AbortSignal, @@ -852,6 +869,7 @@ async function embedAllStale( // D7: thread sourceId so source-scoped runs only count + visit // that source's NULL embeddings. const sourceOpt = sourceId ? { sourceId } : undefined; + const includeNullSig = !!staleOpts?.includeNullSignature; // v0.41.31: re-embed pages whose embedding_signature drifted (model/dims // swap). dry-run must NOT mutate, so it counts signature-stale via the @@ -861,16 +879,46 @@ async function embedAllStale( const invalidated = await engine.invalidateStaleSignatureEmbeddings({ signature, ...(sourceId && { sourceId }), + ...(includeNullSig && { includeNullSignature: true }), }); if (invalidated > 0 && !staleOpts?.quiet) { slog(`[embed] invalidated ${invalidated} chunk(s) embedded under a prior model signature`); } + // #3391: the grandfather clause keeps NULL-signature pages on their OLD + // vectors — two embedding spaces mixed in one index. Loud stderr warning + // with the fix, instead of silent retrieval degradation. + // + // Deliberately NOT gated on `invalidated > 0`: the original bug report's + // shape is a brain where EVERY embedded page predates the signature stamp, + // so nothing drifts, nothing is invalidated — and pre-fix that brain got + // no warning AND no work, the exact silent case #3391 is about. The probe + // below computes the left-behind count directly, which is 0 on a healthy + // brain, so an unaffected run stays quiet. + if (!includeNullSig) { + try { + const wide = await engine.countStaleChunks({ ...sourceOpt, signature, includeNullSignature: true }); + const narrow = await engine.countStaleChunks({ ...sourceOpt, signature }); + const leftBehind = wide - narrow; + if (leftBehind > 0) { + serr( + ` [embed] WARNING: ${leftBehind} embedded chunk(s) sit on pages with no recorded ` + + `embedding signature and were NOT invalidated — they remain in the previous model's ` + + `embedding space. Re-run with --include-null-signature (or use ` + + `\`gbrain migrate embeddings\`) to re-embed them.`, + ); + } + } catch { + // The warning probe is best-effort; never break the embed run. + } + } } // Pre-flight: 0 stale chunks → nothing to do, no further DB reads. // dry-run includes signature-drift in the count without mutating. const staleCount = await engine.countStaleChunks( - dryRun && signature ? { ...sourceOpt, signature } : sourceOpt, + dryRun && signature + ? { ...sourceOpt, signature, ...(includeNullSig && { includeNullSignature: true }) } + : sourceOpt, ); if (staleCount === 0) { if (!staleOpts?.quiet) { @@ -1138,7 +1186,9 @@ async function embedAllStale( // as a clean run — re-running won't help until the underlying failure is fixed. if (staleOpts?.catchUp && !effectiveSignal.aborted && embedFailures > 0) { const remaining = await engine.countStaleChunks( - signature ? { signature, ...(sourceId ? { sourceId } : {}) } : (sourceId ? { sourceId } : undefined), + signature + ? { signature, ...(sourceId ? { sourceId } : {}), ...(includeNullSig && { includeNullSignature: true }) } + : (sourceId ? { sourceId } : undefined), ); if (remaining > 0) { serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${embedFailures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`); diff --git a/src/commands/migrate-embeddings.ts b/src/commands/migrate-embeddings.ts new file mode 100644 index 000000000..3feac7dc1 --- /dev/null +++ b/src/commands/migrate-embeddings.ts @@ -0,0 +1,402 @@ +/** + * `gbrain migrate embeddings --to ` (#3390) — the + * provider-agnostic forward migration off any embedding provider, built for + * the ZeroEntropy 2026-09-04 sunset but not keyed to it. + * + * Also reachable as `gbrain retrieval-upgrade` — the command README.md and + * doctor.ts have promised since v0.36 but which never had a dispatch branch. + * + * Flow (everything heavy is reused, see src/core/embedding-migration.ts): + * 1. plan — chunk/char counts via the widened stale predicates, + * cost estimate from embedding-pricing.ts + * 2. preflight— print estimate; require --yes or interactive confirm + * (non-TTY without --yes refuses with exit 2, mirroring the + * reindex-code cost gate in docs/operations/spend-controls.md) + * 3. probe — one live embed against the TARGET provider BEFORE any + * mutation (validates key + model + dims in one shot) + * 4. apply — schema transition (dim change), config (DB + file plane), + * #3391 NULL-signature-inclusive invalidation, cache purge + * 5. re-embed — runEmbedCore --stale --catch-up with single-flight locks, + * pacing (--pace), progress reporting. Resumable: a killed + * run re-runs the SAME command; the NULL-embedding cursor is + * the checkpoint and steps 3-4 no-op on the second pass. + */ + +import type { BrainEngine } from '../core/engine.ts'; +import { serr, slog } from '../core/console-prefix.ts'; +import { + planEmbeddingMigration, + applyEmbeddingMigration, + completeEmbeddingMigration, + reconcilePageSignatures, + MIGRATION_STATE_KEY, + type EmbeddingMigrationPlan, +} from '../core/embedding-migration.ts'; +import { formatEnvOverrideWarning } from '../core/retrieval-upgrade-planner.ts'; +import { parsePaceArgs, runEmbedCore } from './embed.ts'; + +export interface MigrateEmbeddingsFlags { + to?: string; + dim?: number; + yes: boolean; + dryRun: boolean; + json: boolean; + noEmbed: boolean; + ignoreEnvOverride: boolean; + batchSize?: number; + pace?: ReturnType; +} + +export function parseMigrateEmbeddingsFlags(args: string[]): MigrateEmbeddingsFlags { + const toIdx = args.indexOf('--to'); + const dimIdx = args.indexOf('--dim'); + const dimRaw = dimIdx >= 0 ? parseInt(args[dimIdx + 1] ?? '', 10) : NaN; + const bsIdx = args.indexOf('--batch-size'); + const bsRaw = bsIdx >= 0 ? parseInt(args[bsIdx + 1] ?? '', 10) : NaN; + const batchSize = Number.isFinite(bsRaw) && bsRaw > 0 ? Math.min(10_000, bsRaw) : undefined; + return { + to: toIdx >= 0 ? args[toIdx + 1] : undefined, + dim: Number.isFinite(dimRaw) && dimRaw > 0 ? dimRaw : undefined, + yes: args.includes('--yes') || args.includes('--non-interactive'), + dryRun: args.includes('--dry-run'), + json: args.includes('--json'), + noEmbed: args.includes('--no-embed'), + ignoreEnvOverride: args.includes('--ignore-env-override'), + ...(batchSize !== undefined && { batchSize }), + pace: parsePaceArgs(args), + }; +} + +function printHelp(): void { + process.stdout.write(`Usage: gbrain migrate embeddings --to [flags] + +Re-embed the whole brain onto a different embedding provider/model. Handles +dimension changes (schema transition), pages without a recorded embedding +signature (#3391), the query cache, and resume-after-kill. The forward path +off a sunsetting provider. + +Flags: + --to Target embedding model (e.g. openai:text-embedding-3-small). + --dim Target dimensions. Defaults to the provider recipe's + declared width; required when the recipe declares none. + --dry-run Plan + cost estimate only; change nothing. + --yes Skip the confirm prompt (required non-interactively). + --json Machine-readable envelope on stdout. + --no-embed Apply schema + config + invalidation, but skip the + re-embed pass (run \`gbrain embed --stale --include-null-signature\` + or \`... --background\` yourself). + --batch-size Stale-chunk batch size for the re-embed (default 2000). + --pace[=mode] DB-contention pacing for the re-embed (off|gentle|balanced|aggressive). + --ignore-env-override Proceed even when GBRAIN_EMBEDDING_* env vars would + override the target at runtime (you know why). + --help Show this help. + +A killed run is resumable: re-run the same command. Already-migrated chunks +are never re-embedded twice. +`); +} + +function renderPlan(plan: EmbeddingMigrationPlan): string { + const lines: string[] = []; + lines.push('Embedding migration plan'); + lines.push(` From: ${plan.from_model} (${plan.from_dims}d${plan.column_dims !== null && plan.column_dims !== plan.from_dims ? `; column is actually ${plan.column_dims}d` : ''})`); + lines.push(` To: ${plan.to_model} (${plan.to_dims}d)`); + if (plan.dim_change) { + lines.push(` DESTRUCTIVE: the embedding column is rebuilt at ${plan.to_dims}d, which DELETES`); + lines.push(' every stored embedding vector in this brain. They are not recoverable —'); + lines.push(' going back to the old provider means paying for a second full re-embed.'); + lines.push(' Until the re-embed finishes, semantic search is degraded to lexical-only.'); + lines.push(` The query cache and fact embeddings are rebuilt at ${plan.to_dims}d too`); + lines.push(' (cache refills on next query; facts re-embed on their next write).'); + } + lines.push(` Chunks to re-embed: ${plan.chunks_to_embed}${plan.null_signature_chunks > 0 ? ` (includes ${plan.null_signature_chunks} on pages with no recorded embedding signature)` : ''}`); + lines.push( + plan.price_known + ? ` Estimated cost: $${plan.est_cost_usd.toFixed(2)} (${plan.total_chars} chars at the ${plan.to_model} rate)` + : ` Estimated cost: unknown — no pricing entry for ${plan.to_model}. Check the provider's pricing before proceeding.`, + ); + if (plan.resuming) { + lines.push(' Resuming: a prior migration to this target was interrupted; continuing it.'); + } + if (plan.reranker_warning) { + lines.push(` WARNING: ${plan.reranker_warning}`); + } + return lines.join('\n'); +} + +/** Single-keypress y/N confirm on stdin. Injectable for tests. */ +async function defaultConfirm(question: string): Promise { + process.stderr.write(`${question} [y/N] `); + const stdin = process.stdin; + stdin.setRawMode?.(true); + stdin.resume(); + const key: string = await new Promise((resolve) => { + stdin.once('data', (d) => resolve(d.toString())); + }); + stdin.setRawMode?.(false); + stdin.pause(); + process.stderr.write('\n'); + return key.trim().toLowerCase().startsWith('y'); +} + +/** + * One tiny embed against the TARGET provider, BEFORE any mutation: validates + * the API key, the model id, and dimension support in a single call, so a bad + * target fails with the brain untouched instead of after the column is + * dropped. Shared by the CLI and the `migrate_embeddings` op (the op used to + * skip it, which let `yes:true` drop the column against a bad key). + */ +export async function probeTargetProvider( + toModel: string, + toDims: number, +): Promise<{ ok: true } | { ok: false; message: string }> { + try { + const { embed } = await import('../core/ai/gateway.ts'); + const vecs = await embed(['gbrain embedding migration probe'], { + embeddingModel: toModel, + dimensions: toDims, + }); + const got = vecs[0]?.length ?? 0; + if (got !== toDims) { + return { + ok: false, + message: `Target provider returned ${got}-dim vectors, expected ${toDims}. Pass a valid --dim for ${toModel}.`, + }; + } + return { ok: true }; + } catch (e) { + return { + ok: false, + message: `Preflight embed against ${toModel} failed — nothing was changed:\n ${e instanceof Error ? e.message : String(e)}`, + }; + } +} + +/** + * Persist the target model+dims to the FILE plane and reconfigure the + * in-process gateway. The gateway reads file/env config, not the DB plane — + * without this the re-embed would silently run against the OLD provider. + * Shared by the CLI command and the `migrate_embeddings` op handler. + */ +export async function persistEmbeddingFileConfig( + toModel: string, + toDims: number, +): Promise { + const { loadConfig, saveConfig } = await import('../core/config.ts'); + const { configureGateway } = await import('../core/ai/gateway.ts'); + const { buildGatewayConfig } = await import('../core/ai/build-gateway-config.ts'); + const cfg = loadConfig(); + if (!cfg) { + // REFUSE rather than warn-and-proceed. Without a file plane to write, the + // switch would not survive this process: the next `gbrain` invocation + // reads file/env config, sees the OLD provider, and re-embeds the brain + // back into the old space (paying twice) — or fails outright against a + // column that is now the new width. Thrown from inside + // applyEmbeddingMigration's try, so it surfaces as status: 'failed' + // BEFORE the config/cache steps and the caller exits non-zero. + throw new Error( + 'No ~/.gbrain/config.json found — refusing to migrate.\n' + + ' The embed pipeline reads file/env config, so without a file plane this switch\n' + + ' would not survive the process and the next run would re-embed into the old space.\n' + + ' Fix: run `gbrain init` (or set GBRAIN_EMBEDDING_MODEL + GBRAIN_EMBEDDING_DIMENSIONS\n' + + ' in the environment of every gbrain process) and re-run.', + ); + } + cfg.embedding_model = toModel; + cfg.embedding_dimensions = toDims; + saveConfig(cfg); + configureGateway(buildGatewayConfig(cfg)); +} + +export interface RunMigrateEmbeddingsOpts { + /** Test seams. */ + confirm?: (question: string) => Promise; + isTTY?: boolean; + exit?: (code: number) => never; +} + +export async function runMigrateEmbeddings( + engine: BrainEngine, + args: string[], + opts: RunMigrateEmbeddingsOpts = {}, +): Promise { + // Explicit `never` annotation so TS control-flow analysis treats every + // exit() call as terminal (required for narrowing after the guard blocks). + const exit: (code: number) => never = opts.exit ?? ((code: number) => process.exit(code)); + if (args.includes('--help') || args.includes('-h')) { + printHelp(); + exit(0); + } + const flags = parseMigrateEmbeddingsFlags(args); + if (!flags.to) { + serr('Missing --to . Example: gbrain migrate embeddings --to openai:text-embedding-3-small'); + serr('Run with --help for all flags.'); + exit(1); + } + + // From-state as the gateway resolved it (file/env config + defaults) — + // the truth for what embeds run under TODAY. + let fromModel: string | undefined; + let fromDims: number | undefined; + try { + const { getEmbeddingModel, getEmbeddingDimensions } = await import('../core/ai/gateway.ts'); + fromModel = getEmbeddingModel(); + fromDims = getEmbeddingDimensions(); + } catch { + // Gateway unconfigured — plan falls back to shipped defaults. + } + + let plan: EmbeddingMigrationPlan; + try { + plan = await planEmbeddingMigration(engine, { + to: flags.to!, + ...(flags.dim !== undefined && { dim: flags.dim }), + ...(fromModel !== undefined && { fromModel }), + ...(fromDims !== undefined && { fromDims }), + }); + } catch (e) { + serr(e instanceof Error ? e.message : String(e)); + exit(1); + return; // unreachable; keeps TS happy for injected exit seams + } + + if (flags.json) { + // Human plan goes to stderr so stdout stays JSON-clean. + serr(renderPlan(plan)); + } else { + console.log(renderPlan(plan)); + } + + if (plan.chunks_to_embed === 0 && !plan.dim_change && plan.from_model === plan.to_model) { + if (flags.json) console.log(JSON.stringify({ status: 'skipped_no_work', plan }, null, 2)); + else console.log('Nothing to migrate — brain is already on the target model.'); + exit(0); + } + + if (flags.dryRun) { + if (flags.json) console.log(JSON.stringify({ status: 'planned', plan }, null, 2)); + exit(0); + } + + // ── Consent gate. Unlike the pure cost gates in + // docs/operations/spend-controls.md, `spend.posture=tokenmax` does NOT + // bypass this one: posture waives the SPEND ceiling, and this gate also + // guards a destructive schema rebuild (existing vectors are dropped, and + // retrieval is degraded until the re-embed finishes). We honor the posture + // by marking the dollar figure informational, and still ask. + if (!flags.yes) { + const { resolveSpendPosture } = await import('../core/spend-posture.ts'); + const posture = await resolveSpendPosture(engine); + if (posture === 'tokenmax') { + serr(' [migrate] spend.posture=tokenmax: the cost estimate above is informational.'); + serr(' [migrate] Confirmation is still required — this rebuilds the embedding column (destructive, not just costly).'); + } + const isTTY = opts.isTTY ?? Boolean(process.stdin.isTTY); + if (!isTTY) { + serr('Refusing to migrate without confirmation in a non-TTY environment. Re-run with --yes.'); + exit(2); + } + const confirm = opts.confirm ?? defaultConfirm; + const priceNote = plan.price_known ? `~$${plan.est_cost_usd.toFixed(2)}` : 'an UNKNOWN amount'; + const ok = await confirm(`Re-embed ${plan.chunks_to_embed} chunks (${priceNote})?`); + if (!ok) { + serr('Aborted. Nothing was changed.'); + exit(1); + } + } + + // ── Live probe BEFORE any mutation: one tiny embed against the TARGET + // provider validates API key, model id, and dimension support in one call. + const probe = await probeTargetProvider(plan.to_model, plan.to_dims); + if (!probe.ok) { + serr(probe.message); + exit(1); + } + + // ── Apply: schema + config + invalidation + cache purge. + const applied = await applyEmbeddingMigration(engine, plan, { + ignoreEnvOverride: flags.ignoreEnvOverride, + persistConfig: (toModel, toDims) => persistEmbeddingFileConfig(toModel, toDims), + }); + + if (applied.status === 'refused') { + if (flags.json) console.log(JSON.stringify(applied, null, 2)); + else serr(formatEnvOverrideWarning(applied.warning)); + exit(1); + } + if (applied.status === 'failed') { + if (flags.json) console.log(JSON.stringify(applied, null, 2)); + else serr(`Migration apply failed: ${applied.reason}`); + exit(1); + } + + serr(` [migrate] schema ${applied.schema_transitioned ? `rebuilt at ${plan.to_dims}d` : 'unchanged'}; ` + + `${applied.invalidated} chunk(s) invalidated; query cache purged (${applied.cache_cleared} row(s)).`); + + if (flags.noEmbed) { + const msg = 'Config + schema migrated. Re-embed deferred — run: gbrain embed --stale --catch-up --include-null-signature'; + if (flags.json) console.log(JSON.stringify({ ...applied, status: 'applied_no_embed', plan }, null, 2)); + else console.log(msg); + exit(0); + } + + // ── Re-embed. All the machinery (locks, pacing, backoff, progress, + // signature stamping) is the standard embed pipeline. + const { createProgress } = await import('../core/progress.ts'); + const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts'); + const progress = createProgress(cliOptsToProgressOptions(getCliOptions())); + let progressStarted = false; + const embedResult = await runEmbedCore(engine, { + stale: true, + catchUp: true, + singleFlight: true, + includeNullSignature: true, + quiet: flags.json, + ...(flags.batchSize !== undefined && { batchSize: flags.batchSize }), + ...(flags.pace && { pace: flags.pace }), + onProgress: (done, total) => { + if (!progressStarted) { + progress.start('migrate.reembed', total); + progressStarted = true; + } + progress.tick(1); + }, + }); + if (progressStarted) progress.finish(); + + // Reconcile signatures BEFORE the completion probe: pages straddling a + // stale-batch boundary are embedded correctly but left unstamped by the + // embed loop's all-or-nothing stamp rule. Without this the probe would call + // a fully-migrated brain "incomplete" and the re-run would pay again. + const reconciled = await reconcilePageSignatures(engine, plan); + if (reconciled > 0) { + serr(` [migrate] reconciled the embedding signature on ${reconciled} fully-embedded page(s) (batch-boundary pages).`); + } + + const remaining = await engine.countStaleChunks({ + signature: `${plan.to_model}:${plan.to_dims}`, + includeNullSignature: true, + }); + + if (remaining === 0) { + await completeEmbeddingMigration(engine, plan); + if (flags.json) { + console.log(JSON.stringify({ status: 'completed', plan, embedded: embedResult.embedded, remaining: 0 }, null, 2)); + } else { + slog(`Migration complete: ${embedResult.embedded} chunk(s) embedded on ${plan.to_model} (${plan.to_dims}d).`); + if (plan.reranker_warning) serr(` [migrate] reminder: ${plan.reranker_warning}`); + } + exit(0); + } else { + if (flags.json) { + console.log(JSON.stringify({ status: 'incomplete', plan, embedded: embedResult.embedded, remaining }, null, 2)); + } else { + serr(`Migration incomplete: ${remaining} chunk(s) still stale (embed failures or an interrupted run).`); + serr('Re-run the same command to resume — completed chunks are never re-embedded.'); + } + exit(1); + } +} + +/** Re-export for the op handler + tests. */ +export { MIGRATION_STATE_KEY }; diff --git a/src/commands/upgrade.ts b/src/commands/upgrade.ts index c71882b77..313ab4120 100644 --- a/src/commands/upgrade.ts +++ b/src/commands/upgrade.ts @@ -462,6 +462,53 @@ export async function runPostUpgrade(args: string[] = []): Promise { // Banner is cosmetic; never block the upgrade. } + // #3390: ZeroEntropy sunset notice. ZE announced (2026-07-24) that + // its hosted endpoints — including /models/embed and /models/rerank — + // shut down on 2026-09-04. Any brain resolving to a zeroentropyai:* + // embedding model (including default-config brains that never set + // one) loses SEMANTIC RETRIEVAL ENTIRELY on that date: the query + // embedding uses the same endpoint, so existing vectors become + // unqueryable. One-shot per install, gated by + // `ze_sunset_notice_shown` (same pattern as the search-mode banner). + try { + const shown = await engine.getConfig('ze_sunset_notice_shown'); + const { DEFAULT_EMBEDDING_MODEL } = await import('../core/ai/defaults.ts'); + const effectiveModel = cfgSchema.embedding_model ?? DEFAULT_EMBEDDING_MODEL; + const rerankerModel = await engine.getConfig('search.reranker.model'); + const onZeEmbedding = effectiveModel.startsWith('zeroentropyai:'); + const onZeReranker = !!rerankerModel?.startsWith('zeroentropyai:'); + if (shown !== 'true' && (onZeEmbedding || onZeReranker)) { + console.log(''); + console.log('═══════════════════════════════════════════════════════════════'); + console.log('[gbrain] ACTION REQUIRED: ZeroEntropy hosted API sunsets 2026-09-04.'); + if (onZeEmbedding) { + console.log(`[gbrain] This brain embeds with ${effectiveModel}. After the sunset,`); + console.log('[gbrain] semantic retrieval STOPS WORKING (queries can no longer be'); + console.log('[gbrain] embedded against your existing vectors).'); + } + if (onZeReranker) { + console.log(`[gbrain] The reranker (${rerankerModel}) also sunsets; search falls`); + console.log('[gbrain] back to unreranked ordering.'); + } + console.log('═══════════════════════════════════════════════════════════════'); + console.log(''); + console.log('Migrate before the sunset (resumable; preview cost first):'); + console.log(' gbrain migrate embeddings --to --dry-run'); + console.log(' gbrain migrate embeddings --to '); + console.log(''); + console.log('Self-hosting zembed-1 (weights are Apache-2.0) via llama-server /'); + console.log('ollama also works and preserves your existing vectors — point'); + console.log('embedding at the local endpoint instead of migrating.'); + if (onZeReranker) { + console.log('Reranker: gbrain config set search.reranker.enabled false (or pick another).'); + } + console.log(''); + await engine.setConfig('ze_sunset_notice_shown', 'true'); + } + } catch { + // Banner is cosmetic; never block the upgrade. + } + // PR1: skill-catalog publish consent. New installs default ON at // `gbrain init`; EXISTING installs stay OFF (default-OFF runtime = no // silent capability grant on upgrade) until the owner opts in HERE. diff --git a/src/core/embedding-migration.ts b/src/core/embedding-migration.ts new file mode 100644 index 000000000..18b9095de --- /dev/null +++ b/src/core/embedding-migration.ts @@ -0,0 +1,340 @@ +/** + * Provider-agnostic embedding migration (#3390). + * + * `gbrain migrate embeddings --to ` re-embeds a brain onto + * any configured provider — the forward path off a sunsetting provider that + * `ze-switch` (ZE-only target) and `ze-switch --undo` (needs a snapshot fresh + * installs don't have) cannot cover. + * + * Deliberately thin: everything heavy is reused — + * - runSchemaTransition (retrieval-upgrade-planner.ts) for dimension changes + * - invalidateStaleSignatureEmbeddings + the NULL-embedding cursor for + * staleness + resume (the NULL column IS the checkpoint: a killed run + * re-runs the same command and continues where it stopped) + * - the embed pipeline (src/commands/embed.ts) for the actual re-embed, + * with pacing, backfill locks, rate-limit backoff, and progress + * - lookupEmbeddingPrice / estimateCostFromChars for the preflight estimate + * - detectEnvOverride (the #1421 damage-class gate) before any mutation + * + * #3391 companion fix: the migration widens staleness with + * `includeNullSignature: true` so pages that predate the v108 signature stamp + * are re-embedded too, instead of silently staying in the old embedding space. + * + * The command layer (src/commands/migrate-embeddings.ts) owns everything + * process-shaped: confirm prompts, file-plane config persistence (the gateway + * reads file/env, not the DB plane), gateway reconfiguration, and the embed + * catch-up run. This module is engine-pure so both engines and the op handler + * share one implementation. + */ + +import type { BrainEngine } from './engine.ts'; +import { resolveRecipe, embeddingDimsForModel } from './ai/model-resolver.ts'; +import { lookupEmbeddingPrice, estimateCostFromChars } from './embedding-pricing.ts'; +import { detectEnvOverride, type EnvOverrideWarning } from './retrieval-upgrade-planner.ts'; +import { runSchemaTransition } from './retrieval-upgrade-planner.ts'; +import { readContentChunksEmbeddingDim } from './embedding-dim-check.ts'; +import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts'; + +/** + * Resume/state marker (DB plane). Present while a migration is in flight so + * a re-run can detect + resume; cleared when the re-embed drains to zero. + */ +export const MIGRATION_STATE_KEY = 'embedding_migration.state'; +/** ISO timestamp + summary of the last completed migration (DB plane). */ +export const MIGRATION_COMPLETED_KEY = 'embedding_migration.completed'; + +export interface MigrationState { + to_model: string; + to_dims: number; + from_model: string; + from_dims: number; + started_at: string; +} + +export interface EmbeddingMigrationPlan { + from_model: string; + from_dims: number; + /** Actual `content_chunks.embedding` vector(N) width (null = column absent). */ + column_dims: number | null; + to_model: string; + to_dims: number; + /** True when the schema column must be rebuilt at a new width. */ + dim_change: boolean; + /** Chunks not yet in the target embedding space (the migration workload). */ + chunks_to_embed: number; + /** Characters across those chunks (feeds the cost estimate). */ + total_chars: number; + /** + * #3391 visibility: embedded chunks on pages with NO recorded signature + * (pre-v108). Included in chunks_to_embed via includeNullSignature. + */ + null_signature_chunks: number; + est_cost_usd: number; + /** False when the target model has no entry in EMBEDDING_PRICING. */ + price_known: boolean; + /** True when a prior in-flight migration state matches this target. */ + resuming: boolean; + /** Set when the brain's reranker is also on the outgoing provider. */ + reranker_warning: string | null; +} + +export type MigrationApplyResult = + | { status: 'applied'; invalidated: number; cache_cleared: number; schema_transitioned: boolean } + | { status: 'refused'; reason: 'env_override'; warning: EnvOverrideWarning } + | { status: 'failed'; reason: string }; + +/** `:` — must match currentEmbeddingSignature()'s shape. */ +export function migrationSignature(toModel: string, toDims: number): string { + return `${toModel}:${toDims}`; +} + +/** + * Resolve + validate the target `provider:model` and dimensions. + * Throws with a paste-ready message on an unknown provider or when the + * recipe declares no default dims and the caller passed none. + */ +export function resolveMigrationTarget(to: string, dimFlag?: number): { toModel: string; toDims: number } { + if (!to.includes(':')) { + throw new Error( + `--to must be provider:model (e.g. openai:text-embedding-3-small). Got: ${to}`, + ); + } + // Throws AIConfigError with provider list on an unknown provider. + const { recipe } = resolveRecipe(to); + if (!recipe.touchpoints.embedding) { + throw new Error(`Provider ${recipe.id} has no embedding support. Pick an embedding-capable provider:model.`); + } + const toDims = dimFlag ?? embeddingDimsForModel(recipe, to); + if (!toDims || toDims <= 0) { + throw new Error( + `No default dimension known for ${to}. Pass --dim explicitly (see the provider's docs for valid values).`, + ); + } + return { toModel: to, toDims }; +} + +/** + * Pure read: compute the migration workload. Uses the stale-chunk predicates + * with the TARGET signature + includeNullSignature so the count is + * resume-aware — a re-plan mid-migration counts only what remains. + */ +export async function planEmbeddingMigration( + engine: BrainEngine, + opts: { to: string; dim?: number; fromModel?: string; fromDims?: number }, +): Promise { + const { toModel, toDims } = resolveMigrationTarget(opts.to, opts.dim); + + // From-state: caller (CLI) passes the gateway-resolved values; fall back + // to the shipped defaults for gateway-less contexts (unit tests, op probe). + const fromModel = opts.fromModel ?? DEFAULT_EMBEDDING_MODEL; + const fromDims = opts.fromDims ?? DEFAULT_EMBEDDING_DIMENSIONS; + + const col = await readContentChunksEmbeddingDim(engine); + + const sig = migrationSignature(toModel, toDims); + const wide = await engine.countStaleChunks({ signature: sig, includeNullSignature: true }); + const narrow = await engine.countStaleChunks({ signature: sig }); + const totalChars = await engine.sumStaleChunkChars({ signature: sig, includeNullSignature: true }); + + const price = lookupEmbeddingPrice(toModel); + const estCostUsd = price.kind === 'known' + ? estimateCostFromChars(totalChars, price.pricePerMTok) + : 0; + + let resuming = false; + try { + const stateStr = await engine.getConfig(MIGRATION_STATE_KEY); + if (stateStr) { + const state = JSON.parse(stateStr) as MigrationState; + resuming = state.to_model === toModel && state.to_dims === toDims; + } + } catch { + // Corrupt state marker — treat as fresh. + } + + // Sunset companion warning: migrating embeddings off a provider whose + // reranker is still configured leaves rerank on the outgoing provider. + let rerankerWarning: string | null = null; + try { + const rr = await engine.getConfig('search.reranker.model'); + const outgoingProvider = fromModel.split(':')[0]; + const targetProvider = toModel.split(':')[0]; + if (rr && outgoingProvider !== targetProvider && rr.startsWith(`${outgoingProvider}:`)) { + rerankerWarning = + `search.reranker.model is still ${rr} (the outgoing provider). ` + + `If that provider is sunsetting, also update or disable the reranker: ` + + `gbrain config set search.reranker.enabled false`; + } + } catch { + // Reranker warning is cosmetic. + } + + return { + from_model: fromModel, + from_dims: fromDims, + column_dims: col.dims, + to_model: toModel, + to_dims: toDims, + dim_change: col.dims !== null && col.dims !== toDims, + chunks_to_embed: wide, + total_chars: totalChars, + null_signature_chunks: wide - narrow, + est_cost_usd: estCostUsd, + price_known: price.kind === 'known', + resuming, + reranker_warning: rerankerWarning, + }; +} + +/** + * Apply the non-embed half of the migration: env gate, state marker, schema + * transition (dim changes only), DB-plane config, file-plane persistence + * (via callback — the core module never touches ~/.gbrain), stale-signature + * invalidation (#3391: includeNullSignature), and query-cache purge. + * + * Ordering makes every step idempotent under a crash + re-run: + * state marker → schema → config → invalidate → cache purge. + * A crash anywhere leaves the state marker set; the re-run re-executes the + * remaining steps (schema transition no-ops when the column is already at + * the target width via the actual-width probe; invalidation matches nothing + * the second time). + */ +export async function applyEmbeddingMigration( + engine: BrainEngine, + plan: EmbeddingMigrationPlan, + opts: { + ignoreEnvOverride?: boolean; + /** Persist target model+dims to the file plane + reconfigure the gateway. */ + persistConfig?: (toModel: string, toDims: number) => void | Promise; + } = {}, +): Promise { + const envWarning = detectEnvOverride(plan.to_model, plan.to_dims); + if (envWarning.triggered && !opts.ignoreEnvOverride) { + return { status: 'refused', reason: 'env_override', warning: envWarning }; + } + + try { + // 1. State marker FIRST — a crash after any later step is resumable. + const state: MigrationState = { + to_model: plan.to_model, + to_dims: plan.to_dims, + from_model: plan.from_model, + from_dims: plan.from_dims, + started_at: new Date().toISOString(), + }; + await engine.setConfig(MIGRATION_STATE_KEY, JSON.stringify(state)); + + // 2. Schema transition when the ACTUAL column width differs from the + // target (probe again — the plan may be stale after a resume). + let schemaTransitioned = false; + const col = await readContentChunksEmbeddingDim(engine); + if (col.dims !== plan.to_dims) { + await runSchemaTransition(engine, plan.to_dims); + schemaTransitioned = true; + } + + // 3. #3391: mark EVERYTHING not in the target space as stale, including + // NULL-signature (pre-v108) pages. After a schema transition this is + // a cheap no-op (the column rebuild already nulled every embedding). + // + // ORDERING (adversarial review): invalidation MUST precede the config + // writes below. On a SAME-dim provider swap there is no schema + // transition to null the vectors, so a crash between "config says new + // provider" and "old vectors invalidated" would leave NEW-space query + // embeddings scored against OLD-space document vectors — silently + // WRONG results. Invalidating first makes the crash window safe: + // config still says the old provider, and the rows are merely stale + // (empty/degraded results, never wrong ones). + const invalidated = await engine.invalidateStaleSignatureEmbeddings({ + signature: migrationSignature(plan.to_model, plan.to_dims), + includeNullSignature: true, + }); + + // 4. DB-plane config (doctor's embedding_width_consistency reads these). + await engine.setConfig('embedding_model', plan.to_model); + await engine.setConfig('embedding_dimensions', String(plan.to_dims)); + + // 5. File plane + gateway (the embed pipeline reads file/env, not DB). + await opts.persistConfig?.(plan.to_model, plan.to_dims); + + // 6. Purge the semantic query cache. The knobs hash folds provider:model + // for callers that thread KnobsHashContext, but legacy callers fall + // back to 'default' — a row they wrote pre-migration must not be + // served post-migration. Best-effort (cache must never block). + let cacheCleared = 0; + try { + const { SemanticQueryCache } = await import('./search/query-cache.ts'); + cacheCleared = await new SemanticQueryCache(engine).clear({}); + } catch { + // Table may not exist on old brains; a miss here is harmless. + } + + return { status: 'applied', invalidated, cache_cleared: cacheCleared, schema_transitioned: schemaTransitioned }; + } catch (err) { + return { status: 'failed', reason: err instanceof Error ? err.message : String(err) }; + } +} + +/** + * Stamp the target signature on every page that is fully embedded but not yet + * stamped. Call after the re-embed drain, BEFORE the completion probe. + * + * Why this exists (adversarial review): the embed loop only stamps a page when + * `stale.length === existing.length` — i.e. when every one of the page's + * chunks was in the SAME batch. `listStaleChunks` is a plain keyset LIMIT with + * no page alignment, so on any corpus larger than one batch (default 2000 + * chunks) the page straddling each boundary is embedded correctly but never + * stamped. Without this reconcile the command reports "incomplete" + exit 1 on + * a perfectly-migrated brain, and the re-run re-invalidates and PAYS AGAIN for + * those pages — breaking the "already-migrated chunks are never re-embedded" + * contract. + * + * Safety: this is only sound because `applyEmbeddingMigration` invalidated + * (NULLed) every chunk that was NOT already in the target space. So "page has + * zero NULL-embedding chunks" ⇒ "every chunk on this page was embedded in the + * target space during this run". Pages with any remaining NULL chunk (a real + * embed failure) are deliberately left unstamped so the completion probe still + * reports them. + * + * Returns the number of pages stamped. + */ +export async function reconcilePageSignatures( + engine: BrainEngine, + plan: EmbeddingMigrationPlan, +): Promise { + const sig = migrationSignature(plan.to_model, plan.to_dims); + const rows = await engine.executeRaw<{ slug: string }>( + `UPDATE pages p + SET embedding_signature = $1 + WHERE p.deleted_at IS NULL + AND (p.embedding_signature IS DISTINCT FROM $1) + AND EXISTS (SELECT 1 FROM content_chunks c WHERE c.page_id = p.id) + AND NOT EXISTS ( + SELECT 1 FROM content_chunks c + WHERE c.page_id = p.id AND c.embedding IS NULL + ) + RETURNING p.slug`, + [sig], + ); + return (rows as unknown[]).length; +} + +/** + * Finish bookkeeping after the re-embed drains: clear the in-flight marker, + * stamp the completion record. Call ONLY when countStaleChunks() === 0. + */ +export async function completeEmbeddingMigration( + engine: BrainEngine, + plan: EmbeddingMigrationPlan, +): Promise { + await engine.unsetConfig(MIGRATION_STATE_KEY); + await engine.setConfig( + MIGRATION_COMPLETED_KEY, + JSON.stringify({ + to_model: plan.to_model, + to_dims: plan.to_dims, + from_model: plan.from_model, + completed_at: new Date().toISOString(), + }), + ); +} diff --git a/src/core/engine.ts b/src/core/engine.ts index f87ae5d38..b0b78acd4 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -1005,8 +1005,13 @@ export interface BrainEngine { * counts across every source in the brain. Operators running * `gbrain embed --stale --source media-corpus` expect only that * source's NULLs touched; the caller threads `sourceId` here. + * + * `includeNullSignature` (only meaningful with `signature`, #3391): also + * count embedded chunks whose page has NO recorded signature (v108 + * grandfathered). Provider-migration paths set this so pre-stamp pages + * aren't silently left in the old embedding space. */ - countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise; + countStaleChunks(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise; /** * Sum of LENGTH(chunk_text) over stale chunks — the character-count * backlog the embed phase / embed-backfill will process. Sibling of @@ -1020,8 +1025,10 @@ export interface BrainEngine { * model signature (a model/dims swap). NULL signature is GRANDFATHERED * (never counted) so the post-migration corpus isn't flagged en masse. * Omit `signature` for the legacy `embedding IS NULL`-only count. + * `includeNullSignature` lifts the grandfather clause (#3391) — see + * countStaleChunks. */ - sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise; + sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise; /** * Stamp `pages.embedding_signature = signature` for one page. Called after * a page's chunks are (re)embedded so a later model swap can detect it as @@ -1036,8 +1043,15 @@ export interface BrainEngine { * drift pages flow through the existing NULL-embedding cursor (keeps * listStaleChunks's keyset pagination untouched). GRANDFATHER: NULL * signature is never invalidated. `sourceId` scopes the sweep. + * + * `includeNullSignature` (#3391): ALSO invalidate embedded chunks whose + * page signature is NULL (pre-v108 pages that predate the stamp). After a + * provider/model swap those vectors are in the old embedding space; the + * default grandfather clause would silently keep them mixed into the new + * index. `gbrain migrate embeddings` and `embed --stale + * --include-null-signature` set this. */ - invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise; + invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise; /** * Return every chunk where embedding IS NULL, with the metadata needed * to call embedBatch + upsertChunks. The `embedding` column is omitted diff --git a/src/core/operations.ts b/src/core/operations.ts index ff6dfc139..04fb9856e 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -4555,6 +4555,90 @@ const code_traversal_cache_clear: Operation = { cliHints: { name: 'code_traversal_cache_clear', hidden: true }, }; +// --- #3390: provider-agnostic embedding migration --- + +const migrate_embeddings: Operation = { + name: 'migrate_embeddings', + description: 'Re-embed the brain onto a different embedding provider/model (#3390): schema dimension transition, NULL-signature (#3391) invalidation, query-cache purge, resumable re-embed. Without yes=true returns the plan + cost estimate only. Local-only admin op; the primary surface is `gbrain migrate embeddings`.', + params: { + to: { type: 'string', required: true, description: 'Target provider:model (e.g. openai:text-embedding-3-small).' }, + dim: { type: 'number', description: "Target dimensions. Defaults to the provider recipe's declared width; required when the recipe declares none." }, + dry_run: { type: 'boolean', description: 'Plan + cost estimate only; change nothing.' }, + yes: { type: 'boolean', description: 'Confirm the re-embed spend + destructive schema change. Required for a live run.' }, + }, + mutating: true, + scope: 'admin', + localOnly: true, + handler: async (ctx, p) => { + // Belt-and-braces on top of localOnly (the get_recent_transcripts + // pattern): a schema-rebuilding, money-spending op must never be + // reachable from a remote transport even if a future dispatch path + // forgets the localOnly filter. + if (ctx.remote !== false) { + throw new Error('migrate_embeddings is local-only. Run `gbrain migrate embeddings` on the host.'); + } + const { + planEmbeddingMigration, applyEmbeddingMigration, completeEmbeddingMigration, + reconcilePageSignatures, migrationSignature, + } = await import('./embedding-migration.ts'); + const to = p.to as string; + const dim = p.dim as number | undefined; + let fromModel: string | undefined; + let fromDims: number | undefined; + try { + const { getEmbeddingModel, getEmbeddingDimensions } = await import('./ai/gateway.ts'); + fromModel = getEmbeddingModel(); + fromDims = getEmbeddingDimensions(); + } catch { /* gateway unconfigured — plan falls back to defaults */ } + const plan = await planEmbeddingMigration(ctx.engine, { + to, + ...(dim !== undefined && { dim }), + ...(fromModel !== undefined && { fromModel }), + ...(fromDims !== undefined && { fromDims }), + }); + if (ctx.dryRun || p.dry_run === true || p.yes !== true) { + return { status: p.yes === true || p.dry_run === true ? 'planned' : 'needs_confirmation', plan }; + } + const { persistEmbeddingFileConfig, probeTargetProvider } = await import('../commands/migrate-embeddings.ts'); + // Safety parity with the CLI path: probe the target provider BEFORE any + // mutation. Without this, `yes:true` would drop the embedding column and + // only then discover the key/model/dim is wrong. + const probe = await probeTargetProvider(plan.to_model, plan.to_dims); + if (!probe.ok) return { status: 'failed', reason: probe.message, plan }; + const applied = await applyEmbeddingMigration(ctx.engine, plan, { + persistConfig: (m, d) => persistEmbeddingFileConfig(m, d), + }); + if (applied.status !== 'applied') return { ...applied, plan }; + const { runEmbedCore } = await import('../commands/embed.ts'); + // singleFlight parity with the CLI path: takes the same per-source + // embed-backfill lock so this can't race a queued embed-backfill job on + // the NULL→non-NULL upsert (the TODOS:2299 class). + const embedResult = await runEmbedCore(ctx.engine, { + stale: true, catchUp: true, singleFlight: true, includeNullSignature: true, quiet: true, + }); + // Stamp batch-boundary pages before probing for completion (see + // reconcilePageSignatures — the embed loop's all-or-nothing stamp rule + // skips any page split across two stale batches). + const reconciled = await reconcilePageSignatures(ctx.engine, plan); + const remaining = await ctx.engine.countStaleChunks({ + signature: migrationSignature(plan.to_model, plan.to_dims), + includeNullSignature: true, + }); + if (remaining === 0) await completeEmbeddingMigration(ctx.engine, plan); + return { + status: remaining === 0 ? 'completed' : 'incomplete', + plan, + embedded: embedResult.embedded, + remaining, + signatures_reconciled: reconciled, + invalidated: applied.invalidated, + schema_transitioned: applied.schema_transitioned, + cache_cleared: applied.cache_cleared, + }; + }, + cliHints: { name: 'migrate-embeddings', hidden: true }, +}; + // --- v0.36 Phase 2: search_by_image (image-as-query) --- const search_by_image: Operation = { @@ -5657,6 +5741,8 @@ export const operations: Operation[] = [ code_blast, code_flow, // v0.34 W3b: code_traversal_cache admin clear op code_traversal_cache_clear, + // #3390: provider-agnostic embedding migration (local-only admin) + migrate_embeddings, // v0.40.6.0 Schema Cathedral v3: 9 new ops — 7 read + 2 admin (NOT // localOnly per D2 so remote agents (your OpenClaw, etc.) can author packs). // schema_apply_mutations is batched per D10 — one MCP tool, N diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 453486116..d930f421a 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -2428,15 +2428,21 @@ export class PGLiteEngine implements BrainEngine { /** * Build the stale-chunk WHERE clause + positional params. embed_skip is * always excluded. `signature` widens "stale" to include embedding_signature - * drift (NULL grandfathered → never stale). Shared by countStaleChunks + + * drift (NULL grandfathered → never stale). `includeNullSignature` (#3391) + * lifts the grandfather clause so pre-stamp pages count as stale too + * (provider-migration paths). Shared by countStaleChunks + * sumStaleChunkChars so they can't drift. */ - private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } { + private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): { where: string; params: unknown[] } { const params: unknown[] = []; const conds: string[] = []; if (opts?.signature !== undefined) { params.push(opts.signature); - conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`); + conds.push( + opts.includeNullSignature + ? `(cc.embedding IS NULL OR p.embedding_signature IS NULL OR p.embedding_signature <> $${params.length})` + : `(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`, + ); } else { conds.push(`cc.embedding IS NULL`); } @@ -2448,7 +2454,7 @@ export class PGLiteEngine implements BrainEngine { return { where: conds.join(' AND '), params }; } - async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise { + async countStaleChunks(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise { // D7: source-scoped count for `gbrain embed --stale --source X`. Always // JOIN pages so embed-skip + signature predicates apply. PGLite is // PostgreSQL 17.5 in WASM and supports the full JSONB operator set. @@ -2464,7 +2470,7 @@ export class PGLiteEngine implements BrainEngine { return Number(count); } - async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise { + async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise { // Sibling of countStaleChunks: same stale predicate, summing chunk_text // length for the sync cost preview. ::bigint guards int4 overflow. const { where, params } = this.buildStaleChunkWhere(opts); @@ -2486,24 +2492,29 @@ export class PGLiteEngine implements BrainEngine { ); } - async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise { + async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise { // NULL out embeddings whose page signature is set AND differs from the - // current model signature. GRANDFATHER: NULL signature untouched. Feeds - // the existing NULL-embedding cursor so listStaleChunks stays unchanged. + // current model signature. GRANDFATHER: NULL signature untouched — + // UNLESS includeNullSignature (#3391): provider migrations must not + // leave pre-stamp pages in the old embedding space. Feeds the existing + // NULL-embedding cursor so listStaleChunks stays unchanged. const params: unknown[] = [opts.signature]; let srcClause = ''; if (opts.sourceId !== undefined) { params.push(opts.sourceId); srcClause = ` AND p.source_id = $${params.length}`; } + const sigClause = opts.includeNullSignature + ? `(p.embedding_signature IS NULL OR p.embedding_signature <> $1)` + : `p.embedding_signature IS NOT NULL + AND p.embedding_signature <> $1`; const { rows } = await this.db.query( `UPDATE content_chunks cc SET embedding = NULL, embedded_at = NULL FROM pages p WHERE cc.page_id = p.id AND cc.embedding IS NOT NULL - AND p.embedding_signature IS NOT NULL - AND p.embedding_signature <> $1${srcClause} + AND ${sigClause}${srcClause} RETURNING cc.page_id`, params, ); diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 1f8657e23..eb830ff26 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -2557,15 +2557,21 @@ export class PostgresEngine implements BrainEngine { /** * Build the stale-chunk WHERE clause + positional params for sql.unsafe. * embed_skip always excluded. `signature` widens "stale" to include - * embedding_signature drift (NULL grandfathered). Shared by - * countStaleChunks + sumStaleChunkChars (parity with the PGLite sibling). + * embedding_signature drift (NULL grandfathered). `includeNullSignature` + * (#3391) lifts the grandfather clause so pre-stamp pages count as stale + * too (provider-migration paths). Shared by countStaleChunks + + * sumStaleChunkChars (parity with the PGLite sibling). */ - private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } { + private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): { where: string; params: unknown[] } { const params: unknown[] = []; const conds: string[] = []; if (opts?.signature !== undefined) { params.push(opts.signature); - conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`); + conds.push( + opts.includeNullSignature + ? `(cc.embedding IS NULL OR p.embedding_signature IS NULL OR p.embedding_signature <> $${params.length})` + : `(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`, + ); } else { conds.push(`cc.embedding IS NULL`); } @@ -2577,10 +2583,11 @@ export class PostgresEngine implements BrainEngine { return { where: conds.join(' AND '), params }; } - async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise { + async countStaleChunks(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise { // Always JOIN pages so the embed_skip + signature predicates apply. // D7: source_id scoping. v0.41.31: optional signature widens staleness - // to embedding_signature drift (NULL grandfathered). + // to embedding_signature drift (NULL grandfathered unless + // includeNullSignature, #3391). const { where, params } = this.buildStaleChunkWhere(opts); // RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING). return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => { @@ -2595,7 +2602,7 @@ export class PostgresEngine implements BrainEngine { }); } - async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise { + async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise { // Sibling of countStaleChunks: same stale predicate, summing chunk_text // length for the sync cost preview. ::bigint guards int4 overflow. const { where, params } = this.buildStaleChunkWhere(opts); @@ -2617,24 +2624,29 @@ export class PostgresEngine implements BrainEngine { `; } - async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise { + async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise { // NULL embeddings whose page signature is set AND differs from current. - // GRANDFATHER: NULL signature untouched. Feeds the NULL-embedding cursor - // so listStaleChunks stays unchanged. RETURNING → row count. + // GRANDFATHER: NULL signature untouched — UNLESS includeNullSignature + // (#3391): provider migrations must not leave pre-stamp pages in the old + // embedding space. Feeds the NULL-embedding cursor so listStaleChunks + // stays unchanged. RETURNING → row count. const params: unknown[] = [opts.signature]; let srcClause = ''; if (opts.sourceId !== undefined) { params.push(opts.sourceId); srcClause = ` AND p.source_id = $${params.length}`; } + const sigClause = opts.includeNullSignature + ? `(p.embedding_signature IS NULL OR p.embedding_signature <> $1)` + : `p.embedding_signature IS NOT NULL + AND p.embedding_signature <> $1`; const rows = await this.sql.unsafe( `UPDATE content_chunks cc SET embedding = NULL, embedded_at = NULL FROM pages p WHERE cc.page_id = p.id AND cc.embedding IS NOT NULL - AND p.embedding_signature IS NOT NULL - AND p.embedding_signature <> $1${srcClause} + AND ${sigClause}${srcClause} RETURNING cc.page_id`, params as Parameters[1], ); diff --git a/src/core/retrieval-upgrade-planner.ts b/src/core/retrieval-upgrade-planner.ts index 402a00835..1edce6522 100644 --- a/src/core/retrieval-upgrade-planner.ts +++ b/src/core/retrieval-upgrade-planner.ts @@ -63,6 +63,7 @@ import type { BrainEngine } from './engine.ts'; import { MARKDOWN_CHUNKER_VERSION } from './chunkers/recursive.ts'; import { lookupEmbeddingPrice, estimateCostFromChars } from './embedding-pricing.ts'; import { computeReembedEstimate } from './post-upgrade-reembed.ts'; +import { hnswIndexExpected } from './vector-index.ts'; // ============================================================================ // Constants @@ -551,8 +552,12 @@ export async function undoRetrievalUpgrade(engine: BrainEngine): Promise< * * IF NOT EXISTS on CREATE INDEX makes the operation safe to re-run during * `--resume`. + * + * Exported (#3390) so the provider-agnostic embedding migration + * (src/core/embedding-migration.ts) reuses the SAME dimension-transition + * path instead of duplicating the DDL sequence. */ -async function runSchemaTransition(engine: BrainEngine, targetDim: number): Promise { +export async function runSchemaTransition(engine: BrainEngine, targetDim: number): Promise { // v0.41 fix: only transition the primary text embedding column. // The embedding_image (v0.27.1) and embedding_multimodal (v0.36 / migration // v78) columns use SEPARATE multimodal models (e.g. voyage-multimodal-3 at @@ -595,9 +600,95 @@ async function runSchemaTransition(engine: BrainEngine, targetDim: number): Prom WHERE embedding_image IS NOT NULL`, ); } + + // #3390: the OTHER two dim-pinned columns that carry TEXT-embedding-space + // vectors. Both are created at brain-birth width (migrate.ts v55 for + // query_cache, v42 for facts) and NO migration ever ALTERs them, so before + // this fix a dimension change left them at the old width: + // - query_cache.embedding stayed narrow → every store() AND lookup() + // silently swallowed the width error (by design, so the cache can + // never break search), i.e. a PERMANENT 0% hit rate. + // - facts.embedding stayed narrow → every per-fact embed write failed + // ($N::vector into the old width), and the doctor check that would + // warn is skipped on PGLite (the DEFAULT engine). + // Both are text-embedding-space columns, so they MUST move with + // content_chunks.embedding. The image/multimodal columns above are the + // deliberate exception (separate models, independent dims). + for (const t of TEXT_EMBEDDING_DIM_PINNED_TABLES) { + await transitionDimPinnedColumn(tx, t.table, t.index, t.indexSql, targetDim); + } }); } +/** + * The dim-pinned TEXT-embedding-space columns outside content_chunks. + * `indexSql` is a factory because each table's index carries its own partial + * WHERE clause + opclass, and the opclass must match the column TYPE + * (vector_cosine_ops vs halfvec_cosine_ops). + */ +const TEXT_EMBEDDING_DIM_PINNED_TABLES: ReadonlyArray<{ + table: string; + index: string; + indexSql: (opclass: string) => string; +}> = [ + { + table: 'query_cache', + index: 'idx_query_cache_embedding_hnsw', + indexSql: (opclass) => + `CREATE INDEX IF NOT EXISTS idx_query_cache_embedding_hnsw + ON query_cache USING hnsw (embedding ${opclass}) + WHERE embedding IS NOT NULL`, + }, + { + table: 'facts', + index: 'idx_facts_embedding_hnsw', + indexSql: (opclass) => + `CREATE INDEX IF NOT EXISTS idx_facts_embedding_hnsw + ON facts USING hnsw (embedding ${opclass}) + WHERE embedding IS NOT NULL AND expired_at IS NULL`, + }, +]; + +/** + * Rebuild one dim-pinned embedding column at `targetDim`, PRESERVING its + * existing column type (`vector` vs `halfvec` — migrate.ts picks halfvec when + * the server supports it, and the HNSW opclass must match). No-op when the + * table or column doesn't exist (fresh/older brains). + * + * Dropping the column discards the stored vectors, which is correct: they are + * in the OLD embedding space and unusable after the swap. query_cache is a + * cache (refills on the next query); facts re-embed on their next write / + * `gbrain extract` pass. + */ +async function transitionDimPinnedColumn( + tx: { executeRaw: (sql: string, params?: unknown[]) => Promise }, + table: string, + indexName: string, + indexSql: (opclass: string) => string, + targetDim: number, +): Promise { + const probe = await tx.executeRaw<{ udt_name: string | null }>( + `SELECT udt_name FROM information_schema.columns + WHERE table_schema = 'public' AND table_name = $1 AND column_name = 'embedding'`, + [table], + ); + const udt = probe[0]?.udt_name; + if (!udt) return; // table or column absent — nothing to transition + // Preserve the column type; anything unexpected falls back to `vector`. + const columnType: 'vector' | 'halfvec' = udt.toLowerCase() === 'halfvec' ? 'halfvec' : 'vector'; + const opclass = columnType === 'halfvec' ? 'halfvec_cosine_ops' : 'vector_cosine_ops'; + + await tx.executeRaw(`DROP INDEX IF EXISTS ${indexName}`); + await tx.executeRaw(`ALTER TABLE ${table} DROP COLUMN IF EXISTS embedding`); + await tx.executeRaw(`ALTER TABLE ${table} ADD COLUMN embedding ${columnType}(${targetDim})`); + // HNSW has a per-type dimension ceiling; above it pgvector refuses the + // index and exact scans remain the (correct, slower) path. Mirrors the + // same guard in migrate.ts's original DDL. + if (hnswIndexExpected(columnType, targetDim)) { + await tx.executeRaw(indexSql(opclass)); + } +} + // ============================================================================ // Helpers // ============================================================================ diff --git a/src/core/search/mode.ts b/src/core/search/mode.ts index cb5a14782..bb4df8d42 100644 --- a/src/core/search/mode.ts +++ b/src/core/search/mode.ts @@ -756,7 +756,17 @@ export function attributeKnob( // slugs written by a process without it, and vice versa. Same one-time // global cold-miss pattern as the bumps above; refills within // cache.ttl_seconds (3600s default). -export const KNOBS_HASH_VERSION = 12; +// +// bump 12→13 (#3390/#3391): embedding-provider migration wave. The `prov=` +// component only isolates callers that thread KnobsHashContext.embeddingModel; +// legacy callers hash `prov=default` before AND after a provider swap, so a +// cache row computed against the pre-migration embedding space could be +// served post-migration. `gbrain migrate embeddings` purges query_cache +// directly at swap time; this version bump is the belt-and-braces for rows +// written between the #3391 stale-fix (which changes which chunks count as +// current) and the operator's migration run. Same one-time global cold-miss +// pattern as the bumps above. +export const KNOBS_HASH_VERSION = 13; /** * v0.36 (D8 / CDX-2) — second-arg context for the cache key. The diff --git a/test/cross-modal-phase1.test.ts b/test/cross-modal-phase1.test.ts index 19689a324..177d2030c 100644 --- a/test/cross-modal-phase1.test.ts +++ b/test/cross-modal-phase1.test.ts @@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => { return resolveSearchMode({ mode: 'balanced' }); } - test('KNOBS_HASH_VERSION is 12 (cross-modal still appended; 11→12 hard-exclude fold #2825)', () => { + test('KNOBS_HASH_VERSION is 13 (cross-modal still appended; 12→13 embedding-provider migration #3390)', () => { // v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3 // with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) + // v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields. @@ -146,7 +146,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => { // v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type // finally reaches asymmetric providers — pre-fix rows were keyed on // document-side query vectors. #2825: 11→12 hard-exclude fold (hx=). - expect(KNOBS_HASH_VERSION).toBe(12); + expect(KNOBS_HASH_VERSION).toBe(13); }); test('flipping unified_multimodal changes the hash', () => { diff --git a/test/e2e/migrate-embeddings-postgres.test.ts b/test/e2e/migrate-embeddings-postgres.test.ts new file mode 100644 index 000000000..f33e67dc2 --- /dev/null +++ b/test/e2e/migrate-embeddings-postgres.test.ts @@ -0,0 +1,247 @@ +/** + * #3390/#3391 — embedding migration on REAL Postgres + pgvector. + * + * Engine-parity pin for the #3391 includeNullSignature predicates + * (identical semantics to the PGLite coverage in + * test/embedding-migration.test.ts) PLUS the migration path that genuinely + * differs on real Postgres: runSchemaTransition's DROP INDEX / DROP COLUMN / + * ADD vector(N) / CREATE hnsw sequence against a native pgvector, followed + * by a full re-embed through the real pipeline with a fake transport. + * + * Gated by DATABASE_URL (docs/TESTING.md: docker pgvector/pgvector:pg16, + * e.g. on :5435). Restores the original column width in afterAll so later + * e2e files see the schema they expect. + * + * Run: DATABASE_URL=postgres://...gbrain_test bun test test/e2e/migrate-embeddings-postgres.test.ts + */ +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import type { PostgresEngine } from '../../src/core/postgres-engine.ts'; +import { hasDatabase, setupDB, teardownDB } from './helpers.ts'; +import { + configureGateway, + resetGateway, + __setEmbedTransportForTests, +} from '../../src/core/ai/gateway.ts'; +import { runEmbedCore } from '../../src/commands/embed.ts'; +import { + planEmbeddingMigration, + applyEmbeddingMigration, + completeEmbeddingMigration, + migrationSignature, + MIGRATION_STATE_KEY, +} from '../../src/core/embedding-migration.ts'; +import { runSchemaTransition } from '../../src/core/retrieval-upgrade-planner.ts'; +import type { ChunkInput } from '../../src/core/types.ts'; + +const RUN = hasDatabase(); +const d = RUN ? describe : describe.skip; + +let engine: PostgresEngine; +let originalDims: number; +let currentDims = 0; // fake-transport vector width, set per phase +const savedEnv: Record = {}; + +async function columnDims(): Promise { + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' + AND attnum > 0 AND NOT attisdropped`, + ); + return Number(rows[0]?.dim); +} + +async function embeddingColWidth(table: string): Promise { + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = $1::regclass AND attname = 'embedding' + AND attnum > 0 AND NOT attisdropped`, + [table], + ); + return Number(rows[0]?.dim); +} + +async function seedEmbedded(slug: string, text: string, signature: string | null): Promise { + await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` }); + const chunks: ChunkInput[] = [ + { chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4 }, + ]; + await engine.upsertChunks(slug, chunks); + await engine.executeRaw( + `UPDATE content_chunks + SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector + WHERE page_id = (SELECT id FROM pages WHERE slug = $2 AND source_id = 'default')`, + [originalDims, slug], + ); + if (signature !== null) { + await engine.setPageEmbeddingSignature(slug, { signature }); + } +} + +d('embedding migration (live Postgres + pgvector)', () => { + beforeAll(async () => { + for (const k of ['GBRAIN_EMBEDDING_MODEL', 'GBRAIN_EMBEDDING_DIMENSIONS']) { + savedEnv[k] = process.env[k]; + delete process.env[k]; + } + engine = await setupDB(); + originalDims = await columnDims(); + + resetGateway(); + configureGateway({ + embedding_model: 'openai:text-embedding-3-small', + embedding_dimensions: 1536, + env: { OPENAI_API_KEY: 'sk-test-fake' }, + }); + __setEmbedTransportForTests(async ({ values }: { values: string[] }) => ({ + embeddings: values.map(() => new Array(currentDims).fill(0).map((_, i) => Math.cos(i) * 0.01 + 0.002)), + usage: { tokens: values.length * 4 }, + }) as never); + }, 60000); + + afterAll(async () => { + __setEmbedTransportForTests(null); + resetGateway(); + // Restore the shared test DB's column width for subsequent e2e files. + if (engine && originalDims && (await columnDims()) !== originalDims) { + await runSchemaTransition(engine, originalDims); + } + await teardownDB(); + for (const [k, v] of Object.entries(savedEnv)) { + if (v === undefined) delete process.env[k]; + else process.env[k] = v; + } + }, 60000); + + test('#3391 predicates behave identically to PGLite (engine parity)', async () => { + await seedEmbedded('parity/legacy', 'abcde', null); // NULL sig + await seedEmbedded('parity/drifted', 'fghij', 'old:model:1'); // mismatched + await seedEmbedded('parity/fresh', 'klmno', 'new:model:1'); // matching + const sig = 'new:model:1'; + + // Grandfathered (no flag): only the drifted page counts. + expect(await engine.countStaleChunks({ signature: sig })).toBe(1); + expect(await engine.sumStaleChunkChars({ signature: sig })).toBe(5); + // Widened (#3391): legacy counts too; matching still excluded. + expect(await engine.countStaleChunks({ signature: sig, includeNullSignature: true })).toBe(2); + expect(await engine.sumStaleChunkChars({ signature: sig, includeNullSignature: true })).toBe(10); + + // Invalidation parity: default grandfathers, flag lifts it, idempotent. + expect(await engine.invalidateStaleSignatureEmbeddings({ signature: sig })).toBe(1); + expect(await engine.invalidateStaleSignatureEmbeddings({ signature: sig, includeNullSignature: true })).toBe(1); // legacy + expect(await engine.invalidateStaleSignatureEmbeddings({ signature: sig, includeNullSignature: true })).toBe(0); + const kept = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE p.slug = 'parity/fresh' AND cc.embedding IS NOT NULL`, + ); + expect(Number(kept[0]?.n)).toBe(1); + + // Clean up parity fixtures so the migration test below starts exact. + await engine.executeRaw(`DELETE FROM pages WHERE slug LIKE 'parity/%'`); + }, 30000); + + test('full migration with a dimension change on real pgvector', async () => { + const targetDims = originalDims === 1536 ? 1024 : 1536; + const toModel = targetDims === 1536 ? 'openai:text-embedding-3-small' : 'voyage:voyage-3-large'; + + // Brain state: one current-signature page, one pre-v108 NULL-signature + // page, one never-embedded page. + await seedEmbedded('mig/current', 'aaaaa', migrationSignature('zeroentropyai:zembed-1', originalDims)); + await seedEmbedded('mig/legacy', 'bbbbb', null); + await engine.putPage('mig/pending', { type: 'note', title: 'pending', compiled_truth: '# pending' }); + await engine.upsertChunks('mig/pending', [ + { chunk_index: 0, chunk_text: 'ccccc', chunk_source: 'compiled_truth', token_count: 2 }, + ]); + + const plan = await planEmbeddingMigration(engine, { + to: toModel, + dim: targetDims, + fromModel: 'zeroentropyai:zembed-1', + fromDims: originalDims, + }); + expect(plan.dim_change).toBe(true); + expect(plan.chunks_to_embed).toBe(3); + expect(plan.null_signature_chunks).toBe(1); + + // Apply: runSchemaTransition on REAL Postgres (native pgvector DDL). + const persisted: Array<[string, number]> = []; + const applied = await applyEmbeddingMigration(engine, plan, { + persistConfig: (m, dd) => { persisted.push([m, dd]); }, + }); + expect(applied.status).toBe('applied'); + if (applied.status !== 'applied') throw new Error('unreachable'); + expect(applied.schema_transitioned).toBe(true); + expect(persisted).toEqual([[toModel, targetDims]]); + expect(await columnDims()).toBe(targetDims); + + // All THREE dim-pinned text-embedding-space columns move together on real + // pgvector (query_cache + facts are created at brain-birth width and no + // migration ever ALTERs them — before the fix they stayed narrow, which + // silently killed the query cache and every per-fact embed write). + expect(await embeddingColWidth('query_cache')).toBe(targetDims); + expect(await embeddingColWidth('facts')).toBe(targetDims); + // And the rebuilt columns actually ACCEPT a vector at the new width. + const nv = `[${new Array(targetDims).fill(0.01).join(',')}]`; + await engine.executeRaw( + `INSERT INTO query_cache (id, query_text, source_id, embedding) + VALUES ('pg-post-migrate', 'q', 'default', $1::vector)`, + [nv], + ); + await engine.executeRaw( + `INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability, source, confidence, embedding) + VALUES ('default', 'pg-e', 'pg-f', 'fact', 'private', 'medium', 'test', 1.0, $1::vector)`, + [nv], + ); + const accepted = await engine.executeRaw<{ qc: number; f: number }>( + `SELECT (SELECT count(*)::int FROM query_cache WHERE id = 'pg-post-migrate') AS qc, + (SELECT count(*)::int FROM facts WHERE entity_slug = 'pg-e') AS f`, + ); + expect(Number(accepted[0]?.qc)).toBe(1); + expect(Number(accepted[0]?.f)).toBe(1); + expect(await engine.getConfig('embedding_model')).toBe(toModel); + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeTruthy(); + + // HNSW index rebuilt inside the same transaction. + const idx = await engine.executeRaw<{ indexname: string }>( + `SELECT indexname FROM pg_indexes WHERE tablename = 'content_chunks' AND indexname = 'idx_chunks_embedding'`, + ); + expect(idx.length).toBe(1); + + // Re-embed through the real pipeline at the new width. NOTE: no + // resetGateway() here — it would clear the installed fake transport. + currentDims = targetDims; + configureGateway({ + embedding_model: toModel, + embedding_dimensions: targetDims, + env: { OPENAI_API_KEY: 'sk-test-fake', VOYAGE_API_KEY: 'va-test-fake' }, + }); + const res = await runEmbedCore(engine, { + stale: true, catchUp: true, includeNullSignature: true, quiet: true, + }); + expect(res.embedded).toBe(3); + + // Everything is in the target space, including the NULL-signature page. + const newSig = migrationSignature(toModel, targetDims); + expect(await engine.countStaleChunks({ signature: newSig, includeNullSignature: true })).toBe(0); + const sigs = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM pages WHERE slug LIKE 'mig/%' AND embedding_signature = $1`, + [newSig], + ); + expect(Number(sigs[0]?.n)).toBe(3); + + // Vector search works against the new column at the new width. + const qvec = `[${new Array(targetDims).fill(0).map((_, i) => (Math.cos(i) * 0.01 + 0.002).toFixed(6)).join(',')}]`; + const rows = await engine.executeRaw<{ slug: string }>( + `SELECT p.slug FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE cc.embedding IS NOT NULL + ORDER BY cc.embedding <=> $1::vector + LIMIT 3`, + [qvec], + ); + expect(rows.length).toBe(3); + + await completeEmbeddingMigration(engine, plan); + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy(); + }, 120000); +}); diff --git a/test/embedding-migration.test.ts b/test/embedding-migration.test.ts new file mode 100644 index 000000000..90e7ddd84 --- /dev/null +++ b/test/embedding-migration.test.ts @@ -0,0 +1,424 @@ +/** + * #3390 — provider-agnostic embedding migration (planner/applier, PGLite) + + * #3391 — NULL-signature rows must be treatable as stale. + * + * Covers: + * - resolveMigrationTarget validation (provider:model shape, unknown + * provider, recipe-default dims, --dim override, dims-required recipes) + * - includeNullSignature widening on countStaleChunks / + * sumStaleChunkChars / invalidateStaleSignatureEmbeddings (#3391) + * - planEmbeddingMigration counts, cost math, price_known, resuming + * - applyEmbeddingMigration: env-override refusal fires BEFORE any + * mutation, schema transition on dim change, DB-plane config writes, + * state marker, NULL-signature-inclusive invalidation, query-cache + * purge, idempotent re-apply (resume) + * - completeEmbeddingMigration bookkeeping + * - migrate_embeddings op contract (admin, localOnly, remote guard, + * needs_confirmation without yes) + * + * Canonical PGLite block (CLAUDE.md R3+R4). Engine parity for the #3391 + * predicates is pinned on real Postgres in + * test/e2e/migrate-embeddings-postgres.test.ts. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { withEnv } from './helpers/with-env.ts'; +import type { ChunkInput } from '../src/core/types.ts'; +import { + resolveMigrationTarget, + planEmbeddingMigration, + applyEmbeddingMigration, + completeEmbeddingMigration, + reconcilePageSignatures, + migrationSignature, + MIGRATION_STATE_KEY, + MIGRATION_COMPLETED_KEY, +} from '../src/core/embedding-migration.ts'; +import { estimateCostFromChars } from '../src/core/embedding-pricing.ts'; +import { operations } from '../src/core/operations.ts'; + +let engine: PGLiteEngine; +let colDim: number; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 30000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`, + ); + colDim = Number(rows[0]?.dim); +}); + +/** Seed a page with one embedded chunk and a given signature (null = pre-v108). */ +async function seedEmbedded(slug: string, text: string, signature: string | null): Promise { + await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` }); + const chunks: ChunkInput[] = [ + { chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4, embedding: undefined }, + ]; + await engine.upsertChunks(slug, chunks); + await engine.executeRaw( + `UPDATE content_chunks + SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector + WHERE page_id = (SELECT id FROM pages WHERE slug = $2 AND source_id = 'default')`, + [colDim, slug], + ); + if (signature !== null) { + await engine.setPageEmbeddingSignature(slug, { signature }); + } +} + +/** Actual vector(N)/halfvec(N) width of a table's `embedding` column. */ +async function embeddingColWidth(table: string): Promise { + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = $1::regclass AND attname = 'embedding' + AND attnum > 0 AND NOT attisdropped`, + [table], + ); + return Number(rows[0]?.dim); +} + +/** Seed a page with one UNembedded chunk (classic NULL-embedding stale). */ +async function seedUnembedded(slug: string, text: string): Promise { + await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` }); + await engine.upsertChunks(slug, [ + { chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4, embedding: undefined }, + ]); +} + +describe('resolveMigrationTarget', () => { + test('requires provider:model shape', () => { + expect(() => resolveMigrationTarget('text-embedding-3-small')).toThrow(/provider:model/); + }); + test('unknown provider throws', () => { + expect(() => resolveMigrationTarget('nosuchprovider:some-model')).toThrow(); + }); + test('recipe default dims resolve (openai → 1536)', () => { + expect(resolveMigrationTarget('openai:text-embedding-3-small')).toEqual({ + toModel: 'openai:text-embedding-3-small', + toDims: 1536, + }); + }); + test('explicit --dim wins over recipe default', () => { + expect(resolveMigrationTarget('openai:text-embedding-3-small', 512).toDims).toBe(512); + }); + test('recipe with default_dims=0 requires --dim', () => { + expect(() => resolveMigrationTarget('litellm:my-custom-model')).toThrow(/--dim/); + expect(resolveMigrationTarget('litellm:my-custom-model', 1024).toDims).toBe(1024); + }); +}); + +describe('#3391 includeNullSignature widening', () => { + test('countStaleChunks / sumStaleChunkChars include NULL-signature rows only with the flag', async () => { + await seedEmbedded('legacy', 'abcde', null); // NULL sig, embedded + await seedEmbedded('drifted', 'fghij', 'old:model:1'); // mismatched sig + await seedEmbedded('fresh', 'klmno', 'new:model:1'); // matching sig + const sig = 'new:model:1'; + + // Default (grandfathered): legacy is invisible. + expect(await engine.countStaleChunks({ signature: sig })).toBe(1); + expect(await engine.sumStaleChunkChars({ signature: sig })).toBe(5); + + // Widened: legacy counts too; matching still excluded. + expect(await engine.countStaleChunks({ signature: sig, includeNullSignature: true })).toBe(2); + expect(await engine.sumStaleChunkChars({ signature: sig, includeNullSignature: true })).toBe(10); + }); + + test('invalidateStaleSignatureEmbeddings with the flag NULLs legacy + drifted, keeps matching', async () => { + await seedEmbedded('legacy', 'abcde', null); + await seedEmbedded('drifted', 'fghij', 'old:model:1'); + await seedEmbedded('fresh', 'klmno', 'new:model:1'); + + const n = await engine.invalidateStaleSignatureEmbeddings({ + signature: 'new:model:1', + includeNullSignature: true, + }); + expect(n).toBe(2); // legacy + drifted; NOT fresh + + expect(await engine.countStaleChunks()).toBe(2); // both now NULL-embedding + const kept = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE p.slug = 'fresh' AND cc.embedding IS NOT NULL`, + ); + expect(Number(kept[0]?.n)).toBe(1); + + // Idempotent. + expect(await engine.invalidateStaleSignatureEmbeddings({ + signature: 'new:model:1', includeNullSignature: true, + })).toBe(0); + }); + + test('default behavior unchanged: NULL signature stays grandfathered without the flag', async () => { + await seedEmbedded('legacy', 'abcde', null); + expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'new:model:1' })).toBe(0); + expect(await engine.countStaleChunks({ signature: 'new:model:1' })).toBe(0); + }); +}); + +describe('planEmbeddingMigration', () => { + test('counts everything not in the target space, splits out NULL-signature chunks, prices it', async () => { + await seedEmbedded('legacy', 'abcde', null); // 5 chars + await seedEmbedded('current', 'fghij', `zeroentropyai:zembed-1:${colDim}`); // 5 chars + await seedUnembedded('pending', 'klmnop'); // 6 chars + + const plan = await planEmbeddingMigration(engine, { + to: 'openai:text-embedding-3-small', + fromModel: 'zeroentropyai:zembed-1', + fromDims: colDim, + }); + + expect(plan.to_model).toBe('openai:text-embedding-3-small'); + expect(plan.to_dims).toBe(1536); + expect(plan.column_dims).toBe(colDim); + expect(plan.dim_change).toBe(colDim !== 1536); + expect(plan.chunks_to_embed).toBe(3); // all three + expect(plan.null_signature_chunks).toBe(1); // 'legacy' only + expect(plan.total_chars).toBe(16); + expect(plan.price_known).toBe(true); + expect(plan.est_cost_usd).toBeCloseTo(estimateCostFromChars(16, 0.02), 10); + expect(plan.resuming).toBe(false); + }); + + test('unknown pricing → price_known false, cost 0', async () => { + await seedUnembedded('p1', 'abc'); + const plan = await planEmbeddingMigration(engine, { to: 'litellm:custom', dim: 1024 }); + expect(plan.price_known).toBe(false); + expect(plan.est_cost_usd).toBe(0); + }); + + test('resuming=true when the state marker matches the target', async () => { + await engine.setConfig(MIGRATION_STATE_KEY, JSON.stringify({ + to_model: 'openai:text-embedding-3-small', to_dims: 1536, + from_model: 'x', from_dims: 1, started_at: 'now', + })); + const plan = await planEmbeddingMigration(engine, { to: 'openai:text-embedding-3-small' }); + expect(plan.resuming).toBe(true); + const other = await planEmbeddingMigration(engine, { to: 'openai:text-embedding-3-large' }); + expect(other.resuming).toBe(false); + }); + + test('reranker on the outgoing provider triggers the warning', async () => { + await engine.setConfig('search.reranker.model', 'zeroentropyai:zerank-2'); + const plan = await planEmbeddingMigration(engine, { + to: 'openai:text-embedding-3-small', + fromModel: 'zeroentropyai:zembed-1', + fromDims: 1280, + }); + expect(plan.reranker_warning).toContain('zeroentropyai:zerank-2'); + }); +}); + +describe('applyEmbeddingMigration', () => { + test('env override refuses BEFORE any mutation', async () => { + await withEnv({ GBRAIN_EMBEDDING_MODEL: 'voyage:voyage-3-large' }, async () => { + const plan = await planEmbeddingMigration(engine, { to: 'openai:text-embedding-3-small' }); + const res = await applyEmbeddingMigration(engine, plan); + expect(res.status).toBe('refused'); + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy(); + expect(await engine.getConfig('embedding_model')).toBeFalsy(); + }); + }); + + test('same-dim swap: no schema transition; invalidates legacy + drifted; writes config + state; purges cache', async () => { + await seedEmbedded('legacy', 'abcde', null); + await seedEmbedded('drifted', 'fghij', `old:model:${colDim}`); + // Seed a query-cache row that must not survive the swap. + await engine.executeRaw( + `INSERT INTO query_cache (id, query_text, source_id) VALUES ('qc1', 'stale query', 'default')`, + ); + + const persisted: Array<[string, number]> = []; + const plan = await planEmbeddingMigration(engine, { + to: 'openai:text-embedding-3-small', dim: colDim, // same width → no DDL + }); + const res = await applyEmbeddingMigration(engine, plan, { + persistConfig: (m, d) => { persisted.push([m, d]); }, + }); + + expect(res.status).toBe('applied'); + if (res.status !== 'applied') throw new Error('unreachable'); + expect(res.schema_transitioned).toBe(false); + expect(res.invalidated).toBe(2); // legacy (#3391) + drifted + expect(res.cache_cleared).toBe(1); + expect(persisted).toEqual([['openai:text-embedding-3-small', colDim]]); + expect(await engine.getConfig('embedding_model')).toBe('openai:text-embedding-3-small'); + expect(await engine.getConfig('embedding_dimensions')).toBe(String(colDim)); + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeTruthy(); + + const qc = await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM query_cache`); + expect(Number(qc[0]?.n)).toBe(0); + }); + + test('dim change: schema transition rebuilds the column at the target width; re-apply is a no-op', async () => { + await seedEmbedded('a', 'abcde', null); + const target = colDim === 512 ? 256 : 512; + const plan = await planEmbeddingMigration(engine, { + to: 'openai:text-embedding-3-small', dim: target, + }); + const res = await applyEmbeddingMigration(engine, plan); + expect(res.status).toBe('applied'); + if (res.status !== 'applied') throw new Error('unreachable'); + expect(res.schema_transitioned).toBe(true); + + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0 AND NOT attisdropped`, + ); + expect(Number(rows[0]?.dim)).toBe(target); + + // Every embedding is NULL after the column rebuild → classic stale. + expect(await engine.countStaleChunks()).toBe(1); + + // Resume: second apply must not transition again (column already at target). + const res2 = await applyEmbeddingMigration(engine, plan); + expect(res2.status).toBe('applied'); + if (res2.status !== 'applied') throw new Error('unreachable'); + expect(res2.schema_transitioned).toBe(false); + expect(res2.invalidated).toBe(0); + }); + + test('dim change transitions ALL THREE text-embedding-space columns (blocker: query_cache + facts were left at the old width)', async () => { + const target = colDim === 512 ? 256 : 512; + + // Pre-condition: all three start at the brain-birth width. + expect(await embeddingColWidth('content_chunks')).toBe(colDim); + expect(await embeddingColWidth('query_cache')).toBe(colDim); + expect(await embeddingColWidth('facts')).toBe(colDim); + + const plan = await planEmbeddingMigration(engine, { + to: 'openai:text-embedding-3-small', dim: target, + }); + const res = await applyEmbeddingMigration(engine, plan); + expect(res.status).toBe('applied'); + + // All three must move together. Before the fix, query_cache and facts + // stayed narrow: the query cache silently accepted ZERO rows forever + // (store() + lookup() swallow the width error by design) and every + // per-fact embed write failed. + expect(await embeddingColWidth('content_chunks')).toBe(target); + expect(await embeddingColWidth('query_cache')).toBe(target); + expect(await embeddingColWidth('facts')).toBe(target); + }); + + test('post-transition the query cache can actually STORE a row at the new width', async () => { + const target = colDim === 512 ? 256 : 512; + const plan = await planEmbeddingMigration(engine, { + to: 'openai:text-embedding-3-small', dim: target, + }); + expect((await applyEmbeddingMigration(engine, plan)).status).toBe('applied'); + + // The real regression symptom: a width-mismatched column makes this + // INSERT throw, which query-cache.ts swallows → permanent 0% hit rate. + const vec = `[${new Array(target).fill(0.01).join(',')}]`; + await engine.executeRaw( + `INSERT INTO query_cache (id, query_text, source_id, embedding) + VALUES ('post-migrate', 'q', 'default', $1::vector)`, + [vec], + ); + const rows = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM query_cache WHERE id = 'post-migrate'`, + ); + expect(Number(rows[0]?.n)).toBe(1); + }); + + test('post-transition a fact embedding at the new width inserts (blocker: facts writes broke)', async () => { + const target = colDim === 512 ? 256 : 512; + const plan = await planEmbeddingMigration(engine, { + to: 'openai:text-embedding-3-small', dim: target, + }); + expect((await applyEmbeddingMigration(engine, plan)).status).toBe('applied'); + + const vec = `[${new Array(target).fill(0.02).join(',')}]`; + await engine.executeRaw( + `INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability, source, confidence, embedding) + VALUES ('default', 'e', 'f', 'fact', 'private', 'medium', 'test', 1.0, $1::vector)`, + [vec], + ); + const rows = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM facts WHERE entity_slug = 'e'`, + ); + expect(Number(rows[0]?.n)).toBe(1); + }); + + test('reconcilePageSignatures stamps fully-embedded pages and leaves partially-embedded ones alone', async () => { + // 'done' — every chunk embedded, but signature still the OLD one (the + // batch-boundary shape: embedded correctly, stamp skipped). + await seedEmbedded('done', 'abcde', 'old:model:1'); + // 'partial' — one embedded chunk + one NULL chunk (a real embed failure). + await seedEmbedded('partial', 'fghij', 'old:model:1'); + await engine.upsertChunks('partial', [ + { chunk_index: 0, chunk_text: 'fghij', chunk_source: 'compiled_truth', token_count: 4 }, + { chunk_index: 1, chunk_text: 'not embedded', chunk_source: 'compiled_truth', token_count: 4 }, + ]); + + const plan = await planEmbeddingMigration(engine, { + to: 'openai:text-embedding-3-small', dim: colDim, + }); + const n = await reconcilePageSignatures(engine, plan); + expect(n).toBe(1); // 'done' only + + const sig = `openai:text-embedding-3-small:${colDim}`; + const rows = await engine.executeRaw<{ slug: string; embedding_signature: string | null }>( + `SELECT slug, embedding_signature FROM pages WHERE slug IN ('done','partial') ORDER BY slug`, + ); + expect(rows.find(r => r.slug === 'done')?.embedding_signature).toBe(sig); + expect(rows.find(r => r.slug === 'partial')?.embedding_signature).toBe('old:model:1'); + }); + + test('completeEmbeddingMigration clears the state marker and stamps completion', async () => { + const plan = await planEmbeddingMigration(engine, { to: 'openai:text-embedding-3-small', dim: colDim }); + await applyEmbeddingMigration(engine, plan); + await completeEmbeddingMigration(engine, plan); + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy(); + const done = JSON.parse((await engine.getConfig(MIGRATION_COMPLETED_KEY))!); + expect(done.to_model).toBe('openai:text-embedding-3-small'); + }); +}); + +describe('migrate_embeddings op contract', () => { + const op = operations.find(o => o.name === 'migrate_embeddings')!; + + test('is admin + localOnly + mutating', () => { + expect(op).toBeDefined(); + expect(op.scope).toBe('admin'); + expect(op.localOnly).toBe(true); + expect(op.mutating).toBe(true); + }); + + test('remote callers are refused even if dispatch forgot the localOnly filter', async () => { + await expect( + op.handler({ engine, remote: true } as never, { to: 'openai:text-embedding-3-small' }), + ).rejects.toThrow(/local-only/); + // remote undefined (not strictly false) is ALSO refused — fail-closed. + await expect( + op.handler({ engine } as never, { to: 'openai:text-embedding-3-small' }), + ).rejects.toThrow(/local-only/); + }); + + test('without yes=true it returns the plan only (no mutation)', async () => { + const res = await op.handler( + { engine, remote: false } as never, + { to: 'openai:text-embedding-3-small', dim: colDim }, + ) as { status: string; plan: { to_model: string } }; + expect(res.status).toBe('needs_confirmation'); + expect(res.plan.to_model).toBe('openai:text-embedding-3-small'); + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy(); + }); + + test('signature helper matches currentEmbeddingSignature shape', () => { + expect(migrationSignature('openai:text-embedding-3-small', 1536)) + .toBe('openai:text-embedding-3-small:1536'); + }); +}); diff --git a/test/migrate-embeddings-boundary.serial.test.ts b/test/migrate-embeddings-boundary.serial.test.ts new file mode 100644 index 000000000..b24308c2b --- /dev/null +++ b/test/migrate-embeddings-boundary.serial.test.ts @@ -0,0 +1,165 @@ +/** + * #3390 regression: page straddling a stale-batch BOUNDARY (adversarial review + * blocker 2). + * + * The embed loop stamps `pages.embedding_signature` only when + * `stale.length === existing.length` — i.e. when every chunk of the page landed + * in the SAME batch. `listStaleChunks` is a plain keyset LIMIT with no page + * alignment, so on any corpus bigger than one batch the page split across the + * boundary is embedded correctly but never stamped. + * + * Pre-fix consequence: the completion probe counted that page stale, the + * command printed "Migration incomplete" and exited 1 on a perfectly-migrated + * brain, and the re-run RE-INVALIDATED and RE-PAID for those pages — + * contradicting the "already-migrated chunks are never re-embedded" contract. + * + * Shape here: 3 pages x 2 chunks = 6 chunks with --batch-size 3, so the + * boundary falls mid-page-2. Asserts exit 0 on the first run, every page + * stamped, and ZERO embed work on the second run. + * + * Named `.serial.test.ts`: holds a temp GBRAIN_HOME + an installed fake embed + * transport for its whole beforeAll→afterAll lifecycle, which withEnv() can't + * wrap. + */ +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { + configureGateway, + resetGateway, + __setEmbedTransportForTests, +} from '../src/core/ai/gateway.ts'; +import { runEmbedCore } from '../src/commands/embed.ts'; +import { runMigrateEmbeddings } from '../src/commands/migrate-embeddings.ts'; +import { MIGRATION_STATE_KEY, MIGRATION_COMPLETED_KEY } from '../src/core/embedding-migration.ts'; + +const FROM_DIMS = 1280; +const TO_DIMS = 1536; +const PAGES = ['b-1', 'b-2', 'b-3']; +const PROBE_TEXT = 'gbrain embedding migration probe'; + +let engine: PGLiteEngine; +let tmpHome: string; +const savedEnv: Record = {}; +let currentDims = FROM_DIMS; +let embeddedTexts: string[] = []; + +class ExitError extends Error { + constructor(public code: number) { super(`exit ${code}`); } +} +const exitSeam = (code: number): never => { throw new ExitError(code); }; + +async function runMigrate(args: string[]): Promise { + try { + await runMigrateEmbeddings(engine, args, { exit: exitSeam }); + throw new Error('runMigrateEmbeddings returned without exiting'); + } catch (e) { + if (e instanceof ExitError) return e.code; + throw e; + } +} + +beforeAll(async () => { + for (const k of ['GBRAIN_HOME', 'GBRAIN_EMBEDDING_MODEL', 'GBRAIN_EMBEDDING_DIMENSIONS', 'OPENAI_API_KEY', 'ZEROENTROPY_API_KEY', 'DATABASE_URL']) { + savedEnv[k] = process.env[k]; + delete process.env[k]; + } + tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-migrate-boundary-')); + process.env.GBRAIN_HOME = tmpHome; + mkdirSync(join(tmpHome, '.gbrain'), { recursive: true }); + writeFileSync(join(tmpHome, '.gbrain', 'config.json'), JSON.stringify({ + engine: 'pglite', + embedding_model: 'zeroentropyai:zembed-1', + embedding_dimensions: FROM_DIMS, + zeroentropy_api_key: 'ze-test-fake', + openai_api_key: 'sk-test-fake', + }, null, 2)); + + resetGateway(); + configureGateway({ + embedding_model: 'zeroentropyai:zembed-1', + embedding_dimensions: FROM_DIMS, + env: { ZEROENTROPY_API_KEY: 'ze-test-fake', OPENAI_API_KEY: 'sk-test-fake' }, + }); + __setEmbedTransportForTests(async ({ values }: { values: string[] }) => { + for (const v of values) if (v !== PROBE_TEXT) embeddedTexts.push(v); + return { + embeddings: values.map(() => new Array(currentDims).fill(0).map((_, i) => Math.sin(i) * 0.01 + 0.003)), + usage: { tokens: values.length * 4 }, + } as never; + }); + + engine = new PGLiteEngine(); + await engine.connect({ embedding_dimensions: FROM_DIMS } as never); + await engine.initSchema(); + + // 3 pages x 2 chunks each = 6 chunks. + for (const slug of PAGES) { + await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` }); + await engine.upsertChunks(slug, [ + { chunk_index: 0, chunk_text: `${slug} chunk zero`, chunk_source: 'compiled_truth', token_count: 4 }, + { chunk_index: 1, chunk_text: `${slug} chunk one`, chunk_source: 'compiled_truth', token_count: 4 }, + ]); + } + await runEmbedCore(engine, { stale: true, quiet: true }); +}, 60000); + +afterAll(async () => { + __setEmbedTransportForTests(null); + resetGateway(); + await engine.disconnect(); + rmSync(tmpHome, { recursive: true, force: true }); + for (const [k, v] of Object.entries(savedEnv)) { + if (v === undefined) delete process.env[k]; + else process.env[k] = v; + } +}); + +describe('migration across a stale-batch boundary', () => { + test('seed is fully embedded at the source width', async () => { + expect(await engine.countStaleChunks()).toBe(0); + const n = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM content_chunks WHERE embedding IS NOT NULL`, + ); + expect(Number(n[0]?.n)).toBe(6); + }); + + test('batch-size 3 splits a page across the boundary yet still exits 0 with every page stamped', async () => { + currentDims = TO_DIMS; + embeddedTexts = []; + + const code = await runMigrate([ + '--to', 'openai:text-embedding-3-small', '--yes', '--batch-size', '3', + ]); + // Pre-fix this was 1 ("Migration incomplete") even though all 6 chunks + // were correctly embedded — the boundary page was never stamped. + expect(code).toBe(0); + + expect(embeddedTexts.length).toBe(6); // all six chunks re-embedded once + expect(await engine.countStaleChunks()).toBe(0); + + // Every page carries the TARGET signature, including the boundary page. + const sig = `openai:text-embedding-3-small:${TO_DIMS}`; + const stamped = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM pages WHERE embedding_signature = $1`, + [sig], + ); + expect(Number(stamped[0]?.n)).toBe(PAGES.length); + + // Completion bookkeeping ran (it only runs when the backlog drained). + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy(); + expect(await engine.getConfig(MIGRATION_COMPLETED_KEY)).toBeTruthy(); + }, 60000); + + test('second run does ZERO work — the "never re-embedded twice" contract holds across a boundary', async () => { + embeddedTexts = []; + const code = await runMigrate([ + '--to', 'openai:text-embedding-3-small', '--yes', '--batch-size', '3', + ]); + expect(code).toBe(0); + // Pre-fix the unstamped boundary page was re-invalidated and PAID FOR again. + expect(embeddedTexts.length).toBe(0); + }, 60000); +}); diff --git a/test/migrate-embeddings-flow.serial.test.ts b/test/migrate-embeddings-flow.serial.test.ts new file mode 100644 index 000000000..6356ff997 --- /dev/null +++ b/test/migrate-embeddings-flow.serial.test.ts @@ -0,0 +1,281 @@ +/** + * #3390 — `gbrain migrate embeddings` END-TO-END on PGLite. + * + * The full command flow with a fake embedding transport: + * 1. Disposable brain seeded via the REAL embed pipeline on a fake 1280d + * "zeroentropyai:zembed-1" provider (the shipped default). + * 2. One page's embedding_signature NULLed (simulates a pre-v108 page, + * the #3391 class). + * 3. Migration to a fake 1536d openai:text-embedding-3-small — with the + * transport failing two specific pages, simulating a killed/partial + * run. Asserts: exit 1 (incomplete), schema at 1536, config file + * swapped, state marker kept, partial progress banked. + * 4. Re-run of the SAME command (the documented resume path). Asserts: + * exit 0, only the two failed pages re-embedded (no duplicate work), + * every chunk embedded at 1536, every page stamped with the new + * signature (including the formerly-NULL one), state marker cleared, + * query cache purged, vector search works against the new column. + * + * Named `.serial.test.ts`: the whole file runs under a temp GBRAIN_HOME + + * an installed fake embed transport for its entire lifecycle (beforeAll → + * afterAll), which withEnv() cannot wrap. GBRAIN_HOME is pointed at a temp dir so the file-plane config write + * (persistEmbeddingFileConfig) never touches the developer's ~/.gbrain. + */ +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { + configureGateway, + resetGateway, + __setEmbedTransportForTests, +} from '../src/core/ai/gateway.ts'; +import { runEmbedCore } from '../src/commands/embed.ts'; +import { runMigrateEmbeddings } from '../src/commands/migrate-embeddings.ts'; +import { + MIGRATION_STATE_KEY, + MIGRATION_COMPLETED_KEY, +} from '../src/core/embedding-migration.ts'; + +const FROM_DIMS = 1280; +const TO_DIMS = 1536; +const PAGES = ['page-1', 'page-2', 'page-3', 'page-4', 'page-5', 'page-6']; +const PROBE_TEXT = 'gbrain embedding migration probe'; + +let engine: PGLiteEngine; +let tmpHome: string; +const savedEnv: Record = {}; + +/** Deterministic fake transport: vector width driven by the test phase. */ +let currentDims = FROM_DIMS; +/** Texts that make the transport throw (simulates a mid-run kill). */ +let failTexts: string[] = []; +/** Every non-probe text the transport embedded, per phase. */ +let embeddedTexts: string[] = []; + +function installTransport(): void { + __setEmbedTransportForTests(async ({ values }: { values: string[] }) => { + for (const v of values) { + if (failTexts.some(f => v.includes(f))) { + throw new Error(`fake transport: simulated failure for "${v.slice(0, 30)}..."`); + } + } + for (const v of values) { + if (v !== PROBE_TEXT) embeddedTexts.push(v); + } + return { + embeddings: values.map(() => new Array(currentDims).fill(0).map((_, i) => Math.sin(i) * 0.01 + 0.001)), + usage: { tokens: values.length * 4 }, + } as never; + }); +} + +class ExitError extends Error { + constructor(public code: number) { super(`exit ${code}`); } +} +const exitSeam = (code: number): never => { throw new ExitError(code); }; + +async function runMigrate(args: string[]): Promise { + try { + await runMigrateEmbeddings(engine, args, { exit: exitSeam }); + throw new Error('runMigrateEmbeddings returned without exiting'); + } catch (e) { + if (e instanceof ExitError) return e.code; + throw e; + } +} + +async function columnDims(): Promise { + const rows = await engine.executeRaw<{ dim: number }>( + `SELECT atttypmod AS dim FROM pg_attribute + WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' + AND attnum > 0 AND NOT attisdropped`, + ); + return Number(rows[0]?.dim); +} + +beforeAll(async () => { + // Isolate the file-plane config. + for (const k of ['GBRAIN_HOME', 'GBRAIN_EMBEDDING_MODEL', 'GBRAIN_EMBEDDING_DIMENSIONS', 'OPENAI_API_KEY', 'ZEROENTROPY_API_KEY', 'DATABASE_URL']) { + savedEnv[k] = process.env[k]; + delete process.env[k]; + } + tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-migrate-e2e-')); + process.env.GBRAIN_HOME = tmpHome; + mkdirSync(join(tmpHome, '.gbrain'), { recursive: true }); + writeFileSync(join(tmpHome, '.gbrain', 'config.json'), JSON.stringify({ + engine: 'pglite', + embedding_model: 'zeroentropyai:zembed-1', + embedding_dimensions: FROM_DIMS, + zeroentropy_api_key: 'ze-test-fake', + openai_api_key: 'sk-test-fake', + }, null, 2)); + + resetGateway(); + configureGateway({ + embedding_model: 'zeroentropyai:zembed-1', + embedding_dimensions: FROM_DIMS, + env: { ZEROENTROPY_API_KEY: 'ze-test-fake', OPENAI_API_KEY: 'sk-test-fake' }, + }); + installTransport(); + + engine = new PGLiteEngine(); + await engine.connect({ embedding_dimensions: FROM_DIMS } as never); + await engine.initSchema(); +}, 60000); + +afterAll(async () => { + __setEmbedTransportForTests(null); + resetGateway(); + await engine.disconnect(); + rmSync(tmpHome, { recursive: true, force: true }); + for (const [k, v] of Object.entries(savedEnv)) { + if (v === undefined) delete process.env[k]; + else process.env[k] = v; + } +}); + +describe('migrate embeddings — full flow on PGLite', () => { + test('seed: 6 pages embedded at 1280d through the real embed pipeline', async () => { + expect(await columnDims()).toBe(FROM_DIMS); + for (const slug of PAGES) { + await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}\n\ncontent for ${slug}` }); + await engine.upsertChunks(slug, [ + { chunk_index: 0, chunk_text: `chunk text for ${slug}`, chunk_source: 'compiled_truth', token_count: 5 }, + ]); + } + const seeded = await runEmbedCore(engine, { stale: true, quiet: true }); + expect(seeded.embedded).toBe(PAGES.length); + expect(await engine.countStaleChunks()).toBe(0); + + // Simulate a pre-v108 page: embedded, but no recorded signature (#3391). + await engine.executeRaw(`UPDATE pages SET embedding_signature = NULL WHERE slug = 'page-1'`); + const sigs = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM pages WHERE embedding_signature = 'zeroentropyai:zembed-1:${FROM_DIMS}'`, + ); + expect(Number(sigs[0]?.n)).toBe(PAGES.length - 1); + + // Seed a query-cache row that must not survive the migration. + await engine.executeRaw( + `INSERT INTO query_cache (id, query_text, source_id) VALUES ('qc-pre', 'old space query', 'default')`, + ); + }, 60000); + + test('dry-run: prints the plan, changes nothing', async () => { + const code = await runMigrate(['--to', 'openai:text-embedding-3-small', '--dry-run', '--json']); + expect(code).toBe(0); + expect(await columnDims()).toBe(FROM_DIMS); + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy(); + // Config file untouched. + const cfg = JSON.parse(readFileSync(join(tmpHome, '.gbrain', 'config.json'), 'utf-8')); + expect(cfg.embedding_model).toBe('zeroentropyai:zembed-1'); + }); + + test('non-TTY without --yes refuses with exit 2 (cost gate)', async () => { + const code = await runMigrate(['--to', 'openai:text-embedding-3-small']); + expect(code).toBe(2); + expect(await columnDims()).toBe(FROM_DIMS); + }); + + test('spend.posture=tokenmax does NOT bypass the gate (guards a destructive rebuild, not just spend)', async () => { + await engine.setConfig('spend.posture', 'tokenmax'); + try { + const code = await runMigrate(['--to', 'openai:text-embedding-3-small']); + expect(code).toBe(2); // posture waives the spend ceiling, not the consent + expect(await columnDims()).toBe(FROM_DIMS); + } finally { + await engine.unsetConfig('spend.posture'); + } + }); + + test('interrupted run: partial progress banks, exit 1, state marker kept', async () => { + currentDims = TO_DIMS; + failTexts = ['page-4', 'page-5']; // simulate dying mid-run on two pages + embeddedTexts = []; + + const code = await runMigrate(['--to', 'openai:text-embedding-3-small', '--yes']); + expect(code).toBe(1); // incomplete + + // Schema + config swapped BEFORE the re-embed, so the partial run is + // already in the new space. + expect(await columnDims()).toBe(TO_DIMS); + const cfg = JSON.parse(readFileSync(join(tmpHome, '.gbrain', 'config.json'), 'utf-8')); + expect(cfg.embedding_model).toBe('openai:text-embedding-3-small'); + expect(cfg.embedding_dimensions).toBe(TO_DIMS); + expect(await engine.getConfig('embedding_model')).toBe('openai:text-embedding-3-small'); + + // 4 of 6 pages embedded; the 2 failed pages remain stale (= the checkpoint). + expect(embeddedTexts.length).toBe(PAGES.length - 2); + expect(await engine.countStaleChunks()).toBe(2); + + // In-flight marker survives so doctor/status can see the migration. + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeTruthy(); + expect(await engine.getConfig(MIGRATION_COMPLETED_KEY)).toBeFalsy(); + + // Query cache was purged at swap time. + const qc = await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM query_cache`); + expect(Number(qc[0]?.n)).toBe(0); + }, 60000); + + test('resume: re-running the same command finishes without redoing work', async () => { + failTexts = []; + embeddedTexts = []; + + const code = await runMigrate(['--to', 'openai:text-embedding-3-small', '--yes']); + expect(code).toBe(0); + + // Only the two previously-failed pages were embedded this pass. + expect(embeddedTexts.length).toBe(2); + expect(embeddedTexts.join(' ')).toContain('page-4'); + expect(embeddedTexts.join(' ')).toContain('page-5'); + + // Everything is in the target space now. + expect(await engine.countStaleChunks()).toBe(0); + expect(await engine.countStaleChunks({ + signature: `openai:text-embedding-3-small:${TO_DIMS}`, + includeNullSignature: true, + })).toBe(0); + + // Every page — including the formerly NULL-signature page-1 — is stamped. + const sigs = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM pages + WHERE embedding_signature = 'openai:text-embedding-3-small:${TO_DIMS}'`, + ); + expect(Number(sigs[0]?.n)).toBe(PAGES.length); + + // Bookkeeping: marker cleared, completion stamped. + expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy(); + const done = JSON.parse((await engine.getConfig(MIGRATION_COMPLETED_KEY))!); + expect(done.to_model).toBe('openai:text-embedding-3-small'); + }, 60000); + + test('post-migration: vector search works against the new 1536d column', async () => { + // HNSW index was rebuilt by the schema transition. + const idx = await engine.executeRaw<{ indexname: string }>( + `SELECT indexname FROM pg_indexes WHERE tablename = 'content_chunks' AND indexname = 'idx_chunks_embedding'`, + ); + expect(idx.length).toBe(1); + + // A cosine query at the new width returns results (all 6 chunks embedded). + const qvec = `[${new Array(TO_DIMS).fill(0).map((_, i) => (Math.sin(i) * 0.01 + 0.001).toFixed(6)).join(',')}]`; + const rows = await engine.executeRaw<{ slug: string }>( + `SELECT p.slug FROM content_chunks cc + JOIN pages p ON p.id = cc.page_id + WHERE cc.embedding IS NOT NULL + ORDER BY cc.embedding <=> $1::vector + LIMIT 3`, + [qvec], + ); + expect(rows.length).toBe(3); + }); + + test('re-run on an already-migrated brain is a clean no-op', async () => { + embeddedTexts = []; + const code = await runMigrate(['--to', 'openai:text-embedding-3-small', '--yes']); + expect(code).toBe(0); + // Nothing re-embedded (probe excluded from embeddedTexts by design). + expect(embeddedTexts.length).toBe(0); + expect(await columnDims()).toBe(TO_DIMS); + }); +}); diff --git a/test/operations-trust-boundary.test.ts b/test/operations-trust-boundary.test.ts index 5fe5db4dd..3940f8a51 100644 --- a/test/operations-trust-boundary.test.ts +++ b/test/operations-trust-boundary.test.ts @@ -153,6 +153,7 @@ describe('mcpOperations filter — localOnly ops are excluded from the HTTP-expo 'purge_deleted_pages', 'get_recent_transcripts', 'code_traversal_cache_clear', + 'migrate_embeddings', ]; const lookup = new Map(operations.map(op => [op.name, op] as const)); for (const name of KNOWN_LOCAL_ONLY) { diff --git a/test/search-alias-resolved-boost.test.ts b/test/search-alias-resolved-boost.test.ts index 224d2d988..943c3d5fb 100644 --- a/test/search-alias-resolved-boost.test.ts +++ b/test/search-alias-resolved-boost.test.ts @@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => { }); describe('KNOBS_HASH_VERSION', () => { - it('is 12 (11→12 hard-exclude fold invalidates rows written under a different exclude policy, #2825)', () => { - expect(KNOBS_HASH_VERSION).toBe(12); + it('is 13 (12→13 embedding-provider migration invalidates rows written against the prior embedding space, #3390)', () => { + expect(KNOBS_HASH_VERSION).toBe(13); }); }); diff --git a/test/search-mode.test.ts b/test/search-mode.test.ts index 223d112d9..ae8d509bb 100644 --- a/test/search-mode.test.ts +++ b/test/search-mode.test.ts @@ -410,7 +410,10 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => { // #2825: bumped 11→12 to fold the resolved hard-exclude prefix list // (hx=) — cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across // processes. - expect(KNOBS_HASH_VERSION).toBe(12); + // #3390/#3391: bumped 12→13 for the embedding-provider migration wave — + // legacy callers hash prov=default before AND after a provider swap, so + // pre-migration cache rows must become unreachable on upgrade. + expect(KNOBS_HASH_VERSION).toBe(13); }); test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => { @@ -575,8 +578,8 @@ describe('v0.40.4 — graph_signals knob', () => { }); describe('v0.42.3.0 — autocut knobs', () => { - test('KNOBS_HASH_VERSION is 12 (11→12 hard-exclude fold, #2825)', () => { - expect(KNOBS_HASH_VERSION).toBe(12); + test('KNOBS_HASH_VERSION is 13 (12→13 embedding-migration wave, #3390/#3391)', () => { + expect(KNOBS_HASH_VERSION).toBe(13); }); test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => { diff --git a/test/search/knobs-hash-reranker.test.ts b/test/search/knobs-hash-reranker.test.ts index 9f73ac394..73493ac0e 100644 --- a/test/search/knobs-hash-reranker.test.ts +++ b/test/search/knobs-hash-reranker.test.ts @@ -44,7 +44,7 @@ function baseKnobs(): ResolvedSearchKnobs { } describe('KNOBS_HASH_VERSION + version invariants', () => { - test('version is 12 (…; 9→10 relational recall; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825)', () => { + test('version is 13 (…; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825; 12→13 embedding-provider migration #3390)', () => { // v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold // floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs // (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation). @@ -64,7 +64,7 @@ describe('KNOBS_HASH_VERSION + version invariants', () => { // pre-fix document-side query vectors must not be served. // #2825: 11→12 to fold the resolved hard-exclude prefix list (hx=) — // cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across processes. - expect(KNOBS_HASH_VERSION).toBe(12); + expect(KNOBS_HASH_VERSION).toBe(13); }); test('hash is 16 hex chars regardless of reranker config', () => {