mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-29 19:01:39 +00:00
v0.42.67.0 feat(migrate): provider-agnostic embedding migration — the path off ZeroEntropy (#3390, fixes #3391) (#3459)
* feat(migrate): provider-agnostic embedding migration service — the path off ZeroEntropy (#3390) - gbrain migrate embeddings --to <provider:model> (alias: retrieval-upgrade): plan + cost preflight, consent gate (--yes / TTY confirm / non-TTY exit 2), live probe against the target provider before any mutation, env-override gate, schema dimension transition via the shared runSchemaTransition, dual-plane config write, NULL-signature-inclusive invalidation, query-cache purge, resumable re-embed through the standard embed pipeline (single-flight locks, backoff, pacing, stderr progress). Killed runs resume by re-running the same command; the NULL-embedding column is the checkpoint. - #3391 root-cause fix (both engines): countStaleChunks / sumStaleChunkChars / invalidateStaleSignatureEmbeddings accept includeNullSignature to lift the v108 grandfather clause; embed --stale warns loudly when a model swap leaves NULL-signature pages in the old embedding space, and --include-null-signature re-embeds them. Default sweep behavior unchanged. - knobs_hash v=12 → v=13 (prov=default legacy callers must not be served pre-migration cache rows). - migrate_embeddings op: scope admin, localOnly, hidden cliHints, hard remote refusal, needs_confirmation without yes=true. - One-shot post-upgrade ZE-sunset banner (ze_sunset_notice_shown) for brains resolving to a zeroentropyai:* embedding model or reranker. - doctor's dimension-mismatch repair hint now names the real command. - Docs: docs/guides/embedding-migration.md, KEY_FILES entries, spend-controls gate row. Tests: PGLite unit + full-lifecycle flow (interrupted-run resume), real-Postgres e2e (pgvector DDL path + #3391 predicate parity). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(test): satisfy check:test-isolation + bump the remaining knobs_hash pins - test/migrate-embeddings-flow.test.ts → .serial.test.ts: the file holds a temp GBRAIN_HOME + an installed fake embed transport for its whole lifecycle (beforeAll → afterAll), which withEnv() can't wrap. This also fixes the CI shard-pollution failure in test/ai/recipes-existing-regression.test.ts (that file passes solo on both master and this branch; the flow test's configureGateway + provider-key deletion was leaking into it inside the same shard process). - test/embedding-migration.test.ts: env-override case now uses withEnv(). - Bump the three remaining KNOBS_HASH_VERSION pins to 13 (cross-modal-phase1, search-alias-resolved-boost, search/knobs-hash-reranker). - Docs + llms bundles follow the test rename. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(test): wire the new Postgres e2e into the smart e2e selector map Changes to embed.ts / embedding-migration.ts / retrieval-upgrade-planner.ts / postgres-engine.ts now trigger test/e2e/migrate-embeddings-postgres.test.ts — the #3391 stale predicates and runSchemaTransition's DDL path behave differently on real pgvector than on PGLite, so the smart selector has to know. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(migrate): consult spend.posture in the embedding-migration consent gate The brief asked the gate to honor spend.posture; it previously didn't read it at all. Now it does — but deliberately does NOT bypass on tokenmax: posture waives the spend CEILING, and this gate also guards a destructive schema rebuild (existing vectors dropped, retrieval degraded until the re-embed finishes). Under tokenmax the dollar figure is marked informational on stderr and the confirmation is still asked; --yes stays the single scripted bypass. Pinned by a new case in the flow test so a later refactor can't quietly turn posture into a bypass. Guide + spend-controls table updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * wip: blocker fixes --------- Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Time Attakc
co-authored by
Claude Fable 5
Garry Tan
Claude Fable 5
Garry Tan
parent
5dfd2696d1
commit
0bbaed2e48
@@ -191,7 +191,10 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling.
|
||||
- `src/commands/enrich.ts` + `src/core/enrich/thin.ts` + `src/core/cycle/enrich-thin.ts` — `gbrain enrich --thin`: batch-develops stub (thin) pages via **brain-internal grounded synthesis**. gbrain's model tooling sees only brain-internal context (search / get_page / facts / backlinks), not the web, so enrich consolidates what the brain ALREADY knows about an entity (scattered across meetings, other pages, deals, facts) into one cited page via ONE `gateway.chat` call per page; web research stays the agent-driven `enrich` SKILL's job. `runEnrichCore(engine, opts, signal)` (strict per-source; multi-source iteration is the caller's job) drives `enrichOne` per candidate: `withRefreshingLock('enrich:<src>:<slug>')` → `getPage` → deterministic retrieve (hybridSearch + getBacklinks + facts + raw_data, source-scoped, sanitized via `INJECTION_PATTERNS`) → `assessGrounding` gate (skip < `MIN_CONTEXT_CHARS`, no LLM) → `buildEnrichPrompt` (grounded dossier, `[Source: slug]` citations, SKIP sentinel) → synth → `put_page` handler (`remote:false`, auto-link + write-through) stamping `enriched_at` + `enriched_by:'cli:enrich'`. Candidate selection is the SQL-native `engine.listEnrichCandidates(opts)` (`src/core/engine.ts` interface + `EnrichCandidate`/`EnrichCandidatesOpts`/`ENRICH_ORDER_SQL` in `src/core/types.ts` + pg/pglite impls): thin-filter + per-page source-correct inbound count (`to_page_id = p.id`, `mentions` excluded) + `enriched_at` recency guard + whitelisted ORDER BY + LIMIT, lightweight projection (NO bodies). Resume via `src/core/op-checkpoint.ts` (local `enrichFingerprint`); budget via `BudgetTracker` + `withBudgetTracker` (best-effort under `--workers > 1` — `runSlidingPool` aborts new claims on `BUDGET_EXHAUSTED` but does NOT cancel in-flight `gateway.chat`; pin `--workers 1` for a hard ceiling). `sanitizeContext` (thin.ts) neutralizes the `<context>…</context>` data-envelope delimiters (injection escape, mirrors the `</trajectory>` convention); the `--background` multi-source fan-out idempotency key carries the run fingerprint via exported `backgroundIdempotencyKey(sid, args)` (a bare `enrich:${sid}` would return stale completed jobs); `runEnrichCore` flags `budget_exhausted` post-hoc when `tracker.totalSpent > tracker.cap` even when the gateway swallowed the final-call throw (via read-only `BudgetTracker.cap` getter); `body()` flushes the checkpoint on `BudgetExhausted` before it propagates so resume doesn't re-charge. The opt-in `enrich_thin` cycle phase (default OFF via `cycle.enrich_thin.enabled`) trickles `max_pages_per_tick` (default 3) per source with per-source cost cap enforced as `min(per_source_cap, brain_wide_remaining)` + brain-wide total + walltime caps. Wired into `cycle.ts` (`CyclePhase`/`ALL_PHASES` between `conversation_facts_backfill` and `skillopt`/`embed`; `PHASE_SCOPE='source'`; `NEEDS_LOCK`; dispatch), `cli.ts` (`CLI_ONLY` + `CLI_ONLY_SELF_HELP` + `THIN_CLIENT_REFUSED_COMMANDS` + dispatch), `jobs.ts` (Minion `enrich` handler, strict per-source, NOT in `PROTECTED_JOB_NAMES`). DI seam `opts.synthesizeFn` keeps tests hermetic (no API key, no mock.module). Pinned by `test/enrich/thin.test.ts`, `test/enrich/idempotency.test.ts`, `test/enrich-cycle-phase.test.ts`, `test/e2e/enrich-pglite.test.ts` (grew-cited, skip, ordering, multi-source, recency, resume, budget abort + checkpoint flush, final-call overage, lock-skip, provenance), `test/e2e/engine-parity.test.ts` (`listEnrichCandidates` pg↔pglite parity).
|
||||
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping.
|
||||
- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[<source-id>] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed <slug>` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything.
|
||||
- `src/commands/embed.ts` — `gbrain embed [--stale|--all] [--slugs ...]`. `--stale` starts with `engine.countStaleChunks()` (single SELECT count(*) WHERE embedding IS NULL, ~50 bytes wire) so a fully-embedded brain short-circuits with no further reads. When stale chunks exist, `engine.listStaleChunks()` returns just the chunks needing embeddings (slug + chunk_index + chunk_text + metadata, no `vector(1536)` payload); caller groups by slug, embeds, re-upserts via `upsertChunks`. All `console.log`/`console.error` call sites use `slog`/`serr` from `src/core/console-prefix.ts` so when `runEmbedCore` runs inside a per-source `withSourcePrefix` scope (installed by the `gbrain sync --all` worker pool) every line carries the `[<source-id>] ` prefix; standalone callers see identical output because slog/serr fall through to bare console fns outside the wrap. Every embed-write path stamps `pages.embedding_signature` via `engine.setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` so a later model/dims swap is detectable as stale. The per-slug path (`embedPage`, used by `gbrain embed <slug>` AND sync's post-import embed step) and the full-re-embed path (`embedAll`) stamp unconditionally per page. The stale path (`embedAllStale`) first calls `invalidateStaleSignatureEmbeddings` on a live run so signature-drifted pages flow through the NULL cursor, then stamps each page — but ONLY when EVERY chunk was stale this pass (a partially-stale page keeps preserved chunks of unknown provenance, so it stays unstamped rather than falsely marked current; `embed --all` fully re-embeds + stamps those). dry-run never mutates: it counts signature-drift via the widened `countStaleChunks({signature})` predicate without NULLing anything. `--include-null-signature` (#3391) lifts the NULL-signature grandfather clause: threads `includeNullSignature: true` into the invalidation + counts so pages that predate the v108 stamp re-embed too after a model swap (both engines' `countStaleChunks`/`sumStaleChunkChars`/`invalidateStaleSignatureEmbeddings` accept the flag; predicate becomes `sig IS NULL OR sig <> current`). Without the flag, a live stale run that just invalidated drifted rows probes for left-behind NULL-signature chunks and emits a loud stderr warning naming the count + the fix — mixed embedding spaces in one index are never silent. Pinned by `test/embedding-migration.test.ts` + `test/e2e/migrate-embeddings-postgres.test.ts`.
|
||||
- `src/core/retrieval-upgrade-planner.ts` — `runSchemaTransition(engine, targetDim)` (exported) is the ONE atomic dimension-transition path, shared by `ze-switch` and `gbrain migrate embeddings`. In a single transaction it rebuilds ALL THREE dim-pinned text-embedding-space columns at `targetDim` — `content_chunks.embedding`, `query_cache.embedding`, `facts.embedding` — preserving each column's declared type (`vector` vs `halfvec`, probed from `information_schema`) and recreating its HNSW index with the matching opclass, gated on `hnswIndexExpected` (above the per-type dim ceiling pgvector refuses the index and exact scans remain the path). query_cache + facts are created at brain-birth width by `migrate.ts` and NO migration ever ALTERs them, so omitting either leaves it silently broken: a narrow `query_cache.embedding` makes every `store()`/`lookup()` fail inside the cache's own error-swallowing (permanent 0% hit rate), and a narrow `facts.embedding` fails every per-fact embed write (the doctor check that would warn is skipped on PGLite, the default engine). `content_chunks.embedding_image` / `embedding_multimodal` are the deliberate exception — separate multimodal models, dimensions independent of the text model. Pinned by `test/embedding-migration.test.ts` (all three widths + a real INSERT at the new width into each) and `test/e2e/migrate-embeddings-postgres.test.ts`.
|
||||
- `src/core/embedding-migration.ts` — provider-agnostic embedding migration core (#3390): `planEmbeddingMigration` (workload counts via the widened stale predicates with the TARGET signature + `includeNullSignature`, so a mid-migration re-plan counts only what remains; cost via `embedding-pricing.ts`; `null_signature_chunks` split out for #3391 visibility; reranker-on-outgoing-provider warning), `applyEmbeddingMigration` (env-override gate BEFORE any mutation → in-flight state marker `embedding_migration.state` → `runSchemaTransition` when the ACTUAL column width differs from target → DB-plane `embedding_model`/`embedding_dimensions` → `persistConfig` callback for the file plane → `invalidateStaleSignatureEmbeddings({includeNullSignature: true})` → `SemanticQueryCache.clear()`), `completeEmbeddingMigration` (clears the marker + stamps `embedding_migration.completed`; call only at zero backlog), `resolveMigrationTarget` (validates `provider:model` via `resolveRecipe`, dims via `embeddingDimsForModel` or explicit `--dim`), `migrationSignature` (matches `currentEmbeddingSignature()` shape). Engine-pure; every step idempotent under crash + re-run — the NULL-embedding column is the checkpoint. Reuses `runSchemaTransition` (now exported from `retrieval-upgrade-planner.ts`) so ze-switch and the migration share ONE dimension-transition path. `reconcilePageSignatures(engine, plan)` runs after the re-embed drain and BEFORE the completion probe: it stamps the target signature on every page that has zero NULL-embedding chunks, covering pages whose chunks straddle a `listStaleChunks` batch boundary (the embed loop only stamps when `stale.length === existing.length`, so a split page is embedded correctly but never stamped — without the reconcile a >1-batch brain reports "incomplete" and the re-run re-invalidates and re-pays for those pages). Sound only because apply() invalidated everything not already in the target space; pages with a remaining NULL chunk stay unstamped so a real embed failure still surfaces. Invalidation is ordered BEFORE the config writes so a crash on a same-dim swap leaves rows merely stale (empty results) rather than new-space queries scored against old-space vectors (silently wrong). Pinned by `test/embedding-migration.test.ts` (PGLite) + `test/e2e/migrate-embeddings-postgres.test.ts` (real pgvector).
|
||||
- `src/commands/migrate-embeddings.ts` — `gbrain migrate embeddings --to <provider:model> [--dim N] [--dry-run] [--yes] [--json] [--no-embed] [--pace[=mode]] [--ignore-env-override]` (alias: `gbrain retrieval-upgrade`, the command README/doctor promised since v0.36). Flow: plan → render (stderr when `--json` so stdout stays JSON-clean) → consent gate (TTY y/N prompt or `--yes`; non-TTY without `--yes` refuses exit 2, mirroring the reindex-code cost gate) → live probe (one embed against the TARGET model/dims BEFORE any mutation — bad key/model/dim fails with nothing changed) → `applyEmbeddingMigration` with `persistEmbeddingFileConfig` (writes `~/.gbrain/config.json` + reconfigures the in-process gateway — the gateway reads file/env, NOT the DB plane) → `runEmbedCore({stale, catchUp, singleFlight, includeNullSignature, pace})` → drain check → `completeEmbeddingMigration` or exit 1 with the resume hint (re-run the same command). Also surfaced as the `migrate_embeddings` op (scope admin, localOnly, hidden cliHints; handler hard-refuses `ctx.remote !== false` and returns `needs_confirmation` + plan without `yes: true`). Pinned by `test/migrate-embeddings-flow.serial.test.ts` (full lifecycle incl. interrupted-run resume on PGLite).
|
||||
- `src/core/conversation-parser/` — 17-pattern built-in chat-format registry + opt-in LLM polish/fallback. Modules: `types.ts` (PatternEntry + ParseResult + DateContext + CaptureMap + TimezonePolicy), `builtins.ts` (17 hand-vetted patterns sourced from public format docs — iMessage/Slack, Telegram bracket + text-export, bold-paren-time, bold-time-dash, bold-name-no-time, Discord classic + export, WhatsApp ISO + US, Signal, Matrix/Element, IRC classic + weechat, Teams export; module-load validation runs every `test_positive[]` + `test_negative[]` sample at startup so a typo in any built-in regex makes gbrain refuse to start; `DEFAULT_SPEAKER_CLEAN` exported as a module-level default), `parse.ts` (orchestrator with pattern-priority scoring across the first 10 lines + date derivation chain `explicit > frontmatter.date > effective_date > '1970-01-01'` + multi-line continuation + timezone warning), `llm-base.ts` (shared `runLlmCall<T>` with content-hash cache in-process + DB-persistent via migration v97 + 4-strategy JSON repair + Anthropic-key probe; polish and fallback are thin wrappers), `llm-polish.ts` (opt-IN; headroom guard skips when tracker within $0.10 of cap; pure `applyPolish` for merge/drop/edit ops), `llm-fallback.ts` (opt-IN; NO regex inference + NO persistence), `eval.ts` (`scoreFixture` + `aggregateScores` + `parseFixtureJsonl` for the fixture-corpus CI gate), `nightly-probe.ts` (DI-stubbed; mode-gated default tokenmax=ON, conservative/balanced opt-in; adversarial false-positive detection). Pattern `bold-name-no-time` (regex `/^\*\*(?!\[)(.+?):\*\*\s*(.*)$/`, ordered after the time-bearing bold patterns) parses `**Speaker:** text` with NO per-line timestamp (Circleback/Granola/Zoom), anchoring every message at `T00:00:00Z` of the frontmatter date (line order preserves sequence, same no-time convention as `irc-classic`); the `(?!\[)` lookahead rejects telegram-bracket `**[18:37] Name:**`; non-shadow is the colon-INSIDE-bold regex (NOT declaration order — `parse.ts` scores every candidate independently, order is only the tie-break). Because `**Label:** text` is a common prose idiom, the pattern sets optional `PatternEntry.score_full_body: true` so `parse.ts` recomputes the winner's acceptance score over the FULL body before the `SCORING_MIN_ACCEPTANCE` floor, keeping a bold-label notes page at `no_match`. Pattern `bold-paren-time` parses `**Speaker** (HH:MM): text` and `(HH:MM:SS)` (date_source: frontmatter). Fallback gates: `SCORING_HEAD_TRIGGER_THRESHOLD = 0.3` triggers a full-body re-score when the head pass scores below that; `SCORING_MIN_ACCEPTANCE = 0.05` blocks essay false-positives. Exported `scorePatternFull(body, entry)`; private `getNonBlankLines(body, headCap?)` + `scoreFromLines(lines, entry)` DRY the quick_reject+regex loop. CLI surfaces at `src/commands/eval-conversation-parser.ts` (`gbrain eval conversation-parser <fixture.jsonl>` exit 0/1/2, wired into `bun run verify` via `check:conversation-parser`) and `src/commands/conversation-parser.ts` (`scan <slug>` debug, `list-builtins`, `validate <file>`). Doctor checks: `conversation_format_coverage`, `progressive_batch_audit_health`, `conversation_parser_probe_health`. Pinned by `test/conversation-parser/{parse,llm-base,llm-fallback,llm-polish,nightly-probe}.test.ts` + the 27-case baseline at `test/extract-conversation-facts.test.ts` (back-compat invariant). Migration v97 (`conversation_parser_llm_cache_table`). Fixtures at `test/fixtures/conversation-formats/{imessage,imessage-time-only-12h,telegram-bracket,whatsapp-iso,whatsapp-us,signal-export,irc-classic,irc-weechat,matrix-element,teams-export,all,adversarial,bold-name-no-time,bold-time-dash}.jsonl` with `scripts/check-fixture-privacy.sh` banning real-name leaks. Maintainer guidance: [conversation parser patterns](conversation-parser-patterns.md).
|
||||
- `src/core/progressive-batch/` — shared ramp-up + cost-cap + verification primitive (trial 10 → ramp 100 → ramp 500 → full, with verification at each stage), with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). Modules: `types.ts` (Stage, StageVerdict, AbortReason, discriminated `Verifier` union `OutputCountVerifier | IdempotentMutationVerifier | NoopVerifier`, Policy, StageReport), `orchestrator.ts` (`runProgressiveBatch(items, verifier, policy, runner)` — reads `getCurrentBudgetTracker()` ahead of `Policy.maxCostUsd` fail-closed; null both ways triggers `abort_cost_cap reason='no_budget_safety_net'`), `audit.ts` (ISO-week JSONL at `~/.gbrain/audit/progressive-batch-YYYY-Www.jsonl` via the shared `audit-writer` primitive), `stage-report.ts` (ASCII formatter for the default `Policy.onStageReport`). Env knobs: `GBRAIN_PROGRESSIVE_BATCH_DISABLED=1`, `GBRAIN_PROGRESSIVE_BATCH_AUTO=1` (skip Ctrl-C grace), `GBRAIN_PROGRESSIVE_BATCH_STAGES=10,100,500`. Sites that "jump straight to full" stay that way by default; ramp is opt-in per-site via `Policy.interactiveAbortMs > 0`. Pinned by `test/progressive-batch/orchestrator.test.ts` (35 cases, every verdict path).
|
||||
- `src/commands/extract-conversation-facts.ts` + `src/core/cycle/conversation-facts-backfill.ts` — bulk fact extraction for long-form conversation pages. Walks conversation/meeting/slack/email/imessage/imessage-daily pages, splits them into time-windowed segments (30-min gap or 30-msg cap), prepends a topical/temporal header, and uses the strict `extractFactsFromTurnWithOutcome()` path so provider and output failures remain retryable instead of becoming successful empty pages. Invariants: **strict per-source core** (`runExtractConversationFactsCore({sourceId, ...})` always takes one sourceId; CLI + cycle phase each do their own multi-source iteration because `PHASE_SCOPE='source'` is taxonomy-only); **bounded two-phase enumeration** (paginated `listPages({type, sourceId, limit:10})`; per-page body cap `MAX_PAGE_BODY_BYTES=25MB`); **page-global `row_num` accumulator** (the facts unique index is `(source_id, source_markdown_slug, row_num)`); **versioned snapshot-bound outcomes** (`cli:extract-conversation-facts:terminal:v2` for complete pages and a separate `non-extractable:v2` source for recognized pages with no eligible segment); **operation checkpoints are scheduling hints only** and never suppress a replay without a matching v2 outcome; **optional `opts.budgetTracker?`** is used as-is, while an absent tracker is created with `maxCostUsd`; **body reads cover compiled truth, timeline, and configured raw-transcript sidecars**; **`facts.extraction_enabled` kill-switch** with `--override-disabled`; **`--types LIST` allowlist** (`conversation,meeting,slack,email,imessage,imessage-daily`); and **`--background` via `maybeBackground`**. The companion `conversation_facts_backfill` cycle phase is default-off, iterates every source, and enforces per-source plus brain-wide cost and wall-time caps. Migration v94 provides the partial facts index used by outcome lookups. `computeConversationFactsBacklogCheck` reports fresh completed, scanned-not-extractable, and unfinished counts separately, warning when more than 10 eligible pages lack a fresh v2 outcome. `sources audit` exposes `facts_backfill_estimate: {pages, est_segments, est_cost_usd, types}`. Pinned by `test/extract-conversation-facts.test.ts` and `test/doctor-conversation-facts-backlog.test.ts`.
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
# Embedding migration — moving a brain to another embedding provider
|
||||
|
||||
`gbrain migrate embeddings` re-embeds an entire brain onto a different
|
||||
embedding provider/model, safely and resumably. It is the forward path off a
|
||||
sunsetting provider (for example ZeroEntropy's hosted API, which shuts down
|
||||
2026-09-04 and is the shipped default for brains that never picked a model) —
|
||||
but it is provider-agnostic: any configured `provider:model` works as a
|
||||
target.
|
||||
|
||||
Also reachable as `gbrain retrieval-upgrade` (the name `doctor` and the
|
||||
README reference).
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# Preview the work + cost. Changes nothing.
|
||||
gbrain migrate embeddings --to openai:text-embedding-3-small --dry-run
|
||||
|
||||
# Run it (interactive confirm shows chunk count + $ estimate first).
|
||||
gbrain migrate embeddings --to openai:text-embedding-3-small
|
||||
|
||||
# Non-interactive (cron / scripts): --yes is required, else exit 2.
|
||||
gbrain migrate embeddings --to voyage:voyage-3-large --yes
|
||||
```
|
||||
|
||||
`--dim <N>` overrides the target width; it defaults to the provider recipe's
|
||||
declared width and is required for recipes that don't declare one (litellm,
|
||||
llama-server, and other bring-your-own-model providers).
|
||||
|
||||
## What it does, in order
|
||||
|
||||
1. **Plan.** Counts every chunk not already in the target embedding space —
|
||||
including chunks on pages with **no recorded embedding signature**
|
||||
(pages embedded before the v108 provenance stamp). Prices the re-embed
|
||||
from the pricing table; unknown providers print "estimate unavailable"
|
||||
instead of a fabricated number.
|
||||
2. **Consent gate.** Prints the plan; requires an interactive `y` or `--yes`.
|
||||
Non-TTY without `--yes` refuses with exit 2 (mirrors the `reindex-code`
|
||||
gate in [spend-controls](../operations/spend-controls.md)). Unlike the pure
|
||||
cost gates there, `spend.posture=tokenmax` does **not** bypass this one:
|
||||
posture waives the spend *ceiling*, and this gate also guards a
|
||||
destructive schema rebuild. Under `tokenmax` the dollar figure is marked
|
||||
informational and the confirmation is still asked. `--yes` is the single
|
||||
scripted bypass.
|
||||
3. **Live probe.** One tiny embed against the TARGET provider before any
|
||||
mutation — validates the API key, model id, and dimension support in a
|
||||
single call. A bad key fails here, with nothing changed.
|
||||
4. **Env-override gate.** Refuses when `GBRAIN_EMBEDDING_MODEL` /
|
||||
`GBRAIN_EMBEDDING_DIMENSIONS` would silently defeat the switch at
|
||||
runtime (the same guard `ze-switch` uses). `--ignore-env-override` for
|
||||
people running deliberate experiments.
|
||||
5. **Apply.** When the target width differs from the actual column width,
|
||||
runs the same atomic schema transition `ze-switch` uses, in one
|
||||
transaction. It rebuilds **all three dim-pinned text-embedding-space
|
||||
columns** — `content_chunks.embedding`, `query_cache.embedding`, and
|
||||
`facts.embedding` — at the new width, preserving each column's type
|
||||
(`vector` vs `halfvec`) and recreating its HNSW index. Missing any of the
|
||||
three leaves it silently broken: a narrow `query_cache.embedding` makes
|
||||
every cache write and read fail *by design* (the cache swallows errors so
|
||||
it can never break search) for a permanent 0% hit rate, and a narrow
|
||||
`facts.embedding` fails every per-fact embed write. The image/multimodal
|
||||
columns ARE deliberately untouched — they use separate models whose
|
||||
dimensions are independent of the text embedding model.
|
||||
Writes `embedding_model` + `embedding_dimensions` to BOTH config planes
|
||||
(file plane for the runtime gateway, DB plane for doctor), invalidates
|
||||
every chunk still in the old space — **including NULL-signature pages** —
|
||||
and purges the semantic query cache so stale cached results can't be
|
||||
served across the swap.
|
||||
6. **Re-embed.** The standard embed pipeline (`embed --stale --catch-up`)
|
||||
with per-source single-flight locks, rate-limit backoff, stderr progress,
|
||||
and optional DB-contention pacing (`--pace[=mode]`).
|
||||
|
||||
## What the rebuild deletes
|
||||
|
||||
The dimension change **deletes every stored embedding vector** in the brain —
|
||||
they are in the old model's space and unusable. They are not recoverable:
|
||||
going back to the previous provider means paying for a second full re-embed.
|
||||
`content_chunks` vectors are rebuilt by the re-embed pass, the query cache
|
||||
refills on the next query, and fact embeddings are rewritten on their next
|
||||
write (or a `gbrain extract` pass).
|
||||
|
||||
## Resume after a kill
|
||||
|
||||
The NULL-embedding column is the checkpoint. If the run is killed (or some
|
||||
pages fail to embed), re-run the **same command**: chunks already embedded on
|
||||
the target are never re-embedded, the schema/config steps no-op, and the run
|
||||
continues where it stopped. An in-flight marker (`embedding_migration.state`
|
||||
in DB config) records the target; it is cleared only when the backlog drains
|
||||
to zero.
|
||||
|
||||
A page whose chunks straddle two stale batches is embedded correctly but not
|
||||
stamped by the embed loop (which only stamps all-or-nothing per batch), so the
|
||||
migration runs one reconcile pass after the drain that stamps every
|
||||
fully-embedded page. Without it a large brain would report "incomplete" and the
|
||||
re-run would pay again for those pages. `--batch-size N` tunes the batch
|
||||
(default 2000).
|
||||
|
||||
`--no-embed` applies schema + config + invalidation and stops, so you can run
|
||||
the (potentially long) re-embed later or in the background:
|
||||
|
||||
```bash
|
||||
gbrain migrate embeddings --to openai:text-embedding-3-small --yes --no-embed
|
||||
gbrain embed --stale --catch-up --include-null-signature --background
|
||||
```
|
||||
|
||||
## During the migration
|
||||
|
||||
While the re-embed runs, semantic search returns degraded (lexical-arm-only)
|
||||
results for not-yet-re-embedded content. Pick a quiet window for large
|
||||
brains, or use `--pace` to keep the DB responsive.
|
||||
|
||||
## Pages without an embedding signature (#3391)
|
||||
|
||||
Pages embedded before provenance stamping have `embedding_signature IS NULL`
|
||||
and are grandfathered by the routine stale sweep (so an upgrade never
|
||||
surprise-re-embeds a whole corpus). After a provider swap that grandfather
|
||||
clause would silently leave those pages in the OLD embedding space — mixed
|
||||
vector spaces in one index, degrading retrieval with nothing in the logs.
|
||||
|
||||
- `gbrain migrate embeddings` always includes them.
|
||||
- Plain `gbrain embed --stale` warns when a model swap leaves NULL-signature
|
||||
pages behind, and `gbrain embed --stale --include-null-signature` re-embeds
|
||||
them.
|
||||
|
||||
## Reranker
|
||||
|
||||
Migrating embeddings does not touch the reranker. If
|
||||
`search.reranker.model` points at the outgoing provider, the plan prints a
|
||||
warning; disable it (`gbrain config set search.reranker.enabled false`) or
|
||||
point it at another provider.
|
||||
|
||||
## Self-hosting instead of migrating
|
||||
|
||||
If the outgoing model's weights are available (zembed-1's are Apache-2.0),
|
||||
serving them locally via `llama-server` / `ollama` / a LiteLLM proxy
|
||||
preserves your existing vectors — no re-embed at all. Point
|
||||
`embedding_model` at the local recipe and keep the same dimensions. The
|
||||
migration command is for when you'd rather move to a hosted provider.
|
||||
@@ -49,6 +49,7 @@ The USD-limit knobs accept `off`, `unlimited`, or `none` (case-insensitive) to m
|
||||
| Backfill per-job budget | `embed.backfill_max_usd` | `10` | caps the job's tracker | `off` (`0` → default) | uncapped (still ledgered) |
|
||||
| Backfill cooldown | `embed.backfill_cooldown_min` | `10` | skips re-submission inside window | — (latency knob, not spend) | **not** bypassed |
|
||||
| `reindex-code` cost gate | — (preview before re-embed) | — | TTY prompt / non-TTY refuse + exit 2 | `--max-cost off` | informational |
|
||||
| `migrate embeddings` consent gate | — (plan + estimate before provider migration) | — | TTY y/N prompt / non-TTY refuse + exit 2 | `--yes` | estimate marked informational, but **still prompts** (guards a destructive schema rebuild, not just spend) |
|
||||
| `enrich` / `onboard --auto` | `--max-usd` (per-call) | — | refuse without a cap (non-TTY) | `--max-usd off` | runs uncapped (still ledgered) |
|
||||
|
||||
### Sync inline-embed cost gate
|
||||
|
||||
@@ -140,6 +140,9 @@ Stable phase names shipped in v0.15.2:
|
||||
- `import.files`
|
||||
- `sync.deletes`, `sync.renames`, `sync.imports`
|
||||
- `migrate.copy_pages`, `migrate.copy_links`
|
||||
- `migrate.reembed` (the re-embed pass of `gbrain migrate embeddings`; total is the
|
||||
stale-chunk backlog at the start of the pass, so it can grow slightly if a
|
||||
writer adds chunks mid-run)
|
||||
- `repair_jsonb.run`, `repair_jsonb.<table>.<column>`
|
||||
- `backlinks.scan`
|
||||
- `lint.pages`
|
||||
|
||||
+11
-1
@@ -46,7 +46,15 @@ export const E2E_TEST_MAP: Record<string, string[]> = {
|
||||
"test/e2e/multi-source-bug-class.test.ts",
|
||||
"test/e2e/synthesize-bigint-job-id-postgres.test.ts",
|
||||
],
|
||||
"src/commands/embed.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/commands/embed.ts": [
|
||||
"test/e2e/multi-source-bug-class.test.ts",
|
||||
// #3391: the NULL-signature stale predicates differ per engine.
|
||||
"test/e2e/migrate-embeddings-postgres.test.ts",
|
||||
],
|
||||
// #3390: runSchemaTransition's DDL path + the stale predicates behave
|
||||
// differently on real pgvector than on PGLite.
|
||||
"src/core/embedding-migration.ts": ["test/e2e/migrate-embeddings-postgres.test.ts"],
|
||||
"src/core/retrieval-upgrade-planner.ts": ["test/e2e/migrate-embeddings-postgres.test.ts"],
|
||||
"src/commands/extract.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
"src/commands/migrate-engine.ts": ["test/e2e/multi-source-bug-class.test.ts"],
|
||||
// Any minions queue/worker/handler change exercises all minion E2E.
|
||||
@@ -64,6 +72,8 @@ export const E2E_TEST_MAP: Record<string, string[]> = {
|
||||
"test/e2e/jsonb-roundtrip.test.ts",
|
||||
"test/e2e/engine-parity.test.ts",
|
||||
"test/e2e/schema-drift.test.ts",
|
||||
// #3391: includeNullSignature stale predicates (engine parity).
|
||||
"test/e2e/migrate-embeddings-postgres.test.ts",
|
||||
],
|
||||
// PGLite bootstrap path + parity guard.
|
||||
"src/core/pglite-engine.ts": [
|
||||
|
||||
+31
-2
@@ -55,7 +55,7 @@ export function bigintToStringReplacer(_key: string, value: unknown): unknown {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
|
||||
export const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'maintain', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'reconcile-links', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'calibration', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'retrieval-upgrade', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'advisor', 'watch', 'reindex-search-vector']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
@@ -107,6 +107,10 @@ const CLI_ONLY_SELF_HELP = new Set([
|
||||
// `gbrain connect --help` prints its own usage (flags + examples) from
|
||||
// runConnect; route around the generic one-line short-circuit.
|
||||
'connect',
|
||||
// #3390 — `gbrain migrate embeddings --help` / `gbrain retrieval-upgrade
|
||||
// --help` print the migration flags from runMigrateEmbeddings. `migrate`
|
||||
// (engine transfer) keeps its own dispatch too.
|
||||
'migrate', 'retrieval-upgrade',
|
||||
]);
|
||||
|
||||
// v114 (#1941): alias -> operation lookup, kept separate from `cliOps` so
|
||||
@@ -1055,7 +1059,7 @@ export function formatResult(opName: string, result: unknown): string {
|
||||
* `runRemoteDoctor` for thin-client installs.
|
||||
*/
|
||||
const THIN_CLIENT_REFUSED_COMMANDS = new Set([
|
||||
'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'apply-migrations',
|
||||
'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'retrieval-upgrade', 'apply-migrations',
|
||||
'repair-jsonb', 'orphans', 'integrity', 'serve',
|
||||
// v0.43 (#2095): watch streams against a LOCAL engine; thin clients get
|
||||
// the volunteer_context MCP op instead.
|
||||
@@ -1102,6 +1106,7 @@ const THIN_CLIENT_REFUSE_HINTS: Record<string, string> = {
|
||||
'extract-conversation-facts': 'extract-conversation-facts runs on the host (requires local engine + chat gateway). Run on the host machine.',
|
||||
enrich: 'enrich runs on the host (requires local engine + chat gateway for grounded synthesis). Run on the host machine.',
|
||||
migrate: "migrate runs on the host's local engine. Run on the host machine.",
|
||||
'retrieval-upgrade': "retrieval-upgrade (embedding migration) rebuilds the host brain's schema + re-embeds. Run on the host machine.",
|
||||
'apply-migrations': 'schema migrations run on the host. SSH and run there.',
|
||||
'repair-jsonb': 'repair-jsonb operates on the local DB only.',
|
||||
integrity: 'integrity scans local files. Run on the host machine.',
|
||||
@@ -1752,10 +1757,33 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
// doctor is handled before connectEngine() above
|
||||
case 'migrate': {
|
||||
// #3390: `gbrain migrate embeddings --to <provider:model>` — the
|
||||
// provider-agnostic embedding migration. Everything else stays the
|
||||
// engine-transfer path (`migrate --to <supabase|pglite>`).
|
||||
if (args[0] === 'embeddings') {
|
||||
const { runMigrateEmbeddings } = await import('./commands/migrate-embeddings.ts');
|
||||
await runMigrateEmbeddings(engine, args.slice(1));
|
||||
break;
|
||||
}
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
console.log('Usage: gbrain migrate --to <supabase|pglite> [--url <url>] [--path <path>] [--force]');
|
||||
console.log(' gbrain migrate embeddings --to <provider:model> [--dim N] [--dry-run] [--yes]');
|
||||
console.log('');
|
||||
console.log('The first form transfers the brain between engines; the second re-embeds');
|
||||
console.log('onto a different embedding provider (run `gbrain migrate embeddings --help`).');
|
||||
break;
|
||||
}
|
||||
const { runMigrateEngine } = await import('./commands/migrate-engine.ts');
|
||||
await runMigrateEngine(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'retrieval-upgrade': {
|
||||
// The command README.md + doctor.ts promised since v0.36 but never
|
||||
// dispatched. Alias for `migrate embeddings` (#3390).
|
||||
const { runMigrateEmbeddings } = await import('./commands/migrate-embeddings.ts');
|
||||
await runMigrateEmbeddings(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'eval': {
|
||||
// v0.32 EXP-5: `eval takes-quality {run,trend,regress}` requires a
|
||||
// brain (samples takes from DB / reads runs table). `replay` was
|
||||
@@ -2352,6 +2380,7 @@ USAGE
|
||||
SETUP
|
||||
init [--pglite|--supabase|--url] Create brain (PGLite default, no server)
|
||||
migrate --to <supabase|pglite> Transfer brain between engines
|
||||
migrate embeddings --to <p:model> Re-embed onto another embedding provider
|
||||
upgrade Self-update
|
||||
check-update [--json] Check for new versions
|
||||
doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings)
|
||||
|
||||
@@ -5923,7 +5923,7 @@ export async function buildChecks(
|
||||
// that doesn't match the gateway's resolved default. Empty-brain vs
|
||||
// non-empty-brain branching determines the repair hint:
|
||||
// - empty brain (no embedded chunks) → `gbrain init --force --embedding-model …`
|
||||
// - non-empty brain → `gbrain retrieval-upgrade --to … --reindex`
|
||||
// - non-empty brain → `gbrain migrate embeddings --to … --dim …` (#3390)
|
||||
// The bug-reporter's `rm -rf ~/.gbrain` recovery is never the right answer.
|
||||
let surfacedUnconfiguredDrift = false;
|
||||
try {
|
||||
@@ -5954,7 +5954,7 @@ export async function buildChecks(
|
||||
if (totalChunks > 0) {
|
||||
const fix = embeddedCount === 0
|
||||
? `No embeddings yet — drop the empty schema and re-init at the right dim:\n gbrain init --force --pglite --embedding-model ${configuredModel} --embedding-dimensions ${configuredDims}`
|
||||
: `Non-empty brain (${embeddedCount} embedded chunks). Migrate cleanly:\n gbrain retrieval-upgrade --to ${configuredModel} --reindex`;
|
||||
: `Non-empty brain (${embeddedCount} embedded chunks). Migrate cleanly:\n gbrain migrate embeddings --to ${configuredModel} --dim ${configuredDims}`;
|
||||
|
||||
checks.push({
|
||||
name: 'embedding_provider',
|
||||
|
||||
+54
-4
@@ -115,6 +115,16 @@ export interface EmbedOpts {
|
||||
* Errors/warnings still go to stderr regardless.
|
||||
*/
|
||||
quiet?: boolean;
|
||||
/**
|
||||
* #3391: widen signature-drift invalidation to pages with NO recorded
|
||||
* embedding_signature (pre-v108). By default those are grandfathered
|
||||
* (never invalidated) so a routine upgrade doesn't surprise-re-embed a
|
||||
* whole corpus — but after a provider/model swap the grandfather clause
|
||||
* silently leaves them in the OLD embedding space, mixing two vector
|
||||
* spaces in one index. `gbrain migrate embeddings` and
|
||||
* `gbrain embed --stale --include-null-signature` set this.
|
||||
*/
|
||||
includeNullSignature?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -356,6 +366,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
|
||||
pacer,
|
||||
paceMaxConcurrency,
|
||||
quiet: opts.quiet,
|
||||
includeNullSignature: opts.includeNullSignature,
|
||||
}, opts.signal);
|
||||
} finally {
|
||||
// E1: surface pacing telemetry (human + structured) when pacing was on.
|
||||
@@ -469,6 +480,8 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
|
||||
const priorityRaw = priorityIdx >= 0 ? args[priorityIdx + 1] : undefined;
|
||||
const priority = priorityRaw === 'recent' ? 'recent' as const : undefined;
|
||||
const catchUp = args.includes('--catch-up');
|
||||
// #3391: re-embed pages that predate the embedding_signature stamp too.
|
||||
const includeNullSignature = args.includes('--include-null-signature');
|
||||
const pace = parsePaceArgs(args);
|
||||
|
||||
let opts: EmbedOpts;
|
||||
@@ -476,11 +489,11 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
|
||||
opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId, batchSize, priority, catchUp };
|
||||
} else if (all || stale) {
|
||||
// E-2: CLI-only single-flight for stale runs (the minion path locks itself).
|
||||
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }) };
|
||||
opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }), ...(includeNullSignature && { includeNullSignature: true }) };
|
||||
} else {
|
||||
const slug = args.find(a => !a.startsWith('--'));
|
||||
if (!slug) {
|
||||
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up]');
|
||||
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run] [--batch-size N] [--priority recent] [--catch-up] [--include-null-signature]');
|
||||
process.exit(1);
|
||||
}
|
||||
opts = { slug, dryRun, sourceId, batchSize, priority, catchUp };
|
||||
@@ -657,6 +670,8 @@ async function embedAll(
|
||||
paceMaxConcurrency?: number;
|
||||
/** #394: suppress human stdout summaries (structured-output callers). */
|
||||
quiet?: boolean;
|
||||
/** #3391: lift the NULL-signature grandfather clause (see EmbedOpts). */
|
||||
includeNullSignature?: boolean;
|
||||
},
|
||||
signal?: AbortSignal,
|
||||
) {
|
||||
@@ -845,6 +860,8 @@ async function embedAllStale(
|
||||
paceMaxConcurrency?: number;
|
||||
/** #394: suppress human stdout summaries (structured-output callers). */
|
||||
quiet?: boolean;
|
||||
/** #3391: lift the NULL-signature grandfather clause (see EmbedOpts). */
|
||||
includeNullSignature?: boolean;
|
||||
},
|
||||
signature?: string,
|
||||
externalSignal?: AbortSignal,
|
||||
@@ -852,6 +869,7 @@ async function embedAllStale(
|
||||
// D7: thread sourceId so source-scoped runs only count + visit
|
||||
// that source's NULL embeddings.
|
||||
const sourceOpt = sourceId ? { sourceId } : undefined;
|
||||
const includeNullSig = !!staleOpts?.includeNullSignature;
|
||||
|
||||
// v0.41.31: re-embed pages whose embedding_signature drifted (model/dims
|
||||
// swap). dry-run must NOT mutate, so it counts signature-stale via the
|
||||
@@ -861,16 +879,46 @@ async function embedAllStale(
|
||||
const invalidated = await engine.invalidateStaleSignatureEmbeddings({
|
||||
signature,
|
||||
...(sourceId && { sourceId }),
|
||||
...(includeNullSig && { includeNullSignature: true }),
|
||||
});
|
||||
if (invalidated > 0 && !staleOpts?.quiet) {
|
||||
slog(`[embed] invalidated ${invalidated} chunk(s) embedded under a prior model signature`);
|
||||
}
|
||||
// #3391: the grandfather clause keeps NULL-signature pages on their OLD
|
||||
// vectors — two embedding spaces mixed in one index. Loud stderr warning
|
||||
// with the fix, instead of silent retrieval degradation.
|
||||
//
|
||||
// Deliberately NOT gated on `invalidated > 0`: the original bug report's
|
||||
// shape is a brain where EVERY embedded page predates the signature stamp,
|
||||
// so nothing drifts, nothing is invalidated — and pre-fix that brain got
|
||||
// no warning AND no work, the exact silent case #3391 is about. The probe
|
||||
// below computes the left-behind count directly, which is 0 on a healthy
|
||||
// brain, so an unaffected run stays quiet.
|
||||
if (!includeNullSig) {
|
||||
try {
|
||||
const wide = await engine.countStaleChunks({ ...sourceOpt, signature, includeNullSignature: true });
|
||||
const narrow = await engine.countStaleChunks({ ...sourceOpt, signature });
|
||||
const leftBehind = wide - narrow;
|
||||
if (leftBehind > 0) {
|
||||
serr(
|
||||
` [embed] WARNING: ${leftBehind} embedded chunk(s) sit on pages with no recorded ` +
|
||||
`embedding signature and were NOT invalidated — they remain in the previous model's ` +
|
||||
`embedding space. Re-run with --include-null-signature (or use ` +
|
||||
`\`gbrain migrate embeddings\`) to re-embed them.`,
|
||||
);
|
||||
}
|
||||
} catch {
|
||||
// The warning probe is best-effort; never break the embed run.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Pre-flight: 0 stale chunks → nothing to do, no further DB reads.
|
||||
// dry-run includes signature-drift in the count without mutating.
|
||||
const staleCount = await engine.countStaleChunks(
|
||||
dryRun && signature ? { ...sourceOpt, signature } : sourceOpt,
|
||||
dryRun && signature
|
||||
? { ...sourceOpt, signature, ...(includeNullSig && { includeNullSignature: true }) }
|
||||
: sourceOpt,
|
||||
);
|
||||
if (staleCount === 0) {
|
||||
if (!staleOpts?.quiet) {
|
||||
@@ -1138,7 +1186,9 @@ async function embedAllStale(
|
||||
// as a clean run — re-running won't help until the underlying failure is fixed.
|
||||
if (staleOpts?.catchUp && !effectiveSignal.aborted && embedFailures > 0) {
|
||||
const remaining = await engine.countStaleChunks(
|
||||
signature ? { signature, ...(sourceId ? { sourceId } : {}) } : (sourceId ? { sourceId } : undefined),
|
||||
signature
|
||||
? { signature, ...(sourceId ? { sourceId } : {}), ...(includeNullSig && { includeNullSignature: true }) }
|
||||
: (sourceId ? { sourceId } : undefined),
|
||||
);
|
||||
if (remaining > 0) {
|
||||
serr(`\n [embed] catch-up finished but ${remaining} chunk(s) remain stale after ${embedFailures} embed failure(s). These are not embeddable as-is; re-running won't clear them until the underlying error is resolved.`);
|
||||
|
||||
@@ -0,0 +1,402 @@
|
||||
/**
|
||||
* `gbrain migrate embeddings --to <provider:model>` (#3390) — the
|
||||
* provider-agnostic forward migration off any embedding provider, built for
|
||||
* the ZeroEntropy 2026-09-04 sunset but not keyed to it.
|
||||
*
|
||||
* Also reachable as `gbrain retrieval-upgrade` — the command README.md and
|
||||
* doctor.ts have promised since v0.36 but which never had a dispatch branch.
|
||||
*
|
||||
* Flow (everything heavy is reused, see src/core/embedding-migration.ts):
|
||||
* 1. plan — chunk/char counts via the widened stale predicates,
|
||||
* cost estimate from embedding-pricing.ts
|
||||
* 2. preflight— print estimate; require --yes or interactive confirm
|
||||
* (non-TTY without --yes refuses with exit 2, mirroring the
|
||||
* reindex-code cost gate in docs/operations/spend-controls.md)
|
||||
* 3. probe — one live embed against the TARGET provider BEFORE any
|
||||
* mutation (validates key + model + dims in one shot)
|
||||
* 4. apply — schema transition (dim change), config (DB + file plane),
|
||||
* #3391 NULL-signature-inclusive invalidation, cache purge
|
||||
* 5. re-embed — runEmbedCore --stale --catch-up with single-flight locks,
|
||||
* pacing (--pace), progress reporting. Resumable: a killed
|
||||
* run re-runs the SAME command; the NULL-embedding cursor is
|
||||
* the checkpoint and steps 3-4 no-op on the second pass.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { serr, slog } from '../core/console-prefix.ts';
|
||||
import {
|
||||
planEmbeddingMigration,
|
||||
applyEmbeddingMigration,
|
||||
completeEmbeddingMigration,
|
||||
reconcilePageSignatures,
|
||||
MIGRATION_STATE_KEY,
|
||||
type EmbeddingMigrationPlan,
|
||||
} from '../core/embedding-migration.ts';
|
||||
import { formatEnvOverrideWarning } from '../core/retrieval-upgrade-planner.ts';
|
||||
import { parsePaceArgs, runEmbedCore } from './embed.ts';
|
||||
|
||||
export interface MigrateEmbeddingsFlags {
|
||||
to?: string;
|
||||
dim?: number;
|
||||
yes: boolean;
|
||||
dryRun: boolean;
|
||||
json: boolean;
|
||||
noEmbed: boolean;
|
||||
ignoreEnvOverride: boolean;
|
||||
batchSize?: number;
|
||||
pace?: ReturnType<typeof parsePaceArgs>;
|
||||
}
|
||||
|
||||
export function parseMigrateEmbeddingsFlags(args: string[]): MigrateEmbeddingsFlags {
|
||||
const toIdx = args.indexOf('--to');
|
||||
const dimIdx = args.indexOf('--dim');
|
||||
const dimRaw = dimIdx >= 0 ? parseInt(args[dimIdx + 1] ?? '', 10) : NaN;
|
||||
const bsIdx = args.indexOf('--batch-size');
|
||||
const bsRaw = bsIdx >= 0 ? parseInt(args[bsIdx + 1] ?? '', 10) : NaN;
|
||||
const batchSize = Number.isFinite(bsRaw) && bsRaw > 0 ? Math.min(10_000, bsRaw) : undefined;
|
||||
return {
|
||||
to: toIdx >= 0 ? args[toIdx + 1] : undefined,
|
||||
dim: Number.isFinite(dimRaw) && dimRaw > 0 ? dimRaw : undefined,
|
||||
yes: args.includes('--yes') || args.includes('--non-interactive'),
|
||||
dryRun: args.includes('--dry-run'),
|
||||
json: args.includes('--json'),
|
||||
noEmbed: args.includes('--no-embed'),
|
||||
ignoreEnvOverride: args.includes('--ignore-env-override'),
|
||||
...(batchSize !== undefined && { batchSize }),
|
||||
pace: parsePaceArgs(args),
|
||||
};
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
process.stdout.write(`Usage: gbrain migrate embeddings --to <provider:model> [flags]
|
||||
|
||||
Re-embed the whole brain onto a different embedding provider/model. Handles
|
||||
dimension changes (schema transition), pages without a recorded embedding
|
||||
signature (#3391), the query cache, and resume-after-kill. The forward path
|
||||
off a sunsetting provider.
|
||||
|
||||
Flags:
|
||||
--to <provider:model> Target embedding model (e.g. openai:text-embedding-3-small).
|
||||
--dim <N> Target dimensions. Defaults to the provider recipe's
|
||||
declared width; required when the recipe declares none.
|
||||
--dry-run Plan + cost estimate only; change nothing.
|
||||
--yes Skip the confirm prompt (required non-interactively).
|
||||
--json Machine-readable envelope on stdout.
|
||||
--no-embed Apply schema + config + invalidation, but skip the
|
||||
re-embed pass (run \`gbrain embed --stale --include-null-signature\`
|
||||
or \`... --background\` yourself).
|
||||
--batch-size <N> Stale-chunk batch size for the re-embed (default 2000).
|
||||
--pace[=mode] DB-contention pacing for the re-embed (off|gentle|balanced|aggressive).
|
||||
--ignore-env-override Proceed even when GBRAIN_EMBEDDING_* env vars would
|
||||
override the target at runtime (you know why).
|
||||
--help Show this help.
|
||||
|
||||
A killed run is resumable: re-run the same command. Already-migrated chunks
|
||||
are never re-embedded twice.
|
||||
`);
|
||||
}
|
||||
|
||||
function renderPlan(plan: EmbeddingMigrationPlan): string {
|
||||
const lines: string[] = [];
|
||||
lines.push('Embedding migration plan');
|
||||
lines.push(` From: ${plan.from_model} (${plan.from_dims}d${plan.column_dims !== null && plan.column_dims !== plan.from_dims ? `; column is actually ${plan.column_dims}d` : ''})`);
|
||||
lines.push(` To: ${plan.to_model} (${plan.to_dims}d)`);
|
||||
if (plan.dim_change) {
|
||||
lines.push(` DESTRUCTIVE: the embedding column is rebuilt at ${plan.to_dims}d, which DELETES`);
|
||||
lines.push(' every stored embedding vector in this brain. They are not recoverable —');
|
||||
lines.push(' going back to the old provider means paying for a second full re-embed.');
|
||||
lines.push(' Until the re-embed finishes, semantic search is degraded to lexical-only.');
|
||||
lines.push(` The query cache and fact embeddings are rebuilt at ${plan.to_dims}d too`);
|
||||
lines.push(' (cache refills on next query; facts re-embed on their next write).');
|
||||
}
|
||||
lines.push(` Chunks to re-embed: ${plan.chunks_to_embed}${plan.null_signature_chunks > 0 ? ` (includes ${plan.null_signature_chunks} on pages with no recorded embedding signature)` : ''}`);
|
||||
lines.push(
|
||||
plan.price_known
|
||||
? ` Estimated cost: $${plan.est_cost_usd.toFixed(2)} (${plan.total_chars} chars at the ${plan.to_model} rate)`
|
||||
: ` Estimated cost: unknown — no pricing entry for ${plan.to_model}. Check the provider's pricing before proceeding.`,
|
||||
);
|
||||
if (plan.resuming) {
|
||||
lines.push(' Resuming: a prior migration to this target was interrupted; continuing it.');
|
||||
}
|
||||
if (plan.reranker_warning) {
|
||||
lines.push(` WARNING: ${plan.reranker_warning}`);
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/** Single-keypress y/N confirm on stdin. Injectable for tests. */
|
||||
async function defaultConfirm(question: string): Promise<boolean> {
|
||||
process.stderr.write(`${question} [y/N] `);
|
||||
const stdin = process.stdin;
|
||||
stdin.setRawMode?.(true);
|
||||
stdin.resume();
|
||||
const key: string = await new Promise((resolve) => {
|
||||
stdin.once('data', (d) => resolve(d.toString()));
|
||||
});
|
||||
stdin.setRawMode?.(false);
|
||||
stdin.pause();
|
||||
process.stderr.write('\n');
|
||||
return key.trim().toLowerCase().startsWith('y');
|
||||
}
|
||||
|
||||
/**
|
||||
* One tiny embed against the TARGET provider, BEFORE any mutation: validates
|
||||
* the API key, the model id, and dimension support in a single call, so a bad
|
||||
* target fails with the brain untouched instead of after the column is
|
||||
* dropped. Shared by the CLI and the `migrate_embeddings` op (the op used to
|
||||
* skip it, which let `yes:true` drop the column against a bad key).
|
||||
*/
|
||||
export async function probeTargetProvider(
|
||||
toModel: string,
|
||||
toDims: number,
|
||||
): Promise<{ ok: true } | { ok: false; message: string }> {
|
||||
try {
|
||||
const { embed } = await import('../core/ai/gateway.ts');
|
||||
const vecs = await embed(['gbrain embedding migration probe'], {
|
||||
embeddingModel: toModel,
|
||||
dimensions: toDims,
|
||||
});
|
||||
const got = vecs[0]?.length ?? 0;
|
||||
if (got !== toDims) {
|
||||
return {
|
||||
ok: false,
|
||||
message: `Target provider returned ${got}-dim vectors, expected ${toDims}. Pass a valid --dim for ${toModel}.`,
|
||||
};
|
||||
}
|
||||
return { ok: true };
|
||||
} catch (e) {
|
||||
return {
|
||||
ok: false,
|
||||
message: `Preflight embed against ${toModel} failed — nothing was changed:\n ${e instanceof Error ? e.message : String(e)}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist the target model+dims to the FILE plane and reconfigure the
|
||||
* in-process gateway. The gateway reads file/env config, not the DB plane —
|
||||
* without this the re-embed would silently run against the OLD provider.
|
||||
* Shared by the CLI command and the `migrate_embeddings` op handler.
|
||||
*/
|
||||
export async function persistEmbeddingFileConfig(
|
||||
toModel: string,
|
||||
toDims: number,
|
||||
): Promise<void> {
|
||||
const { loadConfig, saveConfig } = await import('../core/config.ts');
|
||||
const { configureGateway } = await import('../core/ai/gateway.ts');
|
||||
const { buildGatewayConfig } = await import('../core/ai/build-gateway-config.ts');
|
||||
const cfg = loadConfig();
|
||||
if (!cfg) {
|
||||
// REFUSE rather than warn-and-proceed. Without a file plane to write, the
|
||||
// switch would not survive this process: the next `gbrain` invocation
|
||||
// reads file/env config, sees the OLD provider, and re-embeds the brain
|
||||
// back into the old space (paying twice) — or fails outright against a
|
||||
// column that is now the new width. Thrown from inside
|
||||
// applyEmbeddingMigration's try, so it surfaces as status: 'failed'
|
||||
// BEFORE the config/cache steps and the caller exits non-zero.
|
||||
throw new Error(
|
||||
'No ~/.gbrain/config.json found — refusing to migrate.\n' +
|
||||
' The embed pipeline reads file/env config, so without a file plane this switch\n' +
|
||||
' would not survive the process and the next run would re-embed into the old space.\n' +
|
||||
' Fix: run `gbrain init` (or set GBRAIN_EMBEDDING_MODEL + GBRAIN_EMBEDDING_DIMENSIONS\n' +
|
||||
' in the environment of every gbrain process) and re-run.',
|
||||
);
|
||||
}
|
||||
cfg.embedding_model = toModel;
|
||||
cfg.embedding_dimensions = toDims;
|
||||
saveConfig(cfg);
|
||||
configureGateway(buildGatewayConfig(cfg));
|
||||
}
|
||||
|
||||
export interface RunMigrateEmbeddingsOpts {
|
||||
/** Test seams. */
|
||||
confirm?: (question: string) => Promise<boolean>;
|
||||
isTTY?: boolean;
|
||||
exit?: (code: number) => never;
|
||||
}
|
||||
|
||||
export async function runMigrateEmbeddings(
|
||||
engine: BrainEngine,
|
||||
args: string[],
|
||||
opts: RunMigrateEmbeddingsOpts = {},
|
||||
): Promise<void> {
|
||||
// Explicit `never` annotation so TS control-flow analysis treats every
|
||||
// exit() call as terminal (required for narrowing after the guard blocks).
|
||||
const exit: (code: number) => never = opts.exit ?? ((code: number) => process.exit(code));
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
printHelp();
|
||||
exit(0);
|
||||
}
|
||||
const flags = parseMigrateEmbeddingsFlags(args);
|
||||
if (!flags.to) {
|
||||
serr('Missing --to <provider:model>. Example: gbrain migrate embeddings --to openai:text-embedding-3-small');
|
||||
serr('Run with --help for all flags.');
|
||||
exit(1);
|
||||
}
|
||||
|
||||
// From-state as the gateway resolved it (file/env config + defaults) —
|
||||
// the truth for what embeds run under TODAY.
|
||||
let fromModel: string | undefined;
|
||||
let fromDims: number | undefined;
|
||||
try {
|
||||
const { getEmbeddingModel, getEmbeddingDimensions } = await import('../core/ai/gateway.ts');
|
||||
fromModel = getEmbeddingModel();
|
||||
fromDims = getEmbeddingDimensions();
|
||||
} catch {
|
||||
// Gateway unconfigured — plan falls back to shipped defaults.
|
||||
}
|
||||
|
||||
let plan: EmbeddingMigrationPlan;
|
||||
try {
|
||||
plan = await planEmbeddingMigration(engine, {
|
||||
to: flags.to!,
|
||||
...(flags.dim !== undefined && { dim: flags.dim }),
|
||||
...(fromModel !== undefined && { fromModel }),
|
||||
...(fromDims !== undefined && { fromDims }),
|
||||
});
|
||||
} catch (e) {
|
||||
serr(e instanceof Error ? e.message : String(e));
|
||||
exit(1);
|
||||
return; // unreachable; keeps TS happy for injected exit seams
|
||||
}
|
||||
|
||||
if (flags.json) {
|
||||
// Human plan goes to stderr so stdout stays JSON-clean.
|
||||
serr(renderPlan(plan));
|
||||
} else {
|
||||
console.log(renderPlan(plan));
|
||||
}
|
||||
|
||||
if (plan.chunks_to_embed === 0 && !plan.dim_change && plan.from_model === plan.to_model) {
|
||||
if (flags.json) console.log(JSON.stringify({ status: 'skipped_no_work', plan }, null, 2));
|
||||
else console.log('Nothing to migrate — brain is already on the target model.');
|
||||
exit(0);
|
||||
}
|
||||
|
||||
if (flags.dryRun) {
|
||||
if (flags.json) console.log(JSON.stringify({ status: 'planned', plan }, null, 2));
|
||||
exit(0);
|
||||
}
|
||||
|
||||
// ── Consent gate. Unlike the pure cost gates in
|
||||
// docs/operations/spend-controls.md, `spend.posture=tokenmax` does NOT
|
||||
// bypass this one: posture waives the SPEND ceiling, and this gate also
|
||||
// guards a destructive schema rebuild (existing vectors are dropped, and
|
||||
// retrieval is degraded until the re-embed finishes). We honor the posture
|
||||
// by marking the dollar figure informational, and still ask.
|
||||
if (!flags.yes) {
|
||||
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
|
||||
const posture = await resolveSpendPosture(engine);
|
||||
if (posture === 'tokenmax') {
|
||||
serr(' [migrate] spend.posture=tokenmax: the cost estimate above is informational.');
|
||||
serr(' [migrate] Confirmation is still required — this rebuilds the embedding column (destructive, not just costly).');
|
||||
}
|
||||
const isTTY = opts.isTTY ?? Boolean(process.stdin.isTTY);
|
||||
if (!isTTY) {
|
||||
serr('Refusing to migrate without confirmation in a non-TTY environment. Re-run with --yes.');
|
||||
exit(2);
|
||||
}
|
||||
const confirm = opts.confirm ?? defaultConfirm;
|
||||
const priceNote = plan.price_known ? `~$${plan.est_cost_usd.toFixed(2)}` : 'an UNKNOWN amount';
|
||||
const ok = await confirm(`Re-embed ${plan.chunks_to_embed} chunks (${priceNote})?`);
|
||||
if (!ok) {
|
||||
serr('Aborted. Nothing was changed.');
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Live probe BEFORE any mutation: one tiny embed against the TARGET
|
||||
// provider validates API key, model id, and dimension support in one call.
|
||||
const probe = await probeTargetProvider(plan.to_model, plan.to_dims);
|
||||
if (!probe.ok) {
|
||||
serr(probe.message);
|
||||
exit(1);
|
||||
}
|
||||
|
||||
// ── Apply: schema + config + invalidation + cache purge.
|
||||
const applied = await applyEmbeddingMigration(engine, plan, {
|
||||
ignoreEnvOverride: flags.ignoreEnvOverride,
|
||||
persistConfig: (toModel, toDims) => persistEmbeddingFileConfig(toModel, toDims),
|
||||
});
|
||||
|
||||
if (applied.status === 'refused') {
|
||||
if (flags.json) console.log(JSON.stringify(applied, null, 2));
|
||||
else serr(formatEnvOverrideWarning(applied.warning));
|
||||
exit(1);
|
||||
}
|
||||
if (applied.status === 'failed') {
|
||||
if (flags.json) console.log(JSON.stringify(applied, null, 2));
|
||||
else serr(`Migration apply failed: ${applied.reason}`);
|
||||
exit(1);
|
||||
}
|
||||
|
||||
serr(` [migrate] schema ${applied.schema_transitioned ? `rebuilt at ${plan.to_dims}d` : 'unchanged'}; ` +
|
||||
`${applied.invalidated} chunk(s) invalidated; query cache purged (${applied.cache_cleared} row(s)).`);
|
||||
|
||||
if (flags.noEmbed) {
|
||||
const msg = 'Config + schema migrated. Re-embed deferred — run: gbrain embed --stale --catch-up --include-null-signature';
|
||||
if (flags.json) console.log(JSON.stringify({ ...applied, status: 'applied_no_embed', plan }, null, 2));
|
||||
else console.log(msg);
|
||||
exit(0);
|
||||
}
|
||||
|
||||
// ── Re-embed. All the machinery (locks, pacing, backoff, progress,
|
||||
// signature stamping) is the standard embed pipeline.
|
||||
const { createProgress } = await import('../core/progress.ts');
|
||||
const { getCliOptions, cliOptsToProgressOptions } = await import('../core/cli-options.ts');
|
||||
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
|
||||
let progressStarted = false;
|
||||
const embedResult = await runEmbedCore(engine, {
|
||||
stale: true,
|
||||
catchUp: true,
|
||||
singleFlight: true,
|
||||
includeNullSignature: true,
|
||||
quiet: flags.json,
|
||||
...(flags.batchSize !== undefined && { batchSize: flags.batchSize }),
|
||||
...(flags.pace && { pace: flags.pace }),
|
||||
onProgress: (done, total) => {
|
||||
if (!progressStarted) {
|
||||
progress.start('migrate.reembed', total);
|
||||
progressStarted = true;
|
||||
}
|
||||
progress.tick(1);
|
||||
},
|
||||
});
|
||||
if (progressStarted) progress.finish();
|
||||
|
||||
// Reconcile signatures BEFORE the completion probe: pages straddling a
|
||||
// stale-batch boundary are embedded correctly but left unstamped by the
|
||||
// embed loop's all-or-nothing stamp rule. Without this the probe would call
|
||||
// a fully-migrated brain "incomplete" and the re-run would pay again.
|
||||
const reconciled = await reconcilePageSignatures(engine, plan);
|
||||
if (reconciled > 0) {
|
||||
serr(` [migrate] reconciled the embedding signature on ${reconciled} fully-embedded page(s) (batch-boundary pages).`);
|
||||
}
|
||||
|
||||
const remaining = await engine.countStaleChunks({
|
||||
signature: `${plan.to_model}:${plan.to_dims}`,
|
||||
includeNullSignature: true,
|
||||
});
|
||||
|
||||
if (remaining === 0) {
|
||||
await completeEmbeddingMigration(engine, plan);
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({ status: 'completed', plan, embedded: embedResult.embedded, remaining: 0 }, null, 2));
|
||||
} else {
|
||||
slog(`Migration complete: ${embedResult.embedded} chunk(s) embedded on ${plan.to_model} (${plan.to_dims}d).`);
|
||||
if (plan.reranker_warning) serr(` [migrate] reminder: ${plan.reranker_warning}`);
|
||||
}
|
||||
exit(0);
|
||||
} else {
|
||||
if (flags.json) {
|
||||
console.log(JSON.stringify({ status: 'incomplete', plan, embedded: embedResult.embedded, remaining }, null, 2));
|
||||
} else {
|
||||
serr(`Migration incomplete: ${remaining} chunk(s) still stale (embed failures or an interrupted run).`);
|
||||
serr('Re-run the same command to resume — completed chunks are never re-embedded.');
|
||||
}
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
/** Re-export for the op handler + tests. */
|
||||
export { MIGRATION_STATE_KEY };
|
||||
@@ -462,6 +462,53 @@ export async function runPostUpgrade(args: string[] = []): Promise<void> {
|
||||
// Banner is cosmetic; never block the upgrade.
|
||||
}
|
||||
|
||||
// #3390: ZeroEntropy sunset notice. ZE announced (2026-07-24) that
|
||||
// its hosted endpoints — including /models/embed and /models/rerank —
|
||||
// shut down on 2026-09-04. Any brain resolving to a zeroentropyai:*
|
||||
// embedding model (including default-config brains that never set
|
||||
// one) loses SEMANTIC RETRIEVAL ENTIRELY on that date: the query
|
||||
// embedding uses the same endpoint, so existing vectors become
|
||||
// unqueryable. One-shot per install, gated by
|
||||
// `ze_sunset_notice_shown` (same pattern as the search-mode banner).
|
||||
try {
|
||||
const shown = await engine.getConfig('ze_sunset_notice_shown');
|
||||
const { DEFAULT_EMBEDDING_MODEL } = await import('../core/ai/defaults.ts');
|
||||
const effectiveModel = cfgSchema.embedding_model ?? DEFAULT_EMBEDDING_MODEL;
|
||||
const rerankerModel = await engine.getConfig('search.reranker.model');
|
||||
const onZeEmbedding = effectiveModel.startsWith('zeroentropyai:');
|
||||
const onZeReranker = !!rerankerModel?.startsWith('zeroentropyai:');
|
||||
if (shown !== 'true' && (onZeEmbedding || onZeReranker)) {
|
||||
console.log('');
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('[gbrain] ACTION REQUIRED: ZeroEntropy hosted API sunsets 2026-09-04.');
|
||||
if (onZeEmbedding) {
|
||||
console.log(`[gbrain] This brain embeds with ${effectiveModel}. After the sunset,`);
|
||||
console.log('[gbrain] semantic retrieval STOPS WORKING (queries can no longer be');
|
||||
console.log('[gbrain] embedded against your existing vectors).');
|
||||
}
|
||||
if (onZeReranker) {
|
||||
console.log(`[gbrain] The reranker (${rerankerModel}) also sunsets; search falls`);
|
||||
console.log('[gbrain] back to unreranked ordering.');
|
||||
}
|
||||
console.log('═══════════════════════════════════════════════════════════════');
|
||||
console.log('');
|
||||
console.log('Migrate before the sunset (resumable; preview cost first):');
|
||||
console.log(' gbrain migrate embeddings --to <provider:model> --dry-run');
|
||||
console.log(' gbrain migrate embeddings --to <provider:model>');
|
||||
console.log('');
|
||||
console.log('Self-hosting zembed-1 (weights are Apache-2.0) via llama-server /');
|
||||
console.log('ollama also works and preserves your existing vectors — point');
|
||||
console.log('embedding at the local endpoint instead of migrating.');
|
||||
if (onZeReranker) {
|
||||
console.log('Reranker: gbrain config set search.reranker.enabled false (or pick another).');
|
||||
}
|
||||
console.log('');
|
||||
await engine.setConfig('ze_sunset_notice_shown', 'true');
|
||||
}
|
||||
} catch {
|
||||
// Banner is cosmetic; never block the upgrade.
|
||||
}
|
||||
|
||||
// PR1: skill-catalog publish consent. New installs default ON at
|
||||
// `gbrain init`; EXISTING installs stay OFF (default-OFF runtime = no
|
||||
// silent capability grant on upgrade) until the owner opts in HERE.
|
||||
|
||||
@@ -0,0 +1,340 @@
|
||||
/**
|
||||
* Provider-agnostic embedding migration (#3390).
|
||||
*
|
||||
* `gbrain migrate embeddings --to <provider:model>` re-embeds a brain onto
|
||||
* any configured provider — the forward path off a sunsetting provider that
|
||||
* `ze-switch` (ZE-only target) and `ze-switch --undo` (needs a snapshot fresh
|
||||
* installs don't have) cannot cover.
|
||||
*
|
||||
* Deliberately thin: everything heavy is reused —
|
||||
* - runSchemaTransition (retrieval-upgrade-planner.ts) for dimension changes
|
||||
* - invalidateStaleSignatureEmbeddings + the NULL-embedding cursor for
|
||||
* staleness + resume (the NULL column IS the checkpoint: a killed run
|
||||
* re-runs the same command and continues where it stopped)
|
||||
* - the embed pipeline (src/commands/embed.ts) for the actual re-embed,
|
||||
* with pacing, backfill locks, rate-limit backoff, and progress
|
||||
* - lookupEmbeddingPrice / estimateCostFromChars for the preflight estimate
|
||||
* - detectEnvOverride (the #1421 damage-class gate) before any mutation
|
||||
*
|
||||
* #3391 companion fix: the migration widens staleness with
|
||||
* `includeNullSignature: true` so pages that predate the v108 signature stamp
|
||||
* are re-embedded too, instead of silently staying in the old embedding space.
|
||||
*
|
||||
* The command layer (src/commands/migrate-embeddings.ts) owns everything
|
||||
* process-shaped: confirm prompts, file-plane config persistence (the gateway
|
||||
* reads file/env, not the DB plane), gateway reconfiguration, and the embed
|
||||
* catch-up run. This module is engine-pure so both engines and the op handler
|
||||
* share one implementation.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { resolveRecipe, embeddingDimsForModel } from './ai/model-resolver.ts';
|
||||
import { lookupEmbeddingPrice, estimateCostFromChars } from './embedding-pricing.ts';
|
||||
import { detectEnvOverride, type EnvOverrideWarning } from './retrieval-upgrade-planner.ts';
|
||||
import { runSchemaTransition } from './retrieval-upgrade-planner.ts';
|
||||
import { readContentChunksEmbeddingDim } from './embedding-dim-check.ts';
|
||||
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
|
||||
|
||||
/**
|
||||
* Resume/state marker (DB plane). Present while a migration is in flight so
|
||||
* a re-run can detect + resume; cleared when the re-embed drains to zero.
|
||||
*/
|
||||
export const MIGRATION_STATE_KEY = 'embedding_migration.state';
|
||||
/** ISO timestamp + summary of the last completed migration (DB plane). */
|
||||
export const MIGRATION_COMPLETED_KEY = 'embedding_migration.completed';
|
||||
|
||||
export interface MigrationState {
|
||||
to_model: string;
|
||||
to_dims: number;
|
||||
from_model: string;
|
||||
from_dims: number;
|
||||
started_at: string;
|
||||
}
|
||||
|
||||
export interface EmbeddingMigrationPlan {
|
||||
from_model: string;
|
||||
from_dims: number;
|
||||
/** Actual `content_chunks.embedding` vector(N) width (null = column absent). */
|
||||
column_dims: number | null;
|
||||
to_model: string;
|
||||
to_dims: number;
|
||||
/** True when the schema column must be rebuilt at a new width. */
|
||||
dim_change: boolean;
|
||||
/** Chunks not yet in the target embedding space (the migration workload). */
|
||||
chunks_to_embed: number;
|
||||
/** Characters across those chunks (feeds the cost estimate). */
|
||||
total_chars: number;
|
||||
/**
|
||||
* #3391 visibility: embedded chunks on pages with NO recorded signature
|
||||
* (pre-v108). Included in chunks_to_embed via includeNullSignature.
|
||||
*/
|
||||
null_signature_chunks: number;
|
||||
est_cost_usd: number;
|
||||
/** False when the target model has no entry in EMBEDDING_PRICING. */
|
||||
price_known: boolean;
|
||||
/** True when a prior in-flight migration state matches this target. */
|
||||
resuming: boolean;
|
||||
/** Set when the brain's reranker is also on the outgoing provider. */
|
||||
reranker_warning: string | null;
|
||||
}
|
||||
|
||||
export type MigrationApplyResult =
|
||||
| { status: 'applied'; invalidated: number; cache_cleared: number; schema_transitioned: boolean }
|
||||
| { status: 'refused'; reason: 'env_override'; warning: EnvOverrideWarning }
|
||||
| { status: 'failed'; reason: string };
|
||||
|
||||
/** `<provider:model>:<dims>` — must match currentEmbeddingSignature()'s shape. */
|
||||
export function migrationSignature(toModel: string, toDims: number): string {
|
||||
return `${toModel}:${toDims}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve + validate the target `provider:model` and dimensions.
|
||||
* Throws with a paste-ready message on an unknown provider or when the
|
||||
* recipe declares no default dims and the caller passed none.
|
||||
*/
|
||||
export function resolveMigrationTarget(to: string, dimFlag?: number): { toModel: string; toDims: number } {
|
||||
if (!to.includes(':')) {
|
||||
throw new Error(
|
||||
`--to must be provider:model (e.g. openai:text-embedding-3-small). Got: ${to}`,
|
||||
);
|
||||
}
|
||||
// Throws AIConfigError with provider list on an unknown provider.
|
||||
const { recipe } = resolveRecipe(to);
|
||||
if (!recipe.touchpoints.embedding) {
|
||||
throw new Error(`Provider ${recipe.id} has no embedding support. Pick an embedding-capable provider:model.`);
|
||||
}
|
||||
const toDims = dimFlag ?? embeddingDimsForModel(recipe, to);
|
||||
if (!toDims || toDims <= 0) {
|
||||
throw new Error(
|
||||
`No default dimension known for ${to}. Pass --dim <N> explicitly (see the provider's docs for valid values).`,
|
||||
);
|
||||
}
|
||||
return { toModel: to, toDims };
|
||||
}
|
||||
|
||||
/**
|
||||
* Pure read: compute the migration workload. Uses the stale-chunk predicates
|
||||
* with the TARGET signature + includeNullSignature so the count is
|
||||
* resume-aware — a re-plan mid-migration counts only what remains.
|
||||
*/
|
||||
export async function planEmbeddingMigration(
|
||||
engine: BrainEngine,
|
||||
opts: { to: string; dim?: number; fromModel?: string; fromDims?: number },
|
||||
): Promise<EmbeddingMigrationPlan> {
|
||||
const { toModel, toDims } = resolveMigrationTarget(opts.to, opts.dim);
|
||||
|
||||
// From-state: caller (CLI) passes the gateway-resolved values; fall back
|
||||
// to the shipped defaults for gateway-less contexts (unit tests, op probe).
|
||||
const fromModel = opts.fromModel ?? DEFAULT_EMBEDDING_MODEL;
|
||||
const fromDims = opts.fromDims ?? DEFAULT_EMBEDDING_DIMENSIONS;
|
||||
|
||||
const col = await readContentChunksEmbeddingDim(engine);
|
||||
|
||||
const sig = migrationSignature(toModel, toDims);
|
||||
const wide = await engine.countStaleChunks({ signature: sig, includeNullSignature: true });
|
||||
const narrow = await engine.countStaleChunks({ signature: sig });
|
||||
const totalChars = await engine.sumStaleChunkChars({ signature: sig, includeNullSignature: true });
|
||||
|
||||
const price = lookupEmbeddingPrice(toModel);
|
||||
const estCostUsd = price.kind === 'known'
|
||||
? estimateCostFromChars(totalChars, price.pricePerMTok)
|
||||
: 0;
|
||||
|
||||
let resuming = false;
|
||||
try {
|
||||
const stateStr = await engine.getConfig(MIGRATION_STATE_KEY);
|
||||
if (stateStr) {
|
||||
const state = JSON.parse(stateStr) as MigrationState;
|
||||
resuming = state.to_model === toModel && state.to_dims === toDims;
|
||||
}
|
||||
} catch {
|
||||
// Corrupt state marker — treat as fresh.
|
||||
}
|
||||
|
||||
// Sunset companion warning: migrating embeddings off a provider whose
|
||||
// reranker is still configured leaves rerank on the outgoing provider.
|
||||
let rerankerWarning: string | null = null;
|
||||
try {
|
||||
const rr = await engine.getConfig('search.reranker.model');
|
||||
const outgoingProvider = fromModel.split(':')[0];
|
||||
const targetProvider = toModel.split(':')[0];
|
||||
if (rr && outgoingProvider !== targetProvider && rr.startsWith(`${outgoingProvider}:`)) {
|
||||
rerankerWarning =
|
||||
`search.reranker.model is still ${rr} (the outgoing provider). ` +
|
||||
`If that provider is sunsetting, also update or disable the reranker: ` +
|
||||
`gbrain config set search.reranker.enabled false`;
|
||||
}
|
||||
} catch {
|
||||
// Reranker warning is cosmetic.
|
||||
}
|
||||
|
||||
return {
|
||||
from_model: fromModel,
|
||||
from_dims: fromDims,
|
||||
column_dims: col.dims,
|
||||
to_model: toModel,
|
||||
to_dims: toDims,
|
||||
dim_change: col.dims !== null && col.dims !== toDims,
|
||||
chunks_to_embed: wide,
|
||||
total_chars: totalChars,
|
||||
null_signature_chunks: wide - narrow,
|
||||
est_cost_usd: estCostUsd,
|
||||
price_known: price.kind === 'known',
|
||||
resuming,
|
||||
reranker_warning: rerankerWarning,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply the non-embed half of the migration: env gate, state marker, schema
|
||||
* transition (dim changes only), DB-plane config, file-plane persistence
|
||||
* (via callback — the core module never touches ~/.gbrain), stale-signature
|
||||
* invalidation (#3391: includeNullSignature), and query-cache purge.
|
||||
*
|
||||
* Ordering makes every step idempotent under a crash + re-run:
|
||||
* state marker → schema → config → invalidate → cache purge.
|
||||
* A crash anywhere leaves the state marker set; the re-run re-executes the
|
||||
* remaining steps (schema transition no-ops when the column is already at
|
||||
* the target width via the actual-width probe; invalidation matches nothing
|
||||
* the second time).
|
||||
*/
|
||||
export async function applyEmbeddingMigration(
|
||||
engine: BrainEngine,
|
||||
plan: EmbeddingMigrationPlan,
|
||||
opts: {
|
||||
ignoreEnvOverride?: boolean;
|
||||
/** Persist target model+dims to the file plane + reconfigure the gateway. */
|
||||
persistConfig?: (toModel: string, toDims: number) => void | Promise<void>;
|
||||
} = {},
|
||||
): Promise<MigrationApplyResult> {
|
||||
const envWarning = detectEnvOverride(plan.to_model, plan.to_dims);
|
||||
if (envWarning.triggered && !opts.ignoreEnvOverride) {
|
||||
return { status: 'refused', reason: 'env_override', warning: envWarning };
|
||||
}
|
||||
|
||||
try {
|
||||
// 1. State marker FIRST — a crash after any later step is resumable.
|
||||
const state: MigrationState = {
|
||||
to_model: plan.to_model,
|
||||
to_dims: plan.to_dims,
|
||||
from_model: plan.from_model,
|
||||
from_dims: plan.from_dims,
|
||||
started_at: new Date().toISOString(),
|
||||
};
|
||||
await engine.setConfig(MIGRATION_STATE_KEY, JSON.stringify(state));
|
||||
|
||||
// 2. Schema transition when the ACTUAL column width differs from the
|
||||
// target (probe again — the plan may be stale after a resume).
|
||||
let schemaTransitioned = false;
|
||||
const col = await readContentChunksEmbeddingDim(engine);
|
||||
if (col.dims !== plan.to_dims) {
|
||||
await runSchemaTransition(engine, plan.to_dims);
|
||||
schemaTransitioned = true;
|
||||
}
|
||||
|
||||
// 3. #3391: mark EVERYTHING not in the target space as stale, including
|
||||
// NULL-signature (pre-v108) pages. After a schema transition this is
|
||||
// a cheap no-op (the column rebuild already nulled every embedding).
|
||||
//
|
||||
// ORDERING (adversarial review): invalidation MUST precede the config
|
||||
// writes below. On a SAME-dim provider swap there is no schema
|
||||
// transition to null the vectors, so a crash between "config says new
|
||||
// provider" and "old vectors invalidated" would leave NEW-space query
|
||||
// embeddings scored against OLD-space document vectors — silently
|
||||
// WRONG results. Invalidating first makes the crash window safe:
|
||||
// config still says the old provider, and the rows are merely stale
|
||||
// (empty/degraded results, never wrong ones).
|
||||
const invalidated = await engine.invalidateStaleSignatureEmbeddings({
|
||||
signature: migrationSignature(plan.to_model, plan.to_dims),
|
||||
includeNullSignature: true,
|
||||
});
|
||||
|
||||
// 4. DB-plane config (doctor's embedding_width_consistency reads these).
|
||||
await engine.setConfig('embedding_model', plan.to_model);
|
||||
await engine.setConfig('embedding_dimensions', String(plan.to_dims));
|
||||
|
||||
// 5. File plane + gateway (the embed pipeline reads file/env, not DB).
|
||||
await opts.persistConfig?.(plan.to_model, plan.to_dims);
|
||||
|
||||
// 6. Purge the semantic query cache. The knobs hash folds provider:model
|
||||
// for callers that thread KnobsHashContext, but legacy callers fall
|
||||
// back to 'default' — a row they wrote pre-migration must not be
|
||||
// served post-migration. Best-effort (cache must never block).
|
||||
let cacheCleared = 0;
|
||||
try {
|
||||
const { SemanticQueryCache } = await import('./search/query-cache.ts');
|
||||
cacheCleared = await new SemanticQueryCache(engine).clear({});
|
||||
} catch {
|
||||
// Table may not exist on old brains; a miss here is harmless.
|
||||
}
|
||||
|
||||
return { status: 'applied', invalidated, cache_cleared: cacheCleared, schema_transitioned: schemaTransitioned };
|
||||
} catch (err) {
|
||||
return { status: 'failed', reason: err instanceof Error ? err.message : String(err) };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Stamp the target signature on every page that is fully embedded but not yet
|
||||
* stamped. Call after the re-embed drain, BEFORE the completion probe.
|
||||
*
|
||||
* Why this exists (adversarial review): the embed loop only stamps a page when
|
||||
* `stale.length === existing.length` — i.e. when every one of the page's
|
||||
* chunks was in the SAME batch. `listStaleChunks` is a plain keyset LIMIT with
|
||||
* no page alignment, so on any corpus larger than one batch (default 2000
|
||||
* chunks) the page straddling each boundary is embedded correctly but never
|
||||
* stamped. Without this reconcile the command reports "incomplete" + exit 1 on
|
||||
* a perfectly-migrated brain, and the re-run re-invalidates and PAYS AGAIN for
|
||||
* those pages — breaking the "already-migrated chunks are never re-embedded"
|
||||
* contract.
|
||||
*
|
||||
* Safety: this is only sound because `applyEmbeddingMigration` invalidated
|
||||
* (NULLed) every chunk that was NOT already in the target space. So "page has
|
||||
* zero NULL-embedding chunks" ⇒ "every chunk on this page was embedded in the
|
||||
* target space during this run". Pages with any remaining NULL chunk (a real
|
||||
* embed failure) are deliberately left unstamped so the completion probe still
|
||||
* reports them.
|
||||
*
|
||||
* Returns the number of pages stamped.
|
||||
*/
|
||||
export async function reconcilePageSignatures(
|
||||
engine: BrainEngine,
|
||||
plan: EmbeddingMigrationPlan,
|
||||
): Promise<number> {
|
||||
const sig = migrationSignature(plan.to_model, plan.to_dims);
|
||||
const rows = await engine.executeRaw<{ slug: string }>(
|
||||
`UPDATE pages p
|
||||
SET embedding_signature = $1
|
||||
WHERE p.deleted_at IS NULL
|
||||
AND (p.embedding_signature IS DISTINCT FROM $1)
|
||||
AND EXISTS (SELECT 1 FROM content_chunks c WHERE c.page_id = p.id)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM content_chunks c
|
||||
WHERE c.page_id = p.id AND c.embedding IS NULL
|
||||
)
|
||||
RETURNING p.slug`,
|
||||
[sig],
|
||||
);
|
||||
return (rows as unknown[]).length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Finish bookkeeping after the re-embed drains: clear the in-flight marker,
|
||||
* stamp the completion record. Call ONLY when countStaleChunks() === 0.
|
||||
*/
|
||||
export async function completeEmbeddingMigration(
|
||||
engine: BrainEngine,
|
||||
plan: EmbeddingMigrationPlan,
|
||||
): Promise<void> {
|
||||
await engine.unsetConfig(MIGRATION_STATE_KEY);
|
||||
await engine.setConfig(
|
||||
MIGRATION_COMPLETED_KEY,
|
||||
JSON.stringify({
|
||||
to_model: plan.to_model,
|
||||
to_dims: plan.to_dims,
|
||||
from_model: plan.from_model,
|
||||
completed_at: new Date().toISOString(),
|
||||
}),
|
||||
);
|
||||
}
|
||||
+17
-3
@@ -1005,8 +1005,13 @@ export interface BrainEngine {
|
||||
* counts across every source in the brain. Operators running
|
||||
* `gbrain embed --stale --source media-corpus` expect only that
|
||||
* source's NULLs touched; the caller threads `sourceId` here.
|
||||
*
|
||||
* `includeNullSignature` (only meaningful with `signature`, #3391): also
|
||||
* count embedded chunks whose page has NO recorded signature (v108
|
||||
* grandfathered). Provider-migration paths set this so pre-stamp pages
|
||||
* aren't silently left in the old embedding space.
|
||||
*/
|
||||
countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number>;
|
||||
countStaleChunks(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number>;
|
||||
/**
|
||||
* Sum of LENGTH(chunk_text) over stale chunks — the character-count
|
||||
* backlog the embed phase / embed-backfill will process. Sibling of
|
||||
@@ -1020,8 +1025,10 @@ export interface BrainEngine {
|
||||
* model signature (a model/dims swap). NULL signature is GRANDFATHERED
|
||||
* (never counted) so the post-migration corpus isn't flagged en masse.
|
||||
* Omit `signature` for the legacy `embedding IS NULL`-only count.
|
||||
* `includeNullSignature` lifts the grandfather clause (#3391) — see
|
||||
* countStaleChunks.
|
||||
*/
|
||||
sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number>;
|
||||
sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number>;
|
||||
/**
|
||||
* Stamp `pages.embedding_signature = signature` for one page. Called after
|
||||
* a page's chunks are (re)embedded so a later model swap can detect it as
|
||||
@@ -1036,8 +1043,15 @@ export interface BrainEngine {
|
||||
* drift pages flow through the existing NULL-embedding cursor (keeps
|
||||
* listStaleChunks's keyset pagination untouched). GRANDFATHER: NULL
|
||||
* signature is never invalidated. `sourceId` scopes the sweep.
|
||||
*
|
||||
* `includeNullSignature` (#3391): ALSO invalidate embedded chunks whose
|
||||
* page signature is NULL (pre-v108 pages that predate the stamp). After a
|
||||
* provider/model swap those vectors are in the old embedding space; the
|
||||
* default grandfather clause would silently keep them mixed into the new
|
||||
* index. `gbrain migrate embeddings` and `embed --stale
|
||||
* --include-null-signature` set this.
|
||||
*/
|
||||
invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number>;
|
||||
invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise<number>;
|
||||
/**
|
||||
* Return every chunk where embedding IS NULL, with the metadata needed
|
||||
* to call embedBatch + upsertChunks. The `embedding` column is omitted
|
||||
|
||||
@@ -4555,6 +4555,90 @@ const code_traversal_cache_clear: Operation = {
|
||||
cliHints: { name: 'code_traversal_cache_clear', hidden: true },
|
||||
};
|
||||
|
||||
// --- #3390: provider-agnostic embedding migration ---
|
||||
|
||||
const migrate_embeddings: Operation = {
|
||||
name: 'migrate_embeddings',
|
||||
description: 'Re-embed the brain onto a different embedding provider/model (#3390): schema dimension transition, NULL-signature (#3391) invalidation, query-cache purge, resumable re-embed. Without yes=true returns the plan + cost estimate only. Local-only admin op; the primary surface is `gbrain migrate embeddings`.',
|
||||
params: {
|
||||
to: { type: 'string', required: true, description: 'Target provider:model (e.g. openai:text-embedding-3-small).' },
|
||||
dim: { type: 'number', description: "Target dimensions. Defaults to the provider recipe's declared width; required when the recipe declares none." },
|
||||
dry_run: { type: 'boolean', description: 'Plan + cost estimate only; change nothing.' },
|
||||
yes: { type: 'boolean', description: 'Confirm the re-embed spend + destructive schema change. Required for a live run.' },
|
||||
},
|
||||
mutating: true,
|
||||
scope: 'admin',
|
||||
localOnly: true,
|
||||
handler: async (ctx, p) => {
|
||||
// Belt-and-braces on top of localOnly (the get_recent_transcripts
|
||||
// pattern): a schema-rebuilding, money-spending op must never be
|
||||
// reachable from a remote transport even if a future dispatch path
|
||||
// forgets the localOnly filter.
|
||||
if (ctx.remote !== false) {
|
||||
throw new Error('migrate_embeddings is local-only. Run `gbrain migrate embeddings` on the host.');
|
||||
}
|
||||
const {
|
||||
planEmbeddingMigration, applyEmbeddingMigration, completeEmbeddingMigration,
|
||||
reconcilePageSignatures, migrationSignature,
|
||||
} = await import('./embedding-migration.ts');
|
||||
const to = p.to as string;
|
||||
const dim = p.dim as number | undefined;
|
||||
let fromModel: string | undefined;
|
||||
let fromDims: number | undefined;
|
||||
try {
|
||||
const { getEmbeddingModel, getEmbeddingDimensions } = await import('./ai/gateway.ts');
|
||||
fromModel = getEmbeddingModel();
|
||||
fromDims = getEmbeddingDimensions();
|
||||
} catch { /* gateway unconfigured — plan falls back to defaults */ }
|
||||
const plan = await planEmbeddingMigration(ctx.engine, {
|
||||
to,
|
||||
...(dim !== undefined && { dim }),
|
||||
...(fromModel !== undefined && { fromModel }),
|
||||
...(fromDims !== undefined && { fromDims }),
|
||||
});
|
||||
if (ctx.dryRun || p.dry_run === true || p.yes !== true) {
|
||||
return { status: p.yes === true || p.dry_run === true ? 'planned' : 'needs_confirmation', plan };
|
||||
}
|
||||
const { persistEmbeddingFileConfig, probeTargetProvider } = await import('../commands/migrate-embeddings.ts');
|
||||
// Safety parity with the CLI path: probe the target provider BEFORE any
|
||||
// mutation. Without this, `yes:true` would drop the embedding column and
|
||||
// only then discover the key/model/dim is wrong.
|
||||
const probe = await probeTargetProvider(plan.to_model, plan.to_dims);
|
||||
if (!probe.ok) return { status: 'failed', reason: probe.message, plan };
|
||||
const applied = await applyEmbeddingMigration(ctx.engine, plan, {
|
||||
persistConfig: (m, d) => persistEmbeddingFileConfig(m, d),
|
||||
});
|
||||
if (applied.status !== 'applied') return { ...applied, plan };
|
||||
const { runEmbedCore } = await import('../commands/embed.ts');
|
||||
// singleFlight parity with the CLI path: takes the same per-source
|
||||
// embed-backfill lock so this can't race a queued embed-backfill job on
|
||||
// the NULL→non-NULL upsert (the TODOS:2299 class).
|
||||
const embedResult = await runEmbedCore(ctx.engine, {
|
||||
stale: true, catchUp: true, singleFlight: true, includeNullSignature: true, quiet: true,
|
||||
});
|
||||
// Stamp batch-boundary pages before probing for completion (see
|
||||
// reconcilePageSignatures — the embed loop's all-or-nothing stamp rule
|
||||
// skips any page split across two stale batches).
|
||||
const reconciled = await reconcilePageSignatures(ctx.engine, plan);
|
||||
const remaining = await ctx.engine.countStaleChunks({
|
||||
signature: migrationSignature(plan.to_model, plan.to_dims),
|
||||
includeNullSignature: true,
|
||||
});
|
||||
if (remaining === 0) await completeEmbeddingMigration(ctx.engine, plan);
|
||||
return {
|
||||
status: remaining === 0 ? 'completed' : 'incomplete',
|
||||
plan,
|
||||
embedded: embedResult.embedded,
|
||||
remaining,
|
||||
signatures_reconciled: reconciled,
|
||||
invalidated: applied.invalidated,
|
||||
schema_transitioned: applied.schema_transitioned,
|
||||
cache_cleared: applied.cache_cleared,
|
||||
};
|
||||
},
|
||||
cliHints: { name: 'migrate-embeddings', hidden: true },
|
||||
};
|
||||
|
||||
// --- v0.36 Phase 2: search_by_image (image-as-query) ---
|
||||
|
||||
const search_by_image: Operation = {
|
||||
@@ -5657,6 +5741,8 @@ export const operations: Operation[] = [
|
||||
code_blast, code_flow,
|
||||
// v0.34 W3b: code_traversal_cache admin clear op
|
||||
code_traversal_cache_clear,
|
||||
// #3390: provider-agnostic embedding migration (local-only admin)
|
||||
migrate_embeddings,
|
||||
// v0.40.6.0 Schema Cathedral v3: 9 new ops — 7 read + 2 admin (NOT
|
||||
// localOnly per D2 so remote agents (your OpenClaw, etc.) can author packs).
|
||||
// schema_apply_mutations is batched per D10 — one MCP tool, N
|
||||
|
||||
+21
-10
@@ -2428,15 +2428,21 @@ export class PGLiteEngine implements BrainEngine {
|
||||
/**
|
||||
* Build the stale-chunk WHERE clause + positional params. embed_skip is
|
||||
* always excluded. `signature` widens "stale" to include embedding_signature
|
||||
* drift (NULL grandfathered → never stale). Shared by countStaleChunks +
|
||||
* drift (NULL grandfathered → never stale). `includeNullSignature` (#3391)
|
||||
* lifts the grandfather clause so pre-stamp pages count as stale too
|
||||
* (provider-migration paths). Shared by countStaleChunks +
|
||||
* sumStaleChunkChars so they can't drift.
|
||||
*/
|
||||
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
|
||||
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): { where: string; params: unknown[] } {
|
||||
const params: unknown[] = [];
|
||||
const conds: string[] = [];
|
||||
if (opts?.signature !== undefined) {
|
||||
params.push(opts.signature);
|
||||
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
|
||||
conds.push(
|
||||
opts.includeNullSignature
|
||||
? `(cc.embedding IS NULL OR p.embedding_signature IS NULL OR p.embedding_signature <> $${params.length})`
|
||||
: `(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`,
|
||||
);
|
||||
} else {
|
||||
conds.push(`cc.embedding IS NULL`);
|
||||
}
|
||||
@@ -2448,7 +2454,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return { where: conds.join(' AND '), params };
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
async countStaleChunks(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number> {
|
||||
// D7: source-scoped count for `gbrain embed --stale --source X`. Always
|
||||
// JOIN pages so embed-skip + signature predicates apply. PGLite is
|
||||
// PostgreSQL 17.5 in WASM and supports the full JSONB operator set.
|
||||
@@ -2464,7 +2470,7 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return Number(count);
|
||||
}
|
||||
|
||||
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number> {
|
||||
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
|
||||
// length for the sync cost preview. ::bigint guards int4 overflow.
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
@@ -2486,24 +2492,29 @@ export class PGLiteEngine implements BrainEngine {
|
||||
);
|
||||
}
|
||||
|
||||
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number> {
|
||||
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise<number> {
|
||||
// NULL out embeddings whose page signature is set AND differs from the
|
||||
// current model signature. GRANDFATHER: NULL signature untouched. Feeds
|
||||
// the existing NULL-embedding cursor so listStaleChunks stays unchanged.
|
||||
// current model signature. GRANDFATHER: NULL signature untouched —
|
||||
// UNLESS includeNullSignature (#3391): provider migrations must not
|
||||
// leave pre-stamp pages in the old embedding space. Feeds the existing
|
||||
// NULL-embedding cursor so listStaleChunks stays unchanged.
|
||||
const params: unknown[] = [opts.signature];
|
||||
let srcClause = '';
|
||||
if (opts.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
srcClause = ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
const sigClause = opts.includeNullSignature
|
||||
? `(p.embedding_signature IS NULL OR p.embedding_signature <> $1)`
|
||||
: `p.embedding_signature IS NOT NULL
|
||||
AND p.embedding_signature <> $1`;
|
||||
const { rows } = await this.db.query(
|
||||
`UPDATE content_chunks cc
|
||||
SET embedding = NULL, embedded_at = NULL
|
||||
FROM pages p
|
||||
WHERE cc.page_id = p.id
|
||||
AND cc.embedding IS NOT NULL
|
||||
AND p.embedding_signature IS NOT NULL
|
||||
AND p.embedding_signature <> $1${srcClause}
|
||||
AND ${sigClause}${srcClause}
|
||||
RETURNING cc.page_id`,
|
||||
params,
|
||||
);
|
||||
|
||||
+24
-12
@@ -2557,15 +2557,21 @@ export class PostgresEngine implements BrainEngine {
|
||||
/**
|
||||
* Build the stale-chunk WHERE clause + positional params for sql.unsafe.
|
||||
* embed_skip always excluded. `signature` widens "stale" to include
|
||||
* embedding_signature drift (NULL grandfathered). Shared by
|
||||
* countStaleChunks + sumStaleChunkChars (parity with the PGLite sibling).
|
||||
* embedding_signature drift (NULL grandfathered). `includeNullSignature`
|
||||
* (#3391) lifts the grandfather clause so pre-stamp pages count as stale
|
||||
* too (provider-migration paths). Shared by countStaleChunks +
|
||||
* sumStaleChunkChars (parity with the PGLite sibling).
|
||||
*/
|
||||
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string }): { where: string; params: unknown[] } {
|
||||
private buildStaleChunkWhere(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): { where: string; params: unknown[] } {
|
||||
const params: unknown[] = [];
|
||||
const conds: string[] = [];
|
||||
if (opts?.signature !== undefined) {
|
||||
params.push(opts.signature);
|
||||
conds.push(`(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`);
|
||||
conds.push(
|
||||
opts.includeNullSignature
|
||||
? `(cc.embedding IS NULL OR p.embedding_signature IS NULL OR p.embedding_signature <> $${params.length})`
|
||||
: `(cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $${params.length}))`,
|
||||
);
|
||||
} else {
|
||||
conds.push(`cc.embedding IS NULL`);
|
||||
}
|
||||
@@ -2577,10 +2583,11 @@ export class PostgresEngine implements BrainEngine {
|
||||
return { where: conds.join(' AND '), params };
|
||||
}
|
||||
|
||||
async countStaleChunks(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
async countStaleChunks(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number> {
|
||||
// Always JOIN pages so the embed_skip + signature predicates apply.
|
||||
// D7: source_id scoping. v0.41.31: optional signature widens staleness
|
||||
// to embedding_signature drift (NULL grandfathered).
|
||||
// to embedding_signature drift (NULL grandfathered unless
|
||||
// includeNullSignature, #3391).
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
// RLS scope binding (opt-in via GBRAIN_RLS_SCOPE_BINDING).
|
||||
return await this.withScopedReadTransaction(undefined, opts?.sourceId, async (tx) => {
|
||||
@@ -2595,7 +2602,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
});
|
||||
}
|
||||
|
||||
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string }): Promise<number> {
|
||||
async sumStaleChunkChars(opts?: { sourceId?: string; signature?: string; includeNullSignature?: boolean }): Promise<number> {
|
||||
// Sibling of countStaleChunks: same stale predicate, summing chunk_text
|
||||
// length for the sync cost preview. ::bigint guards int4 overflow.
|
||||
const { where, params } = this.buildStaleChunkWhere(opts);
|
||||
@@ -2617,24 +2624,29 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
}
|
||||
|
||||
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string }): Promise<number> {
|
||||
async invalidateStaleSignatureEmbeddings(opts: { signature: string; sourceId?: string; includeNullSignature?: boolean }): Promise<number> {
|
||||
// NULL embeddings whose page signature is set AND differs from current.
|
||||
// GRANDFATHER: NULL signature untouched. Feeds the NULL-embedding cursor
|
||||
// so listStaleChunks stays unchanged. RETURNING → row count.
|
||||
// GRANDFATHER: NULL signature untouched — UNLESS includeNullSignature
|
||||
// (#3391): provider migrations must not leave pre-stamp pages in the old
|
||||
// embedding space. Feeds the NULL-embedding cursor so listStaleChunks
|
||||
// stays unchanged. RETURNING → row count.
|
||||
const params: unknown[] = [opts.signature];
|
||||
let srcClause = '';
|
||||
if (opts.sourceId !== undefined) {
|
||||
params.push(opts.sourceId);
|
||||
srcClause = ` AND p.source_id = $${params.length}`;
|
||||
}
|
||||
const sigClause = opts.includeNullSignature
|
||||
? `(p.embedding_signature IS NULL OR p.embedding_signature <> $1)`
|
||||
: `p.embedding_signature IS NOT NULL
|
||||
AND p.embedding_signature <> $1`;
|
||||
const rows = await this.sql.unsafe(
|
||||
`UPDATE content_chunks cc
|
||||
SET embedding = NULL, embedded_at = NULL
|
||||
FROM pages p
|
||||
WHERE cc.page_id = p.id
|
||||
AND cc.embedding IS NOT NULL
|
||||
AND p.embedding_signature IS NOT NULL
|
||||
AND p.embedding_signature <> $1${srcClause}
|
||||
AND ${sigClause}${srcClause}
|
||||
RETURNING cc.page_id`,
|
||||
params as Parameters<typeof this.sql.unsafe>[1],
|
||||
);
|
||||
|
||||
@@ -63,6 +63,7 @@ import type { BrainEngine } from './engine.ts';
|
||||
import { MARKDOWN_CHUNKER_VERSION } from './chunkers/recursive.ts';
|
||||
import { lookupEmbeddingPrice, estimateCostFromChars } from './embedding-pricing.ts';
|
||||
import { computeReembedEstimate } from './post-upgrade-reembed.ts';
|
||||
import { hnswIndexExpected } from './vector-index.ts';
|
||||
|
||||
// ============================================================================
|
||||
// Constants
|
||||
@@ -551,8 +552,12 @@ export async function undoRetrievalUpgrade(engine: BrainEngine): Promise<
|
||||
*
|
||||
* IF NOT EXISTS on CREATE INDEX makes the operation safe to re-run during
|
||||
* `--resume`.
|
||||
*
|
||||
* Exported (#3390) so the provider-agnostic embedding migration
|
||||
* (src/core/embedding-migration.ts) reuses the SAME dimension-transition
|
||||
* path instead of duplicating the DDL sequence.
|
||||
*/
|
||||
async function runSchemaTransition(engine: BrainEngine, targetDim: number): Promise<void> {
|
||||
export async function runSchemaTransition(engine: BrainEngine, targetDim: number): Promise<void> {
|
||||
// v0.41 fix: only transition the primary text embedding column.
|
||||
// The embedding_image (v0.27.1) and embedding_multimodal (v0.36 / migration
|
||||
// v78) columns use SEPARATE multimodal models (e.g. voyage-multimodal-3 at
|
||||
@@ -595,9 +600,95 @@ async function runSchemaTransition(engine: BrainEngine, targetDim: number): Prom
|
||||
WHERE embedding_image IS NOT NULL`,
|
||||
);
|
||||
}
|
||||
|
||||
// #3390: the OTHER two dim-pinned columns that carry TEXT-embedding-space
|
||||
// vectors. Both are created at brain-birth width (migrate.ts v55 for
|
||||
// query_cache, v42 for facts) and NO migration ever ALTERs them, so before
|
||||
// this fix a dimension change left them at the old width:
|
||||
// - query_cache.embedding stayed narrow → every store() AND lookup()
|
||||
// silently swallowed the width error (by design, so the cache can
|
||||
// never break search), i.e. a PERMANENT 0% hit rate.
|
||||
// - facts.embedding stayed narrow → every per-fact embed write failed
|
||||
// ($N::vector into the old width), and the doctor check that would
|
||||
// warn is skipped on PGLite (the DEFAULT engine).
|
||||
// Both are text-embedding-space columns, so they MUST move with
|
||||
// content_chunks.embedding. The image/multimodal columns above are the
|
||||
// deliberate exception (separate models, independent dims).
|
||||
for (const t of TEXT_EMBEDDING_DIM_PINNED_TABLES) {
|
||||
await transitionDimPinnedColumn(tx, t.table, t.index, t.indexSql, targetDim);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The dim-pinned TEXT-embedding-space columns outside content_chunks.
|
||||
* `indexSql` is a factory because each table's index carries its own partial
|
||||
* WHERE clause + opclass, and the opclass must match the column TYPE
|
||||
* (vector_cosine_ops vs halfvec_cosine_ops).
|
||||
*/
|
||||
const TEXT_EMBEDDING_DIM_PINNED_TABLES: ReadonlyArray<{
|
||||
table: string;
|
||||
index: string;
|
||||
indexSql: (opclass: string) => string;
|
||||
}> = [
|
||||
{
|
||||
table: 'query_cache',
|
||||
index: 'idx_query_cache_embedding_hnsw',
|
||||
indexSql: (opclass) =>
|
||||
`CREATE INDEX IF NOT EXISTS idx_query_cache_embedding_hnsw
|
||||
ON query_cache USING hnsw (embedding ${opclass})
|
||||
WHERE embedding IS NOT NULL`,
|
||||
},
|
||||
{
|
||||
table: 'facts',
|
||||
index: 'idx_facts_embedding_hnsw',
|
||||
indexSql: (opclass) =>
|
||||
`CREATE INDEX IF NOT EXISTS idx_facts_embedding_hnsw
|
||||
ON facts USING hnsw (embedding ${opclass})
|
||||
WHERE embedding IS NOT NULL AND expired_at IS NULL`,
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* Rebuild one dim-pinned embedding column at `targetDim`, PRESERVING its
|
||||
* existing column type (`vector` vs `halfvec` — migrate.ts picks halfvec when
|
||||
* the server supports it, and the HNSW opclass must match). No-op when the
|
||||
* table or column doesn't exist (fresh/older brains).
|
||||
*
|
||||
* Dropping the column discards the stored vectors, which is correct: they are
|
||||
* in the OLD embedding space and unusable after the swap. query_cache is a
|
||||
* cache (refills on the next query); facts re-embed on their next write /
|
||||
* `gbrain extract` pass.
|
||||
*/
|
||||
async function transitionDimPinnedColumn(
|
||||
tx: { executeRaw: <T = unknown>(sql: string, params?: unknown[]) => Promise<T[]> },
|
||||
table: string,
|
||||
indexName: string,
|
||||
indexSql: (opclass: string) => string,
|
||||
targetDim: number,
|
||||
): Promise<void> {
|
||||
const probe = await tx.executeRaw<{ udt_name: string | null }>(
|
||||
`SELECT udt_name FROM information_schema.columns
|
||||
WHERE table_schema = 'public' AND table_name = $1 AND column_name = 'embedding'`,
|
||||
[table],
|
||||
);
|
||||
const udt = probe[0]?.udt_name;
|
||||
if (!udt) return; // table or column absent — nothing to transition
|
||||
// Preserve the column type; anything unexpected falls back to `vector`.
|
||||
const columnType: 'vector' | 'halfvec' = udt.toLowerCase() === 'halfvec' ? 'halfvec' : 'vector';
|
||||
const opclass = columnType === 'halfvec' ? 'halfvec_cosine_ops' : 'vector_cosine_ops';
|
||||
|
||||
await tx.executeRaw(`DROP INDEX IF EXISTS ${indexName}`);
|
||||
await tx.executeRaw(`ALTER TABLE ${table} DROP COLUMN IF EXISTS embedding`);
|
||||
await tx.executeRaw(`ALTER TABLE ${table} ADD COLUMN embedding ${columnType}(${targetDim})`);
|
||||
// HNSW has a per-type dimension ceiling; above it pgvector refuses the
|
||||
// index and exact scans remain the (correct, slower) path. Mirrors the
|
||||
// same guard in migrate.ts's original DDL.
|
||||
if (hnswIndexExpected(columnType, targetDim)) {
|
||||
await tx.executeRaw(indexSql(opclass));
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Helpers
|
||||
// ============================================================================
|
||||
|
||||
+11
-1
@@ -756,7 +756,17 @@ export function attributeKnob<K extends keyof ModeBundle>(
|
||||
// slugs written by a process without it, and vice versa. Same one-time
|
||||
// global cold-miss pattern as the bumps above; refills within
|
||||
// cache.ttl_seconds (3600s default).
|
||||
export const KNOBS_HASH_VERSION = 12;
|
||||
//
|
||||
// bump 12→13 (#3390/#3391): embedding-provider migration wave. The `prov=`
|
||||
// component only isolates callers that thread KnobsHashContext.embeddingModel;
|
||||
// legacy callers hash `prov=default` before AND after a provider swap, so a
|
||||
// cache row computed against the pre-migration embedding space could be
|
||||
// served post-migration. `gbrain migrate embeddings` purges query_cache
|
||||
// directly at swap time; this version bump is the belt-and-braces for rows
|
||||
// written between the #3391 stale-fix (which changes which chunks count as
|
||||
// current) and the operator's migration run. Same one-time global cold-miss
|
||||
// pattern as the bumps above.
|
||||
export const KNOBS_HASH_VERSION = 13;
|
||||
|
||||
/**
|
||||
* v0.36 (D8 / CDX-2) — second-arg context for the cache key. The
|
||||
|
||||
@@ -136,7 +136,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
return resolveSearchMode({ mode: 'balanced' });
|
||||
}
|
||||
|
||||
test('KNOBS_HASH_VERSION is 12 (cross-modal still appended; 11→12 hard-exclude fold #2825)', () => {
|
||||
test('KNOBS_HASH_VERSION is 13 (cross-modal still appended; 12→13 embedding-provider migration #3390)', () => {
|
||||
// v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3
|
||||
// with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) +
|
||||
// v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields.
|
||||
@@ -146,7 +146,7 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => {
|
||||
// v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type
|
||||
// finally reaches asymmetric providers — pre-fix rows were keyed on
|
||||
// document-side query vectors. #2825: 11→12 hard-exclude fold (hx=).
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
|
||||
test('flipping unified_multimodal changes the hash', () => {
|
||||
|
||||
@@ -0,0 +1,247 @@
|
||||
/**
|
||||
* #3390/#3391 — embedding migration on REAL Postgres + pgvector.
|
||||
*
|
||||
* Engine-parity pin for the #3391 includeNullSignature predicates
|
||||
* (identical semantics to the PGLite coverage in
|
||||
* test/embedding-migration.test.ts) PLUS the migration path that genuinely
|
||||
* differs on real Postgres: runSchemaTransition's DROP INDEX / DROP COLUMN /
|
||||
* ADD vector(N) / CREATE hnsw sequence against a native pgvector, followed
|
||||
* by a full re-embed through the real pipeline with a fake transport.
|
||||
*
|
||||
* Gated by DATABASE_URL (docs/TESTING.md: docker pgvector/pgvector:pg16,
|
||||
* e.g. on :5435). Restores the original column width in afterAll so later
|
||||
* e2e files see the schema they expect.
|
||||
*
|
||||
* Run: DATABASE_URL=postgres://...gbrain_test bun test test/e2e/migrate-embeddings-postgres.test.ts
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import type { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
||||
import { hasDatabase, setupDB, teardownDB } from './helpers.ts';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__setEmbedTransportForTests,
|
||||
} from '../../src/core/ai/gateway.ts';
|
||||
import { runEmbedCore } from '../../src/commands/embed.ts';
|
||||
import {
|
||||
planEmbeddingMigration,
|
||||
applyEmbeddingMigration,
|
||||
completeEmbeddingMigration,
|
||||
migrationSignature,
|
||||
MIGRATION_STATE_KEY,
|
||||
} from '../../src/core/embedding-migration.ts';
|
||||
import { runSchemaTransition } from '../../src/core/retrieval-upgrade-planner.ts';
|
||||
import type { ChunkInput } from '../../src/core/types.ts';
|
||||
|
||||
const RUN = hasDatabase();
|
||||
const d = RUN ? describe : describe.skip;
|
||||
|
||||
let engine: PostgresEngine;
|
||||
let originalDims: number;
|
||||
let currentDims = 0; // fake-transport vector width, set per phase
|
||||
const savedEnv: Record<string, string | undefined> = {};
|
||||
|
||||
async function columnDims(): Promise<number> {
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding'
|
||||
AND attnum > 0 AND NOT attisdropped`,
|
||||
);
|
||||
return Number(rows[0]?.dim);
|
||||
}
|
||||
|
||||
async function embeddingColWidth(table: string): Promise<number> {
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = $1::regclass AND attname = 'embedding'
|
||||
AND attnum > 0 AND NOT attisdropped`,
|
||||
[table],
|
||||
);
|
||||
return Number(rows[0]?.dim);
|
||||
}
|
||||
|
||||
async function seedEmbedded(slug: string, text: string, signature: string | null): Promise<void> {
|
||||
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` });
|
||||
const chunks: ChunkInput[] = [
|
||||
{ chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4 },
|
||||
];
|
||||
await engine.upsertChunks(slug, chunks);
|
||||
await engine.executeRaw(
|
||||
`UPDATE content_chunks
|
||||
SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector
|
||||
WHERE page_id = (SELECT id FROM pages WHERE slug = $2 AND source_id = 'default')`,
|
||||
[originalDims, slug],
|
||||
);
|
||||
if (signature !== null) {
|
||||
await engine.setPageEmbeddingSignature(slug, { signature });
|
||||
}
|
||||
}
|
||||
|
||||
d('embedding migration (live Postgres + pgvector)', () => {
|
||||
beforeAll(async () => {
|
||||
for (const k of ['GBRAIN_EMBEDDING_MODEL', 'GBRAIN_EMBEDDING_DIMENSIONS']) {
|
||||
savedEnv[k] = process.env[k];
|
||||
delete process.env[k];
|
||||
}
|
||||
engine = await setupDB();
|
||||
originalDims = await columnDims();
|
||||
|
||||
resetGateway();
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-small',
|
||||
embedding_dimensions: 1536,
|
||||
env: { OPENAI_API_KEY: 'sk-test-fake' },
|
||||
});
|
||||
__setEmbedTransportForTests(async ({ values }: { values: string[] }) => ({
|
||||
embeddings: values.map(() => new Array(currentDims).fill(0).map((_, i) => Math.cos(i) * 0.01 + 0.002)),
|
||||
usage: { tokens: values.length * 4 },
|
||||
}) as never);
|
||||
}, 60000);
|
||||
|
||||
afterAll(async () => {
|
||||
__setEmbedTransportForTests(null);
|
||||
resetGateway();
|
||||
// Restore the shared test DB's column width for subsequent e2e files.
|
||||
if (engine && originalDims && (await columnDims()) !== originalDims) {
|
||||
await runSchemaTransition(engine, originalDims);
|
||||
}
|
||||
await teardownDB();
|
||||
for (const [k, v] of Object.entries(savedEnv)) {
|
||||
if (v === undefined) delete process.env[k];
|
||||
else process.env[k] = v;
|
||||
}
|
||||
}, 60000);
|
||||
|
||||
test('#3391 predicates behave identically to PGLite (engine parity)', async () => {
|
||||
await seedEmbedded('parity/legacy', 'abcde', null); // NULL sig
|
||||
await seedEmbedded('parity/drifted', 'fghij', 'old:model:1'); // mismatched
|
||||
await seedEmbedded('parity/fresh', 'klmno', 'new:model:1'); // matching
|
||||
const sig = 'new:model:1';
|
||||
|
||||
// Grandfathered (no flag): only the drifted page counts.
|
||||
expect(await engine.countStaleChunks({ signature: sig })).toBe(1);
|
||||
expect(await engine.sumStaleChunkChars({ signature: sig })).toBe(5);
|
||||
// Widened (#3391): legacy counts too; matching still excluded.
|
||||
expect(await engine.countStaleChunks({ signature: sig, includeNullSignature: true })).toBe(2);
|
||||
expect(await engine.sumStaleChunkChars({ signature: sig, includeNullSignature: true })).toBe(10);
|
||||
|
||||
// Invalidation parity: default grandfathers, flag lifts it, idempotent.
|
||||
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: sig })).toBe(1);
|
||||
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: sig, includeNullSignature: true })).toBe(1); // legacy
|
||||
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: sig, includeNullSignature: true })).toBe(0);
|
||||
const kept = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.slug = 'parity/fresh' AND cc.embedding IS NOT NULL`,
|
||||
);
|
||||
expect(Number(kept[0]?.n)).toBe(1);
|
||||
|
||||
// Clean up parity fixtures so the migration test below starts exact.
|
||||
await engine.executeRaw(`DELETE FROM pages WHERE slug LIKE 'parity/%'`);
|
||||
}, 30000);
|
||||
|
||||
test('full migration with a dimension change on real pgvector', async () => {
|
||||
const targetDims = originalDims === 1536 ? 1024 : 1536;
|
||||
const toModel = targetDims === 1536 ? 'openai:text-embedding-3-small' : 'voyage:voyage-3-large';
|
||||
|
||||
// Brain state: one current-signature page, one pre-v108 NULL-signature
|
||||
// page, one never-embedded page.
|
||||
await seedEmbedded('mig/current', 'aaaaa', migrationSignature('zeroentropyai:zembed-1', originalDims));
|
||||
await seedEmbedded('mig/legacy', 'bbbbb', null);
|
||||
await engine.putPage('mig/pending', { type: 'note', title: 'pending', compiled_truth: '# pending' });
|
||||
await engine.upsertChunks('mig/pending', [
|
||||
{ chunk_index: 0, chunk_text: 'ccccc', chunk_source: 'compiled_truth', token_count: 2 },
|
||||
]);
|
||||
|
||||
const plan = await planEmbeddingMigration(engine, {
|
||||
to: toModel,
|
||||
dim: targetDims,
|
||||
fromModel: 'zeroentropyai:zembed-1',
|
||||
fromDims: originalDims,
|
||||
});
|
||||
expect(plan.dim_change).toBe(true);
|
||||
expect(plan.chunks_to_embed).toBe(3);
|
||||
expect(plan.null_signature_chunks).toBe(1);
|
||||
|
||||
// Apply: runSchemaTransition on REAL Postgres (native pgvector DDL).
|
||||
const persisted: Array<[string, number]> = [];
|
||||
const applied = await applyEmbeddingMigration(engine, plan, {
|
||||
persistConfig: (m, dd) => { persisted.push([m, dd]); },
|
||||
});
|
||||
expect(applied.status).toBe('applied');
|
||||
if (applied.status !== 'applied') throw new Error('unreachable');
|
||||
expect(applied.schema_transitioned).toBe(true);
|
||||
expect(persisted).toEqual([[toModel, targetDims]]);
|
||||
expect(await columnDims()).toBe(targetDims);
|
||||
|
||||
// All THREE dim-pinned text-embedding-space columns move together on real
|
||||
// pgvector (query_cache + facts are created at brain-birth width and no
|
||||
// migration ever ALTERs them — before the fix they stayed narrow, which
|
||||
// silently killed the query cache and every per-fact embed write).
|
||||
expect(await embeddingColWidth('query_cache')).toBe(targetDims);
|
||||
expect(await embeddingColWidth('facts')).toBe(targetDims);
|
||||
// And the rebuilt columns actually ACCEPT a vector at the new width.
|
||||
const nv = `[${new Array(targetDims).fill(0.01).join(',')}]`;
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO query_cache (id, query_text, source_id, embedding)
|
||||
VALUES ('pg-post-migrate', 'q', 'default', $1::vector)`,
|
||||
[nv],
|
||||
);
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability, source, confidence, embedding)
|
||||
VALUES ('default', 'pg-e', 'pg-f', 'fact', 'private', 'medium', 'test', 1.0, $1::vector)`,
|
||||
[nv],
|
||||
);
|
||||
const accepted = await engine.executeRaw<{ qc: number; f: number }>(
|
||||
`SELECT (SELECT count(*)::int FROM query_cache WHERE id = 'pg-post-migrate') AS qc,
|
||||
(SELECT count(*)::int FROM facts WHERE entity_slug = 'pg-e') AS f`,
|
||||
);
|
||||
expect(Number(accepted[0]?.qc)).toBe(1);
|
||||
expect(Number(accepted[0]?.f)).toBe(1);
|
||||
expect(await engine.getConfig('embedding_model')).toBe(toModel);
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeTruthy();
|
||||
|
||||
// HNSW index rebuilt inside the same transaction.
|
||||
const idx = await engine.executeRaw<{ indexname: string }>(
|
||||
`SELECT indexname FROM pg_indexes WHERE tablename = 'content_chunks' AND indexname = 'idx_chunks_embedding'`,
|
||||
);
|
||||
expect(idx.length).toBe(1);
|
||||
|
||||
// Re-embed through the real pipeline at the new width. NOTE: no
|
||||
// resetGateway() here — it would clear the installed fake transport.
|
||||
currentDims = targetDims;
|
||||
configureGateway({
|
||||
embedding_model: toModel,
|
||||
embedding_dimensions: targetDims,
|
||||
env: { OPENAI_API_KEY: 'sk-test-fake', VOYAGE_API_KEY: 'va-test-fake' },
|
||||
});
|
||||
const res = await runEmbedCore(engine, {
|
||||
stale: true, catchUp: true, includeNullSignature: true, quiet: true,
|
||||
});
|
||||
expect(res.embedded).toBe(3);
|
||||
|
||||
// Everything is in the target space, including the NULL-signature page.
|
||||
const newSig = migrationSignature(toModel, targetDims);
|
||||
expect(await engine.countStaleChunks({ signature: newSig, includeNullSignature: true })).toBe(0);
|
||||
const sigs = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM pages WHERE slug LIKE 'mig/%' AND embedding_signature = $1`,
|
||||
[newSig],
|
||||
);
|
||||
expect(Number(sigs[0]?.n)).toBe(3);
|
||||
|
||||
// Vector search works against the new column at the new width.
|
||||
const qvec = `[${new Array(targetDims).fill(0).map((_, i) => (Math.cos(i) * 0.01 + 0.002).toFixed(6)).join(',')}]`;
|
||||
const rows = await engine.executeRaw<{ slug: string }>(
|
||||
`SELECT p.slug FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT 3`,
|
||||
[qvec],
|
||||
);
|
||||
expect(rows.length).toBe(3);
|
||||
|
||||
await completeEmbeddingMigration(engine, plan);
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy();
|
||||
}, 120000);
|
||||
});
|
||||
@@ -0,0 +1,424 @@
|
||||
/**
|
||||
* #3390 — provider-agnostic embedding migration (planner/applier, PGLite) +
|
||||
* #3391 — NULL-signature rows must be treatable as stale.
|
||||
*
|
||||
* Covers:
|
||||
* - resolveMigrationTarget validation (provider:model shape, unknown
|
||||
* provider, recipe-default dims, --dim override, dims-required recipes)
|
||||
* - includeNullSignature widening on countStaleChunks /
|
||||
* sumStaleChunkChars / invalidateStaleSignatureEmbeddings (#3391)
|
||||
* - planEmbeddingMigration counts, cost math, price_known, resuming
|
||||
* - applyEmbeddingMigration: env-override refusal fires BEFORE any
|
||||
* mutation, schema transition on dim change, DB-plane config writes,
|
||||
* state marker, NULL-signature-inclusive invalidation, query-cache
|
||||
* purge, idempotent re-apply (resume)
|
||||
* - completeEmbeddingMigration bookkeeping
|
||||
* - migrate_embeddings op contract (admin, localOnly, remote guard,
|
||||
* needs_confirmation without yes)
|
||||
*
|
||||
* Canonical PGLite block (CLAUDE.md R3+R4). Engine parity for the #3391
|
||||
* predicates is pinned on real Postgres in
|
||||
* test/e2e/migrate-embeddings-postgres.test.ts.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
import type { ChunkInput } from '../src/core/types.ts';
|
||||
import {
|
||||
resolveMigrationTarget,
|
||||
planEmbeddingMigration,
|
||||
applyEmbeddingMigration,
|
||||
completeEmbeddingMigration,
|
||||
reconcilePageSignatures,
|
||||
migrationSignature,
|
||||
MIGRATION_STATE_KEY,
|
||||
MIGRATION_COMPLETED_KEY,
|
||||
} from '../src/core/embedding-migration.ts';
|
||||
import { estimateCostFromChars } from '../src/core/embedding-pricing.ts';
|
||||
import { operations } from '../src/core/operations.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let colDim: number;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 30000);
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0`,
|
||||
);
|
||||
colDim = Number(rows[0]?.dim);
|
||||
});
|
||||
|
||||
/** Seed a page with one embedded chunk and a given signature (null = pre-v108). */
|
||||
async function seedEmbedded(slug: string, text: string, signature: string | null): Promise<void> {
|
||||
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` });
|
||||
const chunks: ChunkInput[] = [
|
||||
{ chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4, embedding: undefined },
|
||||
];
|
||||
await engine.upsertChunks(slug, chunks);
|
||||
await engine.executeRaw(
|
||||
`UPDATE content_chunks
|
||||
SET embedding = ('[' || array_to_string(array_fill(0.0::real, ARRAY[$1::int]), ',') || ']')::vector
|
||||
WHERE page_id = (SELECT id FROM pages WHERE slug = $2 AND source_id = 'default')`,
|
||||
[colDim, slug],
|
||||
);
|
||||
if (signature !== null) {
|
||||
await engine.setPageEmbeddingSignature(slug, { signature });
|
||||
}
|
||||
}
|
||||
|
||||
/** Actual vector(N)/halfvec(N) width of a table's `embedding` column. */
|
||||
async function embeddingColWidth(table: string): Promise<number> {
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = $1::regclass AND attname = 'embedding'
|
||||
AND attnum > 0 AND NOT attisdropped`,
|
||||
[table],
|
||||
);
|
||||
return Number(rows[0]?.dim);
|
||||
}
|
||||
|
||||
/** Seed a page with one UNembedded chunk (classic NULL-embedding stale). */
|
||||
async function seedUnembedded(slug: string, text: string): Promise<void> {
|
||||
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` });
|
||||
await engine.upsertChunks(slug, [
|
||||
{ chunk_index: 0, chunk_text: text, chunk_source: 'compiled_truth', token_count: 4, embedding: undefined },
|
||||
]);
|
||||
}
|
||||
|
||||
describe('resolveMigrationTarget', () => {
|
||||
test('requires provider:model shape', () => {
|
||||
expect(() => resolveMigrationTarget('text-embedding-3-small')).toThrow(/provider:model/);
|
||||
});
|
||||
test('unknown provider throws', () => {
|
||||
expect(() => resolveMigrationTarget('nosuchprovider:some-model')).toThrow();
|
||||
});
|
||||
test('recipe default dims resolve (openai → 1536)', () => {
|
||||
expect(resolveMigrationTarget('openai:text-embedding-3-small')).toEqual({
|
||||
toModel: 'openai:text-embedding-3-small',
|
||||
toDims: 1536,
|
||||
});
|
||||
});
|
||||
test('explicit --dim wins over recipe default', () => {
|
||||
expect(resolveMigrationTarget('openai:text-embedding-3-small', 512).toDims).toBe(512);
|
||||
});
|
||||
test('recipe with default_dims=0 requires --dim', () => {
|
||||
expect(() => resolveMigrationTarget('litellm:my-custom-model')).toThrow(/--dim/);
|
||||
expect(resolveMigrationTarget('litellm:my-custom-model', 1024).toDims).toBe(1024);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3391 includeNullSignature widening', () => {
|
||||
test('countStaleChunks / sumStaleChunkChars include NULL-signature rows only with the flag', async () => {
|
||||
await seedEmbedded('legacy', 'abcde', null); // NULL sig, embedded
|
||||
await seedEmbedded('drifted', 'fghij', 'old:model:1'); // mismatched sig
|
||||
await seedEmbedded('fresh', 'klmno', 'new:model:1'); // matching sig
|
||||
const sig = 'new:model:1';
|
||||
|
||||
// Default (grandfathered): legacy is invisible.
|
||||
expect(await engine.countStaleChunks({ signature: sig })).toBe(1);
|
||||
expect(await engine.sumStaleChunkChars({ signature: sig })).toBe(5);
|
||||
|
||||
// Widened: legacy counts too; matching still excluded.
|
||||
expect(await engine.countStaleChunks({ signature: sig, includeNullSignature: true })).toBe(2);
|
||||
expect(await engine.sumStaleChunkChars({ signature: sig, includeNullSignature: true })).toBe(10);
|
||||
});
|
||||
|
||||
test('invalidateStaleSignatureEmbeddings with the flag NULLs legacy + drifted, keeps matching', async () => {
|
||||
await seedEmbedded('legacy', 'abcde', null);
|
||||
await seedEmbedded('drifted', 'fghij', 'old:model:1');
|
||||
await seedEmbedded('fresh', 'klmno', 'new:model:1');
|
||||
|
||||
const n = await engine.invalidateStaleSignatureEmbeddings({
|
||||
signature: 'new:model:1',
|
||||
includeNullSignature: true,
|
||||
});
|
||||
expect(n).toBe(2); // legacy + drifted; NOT fresh
|
||||
|
||||
expect(await engine.countStaleChunks()).toBe(2); // both now NULL-embedding
|
||||
const kept = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE p.slug = 'fresh' AND cc.embedding IS NOT NULL`,
|
||||
);
|
||||
expect(Number(kept[0]?.n)).toBe(1);
|
||||
|
||||
// Idempotent.
|
||||
expect(await engine.invalidateStaleSignatureEmbeddings({
|
||||
signature: 'new:model:1', includeNullSignature: true,
|
||||
})).toBe(0);
|
||||
});
|
||||
|
||||
test('default behavior unchanged: NULL signature stays grandfathered without the flag', async () => {
|
||||
await seedEmbedded('legacy', 'abcde', null);
|
||||
expect(await engine.invalidateStaleSignatureEmbeddings({ signature: 'new:model:1' })).toBe(0);
|
||||
expect(await engine.countStaleChunks({ signature: 'new:model:1' })).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('planEmbeddingMigration', () => {
|
||||
test('counts everything not in the target space, splits out NULL-signature chunks, prices it', async () => {
|
||||
await seedEmbedded('legacy', 'abcde', null); // 5 chars
|
||||
await seedEmbedded('current', 'fghij', `zeroentropyai:zembed-1:${colDim}`); // 5 chars
|
||||
await seedUnembedded('pending', 'klmnop'); // 6 chars
|
||||
|
||||
const plan = await planEmbeddingMigration(engine, {
|
||||
to: 'openai:text-embedding-3-small',
|
||||
fromModel: 'zeroentropyai:zembed-1',
|
||||
fromDims: colDim,
|
||||
});
|
||||
|
||||
expect(plan.to_model).toBe('openai:text-embedding-3-small');
|
||||
expect(plan.to_dims).toBe(1536);
|
||||
expect(plan.column_dims).toBe(colDim);
|
||||
expect(plan.dim_change).toBe(colDim !== 1536);
|
||||
expect(plan.chunks_to_embed).toBe(3); // all three
|
||||
expect(plan.null_signature_chunks).toBe(1); // 'legacy' only
|
||||
expect(plan.total_chars).toBe(16);
|
||||
expect(plan.price_known).toBe(true);
|
||||
expect(plan.est_cost_usd).toBeCloseTo(estimateCostFromChars(16, 0.02), 10);
|
||||
expect(plan.resuming).toBe(false);
|
||||
});
|
||||
|
||||
test('unknown pricing → price_known false, cost 0', async () => {
|
||||
await seedUnembedded('p1', 'abc');
|
||||
const plan = await planEmbeddingMigration(engine, { to: 'litellm:custom', dim: 1024 });
|
||||
expect(plan.price_known).toBe(false);
|
||||
expect(plan.est_cost_usd).toBe(0);
|
||||
});
|
||||
|
||||
test('resuming=true when the state marker matches the target', async () => {
|
||||
await engine.setConfig(MIGRATION_STATE_KEY, JSON.stringify({
|
||||
to_model: 'openai:text-embedding-3-small', to_dims: 1536,
|
||||
from_model: 'x', from_dims: 1, started_at: 'now',
|
||||
}));
|
||||
const plan = await planEmbeddingMigration(engine, { to: 'openai:text-embedding-3-small' });
|
||||
expect(plan.resuming).toBe(true);
|
||||
const other = await planEmbeddingMigration(engine, { to: 'openai:text-embedding-3-large' });
|
||||
expect(other.resuming).toBe(false);
|
||||
});
|
||||
|
||||
test('reranker on the outgoing provider triggers the warning', async () => {
|
||||
await engine.setConfig('search.reranker.model', 'zeroentropyai:zerank-2');
|
||||
const plan = await planEmbeddingMigration(engine, {
|
||||
to: 'openai:text-embedding-3-small',
|
||||
fromModel: 'zeroentropyai:zembed-1',
|
||||
fromDims: 1280,
|
||||
});
|
||||
expect(plan.reranker_warning).toContain('zeroentropyai:zerank-2');
|
||||
});
|
||||
});
|
||||
|
||||
describe('applyEmbeddingMigration', () => {
|
||||
test('env override refuses BEFORE any mutation', async () => {
|
||||
await withEnv({ GBRAIN_EMBEDDING_MODEL: 'voyage:voyage-3-large' }, async () => {
|
||||
const plan = await planEmbeddingMigration(engine, { to: 'openai:text-embedding-3-small' });
|
||||
const res = await applyEmbeddingMigration(engine, plan);
|
||||
expect(res.status).toBe('refused');
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy();
|
||||
expect(await engine.getConfig('embedding_model')).toBeFalsy();
|
||||
});
|
||||
});
|
||||
|
||||
test('same-dim swap: no schema transition; invalidates legacy + drifted; writes config + state; purges cache', async () => {
|
||||
await seedEmbedded('legacy', 'abcde', null);
|
||||
await seedEmbedded('drifted', 'fghij', `old:model:${colDim}`);
|
||||
// Seed a query-cache row that must not survive the swap.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO query_cache (id, query_text, source_id) VALUES ('qc1', 'stale query', 'default')`,
|
||||
);
|
||||
|
||||
const persisted: Array<[string, number]> = [];
|
||||
const plan = await planEmbeddingMigration(engine, {
|
||||
to: 'openai:text-embedding-3-small', dim: colDim, // same width → no DDL
|
||||
});
|
||||
const res = await applyEmbeddingMigration(engine, plan, {
|
||||
persistConfig: (m, d) => { persisted.push([m, d]); },
|
||||
});
|
||||
|
||||
expect(res.status).toBe('applied');
|
||||
if (res.status !== 'applied') throw new Error('unreachable');
|
||||
expect(res.schema_transitioned).toBe(false);
|
||||
expect(res.invalidated).toBe(2); // legacy (#3391) + drifted
|
||||
expect(res.cache_cleared).toBe(1);
|
||||
expect(persisted).toEqual([['openai:text-embedding-3-small', colDim]]);
|
||||
expect(await engine.getConfig('embedding_model')).toBe('openai:text-embedding-3-small');
|
||||
expect(await engine.getConfig('embedding_dimensions')).toBe(String(colDim));
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeTruthy();
|
||||
|
||||
const qc = await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM query_cache`);
|
||||
expect(Number(qc[0]?.n)).toBe(0);
|
||||
});
|
||||
|
||||
test('dim change: schema transition rebuilds the column at the target width; re-apply is a no-op', async () => {
|
||||
await seedEmbedded('a', 'abcde', null);
|
||||
const target = colDim === 512 ? 256 : 512;
|
||||
const plan = await planEmbeddingMigration(engine, {
|
||||
to: 'openai:text-embedding-3-small', dim: target,
|
||||
});
|
||||
const res = await applyEmbeddingMigration(engine, plan);
|
||||
expect(res.status).toBe('applied');
|
||||
if (res.status !== 'applied') throw new Error('unreachable');
|
||||
expect(res.schema_transitioned).toBe(true);
|
||||
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding' AND attnum > 0 AND NOT attisdropped`,
|
||||
);
|
||||
expect(Number(rows[0]?.dim)).toBe(target);
|
||||
|
||||
// Every embedding is NULL after the column rebuild → classic stale.
|
||||
expect(await engine.countStaleChunks()).toBe(1);
|
||||
|
||||
// Resume: second apply must not transition again (column already at target).
|
||||
const res2 = await applyEmbeddingMigration(engine, plan);
|
||||
expect(res2.status).toBe('applied');
|
||||
if (res2.status !== 'applied') throw new Error('unreachable');
|
||||
expect(res2.schema_transitioned).toBe(false);
|
||||
expect(res2.invalidated).toBe(0);
|
||||
});
|
||||
|
||||
test('dim change transitions ALL THREE text-embedding-space columns (blocker: query_cache + facts were left at the old width)', async () => {
|
||||
const target = colDim === 512 ? 256 : 512;
|
||||
|
||||
// Pre-condition: all three start at the brain-birth width.
|
||||
expect(await embeddingColWidth('content_chunks')).toBe(colDim);
|
||||
expect(await embeddingColWidth('query_cache')).toBe(colDim);
|
||||
expect(await embeddingColWidth('facts')).toBe(colDim);
|
||||
|
||||
const plan = await planEmbeddingMigration(engine, {
|
||||
to: 'openai:text-embedding-3-small', dim: target,
|
||||
});
|
||||
const res = await applyEmbeddingMigration(engine, plan);
|
||||
expect(res.status).toBe('applied');
|
||||
|
||||
// All three must move together. Before the fix, query_cache and facts
|
||||
// stayed narrow: the query cache silently accepted ZERO rows forever
|
||||
// (store() + lookup() swallow the width error by design) and every
|
||||
// per-fact embed write failed.
|
||||
expect(await embeddingColWidth('content_chunks')).toBe(target);
|
||||
expect(await embeddingColWidth('query_cache')).toBe(target);
|
||||
expect(await embeddingColWidth('facts')).toBe(target);
|
||||
});
|
||||
|
||||
test('post-transition the query cache can actually STORE a row at the new width', async () => {
|
||||
const target = colDim === 512 ? 256 : 512;
|
||||
const plan = await planEmbeddingMigration(engine, {
|
||||
to: 'openai:text-embedding-3-small', dim: target,
|
||||
});
|
||||
expect((await applyEmbeddingMigration(engine, plan)).status).toBe('applied');
|
||||
|
||||
// The real regression symptom: a width-mismatched column makes this
|
||||
// INSERT throw, which query-cache.ts swallows → permanent 0% hit rate.
|
||||
const vec = `[${new Array(target).fill(0.01).join(',')}]`;
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO query_cache (id, query_text, source_id, embedding)
|
||||
VALUES ('post-migrate', 'q', 'default', $1::vector)`,
|
||||
[vec],
|
||||
);
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM query_cache WHERE id = 'post-migrate'`,
|
||||
);
|
||||
expect(Number(rows[0]?.n)).toBe(1);
|
||||
});
|
||||
|
||||
test('post-transition a fact embedding at the new width inserts (blocker: facts writes broke)', async () => {
|
||||
const target = colDim === 512 ? 256 : 512;
|
||||
const plan = await planEmbeddingMigration(engine, {
|
||||
to: 'openai:text-embedding-3-small', dim: target,
|
||||
});
|
||||
expect((await applyEmbeddingMigration(engine, plan)).status).toBe('applied');
|
||||
|
||||
const vec = `[${new Array(target).fill(0.02).join(',')}]`;
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO facts (source_id, entity_slug, fact, kind, visibility, notability, source, confidence, embedding)
|
||||
VALUES ('default', 'e', 'f', 'fact', 'private', 'medium', 'test', 1.0, $1::vector)`,
|
||||
[vec],
|
||||
);
|
||||
const rows = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM facts WHERE entity_slug = 'e'`,
|
||||
);
|
||||
expect(Number(rows[0]?.n)).toBe(1);
|
||||
});
|
||||
|
||||
test('reconcilePageSignatures stamps fully-embedded pages and leaves partially-embedded ones alone', async () => {
|
||||
// 'done' — every chunk embedded, but signature still the OLD one (the
|
||||
// batch-boundary shape: embedded correctly, stamp skipped).
|
||||
await seedEmbedded('done', 'abcde', 'old:model:1');
|
||||
// 'partial' — one embedded chunk + one NULL chunk (a real embed failure).
|
||||
await seedEmbedded('partial', 'fghij', 'old:model:1');
|
||||
await engine.upsertChunks('partial', [
|
||||
{ chunk_index: 0, chunk_text: 'fghij', chunk_source: 'compiled_truth', token_count: 4 },
|
||||
{ chunk_index: 1, chunk_text: 'not embedded', chunk_source: 'compiled_truth', token_count: 4 },
|
||||
]);
|
||||
|
||||
const plan = await planEmbeddingMigration(engine, {
|
||||
to: 'openai:text-embedding-3-small', dim: colDim,
|
||||
});
|
||||
const n = await reconcilePageSignatures(engine, plan);
|
||||
expect(n).toBe(1); // 'done' only
|
||||
|
||||
const sig = `openai:text-embedding-3-small:${colDim}`;
|
||||
const rows = await engine.executeRaw<{ slug: string; embedding_signature: string | null }>(
|
||||
`SELECT slug, embedding_signature FROM pages WHERE slug IN ('done','partial') ORDER BY slug`,
|
||||
);
|
||||
expect(rows.find(r => r.slug === 'done')?.embedding_signature).toBe(sig);
|
||||
expect(rows.find(r => r.slug === 'partial')?.embedding_signature).toBe('old:model:1');
|
||||
});
|
||||
|
||||
test('completeEmbeddingMigration clears the state marker and stamps completion', async () => {
|
||||
const plan = await planEmbeddingMigration(engine, { to: 'openai:text-embedding-3-small', dim: colDim });
|
||||
await applyEmbeddingMigration(engine, plan);
|
||||
await completeEmbeddingMigration(engine, plan);
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy();
|
||||
const done = JSON.parse((await engine.getConfig(MIGRATION_COMPLETED_KEY))!);
|
||||
expect(done.to_model).toBe('openai:text-embedding-3-small');
|
||||
});
|
||||
});
|
||||
|
||||
describe('migrate_embeddings op contract', () => {
|
||||
const op = operations.find(o => o.name === 'migrate_embeddings')!;
|
||||
|
||||
test('is admin + localOnly + mutating', () => {
|
||||
expect(op).toBeDefined();
|
||||
expect(op.scope).toBe('admin');
|
||||
expect(op.localOnly).toBe(true);
|
||||
expect(op.mutating).toBe(true);
|
||||
});
|
||||
|
||||
test('remote callers are refused even if dispatch forgot the localOnly filter', async () => {
|
||||
await expect(
|
||||
op.handler({ engine, remote: true } as never, { to: 'openai:text-embedding-3-small' }),
|
||||
).rejects.toThrow(/local-only/);
|
||||
// remote undefined (not strictly false) is ALSO refused — fail-closed.
|
||||
await expect(
|
||||
op.handler({ engine } as never, { to: 'openai:text-embedding-3-small' }),
|
||||
).rejects.toThrow(/local-only/);
|
||||
});
|
||||
|
||||
test('without yes=true it returns the plan only (no mutation)', async () => {
|
||||
const res = await op.handler(
|
||||
{ engine, remote: false } as never,
|
||||
{ to: 'openai:text-embedding-3-small', dim: colDim },
|
||||
) as { status: string; plan: { to_model: string } };
|
||||
expect(res.status).toBe('needs_confirmation');
|
||||
expect(res.plan.to_model).toBe('openai:text-embedding-3-small');
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy();
|
||||
});
|
||||
|
||||
test('signature helper matches currentEmbeddingSignature shape', () => {
|
||||
expect(migrationSignature('openai:text-embedding-3-small', 1536))
|
||||
.toBe('openai:text-embedding-3-small:1536');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,165 @@
|
||||
/**
|
||||
* #3390 regression: page straddling a stale-batch BOUNDARY (adversarial review
|
||||
* blocker 2).
|
||||
*
|
||||
* The embed loop stamps `pages.embedding_signature` only when
|
||||
* `stale.length === existing.length` — i.e. when every chunk of the page landed
|
||||
* in the SAME batch. `listStaleChunks` is a plain keyset LIMIT with no page
|
||||
* alignment, so on any corpus bigger than one batch the page split across the
|
||||
* boundary is embedded correctly but never stamped.
|
||||
*
|
||||
* Pre-fix consequence: the completion probe counted that page stale, the
|
||||
* command printed "Migration incomplete" and exited 1 on a perfectly-migrated
|
||||
* brain, and the re-run RE-INVALIDATED and RE-PAID for those pages —
|
||||
* contradicting the "already-migrated chunks are never re-embedded" contract.
|
||||
*
|
||||
* Shape here: 3 pages x 2 chunks = 6 chunks with --batch-size 3, so the
|
||||
* boundary falls mid-page-2. Asserts exit 0 on the first run, every page
|
||||
* stamped, and ZERO embed work on the second run.
|
||||
*
|
||||
* Named `.serial.test.ts`: holds a temp GBRAIN_HOME + an installed fake embed
|
||||
* transport for its whole beforeAll→afterAll lifecycle, which withEnv() can't
|
||||
* wrap.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__setEmbedTransportForTests,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
import { runEmbedCore } from '../src/commands/embed.ts';
|
||||
import { runMigrateEmbeddings } from '../src/commands/migrate-embeddings.ts';
|
||||
import { MIGRATION_STATE_KEY, MIGRATION_COMPLETED_KEY } from '../src/core/embedding-migration.ts';
|
||||
|
||||
const FROM_DIMS = 1280;
|
||||
const TO_DIMS = 1536;
|
||||
const PAGES = ['b-1', 'b-2', 'b-3'];
|
||||
const PROBE_TEXT = 'gbrain embedding migration probe';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let tmpHome: string;
|
||||
const savedEnv: Record<string, string | undefined> = {};
|
||||
let currentDims = FROM_DIMS;
|
||||
let embeddedTexts: string[] = [];
|
||||
|
||||
class ExitError extends Error {
|
||||
constructor(public code: number) { super(`exit ${code}`); }
|
||||
}
|
||||
const exitSeam = (code: number): never => { throw new ExitError(code); };
|
||||
|
||||
async function runMigrate(args: string[]): Promise<number> {
|
||||
try {
|
||||
await runMigrateEmbeddings(engine, args, { exit: exitSeam });
|
||||
throw new Error('runMigrateEmbeddings returned without exiting');
|
||||
} catch (e) {
|
||||
if (e instanceof ExitError) return e.code;
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
for (const k of ['GBRAIN_HOME', 'GBRAIN_EMBEDDING_MODEL', 'GBRAIN_EMBEDDING_DIMENSIONS', 'OPENAI_API_KEY', 'ZEROENTROPY_API_KEY', 'DATABASE_URL']) {
|
||||
savedEnv[k] = process.env[k];
|
||||
delete process.env[k];
|
||||
}
|
||||
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-migrate-boundary-'));
|
||||
process.env.GBRAIN_HOME = tmpHome;
|
||||
mkdirSync(join(tmpHome, '.gbrain'), { recursive: true });
|
||||
writeFileSync(join(tmpHome, '.gbrain', 'config.json'), JSON.stringify({
|
||||
engine: 'pglite',
|
||||
embedding_model: 'zeroentropyai:zembed-1',
|
||||
embedding_dimensions: FROM_DIMS,
|
||||
zeroentropy_api_key: 'ze-test-fake',
|
||||
openai_api_key: 'sk-test-fake',
|
||||
}, null, 2));
|
||||
|
||||
resetGateway();
|
||||
configureGateway({
|
||||
embedding_model: 'zeroentropyai:zembed-1',
|
||||
embedding_dimensions: FROM_DIMS,
|
||||
env: { ZEROENTROPY_API_KEY: 'ze-test-fake', OPENAI_API_KEY: 'sk-test-fake' },
|
||||
});
|
||||
__setEmbedTransportForTests(async ({ values }: { values: string[] }) => {
|
||||
for (const v of values) if (v !== PROBE_TEXT) embeddedTexts.push(v);
|
||||
return {
|
||||
embeddings: values.map(() => new Array(currentDims).fill(0).map((_, i) => Math.sin(i) * 0.01 + 0.003)),
|
||||
usage: { tokens: values.length * 4 },
|
||||
} as never;
|
||||
});
|
||||
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ embedding_dimensions: FROM_DIMS } as never);
|
||||
await engine.initSchema();
|
||||
|
||||
// 3 pages x 2 chunks each = 6 chunks.
|
||||
for (const slug of PAGES) {
|
||||
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}` });
|
||||
await engine.upsertChunks(slug, [
|
||||
{ chunk_index: 0, chunk_text: `${slug} chunk zero`, chunk_source: 'compiled_truth', token_count: 4 },
|
||||
{ chunk_index: 1, chunk_text: `${slug} chunk one`, chunk_source: 'compiled_truth', token_count: 4 },
|
||||
]);
|
||||
}
|
||||
await runEmbedCore(engine, { stale: true, quiet: true });
|
||||
}, 60000);
|
||||
|
||||
afterAll(async () => {
|
||||
__setEmbedTransportForTests(null);
|
||||
resetGateway();
|
||||
await engine.disconnect();
|
||||
rmSync(tmpHome, { recursive: true, force: true });
|
||||
for (const [k, v] of Object.entries(savedEnv)) {
|
||||
if (v === undefined) delete process.env[k];
|
||||
else process.env[k] = v;
|
||||
}
|
||||
});
|
||||
|
||||
describe('migration across a stale-batch boundary', () => {
|
||||
test('seed is fully embedded at the source width', async () => {
|
||||
expect(await engine.countStaleChunks()).toBe(0);
|
||||
const n = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM content_chunks WHERE embedding IS NOT NULL`,
|
||||
);
|
||||
expect(Number(n[0]?.n)).toBe(6);
|
||||
});
|
||||
|
||||
test('batch-size 3 splits a page across the boundary yet still exits 0 with every page stamped', async () => {
|
||||
currentDims = TO_DIMS;
|
||||
embeddedTexts = [];
|
||||
|
||||
const code = await runMigrate([
|
||||
'--to', 'openai:text-embedding-3-small', '--yes', '--batch-size', '3',
|
||||
]);
|
||||
// Pre-fix this was 1 ("Migration incomplete") even though all 6 chunks
|
||||
// were correctly embedded — the boundary page was never stamped.
|
||||
expect(code).toBe(0);
|
||||
|
||||
expect(embeddedTexts.length).toBe(6); // all six chunks re-embedded once
|
||||
expect(await engine.countStaleChunks()).toBe(0);
|
||||
|
||||
// Every page carries the TARGET signature, including the boundary page.
|
||||
const sig = `openai:text-embedding-3-small:${TO_DIMS}`;
|
||||
const stamped = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM pages WHERE embedding_signature = $1`,
|
||||
[sig],
|
||||
);
|
||||
expect(Number(stamped[0]?.n)).toBe(PAGES.length);
|
||||
|
||||
// Completion bookkeeping ran (it only runs when the backlog drained).
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy();
|
||||
expect(await engine.getConfig(MIGRATION_COMPLETED_KEY)).toBeTruthy();
|
||||
}, 60000);
|
||||
|
||||
test('second run does ZERO work — the "never re-embedded twice" contract holds across a boundary', async () => {
|
||||
embeddedTexts = [];
|
||||
const code = await runMigrate([
|
||||
'--to', 'openai:text-embedding-3-small', '--yes', '--batch-size', '3',
|
||||
]);
|
||||
expect(code).toBe(0);
|
||||
// Pre-fix the unstamped boundary page was re-invalidated and PAID FOR again.
|
||||
expect(embeddedTexts.length).toBe(0);
|
||||
}, 60000);
|
||||
});
|
||||
@@ -0,0 +1,281 @@
|
||||
/**
|
||||
* #3390 — `gbrain migrate embeddings` END-TO-END on PGLite.
|
||||
*
|
||||
* The full command flow with a fake embedding transport:
|
||||
* 1. Disposable brain seeded via the REAL embed pipeline on a fake 1280d
|
||||
* "zeroentropyai:zembed-1" provider (the shipped default).
|
||||
* 2. One page's embedding_signature NULLed (simulates a pre-v108 page,
|
||||
* the #3391 class).
|
||||
* 3. Migration to a fake 1536d openai:text-embedding-3-small — with the
|
||||
* transport failing two specific pages, simulating a killed/partial
|
||||
* run. Asserts: exit 1 (incomplete), schema at 1536, config file
|
||||
* swapped, state marker kept, partial progress banked.
|
||||
* 4. Re-run of the SAME command (the documented resume path). Asserts:
|
||||
* exit 0, only the two failed pages re-embedded (no duplicate work),
|
||||
* every chunk embedded at 1536, every page stamped with the new
|
||||
* signature (including the formerly-NULL one), state marker cleared,
|
||||
* query cache purged, vector search works against the new column.
|
||||
*
|
||||
* Named `.serial.test.ts`: the whole file runs under a temp GBRAIN_HOME +
|
||||
* an installed fake embed transport for its entire lifecycle (beforeAll →
|
||||
* afterAll), which withEnv() cannot wrap. GBRAIN_HOME is pointed at a temp dir so the file-plane config write
|
||||
* (persistEmbeddingFileConfig) never touches the developer's ~/.gbrain.
|
||||
*/
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import {
|
||||
configureGateway,
|
||||
resetGateway,
|
||||
__setEmbedTransportForTests,
|
||||
} from '../src/core/ai/gateway.ts';
|
||||
import { runEmbedCore } from '../src/commands/embed.ts';
|
||||
import { runMigrateEmbeddings } from '../src/commands/migrate-embeddings.ts';
|
||||
import {
|
||||
MIGRATION_STATE_KEY,
|
||||
MIGRATION_COMPLETED_KEY,
|
||||
} from '../src/core/embedding-migration.ts';
|
||||
|
||||
const FROM_DIMS = 1280;
|
||||
const TO_DIMS = 1536;
|
||||
const PAGES = ['page-1', 'page-2', 'page-3', 'page-4', 'page-5', 'page-6'];
|
||||
const PROBE_TEXT = 'gbrain embedding migration probe';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let tmpHome: string;
|
||||
const savedEnv: Record<string, string | undefined> = {};
|
||||
|
||||
/** Deterministic fake transport: vector width driven by the test phase. */
|
||||
let currentDims = FROM_DIMS;
|
||||
/** Texts that make the transport throw (simulates a mid-run kill). */
|
||||
let failTexts: string[] = [];
|
||||
/** Every non-probe text the transport embedded, per phase. */
|
||||
let embeddedTexts: string[] = [];
|
||||
|
||||
function installTransport(): void {
|
||||
__setEmbedTransportForTests(async ({ values }: { values: string[] }) => {
|
||||
for (const v of values) {
|
||||
if (failTexts.some(f => v.includes(f))) {
|
||||
throw new Error(`fake transport: simulated failure for "${v.slice(0, 30)}..."`);
|
||||
}
|
||||
}
|
||||
for (const v of values) {
|
||||
if (v !== PROBE_TEXT) embeddedTexts.push(v);
|
||||
}
|
||||
return {
|
||||
embeddings: values.map(() => new Array(currentDims).fill(0).map((_, i) => Math.sin(i) * 0.01 + 0.001)),
|
||||
usage: { tokens: values.length * 4 },
|
||||
} as never;
|
||||
});
|
||||
}
|
||||
|
||||
class ExitError extends Error {
|
||||
constructor(public code: number) { super(`exit ${code}`); }
|
||||
}
|
||||
const exitSeam = (code: number): never => { throw new ExitError(code); };
|
||||
|
||||
async function runMigrate(args: string[]): Promise<number> {
|
||||
try {
|
||||
await runMigrateEmbeddings(engine, args, { exit: exitSeam });
|
||||
throw new Error('runMigrateEmbeddings returned without exiting');
|
||||
} catch (e) {
|
||||
if (e instanceof ExitError) return e.code;
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
async function columnDims(): Promise<number> {
|
||||
const rows = await engine.executeRaw<{ dim: number }>(
|
||||
`SELECT atttypmod AS dim FROM pg_attribute
|
||||
WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding'
|
||||
AND attnum > 0 AND NOT attisdropped`,
|
||||
);
|
||||
return Number(rows[0]?.dim);
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
// Isolate the file-plane config.
|
||||
for (const k of ['GBRAIN_HOME', 'GBRAIN_EMBEDDING_MODEL', 'GBRAIN_EMBEDDING_DIMENSIONS', 'OPENAI_API_KEY', 'ZEROENTROPY_API_KEY', 'DATABASE_URL']) {
|
||||
savedEnv[k] = process.env[k];
|
||||
delete process.env[k];
|
||||
}
|
||||
tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-migrate-e2e-'));
|
||||
process.env.GBRAIN_HOME = tmpHome;
|
||||
mkdirSync(join(tmpHome, '.gbrain'), { recursive: true });
|
||||
writeFileSync(join(tmpHome, '.gbrain', 'config.json'), JSON.stringify({
|
||||
engine: 'pglite',
|
||||
embedding_model: 'zeroentropyai:zembed-1',
|
||||
embedding_dimensions: FROM_DIMS,
|
||||
zeroentropy_api_key: 'ze-test-fake',
|
||||
openai_api_key: 'sk-test-fake',
|
||||
}, null, 2));
|
||||
|
||||
resetGateway();
|
||||
configureGateway({
|
||||
embedding_model: 'zeroentropyai:zembed-1',
|
||||
embedding_dimensions: FROM_DIMS,
|
||||
env: { ZEROENTROPY_API_KEY: 'ze-test-fake', OPENAI_API_KEY: 'sk-test-fake' },
|
||||
});
|
||||
installTransport();
|
||||
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ embedding_dimensions: FROM_DIMS } as never);
|
||||
await engine.initSchema();
|
||||
}, 60000);
|
||||
|
||||
afterAll(async () => {
|
||||
__setEmbedTransportForTests(null);
|
||||
resetGateway();
|
||||
await engine.disconnect();
|
||||
rmSync(tmpHome, { recursive: true, force: true });
|
||||
for (const [k, v] of Object.entries(savedEnv)) {
|
||||
if (v === undefined) delete process.env[k];
|
||||
else process.env[k] = v;
|
||||
}
|
||||
});
|
||||
|
||||
describe('migrate embeddings — full flow on PGLite', () => {
|
||||
test('seed: 6 pages embedded at 1280d through the real embed pipeline', async () => {
|
||||
expect(await columnDims()).toBe(FROM_DIMS);
|
||||
for (const slug of PAGES) {
|
||||
await engine.putPage(slug, { type: 'note', title: slug, compiled_truth: `# ${slug}\n\ncontent for ${slug}` });
|
||||
await engine.upsertChunks(slug, [
|
||||
{ chunk_index: 0, chunk_text: `chunk text for ${slug}`, chunk_source: 'compiled_truth', token_count: 5 },
|
||||
]);
|
||||
}
|
||||
const seeded = await runEmbedCore(engine, { stale: true, quiet: true });
|
||||
expect(seeded.embedded).toBe(PAGES.length);
|
||||
expect(await engine.countStaleChunks()).toBe(0);
|
||||
|
||||
// Simulate a pre-v108 page: embedded, but no recorded signature (#3391).
|
||||
await engine.executeRaw(`UPDATE pages SET embedding_signature = NULL WHERE slug = 'page-1'`);
|
||||
const sigs = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM pages WHERE embedding_signature = 'zeroentropyai:zembed-1:${FROM_DIMS}'`,
|
||||
);
|
||||
expect(Number(sigs[0]?.n)).toBe(PAGES.length - 1);
|
||||
|
||||
// Seed a query-cache row that must not survive the migration.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO query_cache (id, query_text, source_id) VALUES ('qc-pre', 'old space query', 'default')`,
|
||||
);
|
||||
}, 60000);
|
||||
|
||||
test('dry-run: prints the plan, changes nothing', async () => {
|
||||
const code = await runMigrate(['--to', 'openai:text-embedding-3-small', '--dry-run', '--json']);
|
||||
expect(code).toBe(0);
|
||||
expect(await columnDims()).toBe(FROM_DIMS);
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy();
|
||||
// Config file untouched.
|
||||
const cfg = JSON.parse(readFileSync(join(tmpHome, '.gbrain', 'config.json'), 'utf-8'));
|
||||
expect(cfg.embedding_model).toBe('zeroentropyai:zembed-1');
|
||||
});
|
||||
|
||||
test('non-TTY without --yes refuses with exit 2 (cost gate)', async () => {
|
||||
const code = await runMigrate(['--to', 'openai:text-embedding-3-small']);
|
||||
expect(code).toBe(2);
|
||||
expect(await columnDims()).toBe(FROM_DIMS);
|
||||
});
|
||||
|
||||
test('spend.posture=tokenmax does NOT bypass the gate (guards a destructive rebuild, not just spend)', async () => {
|
||||
await engine.setConfig('spend.posture', 'tokenmax');
|
||||
try {
|
||||
const code = await runMigrate(['--to', 'openai:text-embedding-3-small']);
|
||||
expect(code).toBe(2); // posture waives the spend ceiling, not the consent
|
||||
expect(await columnDims()).toBe(FROM_DIMS);
|
||||
} finally {
|
||||
await engine.unsetConfig('spend.posture');
|
||||
}
|
||||
});
|
||||
|
||||
test('interrupted run: partial progress banks, exit 1, state marker kept', async () => {
|
||||
currentDims = TO_DIMS;
|
||||
failTexts = ['page-4', 'page-5']; // simulate dying mid-run on two pages
|
||||
embeddedTexts = [];
|
||||
|
||||
const code = await runMigrate(['--to', 'openai:text-embedding-3-small', '--yes']);
|
||||
expect(code).toBe(1); // incomplete
|
||||
|
||||
// Schema + config swapped BEFORE the re-embed, so the partial run is
|
||||
// already in the new space.
|
||||
expect(await columnDims()).toBe(TO_DIMS);
|
||||
const cfg = JSON.parse(readFileSync(join(tmpHome, '.gbrain', 'config.json'), 'utf-8'));
|
||||
expect(cfg.embedding_model).toBe('openai:text-embedding-3-small');
|
||||
expect(cfg.embedding_dimensions).toBe(TO_DIMS);
|
||||
expect(await engine.getConfig('embedding_model')).toBe('openai:text-embedding-3-small');
|
||||
|
||||
// 4 of 6 pages embedded; the 2 failed pages remain stale (= the checkpoint).
|
||||
expect(embeddedTexts.length).toBe(PAGES.length - 2);
|
||||
expect(await engine.countStaleChunks()).toBe(2);
|
||||
|
||||
// In-flight marker survives so doctor/status can see the migration.
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeTruthy();
|
||||
expect(await engine.getConfig(MIGRATION_COMPLETED_KEY)).toBeFalsy();
|
||||
|
||||
// Query cache was purged at swap time.
|
||||
const qc = await engine.executeRaw<{ n: number }>(`SELECT count(*)::int AS n FROM query_cache`);
|
||||
expect(Number(qc[0]?.n)).toBe(0);
|
||||
}, 60000);
|
||||
|
||||
test('resume: re-running the same command finishes without redoing work', async () => {
|
||||
failTexts = [];
|
||||
embeddedTexts = [];
|
||||
|
||||
const code = await runMigrate(['--to', 'openai:text-embedding-3-small', '--yes']);
|
||||
expect(code).toBe(0);
|
||||
|
||||
// Only the two previously-failed pages were embedded this pass.
|
||||
expect(embeddedTexts.length).toBe(2);
|
||||
expect(embeddedTexts.join(' ')).toContain('page-4');
|
||||
expect(embeddedTexts.join(' ')).toContain('page-5');
|
||||
|
||||
// Everything is in the target space now.
|
||||
expect(await engine.countStaleChunks()).toBe(0);
|
||||
expect(await engine.countStaleChunks({
|
||||
signature: `openai:text-embedding-3-small:${TO_DIMS}`,
|
||||
includeNullSignature: true,
|
||||
})).toBe(0);
|
||||
|
||||
// Every page — including the formerly NULL-signature page-1 — is stamped.
|
||||
const sigs = await engine.executeRaw<{ n: number }>(
|
||||
`SELECT count(*)::int AS n FROM pages
|
||||
WHERE embedding_signature = 'openai:text-embedding-3-small:${TO_DIMS}'`,
|
||||
);
|
||||
expect(Number(sigs[0]?.n)).toBe(PAGES.length);
|
||||
|
||||
// Bookkeeping: marker cleared, completion stamped.
|
||||
expect(await engine.getConfig(MIGRATION_STATE_KEY)).toBeFalsy();
|
||||
const done = JSON.parse((await engine.getConfig(MIGRATION_COMPLETED_KEY))!);
|
||||
expect(done.to_model).toBe('openai:text-embedding-3-small');
|
||||
}, 60000);
|
||||
|
||||
test('post-migration: vector search works against the new 1536d column', async () => {
|
||||
// HNSW index was rebuilt by the schema transition.
|
||||
const idx = await engine.executeRaw<{ indexname: string }>(
|
||||
`SELECT indexname FROM pg_indexes WHERE tablename = 'content_chunks' AND indexname = 'idx_chunks_embedding'`,
|
||||
);
|
||||
expect(idx.length).toBe(1);
|
||||
|
||||
// A cosine query at the new width returns results (all 6 chunks embedded).
|
||||
const qvec = `[${new Array(TO_DIMS).fill(0).map((_, i) => (Math.sin(i) * 0.01 + 0.001).toFixed(6)).join(',')}]`;
|
||||
const rows = await engine.executeRaw<{ slug: string }>(
|
||||
`SELECT p.slug FROM content_chunks cc
|
||||
JOIN pages p ON p.id = cc.page_id
|
||||
WHERE cc.embedding IS NOT NULL
|
||||
ORDER BY cc.embedding <=> $1::vector
|
||||
LIMIT 3`,
|
||||
[qvec],
|
||||
);
|
||||
expect(rows.length).toBe(3);
|
||||
});
|
||||
|
||||
test('re-run on an already-migrated brain is a clean no-op', async () => {
|
||||
embeddedTexts = [];
|
||||
const code = await runMigrate(['--to', 'openai:text-embedding-3-small', '--yes']);
|
||||
expect(code).toBe(0);
|
||||
// Nothing re-embedded (probe excluded from embeddedTexts by design).
|
||||
expect(embeddedTexts.length).toBe(0);
|
||||
expect(await columnDims()).toBe(TO_DIMS);
|
||||
});
|
||||
});
|
||||
@@ -153,6 +153,7 @@ describe('mcpOperations filter — localOnly ops are excluded from the HTTP-expo
|
||||
'purge_deleted_pages',
|
||||
'get_recent_transcripts',
|
||||
'code_traversal_cache_clear',
|
||||
'migrate_embeddings',
|
||||
];
|
||||
const lookup = new Map(operations.map(op => [op.name, op] as const));
|
||||
for (const name of KNOWN_LOCAL_ONLY) {
|
||||
|
||||
@@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => {
|
||||
});
|
||||
|
||||
describe('KNOBS_HASH_VERSION', () => {
|
||||
it('is 12 (11→12 hard-exclude fold invalidates rows written under a different exclude policy, #2825)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
it('is 13 (12→13 embedding-provider migration invalidates rows written against the prior embedding space, #3390)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -410,7 +410,10 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => {
|
||||
// #2825: bumped 11→12 to fold the resolved hard-exclude prefix list
|
||||
// (hx=) — cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across
|
||||
// processes.
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
// #3390/#3391: bumped 12→13 for the embedding-provider migration wave —
|
||||
// legacy callers hash prov=default before AND after a provider swap, so
|
||||
// pre-migration cache rows must become unreachable on upgrade.
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
|
||||
test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => {
|
||||
@@ -575,8 +578,8 @@ describe('v0.40.4 — graph_signals knob', () => {
|
||||
});
|
||||
|
||||
describe('v0.42.3.0 — autocut knobs', () => {
|
||||
test('KNOBS_HASH_VERSION is 12 (11→12 hard-exclude fold, #2825)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
test('KNOBS_HASH_VERSION is 13 (12→13 embedding-migration wave, #3390/#3391)', () => {
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
|
||||
test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => {
|
||||
|
||||
@@ -44,7 +44,7 @@ function baseKnobs(): ResolvedSearchKnobs {
|
||||
}
|
||||
|
||||
describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
test('version is 12 (…; 9→10 relational recall; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825)', () => {
|
||||
test('version is 13 (…; 10→11 asymmetric input_type #1400; 11→12 hard-excludes #2825; 12→13 embedding-provider migration #3390)', () => {
|
||||
// v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold
|
||||
// floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs
|
||||
// (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation).
|
||||
@@ -64,7 +64,7 @@ describe('KNOBS_HASH_VERSION + version invariants', () => {
|
||||
// pre-fix document-side query vectors must not be served.
|
||||
// #2825: 11→12 to fold the resolved hard-exclude prefix list (hx=) —
|
||||
// cached rows leaked GBRAIN_SEARCH_EXCLUDE'd slugs across processes.
|
||||
expect(KNOBS_HASH_VERSION).toBe(12);
|
||||
expect(KNOBS_HASH_VERSION).toBe(13);
|
||||
});
|
||||
|
||||
test('hash is 16 hex chars regardless of reranker config', () => {
|
||||
|
||||
Reference in New Issue
Block a user