gbrain/docs/guides/embedding-migration.md

Embedding migration — moving a brain to another embedding provider

gbrain migrate embeddings re-embeds an entire brain onto a different embedding provider/model, safely and resumably. It is the forward path off a sunsetting provider (for example ZeroEntropy's hosted API, which shuts down 2026-09-04 and is the shipped default for brains that never picked a model) — but it is provider-agnostic: any configured provider:model works as a target.

Also reachable as gbrain retrieval-upgrade (the name doctor and the README reference).

Quick start

# Preview the work + cost. Changes nothing.
gbrain migrate embeddings --to openai:text-embedding-3-small --dry-run

# Run it (interactive confirm shows chunk count + $ estimate first).
gbrain migrate embeddings --to openai:text-embedding-3-small

# Non-interactive (cron / scripts): --yes is required, else exit 2.
gbrain migrate embeddings --to voyage:voyage-3-large --yes

--dim <N> overrides the target width; it defaults to the provider recipe's declared width and is required for recipes that don't declare one (litellm, llama-server, and other bring-your-own-model providers).

What it does, in order

1. Plan. Counts every chunk not already in the target embedding space — including chunks on pages with no recorded embedding signature (pages embedded before the v108 provenance stamp). Prices the re-embed from the pricing table; unknown providers print "estimate unavailable" instead of a fabricated number.
2. Consent gate. Prints the plan; requires an interactive y or --yes. Non-TTY without --yes refuses with exit 2 (mirrors the reindex-code gate in spend-controls). Unlike the pure cost gates there, spend.posture=tokenmax does not bypass this one: posture waives the spend ceiling, and this gate also guards a destructive schema rebuild. Under tokenmax the dollar figure is marked informational and the confirmation is still asked. --yes is the single scripted bypass.
3. Live probe. One tiny embed against the TARGET provider before any mutation — validates the API key, model id, and dimension support in a single call. A bad key fails here, with nothing changed.
4. Env-override gate. Refuses when GBRAIN_EMBEDDING_MODEL / GBRAIN_EMBEDDING_DIMENSIONS would silently defeat the switch at runtime (the same guard ze-switch uses). --ignore-env-override for people running deliberate experiments.
5. Apply. When the target width differs from the actual column width, runs the same atomic schema transition ze-switch uses, in one transaction. It rebuilds all three dim-pinned text-embedding-space columns — content_chunks.embedding, query_cache.embedding, and facts.embedding — at the new width, preserving each column's type (vector vs halfvec) and recreating its HNSW index. Missing any of the three leaves it silently broken: a narrow query_cache.embedding makes every cache write and read fail by design (the cache swallows errors so it can never break search) for a permanent 0% hit rate, and a narrow facts.embedding fails every per-fact embed write. The image/multimodal columns ARE deliberately untouched — they use separate models whose dimensions are independent of the text embedding model. Writes embedding_model + embedding_dimensions to BOTH config planes (file plane for the runtime gateway, DB plane for doctor), invalidates every chunk still in the old space — including NULL-signature pages — and purges the semantic query cache so stale cached results can't be served across the swap.
6. Re-embed. The standard embed pipeline (embed --stale --catch-up) with per-source single-flight locks, rate-limit backoff, stderr progress, and optional DB-contention pacing (--pace[=mode]).

What the rebuild deletes

The dimension change deletes every stored embedding vector in the brain — they are in the old model's space and unusable. They are not recoverable: going back to the previous provider means paying for a second full re-embed. content_chunks vectors are rebuilt by the re-embed pass, the query cache refills on the next query, and fact embeddings are rewritten on their next write (or a gbrain extract pass).

Resume after a kill

The NULL-embedding column is the checkpoint. If the run is killed (or some pages fail to embed), re-run the same command: chunks already embedded on the target are never re-embedded, the schema/config steps no-op, and the run continues where it stopped. An in-flight marker (embedding_migration.state in DB config) records the target; it is cleared only when the backlog drains to zero.

A page whose chunks straddle two stale batches is embedded correctly but not stamped by the embed loop (which only stamps all-or-nothing per batch), so the migration runs one reconcile pass after the drain that stamps every fully-embedded page. Without it a large brain would report "incomplete" and the re-run would pay again for those pages. --batch-size N tunes the batch (default 2000).

--no-embed applies schema + config + invalidation and stops, so you can run the (potentially long) re-embed later or in the background:

gbrain migrate embeddings --to openai:text-embedding-3-small --yes --no-embed
gbrain embed --stale --catch-up --include-null-signature --background

During the migration

While the re-embed runs, semantic search returns degraded (lexical-arm-only) results for not-yet-re-embedded content. Pick a quiet window for large brains, or use --pace to keep the DB responsive.

Pages without an embedding signature (#3391)

Pages embedded before provenance stamping have embedding_signature IS NULL and are grandfathered by the routine stale sweep (so an upgrade never surprise-re-embeds a whole corpus). After a provider swap that grandfather clause would silently leave those pages in the OLD embedding space — mixed vector spaces in one index, degrading retrieval with nothing in the logs.

• gbrain migrate embeddings always includes them.
• Plain gbrain embed --stale warns when a model swap leaves NULL-signature pages behind, and gbrain embed --stale --include-null-signature re-embeds them.

Reranker

Migrating embeddings does not touch the reranker. If search.reranker.model points at the outgoing provider, the plan prints a warning; disable it (gbrain config set search.reranker.enabled false) or point it at another provider.

Self-hosting instead of migrating

If the outgoing model's weights are available (zembed-1's are Apache-2.0), serving them locally via llama-server / ollama / a LiteLLM proxy preserves your existing vectors — no re-embed at all. Point embedding_model at the local recipe and keep the same dimensions. The migration command is for when you'd rather move to a hosted provider.