6 Commits

Include renames

Author	SHA1	Message	Date
3 people	a25209bbb2	v0.42.58.0 fix(ai): provider-agnostic gateway — env clobber, base-URL /v1, embedding dims (#1249 #1250 #1292 #2271 #2209) (#2627) ... * fix(ai): drop empty-string env values before merge so they can't clobber config keys (#1249) Claude Code injects ANTHROPIC_API_KEY='' to neuter subprocess LLM calls; an unconditional process.env spread let that empty string override a valid config.json key, breaking every gateway op with NO_ANTHROPIC_API_KEY. Filter '' / undefined before the merge; '0' and 'false' are preserved. * fix(ai): normalize native provider base URLs + replace embedding guard with a dims-presence check (#1250, #1292) #1250: createAnthropic/createOpenAI were called with no baseURL, so an env-injected bare host (e.g. ANTHROPIC_BASE_URL without /v1) 404'd. Add a shared resolveNativeBaseUrl and pass a normalized baseURL at all anthropic + openai native sites (google deferred until its suffix is verified). #1292/D6: the user_provided_model_unset guard was structurally unreachable as a no-model check (parseModelId throws on a bare provider) and only ever false-positived for litellm:<model>, silently disabling vector search. Replace it with a real dims-presence check for user-provided/zero-default recipes and delete the dead branch in both consumers. Also stop configureGateway from fabricating a default embedding_dimensions, so 'no dims set' stays honest. * fix(ai): trust user-declared embedding dims for local recipes + litellm /v1 hint (#2271, #2209) #2271: a new trust_custom_dims flag adds a passthrough tier so ollama / llama-server / litellm accept a user-supplied --embedding-dimensions instead of being hard-rejected. Fail-closed for fixed-dim providers (openai/voyage/ zeroentropy) and excludes openrouter (declares dims_options). Register modern ollama embed model names. #2209: litellm setup_hint now states the /v1 path convention and the docs pointer is corrected to docs/integrations/embedding-providers.md. * docs+test(ai): KEY_FILES current-state for provider-agnostic gateway + embed-preflight dims-unset test (#1249, #1250, #1292) * fix(ai): point user_provided_dims_unset remediation at 'gbrain init' (config set rejects it) + coverage Pre-landing adversarial review (P1): the new dims-unset guard told users to run 'gbrain config set embedding_dimensions <N>', which config.ts hard-rejects (it's a schema-sizing field). Both consumer messages now point at 'gbrain init --embedding-dimensions'. Adds: pgvector-cap-still-fires regression for the trust_custom_dims passthrough, and a configureGateway backfill-invariant test. * chore: bump version and changelog (v0.42.57.0) Provider-agnostic plumbing wave: #1249 empty-env clobber, #1250 native baseURL normalization, #1292 embedding dims-presence guard, #2271 trust_custom_dims passthrough, #2209 litellm /v1 hint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync embedding-providers guide for provider-agnostic gateway wave (v0.42.57.0) Post-ship doc drift fix for the v0.42.57.0 AI-gateway wave: - LiteLLM section now names the /v1 base-URL convention (#2209). - Ollama section lists the newly-registered modern embedders qwen3-embed-8b + snowflake-arctic-embed-l-v2, and notes dims-trust for local recipes (#2271). - llama-server section notes gbrain trusts the user-declared dimension (#2271). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: post-ship doc sweep for v0.42.57.0 provider-agnostic gateway wave - KEY_FILES.md types.ts entry: document EmbeddingTouchpoint.trust_custom_dims (#2271 passthrough tier, runs after dims_options + Matryoshka allowlists) - ENGINES.md: embedding design-choice note now names the provider-agnostic gateway delegation instead of the stale OpenAI-only parenthetical - embedding-providers.md: drop an exact-duplicate doctor-8c paragraph - llms-full.txt regenerated (ENGINES.md is inlined in the bundle) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: apply codex doc-review findings for v0.42.57.0 (base-URL env note, litellm multimodal) - embedding-providers.md OpenAI section: document OPENAI_BASE_URL / ANTHROPIC_BASE_URL bare-host /v1 normalization (#1250 user-facing surface) - TL;DR table: litellm multimodal is backend-permitting (recipe declares supports_multimodal: true, routed via the openai-compat multimodal path), not "no" Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: pin engine-find-trajectory schema to 1536 + stop gateway-state leaks across shard files CI shard 5 failed 7 findTrajectory tests with 'expected 1280 dimensions, not 1536': engine-find-trajectory hardcodes 1536-d vectors but sizes its schema from AMBIENT gateway state in beforeAll — which runs before the legacy-embedding-preload's per-test 1536 restore. A preceding file that ends with a dimensionless configureGateway (facts-extract-silent-no-op) or a bare resetGateway poisons the next fresh initSchema down to 1280-d columns. The new test files in this PR reshuffled shard bin-packing and exposed the trap. - engine-find-trajectory: pin OpenAI/1536 explicitly before initSchema (the pattern bunfig's preload documents) — deterministic regardless of neighbors - facts-extract-silent-no-op, diagnose-embedding-dims, embed-preflight: restore the legacy 1536 pin in afterAll instead of ending reset/dimensionless Reproduced: synthetic dimensionless-gateway file + old victim = the exact 7 CI failures; with the pin = 0. Verified in-process pair runs both orders. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-10 10:05:23 +09:00
+111 people	7c27fa129b	v0.42.41.0 fix: triage wave — 6 data-loss/availability fixes + 9 community PRs (#2128) ... * fix(oauth): default omitted authorize scope to client's full grant When a client omits `scope` on /authorize, the authorize() grant computed `(params.scopes || []).filter(...)` → the empty set. That empty grant was written to oauth_codes and propagated into the access AND refresh tokens, so every request failed `insufficient_scope` even though the client was registered with e.g. `read write`. Because refresh inherits the stored grant, it never self-healed — reconnecting just minted another empty-scoped token. Some MCP connectors (observed with Claude Desktop) omit `scope` on /authorize, so they hit this on every connection. Fix: when no scope is requested, default to the client's full registered scope (RFC 6749 §3.3 permits a server default). This mirrors exchangeClientCredentials, which already does `requestedScope ? ... : allowedScopes`. The result is still clamped to the allowed set, so an explicit over-broad request cannot escalate. Adds test/oauth-authorize-scope-default.test.ts covering: omitted/empty → inherits full grant; explicit subset honored; clamp preserved (over-broad and disallowed-only requests cannot escalate or trigger inheritance). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): skip Python venv/ in the code walker collectSyncableFiles (first-sync walker) and the incremental PRUNE_DIR_NAMES set skipped node_modules but not Python venv/. On a Python repo the walker descended into venv/ (thousands of files); the resulting slug collisions crashed putPage's INSERT ... ON CONFLICT ... RETURNING with "undefined is not an object (evaluating 'row.deleted_at')". Add `venv` alongside node_modules in both the import.ts inline skip and PRUNE_DIR_NAMES. venv is the Python equivalent of node_modules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): carry asymmetric input_type across the AI SDK to the wire body (#1400) dimsProviderOptions() threads input_type ('query' | 'document') into providerOptions.openaiCompatible for asymmetric models (ZE zembed-1, Voyage v3+), but the AI SDK's openai-compatible adapter validates providerOptions against a fixed schema and silently drops the field before building the HTTP body. Every embedQuery() was therefore encoded document-side: the ZE shim's hard default fired ('document'), Voyage and local openai-compat servers got no input_type at all, and asymmetric retrieval silently collapsed toward surface-token overlap — while the providerOptions-level contract test stayed green. Fix: an AsyncLocalStorage (same pattern as __budgetStore) populated in embedSubBatch() only when providerOptions actually threads an input_type, read at body-rewrite time by the fetch shims: - zeroEntropyCompatFetch: recovers the threaded value; document default preserved for ingest paths. - voyageCompatFetch: opt-in like the dims.ts Voyage branch — inject only when threaded; the field stays off the wire otherwise. - NEW openAICompatAsymmetricFetch: fallthrough default for every other openai-compatible recipe (llama-server, litellm, ollama, ...) — the canonical local/proxy paths for asymmetric models. Strict pass-through when nothing was threaded, so symmetric deployments see zero wire change; recipes with their own compat fetch (azure) keep it via the compat.fetch ?? precedence. KNOBS_HASH_VERSION bumped 10→11: cached query_cache rows were keyed on document-side query vectors; pre-fix rows must not be served to post-fix lookups (same convention as the v=3 embedding-provider bump). One-time global cold-miss on upgrade; refills within cache.ttl_seconds. Tests: test/embed-input-type-wire.test.ts runs the REAL SDK transport with a mocked global fetch and asserts on the outbound body — the only layer where this regression is observable. Covers ZE hosted, llama-server, litellm, ollama (query + document sides) and pins the pass-through for non-asymmetric models and Voyage's opt-in shape. 4 of the original 7 assertions fail on master, proving the pin. One structural pin in test/ai/zeroentropy-compat-fetch.test.ts updated to the new line shape (same semantic); KEY_FILES.md gateway.ts entry updated to the new truth. Supersedes #1400 (closed unmerged) — same ALS mechanism, extended to Voyage + all openai-compatible recipes. Credit to @billy-armstrong for the original diagnosis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sync): honor .gitignore in code walk; prune vendor/dist/build collectSyncableFiles (the full-sync / dry-run enumerator) reimplemented its own directory skip list inline (node_modules || ops), bypassing the canonical pruneDir gate and ignoring .gitignore entirely. On a Laravel/PHP repo this descended into vendor/ (~50k Composer files), storage/, and public/build/, trying to import 52k dependency/build files and flooding the index with library internals (a 35-min sync that never finished, killed by the watchdog at 3%). - collectSyncableFiles now enumerates via `git ls-files --cached --others --exclude-standard` when dir is a git work tree, so the walk honors .gitignore (tracked + untracked-not-ignored). Falls back to the FS walk for non-git dirs. EroLab: 52164 -> 1028 files. - The FS fallback now prunes through the canonical pruneDir() instead of a drifted inline list, so the two skip lists can't diverge again. - PRUNE_DIR_NAMES gains vendor/dist/build (dependency + build-output trees). Addresses #1483 (.gbrainignore), #1159 (--respect-gitignore), and the maintainer's #1942 vendor/dist/build prune. Walker regression suites (sync-walker-symlink, brain-writer-walk-prune, sync, sync-walker-submodule) green: 90 pass. * fix(config): ignore DATABASE_URL auto-loaded from cwd .env (#427) Bun merges .env files from the process cwd into process.env before any user code runs. loadConfig() prefers env DATABASE_URL over ~/.gbrain/config.json, so any gbrain invocation from inside a web-app checkout silently retargets the brain at that app's database — reads go to the wrong DB and apply-migrations can write gbrain's schema into a production app database (#427). effectiveEnvDatabaseUrl() re-parses the .env files Bun auto-loads from cwd and treats a DATABASE_URL whose value matches one of them as file-origin: ignored, with a one-time stderr notice. GBRAIN_DATABASE_URL and genuinely exported DATABASE_URLs are honored unchanged, so the operator escape hatch and the e2e suite's env-provided URL keep working. Applied at loadConfig, getDbUrlSource (doctor parity), init --non-interactive, and migrate --to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): arm the disconnect hard-deadline at teardown entry, not before the op body The 10s force-exit timer in the shared-op dispatch was armed BEFORE the try block, so any op whose handler ran past 10s wall-clock was killed mid-flight with process.exit(0) and zero stdout. On a slow Postgres pooler (6-10s per fresh connection) a healthy `gbrain search` was force-exited every time — an empty 'success' indistinguishable from no results. The v0.42.20.0 exitCode honor can't help: a mid-op kill fires before any error path sets exitCode. Move the arming into the finally (teardown entry), matching the fall-through owner-disconnect site later in main(): the timer still bounds a hung drain/disconnect (the C13 contract) but can no longer kill a slow-but-progressing op. Verified on a transaction-pooler Supabase brain: search went from 0 bytes/exit 0 at 10s to real results at ~21s. * fix(import): stamp source_id on extracted call-graph edges importCodeFile built CodeEdgeInput rows without source_id, so every edge landed NULL. getCallersOf/getCalleesOf filter `AND source_id = <scoped>` whenever a worktree pin or --source is in play — NULL never matches, so scoped call-graph queries silently returned 0 rows on multi-source brains even though the edges existed (2,122 edges, 26 targeting the probed symbol, count 0 returned). One-line fix: carry the sourceId already in scope into the edge input. Existing NULL rows backfill with: UPDATE code_edges_symbol e SET source_id = p.source_id FROM content_chunks c JOIN pages p ON p.id = c.page_id WHERE c.id = e.from_chunk_id AND e.source_id IS NULL; (same for code_edges_chunk). Verified: code-callers returns 21 callers where it returned 0. * docs(migrations): NULL embeddings BEFORE the column-type alter The Postgres recipe ordered ALTER COLUMN TYPE vector(N) before the UPDATE that clears stale embeddings. pgvector refuses to cast existing vectors across dimensions ('expected 1024 dimensions, not 1536'), so the recipe as written aborts the transaction on any brain that has embeddings — which is every brain doing this migration. Swap the steps: NULLs cast fine. * fix: honor legacy token source grants in oauth * fix(cli): bound read-scope op handlers at 180s wallclock (pre-landing review) With the hard-deadline timer correctly scoped to teardown, a genuinely wedged read handler (hung pooler connection mid-query) would hang the CLI forever — the #1633 zombie class the old pre-try timer accidentally bounded at 10s. Reads now get a generous withTimeout (180s default, far above any healthy slow-pooler run; --timeout=Ns overrides; exit 124 with the teardown finally still draining + disconnecting). Writes/admin stay unbounded: a long import/embed must never be killed by a default. * fix(import): stamp unscoped edges 'default', matching the pages-table default Review catch: 'sourceId ?? null' fixed the scoped path but left the unscoped one (reindex --code without --source, importCodeFile callers without opts.sourceId) stranding edges at NULL while their pages land under the schema default (pages.source_id DEFAULT 'default') — so getCallersOf(sym, { sourceId: 'default' }) missed them. Same bug, other door. Fallback is now 'default'. * fix(core): runtime dim-migration recipe NULLs embeddings before the alter Review catch: the doc fix corrected docs/embedding-migrations.md, but embeddingMismatchMessage still PRINTED the broken order — ALTER before UPDATE ... SET embedding = NULL — and linked to the now-contradicting doc. pgvector refuses to cast existing vectors across dimensions, so the printed recipe aborted on any brain that has embeddings. Swap the steps and say why inline. * feat(migrate): v116 — backfill NULL edge source_id + index from_symbol_qualified 1. Backfill: edges written before the stamping fix sit at source_id=NULL and stay invisible to scoped call-graph queries until repaired. Derive each edge's source from its own from_chunk's page (pages.source_id is NOT NULL DEFAULT 'default'). Same SQL verified live on a 2,122-edge production brain. 2. Indexes: getCalleesOf filters both edge tables on from_symbol_qualified, which had no index — every callee lookup was a seq scan, amplified per-BFS-node by the recursive code walk. With NULL edges repaired, scoped walks actually expand, so the latent cost becomes real. Mirrored into src/schema.sql; schema-embedded.ts regenerated. * docs(migrations): align the rationale list with the corrected recipe order The 'Why we don't do this automatically' list still said alter-then-wipe; reorder to wipe-then-alter and replace the fragile 'step 3' numeric cross-reference with a name-based one. * test: regression coverage for edge source_id stamping, timer placement, recipe order - import-code-edges-source-id: scoped import stamps edges + scoped getCallersOf/getCalleesOf match (verified failing pre-fix), plus the unscoped-import case asserting 'default' stamping. - cli-force-exit-teardown-arming: structural pin — the hard-deadline timer arms inside the finally (teardown entry), never before the op body; daemon guard, unref, clearTimeout intact. - embedding-dim-check: recipe order pinned — UPDATE precedes ALTER so the printed SQL can't drift from docs/embedding-migrations.md again. * fix(cli): hard-exit after teardown on wallclock timeout; bound makeContext too Adversarial review, two findings on the new timeout path: 1. On timeout the finally drained, disconnected, then CLEARED the hard-deadline timer — removing the only backstop while the abandoned handler (withTimeout races, it does not cancel) can hold ref'd sockets/SDK timers that keep Bun's loop alive: 'timed out' printed, process immortal — the zombie class this branch exists to kill, resurrected through its own fix. The finally now exits explicitly after teardown completes on the timeout path. 2. makeContext does DB I/O (resolveSourceId) for EVERY op and sat outside any bound — a pooler wedge at context build hung reads, writes, and admin alike. It now shares the same wallclock bound. * fix(import): normalize edge source once — closes the '' door and the unscoped chunk fan-out Adversarial review: txOpts used truthiness while the edge stamp used nullish — sourceId:'' put pages under 'default' but stamped edges '', FK-violating against sources(id) and silently dropping the file's whole call graph in the best-effort catch. The unscoped getChunks could also fan out to same-slug chunks from another source. One normalized edgeSourceId (sourceId || 'default') now drives both the chunk lookup and the stamp. * fix(engine): default edge source_id to 'default' at the insert layer (both engines) Adversarial review: addCodeEdges still wrote e.source_id ?? null, so any future caller that forgets the field reintroduces invisible NULL edges the day after the v116 backfill runs. A NULL source_id is invisible to every scoped call-graph query; default to the schema-default source the way the pages table does. Applied to both engines (parity). * fix(core): facts alter recipe NULLs embeddings before cross-dimension alters Adversarial review: buildFactsAlterRecipe shipped the same defect class this branch fixes for content_chunks 350 lines up — a cross-dimension ALTER ... USING cast that pgvector refuses while rows hold old-width vectors. Dimension changes now wipe first (the facts pipeline re-embeds on next write); same-dim type swaps (halfvec <-> vector) keep the lossless cast and PRESERVE data. Both behaviors pinned by tests. * v0.42.39.0 chore: version bump + CHANGELOG + TODOS Marks the v0.42.20.0 'decouple the op-dispatch force-exit timer' follow-up complete — this branch ships exactly that decoupling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(postgres-engine): atomic JSONB merge in updateSourceConfig — eliminate lost-update race ## Problem `updateSourceConfig` used a read-then-write pattern: read the current `config` row, normalize it in JavaScript, then write the merged result back with `SET config = <normalized> || <patch>`. Under concurrent callers (two background autopilot/cycle paths patching different keys simultaneously), both callers can read the same stale row. The later `SET config = ...` then clobbers the earlier patch, silently dropping whatever keys the first caller wrote. Reproduced at 21/25 lost-update events under real Postgres with parallel callers. ## Fix Fold the normalization and merge into a single atomic `UPDATE … SET config = CASE … END || patch` statement. Because the `SET` expression evaluates against the row-locked latest version of `config`, there is no snapshot window between the read and the write. Concurrent callers now converge correctly (50/50 clean in reproduction test). The `CASE` also normalizes historical bad JSONB shapes inline: - `object` — used as-is - `string` — double-encoded config; inner text parsed with the SQL `IS JSON` guard (Postgres 16+) so unparseable strings fall back to `{}` instead of raising `invalid input syntax for type json` - `array` — array of patch objects aggregated into a flat object via `jsonb_object_agg` - anything else — falls back to `{}` `pglite-engine.updateSourceConfig` already used an atomic `||` merge; this change brings postgres-engine to parity. ## Test Added two assertions to `test/list-all-sources.test.ts`: 1. JSONB string holding non-JSON text normalizes to `{}` (no cast throw) 2. JSONB string holding double-encoded valid JSON is parsed then merged * fix(doctor): five correctness fixes — stale locks, content sanity, graph coverage, exit code, gateway guard ## 1. Stale lock break hints cover gbrain-cycle: keys The doctor stale-lock report only recognized `gbrain-sync:` lock prefixes; everything else fell back to `gbrain sync --break-lock`, which is wrong for dream/autopilot cycle locks. A `gbrain-cycle:<source>` or `gbrain-cycle` lock now suggests `gbrain dream --break-lock [--source <name>]`, and unknown lock shapes fall back to `gbrain doctor` instead of a misleading sync command. ## 2. content_sanity_audit_recent counts reject and quarantine as hard failures v0.42 renamed the hard disposition path: rejected pages emit a `reject` event and quarantined junk pages emit `quarantine`; `hard_block` is now only the pre-v0.42 legacy alias. The status check only counted `hard_block`, so fresh `reject` / `quarantine` events from the new path cleared as `ok` whenever fewer than 10 events existed. The check now sums all three for the hard count, and `soft_block + flag` for the soft count. ## 3. graph_coverage excludes test fixture entity pages from the denominator Brains seeded with code sources (e.g. a sync of the gbrain repo itself) could accumulate test fixture pages typed as `entity` / `person`. Including these in the entity-count denominator diluted coverage and produced spurious warnings ("Entity link coverage 0%, timeline 0%") on knowledge-only brains with no real entity pages. The check now queries a per-entity stats CTE that excludes `tools/gbrain/test/*` slugs and the `templates/new-person` stub, with an additional guard for the all-fixture case (`eligibleEntityCount = 0`). ## 4. process.exitCode instead of process.exit at doctor main exit point `process.exit(hasFail ? 1 : 0)` was a hard kill that prevented cleanup handlers (Bun unload events, open DB connections) from running. Using `process.exitCode = hasFail ? 1 : 0` defers the actual termination until the end of the event loop, allowing cleanup to complete. ## 5. checkSubagentCapability exported for test seams + gateway loop guard The function was private, making it untestable in isolation. It is now exported. Additionally, users running gbrain with a non-Anthropic chat model via `agent.use_gateway_loop=true` no longer receive a spurious warning that `ANTHROPIC_API_KEY` is missing — subagents route via the gateway loop in that configuration and do not need the key directly. ## Tests Doctor test suite: 77 pass, 0 fail (no regressions). * fix(engine): deleteFactsForPage excludeSourcePrefixes (#1928) + reconnect() parity (#2034) Engine-layer API for two cycle/availability fixes that share these files: - deleteFactsForPage gains optional excludeSourcePrefixes so the fence reconcile can protect non-fence facts (e.g. cli: conversation facts). - reconnect(ctx?) is now a first-class BrainEngine method on both engines (PostgresEngine already had it; PGLite gains config capture + reconnect) so callers stop using disconnect()+bare connect(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cycle): stop extract_facts from wiping conversation facts (#1928) The fence reconcile delete-then-reinsert wiped cli:-origin facts (no fence to recreate them); a failed-sync full walk turned it brain-wide (1829 rows, 0 reinserted, status ok). Now: exclude cli: rows from the wipe, do NOT inherit the failed-sync->full-walk fallback for this destructive phase, and warn on net-negative reconcile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(autopilot,supervisor): reconnect() instead of disconnect()+bare connect() (#2034) The autopilot health-probe recovery called connect() with no args after disconnect(), losing the startup config (database_url undefined -> FATAL restart-loop on every DB blip) and opening a null-pool window. Both call sites now use engine.reconnect(), which restores the captured config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(write-through): mirror to the assigned source's local_path, never the global repo (#2018) put_page write-through resolved the disk target from the global sync.repo_path, so a default-source page (local_path NULL) got written into an unrelated federated source's working tree. Now it uses the assigned source's own local_path; NULL local_path skips (no leak); the global path is used only as a sole-source fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pglite-lock): heartbeat + steal-grace so live holders are never stolen (#2058) A live holder's lock was force-removed after 5min age alone, letting a second process share the single-writer data dir -> WAL corruption. The lock now heartbeats while held; a holder is reaped only when its PID is dead OR its heartbeat went stale past the steal grace. Pairs PID liveness with heartbeat age to also defeat PID reuse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(migrate,doctor): self-heal idx_timeline_dedup drift (#2038) A migration renumbered during a merge (v102) could be recorded-as-applied without its DDL running, leaving the 3-column index so every timeline write failed the 4-column ON CONFLICT. runMigrations now always runs a shape-keyed drift repair (dedupe-then-rebuild) even when no migration is pending, and doctor surfaces the drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(timeline): un-silence the swallowed batch catch; pin Date-batch round-trip (#2057) The meetings extractor's bare catch {} hid a brain-wide timeline-write failure (0 entries, no error). It now counts + surfaces batch errors. Adds a Date-bearing batch regression test proving the #1861 jsonb_to_recordset refactor already fixed the original ::text[] cast failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v0.42.41.0) Triage fix wave: 6 authored critical fixes (#1928 facts wipe, #2018 write-through leak, #2034 reconnect loop, #2058 WAL lock, #2038 timeline migration drift, #2057 timeline silent-empty) + community PRs #2064 #2052 #2020 #2033 #2074 #2075 #2009 #2072 #2073. TODOS: deferred #1994 #1963 #2050. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: address adversarial review findings (#1928, #2058, #2038, #2057) Codex as-built review of the authored fixes surfaced 4 real issues: - #2058: add a pid+acquired_at ownership token. A stale holder reaped + replaced past the grace must NOT let its resumed heartbeat refresh, nor releaseLock remove, the NEW owner's lock (re-opened the concurrent-writer hole). Heartbeat and release now verify the on-disk lock is still ours. + regression test. - #1928: the destructive-full-walk guard keyed off phases.includes('sync'), which wrongly suppressed a legitimate full reconcile when sync was SKIPPED (no engine / no brainDir). Key off a syncAttempted flag set only when sync actually ran. - #2038: dedupe keeps MIN(id) not MIN(ctid) — deterministic and consistent with the existing v-migration lower-id rule. - #2057: the extract CLI caller now surfaces batch_errors (stderr + exit 1) instead of printing a clean success over failed inserts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(key-files): sync reference to v0.42.41.0 triage-wave behavior Update KEY_FILES.md to current-state truth for the shipped fixes (no release-history clauses, per the reference-doc discipline): - write-through.ts (#2018): resolves the disk target from the assigned source's own local_path; sole-source falls back to sync.repo_path, multi-source skips with source_has_no_local_path rather than leak. - engine.ts (#2034): reconnect() is now a REQUIRED lifecycle method on both engines; config-restoring, never disconnect()+bare connect(). - migrate.ts (#2073): document v116 edge source_id backfill + callee index, and the always-run (version-counter-blind) timeline dedup self-heal. - new entry for timeline-dedup-repair.ts (#2038) + the timeline_dedup_index doctor check. - new entry for pglite-lock.ts (#2058): heartbeat + steal-grace (GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS) so a live holder is never stolen. - extract-facts.ts (#1928): cli:-fact protection, no failed-sync full-walk inheritance, net_fact_deletion warn floor. bun run build:llms re-run (KEY_FILES is link-only so bundles unchanged); freshness + current-state guards green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(write-through): preserve nested multi-source layout; narrow #2018 leak guard The first #2018 fix skipped any no-local_path source on a multi-source brain, which broke the legitimate nested layout (a source without its own tree nests under the host repo at .sources/<id>/ — pinned by put-page-write-through.test). Narrow the guard: a no-local_path source nests under sync.repo_path as before; only SKIP when sync.repo_path is literally another source's own local_path (the actual leak — writing there pollutes that sibling's repo). Caught by the sharded suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: satisfy test-isolation guard for the new lock/reconnect tests CI `verify` flagged 3 intra-process isolation violations in the tests added this wave (the parallel runner shares one process per shard): - pglite-lock.test.ts: the GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS mutation now goes through withEnv() instead of a raw process.env write (R1). - pglite-reconnect: renamed to *.serial.test.ts — it creates per-test engines to exercise the connect/reconnect lifecycle, which doesn't fit the shared beforeAll-engine model (R3/R4). verify is now 30/30; both files green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pglite): reconnect() is a no-op for in-memory engines (#2034) CI serial-tests + test(5) caught two in-branch regressions from the #2034 PGLite reconnect(): - worker/queue claim-error recovery + their renewLock e2e test assume PGLite reconnect is absent/no-op (queue.ts documents it). Making it a real disconnect+reopen wiped an in-memory engine's state mid-job. reconnect() now no-ops for in-memory (no database_path) — file-backed still re-opens the dir (state persists on disk). Restores the documented worker assumption. - connection-resilience 'Supervisor still has the 3-strikes-then-reconnect path' pinned the removed unsafe-cast text; updated to assert the direct this.engine.reconnect() call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: quarantine embed-input-type-wire to serial lane (CI test(5) leak) #2033's embed-input-type-wire.test.ts configures a 1280-dim embedding gateway; the active dimension survived into engine-find-trajectory when CI's 10-way hash-disjoint sharding co-located them (this branch's added files reshuffled the assignment), failing 7 trajectory tests with 'expected 1280 dimensions, not 1536'. resetGateway() in afterEach clears the gateway but the dimension still leaked. It mutates global gateway/embedding state, so it belongs in the serial lane (own bun process, true isolation) by the repo's own definition. Root-caused by reproducing the exact failing pair locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Austin Arnett <austin@sdsconsultinggroup.org> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Dave MacDonald <djmacdonald@ucdavis.edu> Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com> Co-authored-by: Alex P. <12667893+aphaiboon@users.noreply.github.com> Co-authored-by: Garry Tan <bo.m.liu@gmail.com> Co-authored-by: jbarol <barol.j@gmail.com> Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com> Co-authored-by: PAI <pai@scaffolde.ai>	2026-06-12 06:05:34 -07:00
3 people	ffac8ce0f4	v0.41.27.0 fix: withRetry self-heals on null singleton + facts:absorb drain + disconnect audit (closes #1570) (#1608) ... * merge master: rebump v0.41.25.0 → v0.41.27.0 (queue collision) Master shipped v0.41.25.0 (#1538 batched sync deletes) and v0.41.26.0 (#1571 dream --source fix) while this branch was in flight. Conflict resolution rebumps to the next available slot. - VERSION: 0.41.25.0 → 0.41.27.0 - package.json: synced - CHANGELOG.md: my v0.41.27.0 entry placed above master's v0.41.26.0 and v0.41.25.0; in-entry version references updated 0.41.25.0 → 0.41.27.0 and forward-references bumped to v0.41.28+. - TODOS.md: kept master's v0.41.20.x section + my v0.41.27.0+ follow-ups No source-file conflicts during the merge. * feat(diagnostics): db-disconnect audit + doctor surface (v0.41.27.0) Instruments every db.disconnect() and PostgresEngine.disconnect() call with a JSONL audit record so the next user-reported #1570 cycle gives us the offender's caller stack instead of the symptomatic "No database connection" error. Audit shape (~/.gbrain/audit/db-disconnect-YYYY-Www.jsonl): {ts, engine_kind, connection_style, caller_stack[], command, pid} - src/core/audit/db-disconnect-audit.ts (NEW): the audit writer, built on the v0.40.4.0 createAuditWriter cathedral. Captures a 6-frame stack via new Error().stack so the offender is readable without spending stderr noise. - src/core/db.ts: logDbDisconnect call at the top of disconnect() (best-effort; never blocks the real teardown). - src/core/postgres-engine.ts: same instrumentation in PostgresEngine.disconnect() — distinguishes 'module' vs 'instance' connection_style so we can tell legitimate worker-pool teardowns apart from the load-bearing module-singleton class. - src/commands/doctor.ts: extends batch_retry_health to surface 24h disconnect count + most-recent caller stack. Warns when the caller frame isn't a known CLI-exit frame (e.g. cli.ts's finally block at the end of an op-dispatch). This is the diagnostic that tells v0.41.28+ where to apply the real ownership fix. - test/db-disconnect-audit.test.ts: unit coverage for the audit writer + caller-stack capture + JSONL shape. - test/e2e/db-singleton-shared-recovery.test.ts: real-Postgres regression that exercises the singleton-null path end-to-end. Refs #1570 * feat(retry): self-heal on null singleton — closes #1570 symptom (v0.41.27.0) withRetry gains an opt-in reconnect callback that fires between the isRetryableConnError classification and the inter-attempt sleep. PostgresEngine.batchRetry injects this.reconnect() — race-safe via the existing _reconnecting guard, handles module and instance pools. Closes the production loss reported in #1570: dream cycles on Supabase no longer drop ~150 link rows per cycle when the singleton goes null mid-batch. The retry now rebuilds the connection between attempts so the second try has somewhere to write to. - src/core/retry.ts: WithRetryOpts gains `reconnect?: () => Promise<void>`. Awaited in the catch branch. onRetry is also now awaited (back-compat- safe: every existing in-tree caller is a sync arrow). Reconnect failures propagate as the real cause — replaces the symptomatic "No database connection" error with whatever the connect() throw was, so operators see the truth. - src/core/postgres-engine.ts:batchRetry — injects `reconnect: () => this.reconnect()`. Covers all 9 batch-retry call sites (addLinksBatch, addTimelineEntriesBatch, upsertChunks, plus the 6 caller-supplied auditSite labels in extract / sync / reindex). - test/core/retry-reconnect.test.ts: 8 hermetic cases pinning the contract — reconnect fires before sleep, only on retryable errors, back-compat when omitted, signal-aborted bypasses reconnect, onRetry is awaited, full success path end-to-end. The deeper bug (who's calling disconnect mid-cycle) is left unaddressed in this commit by design — the diagnostic instrumentation in the prior commit will tell us in the next production run. Refs #1570 * feat(facts): drainPending() + CLI await before disconnect (v0.41.27.0) Closes the silent 'No database connection' tail-end errors after gbrain capture / put_page: the facts:absorb fire-and-forget queue sometimes outlived the CLI process's connection lifetime, so absorb attempts after engine.disconnect() landed in stderr as the GBrainError shape. - src/core/facts/queue.ts: new drainPending({timeout: 1000}) method distinct from shutdown(). Stops accepting new enqueues, awaits in-flight settle, bounded by timeout, returns count of unfinished. Semantically different from shutdown() (which aborts in-flight) so the symptom — drop work that hasn't started yet but let in-flight work finish — matches what CLI exit actually needs. - src/cli.ts: op-dispatch finally block awaits the drain BEFORE engine.disconnect(). Bounded 1s. Opt-out env GBRAIN_NO_FACTS_DRAIN for callers that don't enqueue (keeps fast-exit paths fast). Mirrors the v0.41.8.0 awaitPendingLastRetrievedWrites pattern. - test/facts-queue-drain-pending.test.ts: 6 hermetic cases — empty drain returns immediately, single in-flight settles, timeout bounds wait, shutdown-after-drain is idempotent, post-drain enqueues are dropped, signal-aborted skips waiting. Refs #1570 * docs: update project documentation for v0.41.27.0 README.md: added troubleshooting entry for the v0.41.27.0 retry-reconnect + facts:absorb drain fix (closes #1570), pointing operators at `gbrain doctor --json` to find the offending disconnect caller. CLAUDE.md: extended `src/core/retry.ts` entry with the new optional `reconnect` callback (v0.41.27.0); added two new Key Files entries for `src/core/audit/db-disconnect-audit.ts` (the diagnostic half of the "instrument first, fix later" pivot) and `FactsQueue.drainPending`; extended `doctor.ts:checkBatchRetryHealth` entry with the in-place extension that surfaces 24h disconnect-call count. llms-full.txt: regenerated to absorb CLAUDE.md edits. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: rebump v0.41.27.0 → v0.41.28.0 (queue collision with #1573) Master shipped v0.41.27.0 (#1573 git-aware sync_freshness) claiming the same slot. Rebump to the next available version. - VERSION + package.json → 0.41.28.0 - CHANGELOG.md: my entry header + in-entry refs 0.41.27.0 → 0.41.28.0 - TODOS.md: my #1570 follow-up section header + body refs bumped * test: pin gateway in put-page-provenance + embedding-dim-check (CI shard fix) Both files failed on CI shards 1 and 8 under the cross-file gateway-state leak class (CLAUDE.md "Test-isolation lint and helpers"). The v0.41.28.0 merge reshuffled the weight-based shard bin-packing, landing a gateway-mutating sibling ahead of these two victims in the same `bun test` process. Mechanism: - put-page-provenance: put_page embeds via the gateway. A sibling left the gateway configured with OpenAI + the CI placeholder `sk-test` (captured at configureGateway time, survives the withEnv restore as cached gateway state). put_page's embed then fired against live OpenAI and 401'd. The bunfig legacy-embedding preload's beforeEach only re-applies legacy when the gateway was RESET — it does NOT correct a sibling that configured a different LIVE config. - embedding-dim-check: initSchema builds the content_chunks vector column at the gateway's configured dim. A sibling leaking ZE/1280 made the column 1280-d, so `expect(dims).toBe(1536)` failed. Fix (victim-side pinning, the escape hatch the preload documents): - Both: configure the gateway explicitly in beforeAll BEFORE initSchema (OpenAI/1536), resetGateway() in afterAll so neither leaks onward. - put-page-provenance also stubs the embed transport via __setEmbedTransportForTests so embed is deterministic and offline; a dummy OPENAI_API_KEY is supplied in the gateway env because instantiateEmbedding builds the OpenAI client (key check) BEFORE the stubbed transport is reached — the stub then intercepts the actual call so the key never leaves the process. Verified: CI shards 1 (1337 pass) + 8 (905 pass) green with OPENAI_API_KEY unset, plus adversarial sibling orderings (gateway.test / doctor-ze-checks preceding). Typecheck + check-test-isolation clean. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-28 19:04:48 -07:00
3 people	d0d0e2a64a	v0.37.11.0: fresh-install PGLite embedding setup fix wave (#1286) ... * chore(test): preload gateway to OpenAI/1536 so 1536-dim test fixtures keep working The v0.37 fix wave changes the canonical gateway defaults to zeroentropyai:zembed-1 / 1280 (matching what v0.36 already chose as the system default). 20+ test files have hardcoded new Float32Array(1536) fixtures that match the OLD schema default. Without this preload, those tests fail with a vector-dim-mismatch on insert. The preload is gateway-only — it doesn't change which model gbrain ships to production users. Tests that want the new ZE/1280 defaults call configureGateway() explicitly in their own beforeAll. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai): canonical embedding defaults + sweep across schema/engines/registry Closes the v0.36 defaults drift bug class. The gateway shipped zeroentropyai:zembed-1 / 1280 as the system default in v0.36 but eight other places kept hardcoding 1536 / text-embedding-3-large. Fresh gbrain init --pglite sized the column to 1536, the embed pipeline used ZE/1280, and every page failed with dim mismatch. - New src/core/ai/defaults.ts leaf module is the canonical source for DEFAULT_EMBEDDING_MODEL / DEFAULT_EMBEDDING_DIMENSIONS. Schema and registry helpers import from this lean module instead of pulling the full gateway (which loads every provider SDK). - src/core/ai/gateway.ts re-exports the constants for back-compat. - src/core/pglite-schema.ts getPGLiteSchema() defaults track gateway. - src/core/postgres-engine.ts getPostgresSchema() default args track gateway (same drift on the Postgres path — codex round 1 CDX-1). - Both engine.initSchema() fallbacks track gateway constants (no more stale OpenAI/1536 catch-block defaults). - Schema seed stops stripping the provider prefix; full provider:model is stored in the DB config table (codex round 1 CDX-4). - Chunk-row INSERT defaults track gateway (codex round 2 CDX2-4 — pglite-engine:1611 + postgres-engine:1647 were production write sites previously hardcoded to text-embedding-3-large). - src/core/search/embedding-column.ts loadRegistry + isCacheSafe gain the cfg > gateway > DEFAULT resolution chain (codex round 2 CDX2-3). The gateway tier matters because callers that configure the gateway (init paths, tests, programmatic SDK) expect the registry to mirror that state when cfg doesn't have an explicit embedding_model. Tests: - schema-templating: default expectation flips to ZE/1280 (v0.37 truth). - embedding-dim-check: 3 new engine-kind branching cases + updated fresh-brain expectation (under legacy preload). - embedding-column: registry + isCacheSafe expectations match new chain. - v0_28_5-fix-wave E2E: engineKind required arg propagated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(init+config+cli): always-configure gateway, file-only loader, honest config-set, sync/reinit help Closes the "fresh init doesn't work + config-set silently lies" bug class end-to-end. Six related changes that ship together because the file-plane/DB-plane contract only holds when init paths, config-set, the gateway env mapping, and the recipe text all agree. Lane B (init paths): - initPGLite, initPostgres, initMigrateOnly always configureGateway() before engine.initSchema(). Pre-fix the call was gated on flags, so bare `gbrain init --pglite` left the gateway unconfigured and the engine fell through to stale OpenAI/1536 defaults instead of the ZE/1280 the gateway would have resolved. - New configureGatewayWithMergedPrecedence() helper applies the locked precedence chain `CLI > env > existing file > gateway internal`. - printResolvedAIChoice() shows the resolved model/dim at init time + surfaces a ZE setup hint inline when the API key is missing. - B.4: saveConfig merge uses loadConfigFileOnly() so transient env state (DATABASE_URL, etc.) never poisons ~/.gbrain/config.json (codex round 2 CDX-5). - B.5: extend the v0.28.5 dim-mismatch detector so it fires when the gateway-resolved dim differs from the existing column, not only when --embedding-dimensions is explicit (codex round 2 CDX-6). Lane C (config plane): - New `loadConfigFileOnly()` reads ~/.gbrain/config.json only — no env merge, no DATABASE_URL inference. Safe write-back source for init. - GBrainConfig gains `zeroentropy_api_key?: string`. loadConfig merges process.env.ZEROENTROPY_API_KEY. buildGatewayConfig at cli.ts:1401 maps it into env.ZEROENTROPY_API_KEY so ZE recipes finally see it (codex round 2 CDX2-5+6 — the v1 fix landed in the wrong file). - `gbrain config set embedding_model` and `... embedding_dimensions` refuse unconditionally and print a paste-ready wipe-and-reinit recipe. No --force escape (codex round 2 CDX2-13). - migrate-engine.ts adds a contract comment at the DB-plane write site documenting "DB stores schema-applied metadata; file plane is canonical for runtime gateway config" + preserves the existing file-plane config across engine migration. Lane D.1 (recipe text): - embeddingMismatchMessage() takes an `engineKind` arg. PGLite branch emits a wipe-and-reinit recipe using gbrainPath('brain.pglite') or the caller's databasePath override. Postgres branch keeps the SQL ALTER recipe. - The PGLite recipe recommends `gbrain reinit-pglite` (new sugar command below) as the one-line path before falling back to the by-hand mv + init + sync sequence. Lane D.4 (sync help dispatch): - `sync` and `reinit-pglite` added to CLI_ONLY_SELF_HELP so their own --help branches reach the user (pre-fix the generic short-circuit fired first and the dedicated usage was unreachable; codex round 2 CDX2-12). - `gbrain sync --help` short-circuits BEFORE engine bind so users on a fresh tmpdir (no config) can read the help without hitting no-such-config errors. Sugar: - New `gbrain reinit-pglite --embedding-model X --embedding-dimensions N` wraps the wipe + init + sync dance into one command. Backs up the brain to <path>.bak. TTY confirmation unless --yes. --no-sync to defer the resync. --json for scripts. Tests: - test/cli.test.ts sync-help test rewritten for the new per-command-usage output (lists --no-embed which is the v0.37 user-visible flag the wave wanted to surface). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(embed+sync): pre-flight dim-mismatch guard + sync hint at both catch sites embedding-pipeline error UX. Pre-fix, a fresh-install dim mismatch produced raw Postgres "expected N dimensions, not M" errors page after page, surfacing only after the worker pool drained the entire corpus. Sync swallowed embed errors at TWO catch sites and never surfaced the recovery recipe. embed.ts: - New `EmbeddingDimMismatchError` tagged class with the paste-ready recipe baked in. - `runEmbedCore` pre-flights via `readContentChunksEmbeddingDim` + gateway.getEmbeddingDimensions() before the worker pool spins up. On mismatch, throws the typed error which the CLI wrapper catches and prints. Dry-run skips the check (no embed risk). - Catches the headline fresh-install bug class at first call instead of letting it hammer N parallel API calls into dim-rejected inserts. sync.ts: - Both embed catches at sync.ts:990 (incremental) and sync.ts:1129 (first-sync) detect EmbeddingDimMismatchError and surface the recipe + a `--no-embed` tip on stderr (codex round 2 CDX2-8: incremental path was previously silent; only the first-sync path was flagged). - Non-mismatch embed failures still stay best-effort (rate limits, transient network) — those shouldn't break sync. - Sync calls runEmbedCore directly instead of runEmbed (which calls process.exit on error and bypasses sync's catch). - Sync gets a proper --help block listing every meaningful flag: --no-embed, --workers, --source, --skip-failed, --retry-failed, --watch, --interval, --no-pull, --all, --json, --yes, --dry-run. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(doctor): read gateway for schema-sizing checks + provider-aware key lookup Doctor's embedding checks were reading the DB config table for embedding_model / embedding_dimensions / zeroentropy_api_key. Post v0.37 the file plane is canonical (the DB plane is schema-applied metadata, not runtime gateway config) so those reads produced stale verdicts on fresh installs whose DB row hadn't been written. - checkEmbeddingWidthConsistency reads gateway.getEmbeddingDimensions() and gateway.getEmbeddingModel() instead of engine.getConfig(...). Reuses readContentChunksEmbeddingDim from the same shared helper init + embed use. On mismatch, the fix hint threads engineKind + databasePath into the new branched recipe (codex round 1 CDX-8 + Lane E.1/E.2). - checkZeEmbeddingHealth reads gateway for the model + loadConfigFileOnly for the key. Fires when (a) resolved model starts with zeroentropyai: AND (b) ZEROENTROPY_API_KEY is unset in env AND (c) file plane has no zeroentropy_api_key (codex round 2 CDX2-10). - loadRecommendationContext reads gateway for both fields and recognizes the ZE key alongside OpenAI/Anthropic in the hasEmbeddingApiKey check, so brains on ZE no longer look "healthy" just because OPENAI_API_KEY happens to be set (codex round 2 CDX2-11). Tests rewritten for the gateway-source-of-truth contract via configureGateway() in beforeAll. Added a "gateway unconfigured: skips with ok" case so doctor doesn't false-warn on cold-boot brains. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test+docs(v0.37): fix-wave unit coverage + PGLite-first migration recipe + TODOS Lands the v0.37 PGLite fresh-install fix wave's structural tests and the user-facing migration recipe overhaul. test/v0_37_fix_wave.test.ts (new): 22 unit cases pinning the lanes: - Lane A: defaults module exports, getPGLiteSchema/getPostgresSchema default-args, registry + isCacheSafe under the `cfg > gateway > DEFAULT` chain (both gateway-set and gateway-reset branches). - Lane B: loadConfigFileOnly env isolation + DATABASE_URL inference refusal + null-on-missing. - Lane C.3: buildGatewayConfig maps zeroentropy_api_key + process.env wins over config (operator escape hatch contract). - Lane D.2: EmbeddingDimMismatchError shape + tag. - Lane D.4: structural assertion that `sync` is in CLI_ONLY_SELF_HELP. - Deferred-TODO ship: reinit-pglite is registered correctly + embeddingMismatchMessage PGLite branch recommends it. docs/embedding-migrations.md: PGLite section moved to top (the default install). The recommended path is `gbrain reinit-pglite` one-liner; the by-hand mv + init + sync sequence stays as the fallback recipe. Postgres SQL ALTER recipe preserved. New section on `gbrain config set` refusal explains the file-plane vs DB-plane contract so users don't follow stale documentation. TODOS.md: 4 deferred follow-ups filed with concrete file pointers: - gbrain embed --try-fallback (provider auto-switch with consent gate) - Full plane unification for non-schema-sizing fields - Worker-pool shared AbortController for mid-run dim drift - Cleanup of back-compat constants in src/core/embedding.ts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(v0.37): fill behavior gaps + headline fresh-install E2E The structural fix-wave tests in test/v0_37_fix_wave.test.ts pin lane-level invariants (exports, registry chain, signature shapes). The audit found 10+ END-TO-END behaviors that the structural tests didn't actually reach. This file fills the highest-leverage gaps. Unit coverage (test/v0_37_gap_fill.test.ts, 12 cases): - Lane A.7: chunk-row INSERT default tracks DEFAULT_EMBEDDING_MODEL constant (pre-fix this was the literal 'text-embedding-3-large' at pglite-engine.ts:1611 + postgres-engine.ts:1647 — production write sites that were never directly tested; codex round 2 CDX2-4). - Lane A.8: schema seed stores full provider:model in DB config (pre-fix the .split(':') strip dropped the prefix; codex round 1 CDX-4). Asserts a fresh ZE init stores `zeroentropyai:zembed-1` in the config table, not bare `zembed-1`. - Lane B precedence: explicit CLI > env > existing file > default test (codex round 2 CDX2-7 contradiction guard). - Lane C.3 env merge: process.env.ZEROENTROPY_API_KEY threads through loadConfig → cfg.zeroentropy_api_key; loadConfigFileOnly does NOT. - Lane D.2 end-to-end: schema=1536 + gateway=1280 → EmbeddingDimMismatchError fires AND the embed transport is never called (the whole point of pre-flight). Plus dry-run skips the check. - Lane D.3 source-text grep: both sync.ts catch sites detect the typed error + the `--no-embed` tip is present (CDX2-8). - Lane E.4 source-text grep: loadRecommendationContext is provider-aware (reads gateway + branches on ZE/OpenAI key). - reinit-pglite contract: refuses on non-PGLite engines + refuses when required flags are missing. E2E (test/e2e/fresh-install-pglite.test.ts, 2 cases): - Bare `gbrain init --pglite` produces a `vector(1280)` schema, prints the resolved choice, persists defaults to config.json — the headline scenario that v0.37 ships to fix. - init → seed page → embed end-to-end: chunks have non-null embeddings; no dim mismatch despite the wave's defaults change. Both E2E cases are IN-PROCESS (per CDX2-12: CLI-subprocess E2E can't inherit `__setEmbedTransportForTests`). They run with stubbed transport returning synthetic 1280-dim vectors so we never hit real provider APIs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(v0.37): defensive gateway restore in reinit-pglite describe block Adds an afterAll that restores the gateway to OpenAI/1536 (matching the bunfig preload) at the end of the reinit-pglite describe. Belt-and- suspenders: earlier describe blocks in this file already restore, but if the reinit-pglite tests ever start mutating the gateway in the future, this protects downstream test files in the same bun-test shard from inheriting a non-default state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.37.10.0) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs: scrub stale config-set recipes for embedding model (v0.37.10.0) README + topologies + embedding-providers were still pointing users at `gbrain config set embedding_model X` / `embedding_dimensions N`. As of v0.37.10.0 those writes are refused — the schema column has to resize alongside the config. Point at `gbrain reinit-pglite` (PGLite) and the SQL recipe in `docs/embedding-migrations.md` (Postgres) instead. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: bump version to v0.37.11.0 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: quarantine v0.37 fix-wave tests to .serial.test.ts CI's `check:test-isolation` lint flagged R1 violations (direct `process.env.GBRAIN_HOME` mutation) in both new fix-wave test files. Per the documented quarantine pattern in CLAUDE.md, rename to `*.serial.test.ts` instead of refactoring through `withEnv()` — both files use beforeEach/afterEach env wiring that's already serial-safe. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 22:11:00 -07:00
3 people	a55de71221	v0.37.10.0 feat(init): env-detection + interactive picker + preflight invariants (#1278) ... * feat(core): Levenshtein helper + preflight schema-dim resolvers Foundation for v0.37.10.0 env-detection wave. Two pure modules: - src/core/levenshtein.ts: editDistance(a,b) + suggestNearest(input, candidates, maxDistance). Used by config-set "did you mean" suggestions and env-var typo detection at init. - src/core/embedding-dim-check.ts: resolveSchemaEmbeddingDim() + resolveSchemaMultimodalDim() pure functions. Validate resolved dim against recipe default_dims + per-provider Matryoshka allow-lists (OpenAI text-3, Voyage flexible-dim, ZeroEntropy zembed-1) BEFORE any DB write. Plus EmbeddingDisabledError + assertEmbeddingEnabled() runtime guard for the deferred-setup path (D9). New PGVECTOR_COLUMN_MAX_DIMS=16000 exported. Tests: 41 unit cases across both modules. * feat(providers): extract formatRecipeTable + add init provider picker Two changes prepping the env-detection wave: - providers.ts: extract formatRecipeTable() helper from runList(). Picker reuses it so UI can't drift from \`gbrain providers list\`. Also adds the codex finding #10 warn-line to \`providers test\` when the tested model differs from the configured default ("Note: tested X in isolation; gbrain's configured embedding is Y — this test does NOT verify your brain's active path."). envReady() takes an explicit env arg for testing. - init-provider-picker.ts (NEW): interactive picker mirroring init-mode-picker.ts. Filters candidate recipes to env-ready ones (codex finding #3), prompts via readLineSafe, exports printSubagentAnthropicCaveat() for shared use from initPGLite/initPostgres. Tests: 17 unit cases (10 providers + 7 picker). * feat(config): embedding_disabled sentinel + strict unknown-key rejection Two changes for the v0.37.10.0 wave: - src/core/config.ts: add embedding_disabled?:boolean to GBrainConfig (D9 deferred-setup sentinel, mutually exclusive with embedding_model). Export KNOWN_CONFIG_KEYS (60+ canonical keys, file-plane + DB-plane) and KNOWN_CONFIG_KEY_PREFIXES (search., models., dream., cycle., etc.) for validation use. - src/commands/config.ts: D6 strict-default unknown-key rejection. Unknown key + no --force → exit 1 with Levenshtein suggestion against KNOWN_CONFIG_KEYS. Prefix matches accepted without --force. --force escape hatch accepts arbitrary keys with stderr WARN. Closes the silent-no-op class the bug reporter hit (embedding.provider, embedding.model, embedding.dimensions all exit 1 with right suggestion). Tests: 19 unit cases pinning the bug-reporter regression + gate logic. * feat(init): env-detection auto-pick + preflight + atomic persist + --no-embedding Core of the v0.37.10.0 wave (D1-D7, D9-D11). Closes the bug where a fresh \`gbrain init --pglite\` silently produced a broken brain when no provider key matched the v0.36 default. resolveAIOptions rewritten with per-touchpoint env detection: - Explicit flag → shorthand → env auto-pick (group by provider id, codex #2) - Picker fires when multiple providers env-ready (D1+D2 hybrid) - Non-TTY zero-key exits 1 with paste-ready setup hint (D3) + Levenshtein typo detection for OPENAPI_API_KEY → OPENAI_API_KEY (D13) - All three touchpoints covered (embedding + expansion + chat, D4) - Local-only providers (Ollama/llama-server) excluded from auto-pick; picking Ollama silently when user has OPENAI_API_KEY set was wrong UX initPGLite + initPostgres: - Drop conditional configureGateway gate → always call before initSchema - Preflight resolveSchemaEmbeddingDim() BEFORE engine.initSchema() (D11) — invalid dim refuses with paste-ready hint, no disk write - Atomic embedding-config persistence (codex #13): either resolved tuple or embedding_disabled:true sentinel, never partial state - Post-initSchema invariant assertion stays as regression guardrail - --no-embedding opt-in flag (D9) for deferred-setup mode - Subagent-Anthropic caveat (D7) fires post-init when chat_model is non-Anthropic AND ANTHROPIC_API_KEY missing Exported groupReadyByProvider() + findEnvKeyTypos() for unit testing. Tests: 21 unit cases covering provider grouping + typo detection edge cases. * feat(embed,import): refuse cleanly when --no-embedding deferred-setup is active T7 of the v0.37.10.0 wave. Both runEmbedCore and runImport now call assertEmbeddingEnabled(loadConfig()) at entry. When the brain was init'd with --no-embedding (config has embedding_disabled:true), they exit 1 with a paste-ready hint: gbrain config set embedding_model <provider>:<model> gbrain config set embedding_dimensions <N> gbrain init --force --embedding-model <provider>:<model> \`gbrain import --no-embed\` flag still works (chunks land without vectors), so users can still ingest in deferred-setup mode and backfill embeddings later with \`gbrain embed --stale\`. * feat(doctor): empty-config drift detection + subagent-Anthropic caveat extension Two doctor check extensions for v0.37.10.0: T9 — embedding_provider check extended for the v0.36 silent-default repair case. When config is empty AND schema column dim differs from the gateway-resolved default, surface the mismatch with empty-brain vs non-empty-brain repair branching (codex finding #7 nuance): - Empty brain (0 embedded chunks) → \`gbrain init --force --pglite --embedding-model <id> --embedding-dimensions <N>\` (drop and re-init) - Non-empty brain → \`gbrain retrieval-upgrade --to <id> --reindex\` Gated on totalChunks > 0 so pristine empty brains aren't pre-warned. Never recommend rm -rf ~/.gbrain. T10 — subagent_provider check (v0.31.12) extended per D7. When chat_model is non-Anthropic AND ANTHROPIC_API_KEY is missing, warn that subagent features (gbrain dream, gbrain agent run, gbrain autopilot) will fail at job submission. Chat alone (gbrain think) still works. * feat(reindex,test): multimodal preflight + E2E suite for fresh PGLite init T11 — reindex-multimodal.ts: hook resolveSchemaMultimodalDim() preflight BEFORE the reindex sweep. Mirrors the text-side contract from initPGLite — if the configured multimodal model can't produce a dim matching the schema column, fail loud here with a \`gbrain config set\` hint rather than mid-reindex with a vector(N) INSERT error. T12 — test/e2e/init-fresh-pglite.test.ts (NEW, 14 cases): subprocess-driven E2E verification of the bug-reporter's repro scenarios: - Happy path: OPENAI_API_KEY set → auto-pick OpenAI, persists config - D3 non-TTY fail-loud (with and without env-key typos) - D6 regression: bug-reporter's three no-op config keys all exit 1 with Levenshtein suggestions - D9 deferred-setup mode + gbrain import refusal (and --no-embed bypass) - D11 preflight refuses BEFORE any disk write - Explicit --embedding-model wins over env detection Each test uses its own throw-away GBRAIN_HOME for hermetic runs. * docs: env-detection + headless-install + close v0.32 picker TODO T13/T14 docs sync for v0.37.10.0: - docs/integrations/embedding-providers.md: TL;DR table refreshed to reflect ZE as v0.36 default; added "Init resolves your provider from env keys" section explaining the auto-pick → picker → fail-loud chain; added "If first import fails" troubleshooting block pointing at gbrain doctor instead of \`rm -rf ~/.gbrain\`. - docs/operations/headless-install.md (NEW): Docker/CI sequencing guide. Two acceptable patterns — provider key at build time (Pattern 1) or --no-embedding opt-in + runtime config (Pattern 2). Codex finding #11. - README.md: Troubleshooting section with one-paragraph repair hint and links to embedding-providers.md + headless-install.md. - TODOS.md: closed v0.32.x "interactive provider chooser" entry as SUPERSEDED by this wave. Added four follow-up entries (dedicated v0.36 broken-install migration, namespaced ext fields, runtime config-key audit, value-level Levenshtein on config set). * chore: bump version and changelog (v0.37.10.0) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(doctor): empty brain scores 100/100 + hermetic doctor-report-remote test Two fixes coupled because the test couldn't pass without the formula fix: src/core/pglite-engine.ts + src/core/postgres-engine.ts — empty brain (pageCount === 0) now gets FULL marks (100/100), not 0/100. Semantically an empty brain has no coverage problem to penalize — there's nothing to embed, nothing to link, nothing to orphan. Vacuous truth applies. The pre-fix "empty = 0" caused fresh-init brains to score as critically unhealthy on \`gbrain doctor\`, which was a structural surprise to users who'd just run init successfully. Same fix on both engines. test/brain-score-breakdown.test.ts — updated the "empty brain" assertion to match the new contract (was: 0/0/0/0/0/0; is: 100/35/25/15/15/10). test/doctor-report-remote.test.ts → renamed to .serial.test.ts and made hermetic. The pre-fix test pulled audit data from the host ~/.gbrain (reranker_health, sync_failures, etc.), which made the assertion non-deterministic depending on whoever ran the suite. Now isolates GBRAIN_HOME to a tempdir via beforeAll/afterAll; env mutation requires serial-quarantine per scripts/check-test-isolation.sh R1. Closes the master-state flake that was failing on every \`bun run test\` run regardless of my branch contents. * docs: update CLAUDE.md and TODOS.md for v0.37.10.0 empty-brain fix - CLAUDE.md: annotate src/core/pglite-engine.ts + src/core/postgres-engine.ts entries with v0.37.10.0 empty-brain 100/100 contract. Vacuous truth: an empty brain has no coverage to penalize, so getBrainScore returns full marks (35/25/15/15/10 breakdown) when pageCount === 0. Pre-fix 0/100 was structurally surprising on fresh init and caused the v0.37.8.0 doctor-report-remote.test.ts flake. - TODOS.md: mark P0 doctor-report-remote.test.ts:65 TODO completed (resolved by commit 9aa571f3's empty-brain-100/100 fix; test renamed to .serial.test.ts and made hermetic per scripts/check-test-isolation.sh R1). - llms-full.txt: regenerated from updated CLAUDE.md per CLAUDE.md "Auto-derived files" rule. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(init): D5 persisted-config-wins on re-init + CI mechanical E2E Two coupled fixes for the v0.37.10.0 wave's interaction with CI's Tier-1 mechanical E2E suite (which runs without any embedding-provider env var). src/commands/init.ts — Honor D5 properly at resolveAIOptions entry. Pre-fix the env-detection branch fired on EVERY init regardless of persisted config. A non-TTY re-init with no env keys exited 1 (D3 fail-loud) even when ~/.gbrain/config.json already had embedding_model set from a prior successful init. Now resolveAIOptions reads loadConfig() first and seeds out.embedding_model / embedding_dimensions / expansion_model / chat_model from the file plane BEFORE running env detection. Also honors embedding_disabled (D9 sentinel) on re-init so deferred-setup brains don't re-trigger fail-loud. test/e2e/mechanical.test.ts:722 — Setup Journey's first init runs against a fresh DB with no persisted config. Pass --embedding-model explicitly (openai:text-embedding-3-large) so the preflight resolves offline. After this init writes config, subsequent inits in the file (RLS self-heal v24, RLS event-trigger probes, etc.) honor the persisted config via the D5 fix above. Verified locally: full test/e2e/mechanical.test.ts → 78 pass / 0 fail. --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>	2026-05-21 18:57:05 -07:00
6 people	1d78013c07	v0.28.5 fix(wave): PGLite upgrade wedge + embedding dim corruption + bun-link foot-gun (#697) ... * fix(engines): pre-add v0.20 + v0.26.3 forward-reference columns in bootstrap The forward-reference bootstrap (PostgresEngine + PGLiteEngine applyForwardReferenceBootstrap) covered v0.18 + v0.19 + v0.26.5 columns but missed two later groups. Brains upgrading from v0.14-era to current master crash before the migration ladder runs: 1. v0.20 Cathedral II — content_chunks.search_vector, parent_symbol_path, doc_comment, symbol_name_qualified. `CREATE INDEX idx_chunks_search_vector` and `CREATE INDEX idx_chunks_symbol_qualified` in schema.sql/PGLITE_SCHEMA_SQL crash with "column search_vector does not exist" / "column symbol_name_qualified does not exist". 2. v0.26.3 — mcp_request_log.agent_name, params, error_message. `CREATE INDEX idx_mcp_log_agent_time ON mcp_request_log(agent_name,...)` crashes with "column agent_name does not exist". Reproduces deterministically on a v0.13/v0.14 brain upgraded straight to current master. The user hits the wall before any of v15-v36 can run. Both engines now probe for these columns and pre-add them via `ALTER TABLE ADD COLUMN IF NOT EXISTS` before SCHEMA_SQL runs. Migrations v26, v27, v33 still run later via runMigrations and remain idempotent (they handle backfill on top of the bootstrap-added columns). Test coverage extended in test/schema-bootstrap-coverage.test.ts: REQUIRED_BOOTSTRAP_COVERAGE now lists 6 new forward references; the strip-and-rebuild block drops the corresponding indexes/triggers so the test exercises a brain that pre-dates v0.20 + v0.26.3 migrations. Repro: brain on schema v13/v14 + run `gbrain init --migrate-only` against current master → fails. With this patch → succeeds; ladder runs to v36. * fix(engines): pre-add v0.27 subagent_messages.provider_id in bootstrap PR #682 covered v0.20 (chunks) + v0.26.3 (mcp_request_log) but missed v0.27's subagent_messages.provider_id. The composite index `idx_subagent_messages_provider ON subagent_messages (job_id, provider_id)` in PGLITE_SCHEMA_SQL crashes on brains pinned at v0.18-v0.26 because provider_id is the SECOND column in the composite — array-extraction patterns that scan only first-column references miss it entirely. This is the wedge surfaced by issue #670 (v0.22.0 → v0.27.0 init --migrate-only crashes with "column 'provider_id' does not exist") and contributing to #661/#657. Both engines now probe for subagent_messages.provider_id and pre-add the column via ALTER TABLE ADD COLUMN IF NOT EXISTS before SCHEMA_SQL runs. Migration v36 (subagent_provider_neutral_persistence_v0_27) still runs later via runMigrations and remains idempotent. Note on the test side: REQUIRED_BOOTSTRAP_COVERAGE is hand-maintained and just gained a v0.27 entry. v0.28.5's Step 3 replaces this array with a SQL parser that auto-derives coverage from PGLITE_SCHEMA_SQL, including composite-index columns. This commit is the targeted follow-up to PR #682's cherry-pick; A2's parser closes the class permanently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): conditional schema-init on connect (closes #651) Adds `hasPendingMigrations(engine)` next to `runMigrations` in migrate.ts: single getConfig('version') probe, returns true when current < LATEST_VERSION, defensively returns true on getConfig failure (treats wedged-config as pending). `connectEngine` in cli.ts now wraps `engine.initSchema()` in a probe gate: short-lived CLI calls (gbrain stats, query, doctor, etc.) on already-migrated brains skip the bootstrap-probe + SCHEMA_SQL replay + ledger-check entirely. Wedged brains still auto-heal — the probe says "yes pending" and initSchema runs as before. Building on oyi77's investigation in PR #652. Same correctness as #652's unconditional initSchema-on-every-connect, but no perf regression on the hot path. Failure non-fatal: if probe or init throws, log a hint and let subsequent operations surface the real error in context. Test coverage in test/migrate.test.ts: 3 cases covering fully-migrated (false), version-rewound (true), and missing-version-config (defensive true). Pairs with v0.28.5's X1 (post-upgrade auto-apply) — the upgrade path runs initSchema explicitly while every other code path that goes through connectEngine gets the cheap probe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(upgrade): post-upgrade auto-applies pending schema migrations (X1) Prior behavior: `gbrain upgrade` → `gbrain post-upgrade` → `apply-migrations` only WARNs at apply-migrations.ts:296-302 when schema version is behind LATEST_VERSION, telling the user to run `gbrain init --migrate-only`. 11 wedge incidents over 2 years have proven users don't read that WARN — they file an issue instead. This commit makes `runPostUpgrade` explicitly call `engine.initSchema()` after the orchestrator migration pass, mirroring `init --migrate-only`'s flow. Side-effect: `gbrain upgrade` now walks away with a healthy brain in the cluster A wedge case (#670, #661, #657, #651, #625, #615, #609). Defensive: wrapped in try/catch so a connection or DDL failure falls back to the existing user-facing WARN. The hint to run `gbrain init --migrate-only` is preserved as the manual escape hatch. Pairs with v0.28.5's A1 (hasPendingMigrations probe in connectEngine): the upgrade path runs initSchema explicitly here, while every other code path that goes through connectEngine gets the cheap probe. Codex outside-voice review caught this gap during plan review: "the plan still does not prove `upgrade` will actually run schema migrations." This is the load-bearing fix that makes v0.28.5's headline outcome ("run upgrade, brain works") literally true for cluster A. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bootstrap): auto-derive coverage from PGLITE_SCHEMA_SQL (A2) Replaces the hand-maintained REQUIRED_BOOTSTRAP_COVERAGE assertion with a SQL-parser-backed structural check. The new test: 1. parseIndexColumnReferences(PGLITE_SCHEMA_SQL) extracts every column referenced by every CREATE INDEX — including composite-index second and third columns. Codex outside-voice review caught that earlier first-col-only patterns missed v0.27's `idx_subagent_messages_provider ON subagent_messages (job_id, provider_id)`, which is exactly how the v0.28.5 wedge happened. 2. parseBaseTableColumns(PGLITE_SCHEMA_SQL) extracts every column declared in CREATE TABLE bodies (including via ALTER TABLE ADD COLUMN inside the schema blob). 3. parseAlterAddColumns(pglite-engine.ts source) extracts every column that applyForwardReferenceBootstrap adds. 4. Static contract: every (table, column) pair from step 1 must appear in either step 2 or step 3. Otherwise the test fails loud, names every uncovered pair, and points at the bootstrap function for the fix. Self-updating: any future CREATE INDEX added to PGLITE_SCHEMA_SQL on a column that bootstrap doesn't yet provide fails this test at PR time. No human required to remember to update an array. Closes the 11-incident wedge class identified in CLAUDE.md (#239, #243, #266, #357, #366, #374, #375, #378, #395, #396). Helper parsers also have their own unit tests covering composite-index second columns, function-wrapped columns (lower(col)), HNSW operator-class suffixes (vector_cosine_ops), and ALTER TABLE column extraction. Existing REQUIRED_BOOTSTRAP_COVERAGE-based tests preserved as a coarse-grained lower bound; the new parser-based test is the load-bearing structural gate going forward. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: support Voyage 2048d schema setup * fix: harden Voyage schema templating * feat: Voyage 4 embedding support + doctor eval - Add voyage-4-large/4/4-lite/4-nano + domain models to Voyage recipe - Fix AI SDK compatibility: strip encoding_format (Voyage rejects 'float'), patch response to add prompt_tokens from total_tokens - Add embedding_provider doctor check: live smoke test verifying model, API key, dimensions, and DB column alignment - Add embedding provider eval qrels for post-migration quality testing Closes: Voyage AI integration for gbrain embedding pipeline * fix: adaptive embed batch sizing for Voyage token limits Voyage's tokenizer is 3-4x denser than OpenAI tiktoken, causing batches of 50+ texts to exceed the 120K token-per-batch limit even when DB token counts (from tiktoken) suggest they'd fit. Changes: - Add max_batch_tokens to EmbeddingTouchpoint type (provider-declared limit) - Set Voyage recipe to 120K token limit - Gateway embed() now auto-splits batches using conservative char-to-token estimate (1:1 ratio, 80% budget utilization) - On token-limit errors, embedSubBatch recursively halves and retries (down to single-text batches before giving up) - Reduce embedding.ts BATCH_SIZE from 100 to 50 as a secondary guard - Add tests for batch splitting logic and error pattern matching Fixes infinite retry loops where the same oversized batch would fail repeatedly because WHERE embedding IS NULL re-fetches identical rows. * fix(init): error on existing-brain dim mismatch + embedding-migration recipe Adds A4 hard-error path: when `gbrain init --embedding-dimensions N` is run against an existing brain whose `content_chunks.embedding` column is a different `vector(M)`, init exits 1 with an inline four-step ALTER recipe and a pointer to docs/embedding-migrations.md. This kills the silent-corruption pattern surfaced by issue #673: the v0.27 schema seeded `('embedding_dimensions', '1536')` regardless of the flag, so users got a config saying 768 but a column at 1536 — first sync write blew up with "expected 1536, got 768." A4's contract: 1. Connect to engine BEFORE saveConfig so we can read the live column type 2. If column exists AND dim != requested, exit 1 (loud failure) 3. If column doesn't exist (fresh init) OR dim matches, proceed normally Recipe in docs/embedding-migrations.md (and inlined in init's error output) covers all four destructive steps codex's plan-review caught: 1. DROP INDEX IF EXISTS idx_chunks_embedding (HNSW won't survive ALTER) 2. ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(N) 3. UPDATE content_chunks SET embedding = NULL, embedded_at = NULL 4. CREATE INDEX HNSW *only if N <= 2000* (pgvector cap) Step 4 is conditional: dims > 2000 (e.g. Voyage 4 Large 2048d) cannot be HNSW-indexed in pgvector; the recipe explicitly says "Skip reindex" in that case so the user doesn't paste a CREATE INDEX that crashes. Helper `readContentChunksEmbeddingDim` and message builder `embeddingMismatchMessage` live in src/core/embedding-dim-check.ts so doctor 8b (next commit) can reuse the same source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(gateway): correct dim-mismatch error to point at manual ALTER recipe (#672) Previous error message recommended running `gbrain migrate --embedding-model … --embedding-dimensions …`, but `gbrain migrate` only handles engine migration (postgres ↔ pglite), not embedding reconfiguration. Following that hint produced a different error and confused users further. New message: - Names the actual options: change models OR migrate the existing brain - Inlines a one-line quick recipe (DROP INDEX → ALTER → UPDATE NULL → config set → embed --stale) - Points at docs/embedding-migrations.md (added in commit 306fc0e1) for the full four-step recipe with HNSW conditional handling Closes #672. Note: #671 (config show hides embedding_model / dimensions) appears to be already fixed on master — `Object.entries(loadConfig())` in config.ts:24 correctly enumerates all keys including embedding_*. Will close #671 with that note when shipping v0.28.5. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(types): doctor 8b uses portable executeRaw + Voyage fetch-shim cast #665's doctor 8b dim-probe used `engine.sql\`...\`` directly (Postgres template literal) which doesn't typecheck against the BrainEngine interface (only PostgresEngine has the .sql getter; PGLite does not). Refactored to use `readContentChunksEmbeddingDim` from src/core/embedding-dim-check.ts — same helper init's A4 hard-error path uses, runs portably on both engines. #680's Voyage fetch-shim passes a custom fetch handler to `createOpenAICompatible` for the encoding_format + prompt_tokens normalization. The SDK accepts the field at runtime but the typed parameter on the pinned version doesn't expose it. Cast to the parameter type so the shim ships without a type error. Both fixes are mechanical cleanup of cherry-picked PRs that didn't typecheck against current master's stricter shape. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): mark cli.ts executable so bun-linked installs work `package.json` declares `"bin": { "gbrain": "src/cli.ts" }`, and bun's linker creates `~/.bun/bin/gbrain` as a symlink to the file. The shebang `#!/usr/bin/env bun` works only when the target file is executable — otherwise bun runs it as a script (because it sees the script via the shebang interpreter), but executing the symlinked target itself fails: $ ls -la ~/.bun/bin/gbrain lrwxrwxrwx ... -> ../install/global/node_modules/gbrain/src/cli.ts $ ~/.bun/bin/gbrain --version /opt/homebrew/bin/bash: line 1: /Users/brandon/.bun/bin/gbrain: Permission denied This bites the postinstall hook that calls `gbrain apply-migrations` (masked by the `||` fallback) and any subprocess that invokes the binary by absolute path (e.g., subagent_messages migration v0.16's `execSync('gbrain init --migrate-only', ...)`). Setting the mode in-tree to 755 fixes both. No content change. * test(ci): guard against src/cli.ts mode-bit regression (cluster C) Cluster C cherry-pick (#683) restored the executable bit on src/cli.ts. This commit adds scripts/check-cli-executable.sh that asserts the git index mode is 100755 and wires it into `bun run verify` (and check:all). Why a CI guard: bun-link installs symlink to src/cli.ts directly. If the mode bit ever regresses to 100644, the very first `gbrain --version` fails with `permission denied` — the exact symptom that motivated #683. This guard runs in <100ms, fast enough for the inner verify loop. Failure mode: clear instructions on what command to run to fix (`chmod +x src/cli.ts && git add --chmod=+x src/cli.ts`) plus a pointer back to issue #683 so future maintainers know why the guard exists. Note: darwin and linux only. Windows preserves the git-stored mode regardless of filesystem chmod, so the index-mode check works the same on every platform CI uses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(upgrade): detect bun-link, warn on npm squatter (#656, #658) Rewrites detectInstallMethod() in src/commands/upgrade.ts:247 with three layered signals per v0.28.5 plan cluster D + codex finding C1: 1. bun-link signal (closes #656): when argv[1] is a symlink, walk up from realpath(argv[1]) up to 6 levels looking for a .git/config whose contents include `garrytan/gbrain` (case-insensitive substring). Returns 'bun-link'. Best-effort: forks, tarballs, and detached source trees fall through to the existing chain. 2. canonical bun authenticity check (closes #658 detection half): when the install lives in node_modules, read package.json and verify repository.url contains `garrytan/gbrain` OR src/cli.ts coexists (squatter ships compiled binary, not source). On 'suspect' verdict, print printSquatterRecovery() — names both git-clone AND release-binary recovery paths so users without a local clone can still recover. 3. Source-marker fallback inside (2). Codex flagged this is spoofable by a determined squatter; accepted — best-effort warning, not assertion. The structural fix is publishing under @garrytan/gbrain (tracked v0.29 follow-up). The squatter's `name: gbrain` field doesn't disambiguate (codex caught this in plan review of my original heuristic). repository.url is the field a careless squatter is least likely to set correctly; src/cli.ts presence is the secondary signal. bun-link installs return 'bun-link' from the switch in runUpgrade, which prints the source-clone upgrade path (`git pull && bun install && bun link`) instead of trying `bun update gbrain` which doesn't apply. README updated with the corresponding "DO NOT use `bun add -g gbrain`" callout naming both #658 and the v0.29 scoped-name plan. Tests in test/upgrade.test.ts cover return-type extension, bun-link signal shape, classifyBunInstall's two-signal check, and the recovery message contents. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.28.5 release: PGLite upgrade wedge + embedding dim corruption + bun-link foot-gun Fix wave bundling 9 community PRs to unwedge users stuck since v0.27. Cluster A — PGLite upgrade wedge (#670, #661, #657, #651, #625, #615, #609): - Bootstrap now covers v0.20+v0.26.3+v0.27 forward references (both engines) - hasPendingMigrations() probe gates initSchema() in connectEngine - Post-upgrade auto-applies pending schema migrations (X1) - SQL-parser-backed bootstrap coverage replaces hand-maintained array (A2) Cluster B — Embedding dim corruption (#673, #672, #666, #640): - Schema templating cascade fixed end-to-end (#641 from @100yenadmin) - gbrain doctor 8b live embedding-provider probe (#665) - Voyage adaptive batch sizing for 120K-token cap (#680) - gbrain init A4 hard-error on existing-brain dim mismatch - docs/embedding-migrations.md with conditional-HNSW four-step recipe - #672 misleading migrate-suggestion error replaced with inline recipe Cluster C — CLI exec bit (#683, dupe of #655): - src/cli.ts mode 100644 → 100755 (#683 from @brandonlipman) - scripts/check-cli-executable.sh CI guard against future regression Cluster D — bun add -g foot-gun (#656, #658): - 3-signal detectInstallMethod rewrite (bun-link, repo.url, source-marker) - Loud-red recovery message names source-clone AND release-binary paths - README "DO NOT use bun add -g gbrain" callout Contributors: @brandonlipman (#682, #683), @mdcruz88 (#668), @ChenyqThu (#627), @alan-mathison-enigma (#610), @oyi77 (#652 building block), @abkrim (#655), @100yenadmin (#641). VERSION 0.27.0 → 0.28.5 package.json 0.27.0 → 0.28.5 schema-embedded.ts regenerated via bun run build:schema llms-full.txt regenerated via bun run build:llms Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(e2e): v0.28.5 fix-wave end-to-end coverage PGLite-only E2E covering the three regression scenarios v0.28.5 was shipped to fix: 1. cluster A — pre-v0.20 brain (missing v0.20 + v0.26.3 + v0.27 columns) re-runs initSchema cleanly. Strips the column set v0.28.5's bootstrap claims to restore (search_vector, parent_symbol_path, doc_comment, symbol_name_qualified, agent_name, params, error_message, provider_id), resets the version row to 13, then re-runs initSchema. Asserts every column comes back AND version reaches LATEST_VERSION. Closes the gap that pre-v0.28.5 produced 11 wedge incidents. 2. cluster B — fresh init at non-default dims templates the column correctly (768d AND 2048d cases). The 2048d case explicitly verifies idx_chunks_embedding is NOT created (codex finding #8 — pgvector's HNSW cap is 2000). 3. A4 — existing-brain dim mismatch helper produces a recipe that inlines all four steps (DROP INDEX, ALTER TYPE, NULL, conditional reindex). Validates the conditional CREATE INDEX HNSW for dims <= 2000 AND its omission for dims > 2000. The recipe a user copy-pastes won't crash them on Voyage 4 Large. Plus a hasPendingMigrations() lifecycle test covering the four states (fresh / migrated / rewound / re-applied) — pairs with the unit test in test/migrate.test.ts but exercises the engine end-to-end. PGLite-only because none of these cases need real Postgres. Postgres-side bootstrap is covered by test/e2e/postgres-bootstrap.test.ts. Run: bun test test/e2e/v0_28_5-fix-wave.test.ts (no DATABASE_URL needed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: refactor embedding-dim-check.test.ts to canonical PGLite pattern Test-isolation lint (R3+R4) requires PGLiteEngine in beforeAll() context with afterAll() disconnect. Refactored to single-engine-per-file pattern; the fresh-brain test uses a one-off engine inside its own try/finally so the file-level engine stays at LATEST schema for the migrated-brain test. No behavior change to the assertions. `bun run verify` now passes clean (privacy + jsonb + progress + test-isolation + wasm + admin-build + cli-exec + typecheck). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(doctor): make 8b embedding-provider probe non-fatal (CI green) CI Tier 1 was failing on `gbrain doctor exits 0 on healthy DB` because the v0.28.5 doctor 8b check (cherry-picked from #665) pushed `status: 'fail'` in two non-fatal scenarios: 1. No API key configured (`isAvailable('embedding')` returns false) 2. Probe throws (network blip, transient 5xx, DNS, rate limit) Both are noise in CI and on offline workstations — the brain is healthy, the provider just isn't reachable from this environment. The v0.28.5 plan P1 decision called for non-fatal-on-offline behavior: > Doctor 8b probes live every run (taken as-is). Non-fatal on network > failure (warns rather than errors); silently skipped when no API key > configured. This commit aligns the implementation with that decision: - !available → status 'ok' with "Skipped (no provider credentials)" message so the run is visible in --json output without failing exit code - catch block → status 'warn' (was 'fail') so probe failures surface informationally without crashing CI / autopilot's periodic doctor runs The mismatch slipped past plan-time review because #665 was cherry-picked before P1 was finalized; the type-fix pass in 4c26e484 only adjusted the DB-column probe shape, not the API-availability gate. CI Tier 1 (Mechanical) — `test/e2e/mechanical.test.ts:1220` — "gbrain doctor exits 0 on healthy DB" now passes against a fresh Postgres without `OPENAI_API_KEY` / `VOYAGE_API_KEY` set. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Brandon Lipman <brandon@offdeck.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Eva <eva@100yen.org> Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>	2026-05-06 20:58:19 -07:00