1 Commits

Include renames

Author	SHA1	Message	Date
3 people	182900d071	v0.31.8 fix: multi-source threading + doctor wedge hint + voyage cap (P2 follow-ups) (#808) ... * feat(multi-source): thread ctx.sourceId through op handlers + engine read-surface Closes the multi-source threading gaps that the v0.31.1.1-fixwave codex review caught. Multi-source brains were silently misrouting writes from every CLI/MCP-driven op (put_page, add_tag, add_link, add_timeline_entry, revert_version, put_raw_data, etc.) because the op handlers in operations.ts ignored ctx.sourceId. Read-side ops were arbitrary-row under same-slug-across-sources because the engine's read methods had no source filter. Engine layer (D12 + D16 + D21): - engine.ts interface: getLinks/getBacklinks/getTimeline/getRawData/ getVersions/getAllSlugs/revertToVersion/putRawData all take opts?: { sourceId?: string }. - pglite-engine.ts + postgres-engine.ts: two-branch query for each read method. Without opts.sourceId, NO source filter applies (preserves pre-v0.31.8 cross-source semantics for back-link validators and any caller that hasn't threaded sourceId yet). With opts.sourceId, scoped to that source — the new path used by reconcileLinks and ctx.sourceId-aware op handlers. Op-handler layer (D7 + D16 + D20): - operations.ts threads ctx.sourceId through 16+ handler sites: put_page, revert_version, put_raw_data, add_tag, remove_tag, add_link, remove_link, add_timeline_entry, create_version, delete_page, restore_page, get_page, get_tags, get_links, get_backlinks, get_timeline, get_versions, get_raw_data, get_chunks, plus reconcileLinks's tx.getLinks/getBacklinks/ addLink/removeLink and engine.getAllSlugs. - Pattern: const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {}; When ctx.sourceId is unset, engine falls through to cross-source view (back-compat). MCP callers populate ctx.sourceId via the transport layer. CLI wiring (D11 + D22): - cli.ts: makeContext is async, calls resolveSourceId() from src/core/source-resolver.ts:58 (the canonical 6-tier chain: --source flag → GBRAIN_SOURCE env → .gbrain-source dotfile → path-match → brain default → 'default'). Wrapped in try/catch so a fresh pre-init brain still returns a clean ctx with no sourceId set. - commands/call.ts: runCall accepts --source <id> flag. Resolves through the same 6-tier chain and threads to handleToolCall via the new opts.sourceId param. - mcp/server.ts: handleToolCall accepts opts.sourceId and threads to buildOperationContext. Tests (D7 + D16 + D20 regression coverage): - test/source-id-tx-regression.test.ts: 8 new op-handler-layer cases covering add_tag/get_tags/add_link/get_links/delete_page/ put_raw_data routing under ctx.sourceId='X' vs unset, plus D16's two-branch back-compat invariant for getLinks (cross- source view preserved when ctx.sourceId is unset). Closes the codex OV-1/OV-2/OV-3 findings from the v0.31.8 plan review. Back-compat is strictly additive: callers that don't pass opts.sourceId see the same results they did pre-v0.31.8. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(doctor): multi_source_drift check surfaces pre-v0.30.3 misroutes Pre-v0.30.3 putPage misrouted multi-source writes from intended source X to (default, slug). The fix-wave fixed forward-going writes but explicitly deferred backfilling the misrouted rows. Operators have had no signal of this silent corruption. Adds src/core/multi-source-drift.ts exporting findMisroutedPages(engine, sources, opts). The heuristic walks each non-default source's local_path and surfaces slugs that exist at (default, slug) in DB but are MISSING from (X, slug) — unambiguous evidence of the misroute shape. Implementation notes (codex OV12 + OV13 + D17): - FS walk handles BOTH .md and .mdx (matches src/core/sync.ts:133, which treats both as markdown). Walks own helper instead of importing from extract.ts so doctor doesn't crash if local_path is unreadable (try/catch on root statSync; ENOENT/EACCES yields zero files, NOT a thrown error that takes down doctor). - Single batched SQL with VALUES clause: collect all candidate slugs into one array, then ONE LEFT JOIN against pages with source_id IN ('default', X). Materialize into Map<slug, Set<source_id>>. NOT a per-file 20K-round-trip loop. - Bounded by limit (10K files) AND timeoutMs (5s). Bail with walk_truncated=true rather than letting doctor hang. - Heuristic softened per OV12: "appears misrouted to default" with TWO possible causes flagged (pre-v0.30.3 misroute OR source X never completed initial sync). The doctor warning suggests verification ('gbrain sources status'), not a destructive action. Wired into runDoctor (3b-multi-source slot, after sync_failures) AND into doctorReportRemote (D14) so thin-client operators see the check when 'gbrain doctor' routes through the remote MCP path. Single-source brains skip the check entirely. Tests: test/multi-source-drift.test.ts (7 PGLite cases) covers: - Single-source brain → skip - Multi-source no-misroutes → ok - Multi-source 2 misrouted slugs → warn with sample - Healthy same-slug-across-sources NOT a false positive (the codex OV4 redesign case — original heuristic would have false-positived) - FS walk hits limit → walk_truncated=true - Unreadable local_path doesn't crash - .mdx files walked alongside .md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(doctor): wire multi_source_drift + wedge force-retry hint (D14 + D19) Wires the new multi_source_drift check into both runDoctor (local) and doctorReportRemote (thin-client remote MCP path), and extends the existing minions_migration block to detect 3-consecutive-partials wedges and emit gbrain apply-migrations --force-retry <v> hints (D19). Pre-v0.31.8, operators wedged on v0.29.1 (or any future migration that hits the apply-migrations runner's 3-consecutive-partials guard) got the generic "Run: gbrain apply-migrations --yes" hint. That command refuses to advance past the guard — so the hint was wrong. Codex OV-11 (and the v0.31.1.1-fixwave commit message) flagged this, but the prior plan said to delegate to apply-migrations.ts:statusForVersion(), which would have re-opened a separate regression: the existing forward-progress override at doctor.ts:303 (newer completion suppresses old partials) is cross-version and statusForVersion is per-version only. This commit extends the existing block in place rather than replacing it: 1. Keep the forward-progress override (lines 348-356) byte-identical so installs that moved past an old v0.11 partial don't light up with stale wedge alerts. 2. Add a 3-consecutive-partials detector after the stuck filter. Since `stuck` already excludes forward-progress-superseded versions, the wedge counter only fires on actual unresolved partials. 3. Branch the message: - wedged.length > 0 → "WEDGED MIGRATION(s): <v>. Run: gbrain apply-migrations --force-retry <v>" (chain with && for multiple) - else if stuck.length > 0 → existing --yes hint - else → no message Same shape duplicated in doctorReportRemote so thin-client operators see the right command on the brain host. Plus the multi_source_drift wiring (D14): same heuristic from the new src/core/multi-source-drift.ts library, called from both local and remote doctor paths. Single-source brains skip. Engine-null guard on the local path (--fast and DB-down branches pass null). Tests: test/doctor.test.ts gains 4 wedge-hint regression cases: - Both branches present in source (forward-progress override + 3-partials detection coexisting). - Anti-regression guard: NO `import { statusForVersion }` from apply-migrations.ts. The prior plan would have introduced this import; keeping it out means doctor stays decoupled from the migration runner's per-version semantics. - Multiple wedged versions chain force-retry calls with `&&`. - Both branches present in doctorReportRemote (thin-client coverage, D14). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(voyage): Content-Length pre-check + per-item base64 cap (D2 + D10) The voyage compat fetch wrapper at gateway.ts:294 called \`await resp.clone().json()\` BEFORE iterating embeddings. A malicious or compromised Voyage endpoint of arbitrary size was fully parsed into the JS heap before any size check could fire. The original v0.31.8 plan put the cap on per-item base64 length, which fires AFTER the JSON parse — defeating the OOM defense entirely (codex OV8). Two-layer fix sized at MAX_VOYAGE_RESPONSE_BYTES = 256 MB ("unambiguously not legit" rather than tight against typical batches; voyage-3-large × 16K embeddings ≈ 200 MB raw fits within the cap): Layer 1 (PRIMARY) — Content-Length header pre-check, fires BEFORE resp.clone().json(). Throws a descriptive error if the header reports a length over the cap. The JSON.parse OOM vector is now gated. Layer 2 (defense-in-depth) — per-embedding base64 length check inside the iteration. Catches the rare case where Layer 1 was skipped (chunked transfer encoding has no Content-Length) AND a single embedding string is unreasonably large. Estimates decoded size as 0.75 × base64 length (canonical base64 → bytes ratio). Tests: test/voyage-response-cap.test.ts — 5 structural source-pin cases including the critical D10 invariant: "Content-Length pre-check appears BEFORE \`const json: any = await resp.clone().json()\` in the inbound block". A future refactor that moves the cap below the JSON parse fails this test loudly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.31.8) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): exclude *.serial.test.ts from sharded parallel run scripts/test-shard.sh (the GitHub Actions runner) was including *.serial.test.ts files alongside regular tests. Serial files use top-level mock.module(...) which leaks across files in the same Bun process — exactly what the .serial naming convention was meant to quarantine. Concretely: test/eval-takes-quality-runner.serial.test.ts mocks src/core/ai/gateway.ts with `configureGateway: () => undefined` (no-op). Because both files landed in shard 2, the mock leaked into test/voyage-multimodal.test.ts: when its tests called configureVoyageMultimodal() → configureGateway(), the no-op fired and _config stayed null. Then embedMultimodal() called requireConfig() which threw "AI gateway is not configured" — 18 tests failed at gateway.ts:171 with [1.00ms] each. Local fast loop (scripts/run-unit-shard.sh) already excludes *.serial.test.ts AND *.slow.test.ts via the same find-arg pattern. test-shard.sh just hadn't picked up the same exclusion when it was written. This commit: 1. Mirrors run-unit-shard.sh's exclusion pattern in test-shard.sh (`-not -name '*.slow.test.ts' -not -name '*.serial.test.ts'`). 2. Adds a "Run *.serial.test.ts" step to .github/workflows/test.yml on shard 1 only, calling scripts/run-serial-tests.sh (--max-concurrency=1). Shard 1 already runs extra setup work (`bun run verify`), so it has the natural slot for the serial pass without slowing the parallel critical path. Verified locally: shard 2 went from 18 voyage-multimodal failures to 0. Shard 2 file count: 81 → 78 (3 serial files removed). Total test count after fix: 1438 (1437 pass + 1 pre-existing env-sensitive warm-create speed gate flake — unrelated to v0.31.8 or this fix). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-10 13:33:05 -07:00