mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.31.8 fix: multi-source threading + doctor wedge hint + voyage cap (P2 follow-ups) (#808)
* feat(multi-source): thread ctx.sourceId through op handlers + engine read-surface
Closes the multi-source threading gaps that the v0.31.1.1-fixwave codex
review caught. Multi-source brains were silently misrouting writes from
every CLI/MCP-driven op (put_page, add_tag, add_link, add_timeline_entry,
revert_version, put_raw_data, etc.) because the op handlers in
operations.ts ignored ctx.sourceId. Read-side ops were arbitrary-row
under same-slug-across-sources because the engine's read methods had no
source filter.
Engine layer (D12 + D16 + D21):
- engine.ts interface: getLinks/getBacklinks/getTimeline/getRawData/
getVersions/getAllSlugs/revertToVersion/putRawData all take
opts?: { sourceId?: string }.
- pglite-engine.ts + postgres-engine.ts: two-branch query for each
read method. Without opts.sourceId, NO source filter applies
(preserves pre-v0.31.8 cross-source semantics for back-link
validators and any caller that hasn't threaded sourceId yet). With
opts.sourceId, scoped to that source — the new path used by
reconcileLinks and ctx.sourceId-aware op handlers.
Op-handler layer (D7 + D16 + D20):
- operations.ts threads ctx.sourceId through 16+ handler sites:
put_page, revert_version, put_raw_data, add_tag, remove_tag,
add_link, remove_link, add_timeline_entry, create_version,
delete_page, restore_page, get_page, get_tags, get_links,
get_backlinks, get_timeline, get_versions, get_raw_data,
get_chunks, plus reconcileLinks's tx.getLinks/getBacklinks/
addLink/removeLink and engine.getAllSlugs.
- Pattern: const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
When ctx.sourceId is unset, engine falls through to cross-source
view (back-compat). MCP callers populate ctx.sourceId via the
transport layer.
CLI wiring (D11 + D22):
- cli.ts: makeContext is async, calls resolveSourceId() from
src/core/source-resolver.ts:58 (the canonical 6-tier chain:
--source flag → GBRAIN_SOURCE env → .gbrain-source dotfile →
path-match → brain default → 'default'). Wrapped in try/catch
so a fresh pre-init brain still returns a clean ctx with no
sourceId set.
- commands/call.ts: runCall accepts --source <id> flag. Resolves
through the same 6-tier chain and threads to handleToolCall
via the new opts.sourceId param.
- mcp/server.ts: handleToolCall accepts opts.sourceId and threads
to buildOperationContext.
Tests (D7 + D16 + D20 regression coverage):
- test/source-id-tx-regression.test.ts: 8 new op-handler-layer
cases covering add_tag/get_tags/add_link/get_links/delete_page/
put_raw_data routing under ctx.sourceId='X' vs unset, plus
D16's two-branch back-compat invariant for getLinks (cross-
source view preserved when ctx.sourceId is unset).
Closes the codex OV-1/OV-2/OV-3 findings from the v0.31.8 plan
review. Back-compat is strictly additive: callers that don't pass
opts.sourceId see the same results they did pre-v0.31.8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(doctor): multi_source_drift check surfaces pre-v0.30.3 misroutes
Pre-v0.30.3 putPage misrouted multi-source writes from intended source X
to (default, slug). The fix-wave fixed forward-going writes but explicitly
deferred backfilling the misrouted rows. Operators have had no signal of
this silent corruption.
Adds src/core/multi-source-drift.ts exporting findMisroutedPages(engine,
sources, opts). The heuristic walks each non-default source's local_path
and surfaces slugs that exist at (default, slug) in DB but are MISSING
from (X, slug) — unambiguous evidence of the misroute shape.
Implementation notes (codex OV12 + OV13 + D17):
- FS walk handles BOTH .md and .mdx (matches src/core/sync.ts:133, which
treats both as markdown). Walks own helper instead of importing from
extract.ts so doctor doesn't crash if local_path is unreadable
(try/catch on root statSync; ENOENT/EACCES yields zero files, NOT a
thrown error that takes down doctor).
- Single batched SQL with VALUES clause: collect all candidate slugs
into one array, then ONE LEFT JOIN against pages with source_id IN
('default', X). Materialize into Map<slug, Set<source_id>>. NOT a
per-file 20K-round-trip loop.
- Bounded by limit (10K files) AND timeoutMs (5s). Bail with
walk_truncated=true rather than letting doctor hang.
- Heuristic softened per OV12: "appears misrouted to default" with TWO
possible causes flagged (pre-v0.30.3 misroute OR source X never
completed initial sync). The doctor warning suggests verification
('gbrain sources status'), not a destructive action.
Wired into runDoctor (3b-multi-source slot, after sync_failures) AND
into doctorReportRemote (D14) so thin-client operators see the check
when 'gbrain doctor' routes through the remote MCP path. Single-source
brains skip the check entirely.
Tests: test/multi-source-drift.test.ts (7 PGLite cases) covers:
- Single-source brain → skip
- Multi-source no-misroutes → ok
- Multi-source 2 misrouted slugs → warn with sample
- Healthy same-slug-across-sources NOT a false positive (the codex
OV4 redesign case — original heuristic would have false-positived)
- FS walk hits limit → walk_truncated=true
- Unreadable local_path doesn't crash
- .mdx files walked alongside .md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(doctor): wire multi_source_drift + wedge force-retry hint (D14 + D19)
Wires the new multi_source_drift check into both runDoctor (local) and
doctorReportRemote (thin-client remote MCP path), and extends the existing
minions_migration block to detect 3-consecutive-partials wedges and emit
gbrain apply-migrations --force-retry <v> hints (D19).
Pre-v0.31.8, operators wedged on v0.29.1 (or any future migration that
hits the apply-migrations runner's 3-consecutive-partials guard) got the
generic "Run: gbrain apply-migrations --yes" hint. That command refuses
to advance past the guard — so the hint was wrong. Codex OV-11 (and the
v0.31.1.1-fixwave commit message) flagged this, but the prior plan said
to delegate to apply-migrations.ts:statusForVersion(), which would have
re-opened a separate regression: the existing forward-progress override
at doctor.ts:303 (newer completion suppresses old partials) is
cross-version and statusForVersion is per-version only.
This commit extends the existing block in place rather than replacing it:
1. Keep the forward-progress override (lines 348-356) byte-identical so
installs that moved past an old v0.11 partial don't light up with
stale wedge alerts.
2. Add a 3-consecutive-partials detector after the stuck filter. Since
`stuck` already excludes forward-progress-superseded versions, the
wedge counter only fires on actual unresolved partials.
3. Branch the message:
- wedged.length > 0 → "WEDGED MIGRATION(s): <v>. Run: gbrain
apply-migrations --force-retry <v>" (chain with && for multiple)
- else if stuck.length > 0 → existing --yes hint
- else → no message
Same shape duplicated in doctorReportRemote so thin-client operators
see the right command on the brain host.
Plus the multi_source_drift wiring (D14): same heuristic from the
new src/core/multi-source-drift.ts library, called from both local and
remote doctor paths. Single-source brains skip. Engine-null guard on
the local path (--fast and DB-down branches pass null).
Tests: test/doctor.test.ts gains 4 wedge-hint regression cases:
- Both branches present in source (forward-progress override + 3-partials
detection coexisting).
- Anti-regression guard: NO `import { statusForVersion }` from
apply-migrations.ts. The prior plan would have introduced this
import; keeping it out means doctor stays decoupled from the
migration runner's per-version semantics.
- Multiple wedged versions chain force-retry calls with `&&`.
- Both branches present in doctorReportRemote (thin-client coverage,
D14).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(voyage): Content-Length pre-check + per-item base64 cap (D2 + D10)
The voyage compat fetch wrapper at gateway.ts:294 called
\`await resp.clone().json()\` BEFORE iterating embeddings. A
malicious or compromised Voyage endpoint of arbitrary size was
fully parsed into the JS heap before any size check could fire.
The original v0.31.8 plan put the cap on per-item base64 length,
which fires AFTER the JSON parse — defeating the OOM defense
entirely (codex OV8).
Two-layer fix sized at MAX_VOYAGE_RESPONSE_BYTES = 256 MB
("unambiguously not legit" rather than tight against typical
batches; voyage-3-large × 16K embeddings ≈ 200 MB raw fits within
the cap):
Layer 1 (PRIMARY) — Content-Length header pre-check, fires
BEFORE resp.clone().json(). Throws a descriptive error if the
header reports a length over the cap. The JSON.parse OOM vector
is now gated.
Layer 2 (defense-in-depth) — per-embedding base64 length check
inside the iteration. Catches the rare case where Layer 1 was
skipped (chunked transfer encoding has no Content-Length) AND a
single embedding string is unreasonably large. Estimates decoded
size as 0.75 × base64 length (canonical base64 → bytes ratio).
Tests: test/voyage-response-cap.test.ts — 5 structural source-pin
cases including the critical D10 invariant: "Content-Length
pre-check appears BEFORE \`const json: any = await
resp.clone().json()\` in the inbound block". A future refactor
that moves the cap below the JSON parse fails this test loudly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.31.8)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): exclude *.serial.test.ts from sharded parallel run
scripts/test-shard.sh (the GitHub Actions runner) was including
*.serial.test.ts files alongside regular tests. Serial files use
top-level mock.module(...) which leaks across files in the same Bun
process — exactly what the .serial naming convention was meant to
quarantine.
Concretely: test/eval-takes-quality-runner.serial.test.ts mocks
src/core/ai/gateway.ts with `configureGateway: () => undefined`
(no-op). Because both files landed in shard 2, the mock leaked into
test/voyage-multimodal.test.ts: when its tests called
configureVoyageMultimodal() → configureGateway(), the no-op fired and
_config stayed null. Then embedMultimodal() called requireConfig()
which threw "AI gateway is not configured" — 18 tests failed at
gateway.ts:171 with [1.00ms] each.
Local fast loop (scripts/run-unit-shard.sh) already excludes
*.serial.test.ts AND *.slow.test.ts via the same find-arg pattern.
test-shard.sh just hadn't picked up the same exclusion when it was
written. This commit:
1. Mirrors run-unit-shard.sh's exclusion pattern in test-shard.sh
(`-not -name '*.slow.test.ts' -not -name '*.serial.test.ts'`).
2. Adds a "Run *.serial.test.ts" step to .github/workflows/test.yml
on shard 1 only, calling scripts/run-serial-tests.sh
(--max-concurrency=1). Shard 1 already runs extra setup work
(`bun run verify`), so it has the natural slot for the serial
pass without slowing the parallel critical path.
Verified locally: shard 2 went from 18 voyage-multimodal failures to
0. Shard 2 file count: 81 → 78 (3 serial files removed). Total test
count after fix: 1438 (1437 pass + 1 pre-existing env-sensitive
warm-create speed gate flake — unrelated to v0.31.8 or this fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
87840341ea
commit
182900d071
@@ -2,6 +2,83 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.31.8] - 2026-05-10
|
||||
|
||||
**Multi-source brains stop misrouting writes, and `gbrain doctor` finally tells you when it does.**
|
||||
|
||||
Two-pass eng + codex review on the v0.31.1.1-fixwave follow-ups caught 8 issues a single pass would have shipped with. The biggest: every CLI/MCP-driven write op (put_page, add_tag, add_link, add_timeline_entry, revert_version, put_raw_data) was silently ignoring `ctx.sourceId` on multi-source brains, landing rows at `(default, slug)` regardless of caller intent. Read ops on the same brains returned arbitrary rows when the same slug existed across sources. Plus voyage embedding responses hit `JSON.parse` before any size cap, and operators wedged on v0.29.1 got the wrong recovery hint.
|
||||
|
||||
### What you can now do
|
||||
|
||||
**`gbrain link`, `gbrain tag`, `gbrain timeline-add` and every other CLI write op respect your active source.** Set `GBRAIN_SOURCE=jarvis-memory` (or use a `.gbrain-source` dotfile, or pass `--source` to `gbrain call`) and the op lands on the right row. Without a source signal, behavior is identical to pre-v0.31.8 — strictly additive. Closes the silent multi-source misroute that was the headline P2 from the parent fix-wave's adversarial review.
|
||||
|
||||
**`gbrain doctor` warns when a multi-source brain has misrouted pages.** The new `multi_source_drift` check walks each non-default source's `local_path`, finds slugs that exist at `(default, slug)` but NOT at `(X, slug)`, and surfaces them with verification steps. Two possible causes are flagged honestly: pre-v0.30.3 putPage misroute, OR a source X that never finished its initial sync. The warning suggests `gbrain sources status` and `gbrain sync --source X --full` rather than a destructive auto-fix. Skipped on single-source brains. Propagates to `doctorReportRemote` so thin-client operators see the same check.
|
||||
|
||||
**`gbrain doctor` gives operators wedged on v0.29.1 (or any future migration) the hint that actually unsticks them.** Pre-v0.31.8 the message said `Run: gbrain apply-migrations --yes` regardless of how partial the migration was. The apply-migrations runner refuses to advance past 3 consecutive partials, so the hint was wrong. Now the doctor detects the wedge condition in place — without breaking the existing forward-progress override that suppresses stale alerts on installs that moved past — and emits `Run: gbrain apply-migrations --force-retry <v>` chained with `&&` for multiple wedges. Same shape in both local and remote-doctor paths.
|
||||
|
||||
**Voyage embedding responses can't OOM the worker.** A 256 MB Content-Length pre-check fires BEFORE `resp.clone().json()`, gating the JSON.parse vector that pre-fix would have allocated arbitrary heap on a malicious or compromised endpoint. A per-embedding base64 cap stays as defense-in-depth for chunked-encoding responses with no Content-Length header. Sized as "unambiguously not legit" — voyage-3-large × 16K embeddings (~200 MB raw) fits within the cap, anything larger is unambiguous misuse.
|
||||
|
||||
### Numbers that matter
|
||||
|
||||
Two passes of eng + codex review caught **2 + 6 = 8** additional issues beyond what a single review would have shipped with. False-positive count across the entire two-pass flow: **2** (lane B agent's OAuth claim, original P2-4 cwd attack premise — both verified and rejected). 22 AskUserQuestion decisions, 0 unresolved. The `multi_source_drift` check uses a single batched SQL query with a VALUES clause — sub-second on a 10K-file source instead of the 20K-round-trip N+1 the original plan would have written.
|
||||
|
||||
### What this means for operators
|
||||
|
||||
If you run a multi-source brain, run `gbrain doctor` after upgrading. Any `multi_source_drift` warnings are evidence of pre-v0.30.3 silent corruption that the parent fix-wave couldn't backfill. Use `gbrain sources status` to verify, then `gbrain sync --source <id> --full` to populate the right rows.
|
||||
|
||||
### To take advantage of v0.31.8
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Set `GBRAIN_SOURCE=<id>` in your shell** (or write it to `.gbrain-source` in your brain repo root, or pass `--source <id>` to `gbrain call`) if you want CLI ops to operate on a non-default source. Without this, ops continue to default to `'default'` exactly like pre-v0.31.8.
|
||||
3. **Verify the outcome:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
gbrain stats
|
||||
```
|
||||
4. **If `gbrain doctor` reports `multi_source_drift` warnings,** verify with `gbrain sources status` then either re-sync the affected source (`gbrain sync --source <id> --full`) or `gbrain delete <slug>` if the default-source row is the misroute. A `gbrain sources rehome` cleanup command is tracked for v0.32.0.
|
||||
5. **If any step fails or the numbers look wrong,** please file an issue: https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
|
||||
- which step broke
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Multi-source threading (codex OV-1, OV-2, OV-3, OV-9, OV-10 fixed)
|
||||
|
||||
- **`src/core/engine.ts`:** read-surface signatures gain `opts?: { sourceId?: string }`. Affected methods: `getLinks`, `getBacklinks`, `getTimeline`, `getRawData`, `getVersions`, `getAllSlugs`, `revertToVersion`, `putRawData`. Plus `TimelineOpts.sourceId` in `src/core/types.ts`.
|
||||
- **`src/core/pglite-engine.ts` + `src/core/postgres-engine.ts`:** every read method uses a **two-branch query**. Without `opts.sourceId`, NO source filter applies (preserves pre-v0.31.8 cross-source semantics for back-link validators and any caller that hasn't threaded sourceId yet). With `opts.sourceId`, scoped to that source — the new path used by reconcileLinks and ctx.sourceId-aware op handlers. The flat-default-to-`'default'` shape from the original plan would have silently regressed every multi-source brain's getLinks/getBacklinks callers.
|
||||
- **`src/core/operations.ts`:** thread `const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};` through 16+ op handlers including the read surface (`get_page`, `get_tags`, `get_links`, `get_backlinks`, `get_timeline`, `get_versions`, `get_raw_data`, `get_chunks`) plus `runAutoLink`/`reconcileLinks` source-aware reads.
|
||||
- **`src/cli.ts`:** `makeContext` is async, calls `resolveSourceId()` from `src/core/source-resolver.ts:58` (the canonical 6-tier chain: `--source` flag → `GBRAIN_SOURCE` env → `.gbrain-source` dotfile → path-match → brain default → `'default'`).
|
||||
- **`src/commands/call.ts`:** `runCall` accepts `--source <id>` flag.
|
||||
- **`src/mcp/server.ts`:** `handleToolCall` accepts `opts.sourceId`.
|
||||
|
||||
#### Doctor checks
|
||||
|
||||
- **`src/core/multi-source-drift.ts` (new):** `findMisroutedPages` walks each non-default source's `local_path` for `.md` AND `.mdx` files (matches `src/core/sync.ts`), wraps the walk in try/catch so unreadable directories don't crash doctor, and runs ONE batched SQL query with VALUES clause to check existence at default vs intended source. Bounded by 10K files OR 5s timeout.
|
||||
- **`src/commands/doctor.ts`:** new `multi_source_drift` check (local + remote), `minions_migration` extended in place with 3-consecutive-partials wedge detection emitting `gbrain apply-migrations --force-retry` hints. Forward-progress override at the existing block stays byte-identical so installs that moved past an old v0.11 partial don't light up with stale alerts.
|
||||
|
||||
#### Voyage adapter
|
||||
|
||||
- **`src/core/ai/gateway.ts`:** `MAX_VOYAGE_RESPONSE_BYTES = 256 * 1024 * 1024`. Layer 1 Content-Length pre-check fires BEFORE `resp.clone().json()`. Layer 2 per-embedding base64 cap as defense-in-depth.
|
||||
|
||||
#### Tests
|
||||
|
||||
- **`test/source-id-tx-regression.test.ts`:** 8 new op-handler-layer cases extending the v0.18.0+ engine-layer coverage. Covers `add_tag`/`get_tags`/`add_link`/`get_links`/`delete_page`/`put_raw_data` routing under `ctx.sourceId='X'` vs unset, plus D16's two-branch back-compat invariant.
|
||||
- **`test/multi-source-drift.test.ts` (new):** 7 PGLite cases covering single-source skip, no-misroutes ok, 2-misroutes warn, healthy same-slug-across-sources is NOT a false positive (the codex OV4 redesign case), FS walk truncation, unreadable local_path, and `.mdx` walking.
|
||||
- **`test/doctor.test.ts`:** 4 wedge-hint regression cases including the D19 anti-regression guard that no `import { statusForVersion }` from apply-migrations.ts exists. The prior plan would have introduced this import; keeping it out means doctor stays decoupled from per-version semantics that don't encode the cross-version forward-progress override.
|
||||
- **`test/voyage-response-cap.test.ts` (new):** 5 structural source-pin cases including the critical D10 invariant: "Content-Length pre-check appears BEFORE `const json: any = await resp.clone().json()`."
|
||||
|
||||
### For contributors
|
||||
|
||||
Two passes of `/plan-eng-review` + outside-voice codex review caught issues a single pass would have shipped with — D16 (read-surface back-compat regression in the original D12 plan), D17 (N+1 risk in the original D8 batch-query wording), D19 (statusForVersion replacement would have regressed the existing forward-progress override), D20 (more read-side op handlers than D16 alone covered), D21 (`putRawData` engine signature was bare and would have broken compilation), D22 (`gbrain call` had no `--source` flag in its grammar so D11's test cases were unreachable). Documented decisions D1–D22 in the plan file at `~/.claude/plans/system-instruction-you-are-working-polished-lerdorf.md`.
|
||||
|
||||
The deferred `gbrain sources rehome <X>` command stays tracked as a v0.32.0 follow-up alongside `embed.ts` source-aware threading and `reconcile-links.ts` `--source` flag exposure. The new `@garrytan/gbrain` scoped-name npm publishing TODO captures the structural fix for the npm-name-squatter problem that `classifyBunInstall` warns about.
|
||||
|
||||
## [0.31.7] - 2026-05-09
|
||||
|
||||
**`gbrain doctor` stops crying wolf, period — and finds itself on every deployment shape. Five community PRs land one cohesive narrative: every false-positive class on disk today, plus the install-path footgun that bit every hosted-CLI user who ever ran `gbrain doctor` from `~`.**
|
||||
|
||||
@@ -1705,3 +1705,35 @@ doesn't gate on scopes. Adding per-tool scope enforcement would let
|
||||
**Effort estimate:** M (human: ~1 day / CC: ~30 min for the schema-aware gate).
|
||||
**Priority:** P3.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
---
|
||||
|
||||
### `@garrytan/gbrain` scoped-name npm publishing
|
||||
**What:** Publish gbrain to npm under the scoped name `@garrytan/gbrain`
|
||||
instead of the bare `gbrain` name. Provides structural defense against the
|
||||
unrelated `gbrain@1.x` squatter package on npm.
|
||||
|
||||
**Why:** `classifyBunInstall()` at `src/commands/upgrade.ts:395` does a
|
||||
best-effort fingerprint check on `repository.url` + `src/cli.ts` marker, with
|
||||
the comment explicitly accepting that signals are spoofable by a determined
|
||||
squatter. Scoped publishing is the structural answer that closes the loop:
|
||||
`bun add -g @garrytan/gbrain` cannot collide with any non-`@garrytan` package.
|
||||
|
||||
**Pros:** closes the squatter vector; consistent with how high-trust npm
|
||||
packages are published; allows removing `classifyBunInstall`'s spoofable
|
||||
signals later.
|
||||
|
||||
**Cons:** multi-week effort; needs reverse-compatible upgrade path for users
|
||||
on the bare-name install (`bun add -g gbrain` → recovery message pointing
|
||||
at the new scoped name); npm publishing flow changes; CI publish step needs
|
||||
scope-aware tagging.
|
||||
|
||||
**Context:** tracked at `src/commands/upgrade.ts:392-394` since v0.29; reaffirmed
|
||||
during v0.31.8 codex outside-voice review. Issue #658 has the surface-level
|
||||
history.
|
||||
|
||||
**Effort estimate:** L (human: ~1 week / CC: ~half a day for the publishing
|
||||
flow + recovery messaging).
|
||||
**Priority:** P2.
|
||||
**Depends on:** decision on whether to deprecate the bare name or dual-publish
|
||||
during a transition window.
|
||||
|
||||
+1
-2
@@ -1,7 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
|
||||
"version": "0.31.7",
|
||||
"version": "0.31.8",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -5,8 +5,15 @@
|
||||
# shard-index: 1-based (1..N)
|
||||
# total-shards: positive integer
|
||||
#
|
||||
# E2E tests under test/e2e/ are excluded — they need DATABASE_URL and run via
|
||||
# bun run test:e2e separately.
|
||||
# Excluded from sharding:
|
||||
# - test/e2e/* — need DATABASE_URL; run via bun run test:e2e
|
||||
# - *.serial.test.ts — concurrency-unsafe (file-wide mock.module / env
|
||||
# leaks); run via scripts/run-serial-tests.sh on
|
||||
# shard 1 only. Including these here lets their
|
||||
# mock.module() calls leak into the rest of the
|
||||
# shard's bun process and silently break unrelated
|
||||
# tests. See test/eval-takes-quality-runner.serial.test.ts
|
||||
# mocking gateway.ts → voyage-multimodal failures.
|
||||
#
|
||||
# Stable partitioning: a file's shard is `(hash(path) % N) + 1`. Same file
|
||||
# lands in the same shard on every run, regardless of how many other files
|
||||
|
||||
+20
-2
@@ -157,7 +157,7 @@ async function main() {
|
||||
// Local engine path (unchanged behavior for local installs).
|
||||
const engine = await connectEngine();
|
||||
try {
|
||||
const ctx = makeContext(engine, params);
|
||||
const ctx = await makeContext(engine, params);
|
||||
const rawResult = await op.handler(ctx, params);
|
||||
// ENG-2 (renderer parity by data shape): JSON-round-trip the local-engine
|
||||
// path's return value so renderers see the same shape they'd see on the
|
||||
@@ -479,7 +479,24 @@ function parseOpArgs(op: Operation, args: string[]): Record<string, unknown> {
|
||||
return params;
|
||||
}
|
||||
|
||||
function makeContext(engine: BrainEngine, params: Record<string, unknown>): OperationContext {
|
||||
async function makeContext(engine: BrainEngine, params: Record<string, unknown>): Promise<OperationContext> {
|
||||
// v0.31.8 (D11): resolve sourceId via the canonical 6-tier chain. Honors
|
||||
// --source / GBRAIN_SOURCE / .gbrain-source / path-match / brain default /
|
||||
// 'default'. Wrapped in try/catch so a doctor / single-source brain that
|
||||
// never set up sources still returns 'default' silently.
|
||||
let sourceId: string | undefined;
|
||||
try {
|
||||
const { resolveSourceId } = await import('./core/source-resolver.ts');
|
||||
// params.source is set when a CLI flag was parsed for the op (rare; most
|
||||
// CLI ops don't take --source). Falls through to env/dotfile/path-match.
|
||||
const explicit = (params.source as string | undefined) ?? null;
|
||||
sourceId = await resolveSourceId(engine, explicit);
|
||||
} catch {
|
||||
// Source resolution failed (e.g. sources table doesn't exist on a fresh
|
||||
// pre-init brain). Leave sourceId unset; engine read methods fall through
|
||||
// to the cross-source view (D16 back-compat path).
|
||||
sourceId = undefined;
|
||||
}
|
||||
return {
|
||||
engine,
|
||||
config: loadConfig() || { engine: 'postgres' },
|
||||
@@ -489,6 +506,7 @@ function makeContext(engine: BrainEngine, params: Record<string, unknown>): Oper
|
||||
// confinement (e.g., cwd-locked file_upload).
|
||||
remote: false,
|
||||
cliOpts: getCliOptions(),
|
||||
...(sourceId ? { sourceId } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+41
-4
@@ -1,16 +1,53 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { handleToolCall } from '../mcp/server.ts';
|
||||
import { resolveSourceId } from '../core/source-resolver.ts';
|
||||
|
||||
/**
|
||||
* `gbrain call <tool> <json>` — trusted local op-dispatch surface.
|
||||
*
|
||||
* v0.31.8 (D22): grammar accepts an optional `--source <id>` flag before the
|
||||
* tool name. The flag is the highest-priority tier in resolveSourceId()'s
|
||||
* 6-tier chain (--source > GBRAIN_SOURCE > .gbrain-source dotfile > path-match
|
||||
* > brain default > 'default'). Without --source, the chain still resolves —
|
||||
* env / dotfile / path-match all work.
|
||||
*/
|
||||
export async function runCall(engine: BrainEngine, args: string[]) {
|
||||
const tool = args[0];
|
||||
const jsonStr = args[1];
|
||||
// Parse --source <id> from anywhere in args (must come before tool/json
|
||||
// tokens to keep the existing `gbrain call <tool> <json>` shape readable,
|
||||
// but the parser is positional-tolerant for ergonomics).
|
||||
let explicitSource: string | null = null;
|
||||
const rest: string[] = [];
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (a === '--source') {
|
||||
const next = args[i + 1];
|
||||
if (!next || next.startsWith('--')) {
|
||||
console.error('--source requires an id (e.g. --source jarvis-memory)');
|
||||
process.exit(1);
|
||||
}
|
||||
explicitSource = next;
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
if (a.startsWith('--source=')) {
|
||||
explicitSource = a.slice('--source='.length);
|
||||
continue;
|
||||
}
|
||||
rest.push(a);
|
||||
}
|
||||
|
||||
const tool = rest[0];
|
||||
const jsonStr = rest[1];
|
||||
|
||||
if (!tool) {
|
||||
console.error('Usage: gbrain call <tool> \'<json>\'');
|
||||
console.error("Usage: gbrain call [--source <id>] <tool> '<json>'");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const params = jsonStr ? JSON.parse(jsonStr) : {};
|
||||
const result = await handleToolCall(engine, tool, params);
|
||||
// Resolve through the canonical 6-tier chain. resolveSourceId() throws if
|
||||
// an explicit/env/dotfile id refers to a non-registered source.
|
||||
const sourceId = await resolveSourceId(engine, explicitSource);
|
||||
const result = await handleToolCall(engine, tool, params, { sourceId });
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
}
|
||||
|
||||
+170
-1
@@ -197,6 +197,51 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
});
|
||||
}
|
||||
|
||||
// 3b. Migration wedge hint (v0.31.8 — D14 + D19). The brain server's
|
||||
// filesystem holds the migration ledger; the wedge condition (>=3 consecutive
|
||||
// partials with no later complete) needs the force-retry hint, not plain
|
||||
// --yes. Same shape as the local doctor at line ~336.
|
||||
try {
|
||||
const completed = loadCompletedMigrations();
|
||||
const byVersion = new Map<string, { complete: boolean; partial: boolean }>();
|
||||
for (const entry of completed) {
|
||||
const seen = byVersion.get(entry.version) ?? { complete: false, partial: false };
|
||||
if (entry.status === 'complete') seen.complete = true;
|
||||
if (entry.status === 'partial') seen.partial = true;
|
||||
byVersion.set(entry.version, seen);
|
||||
}
|
||||
const completedVersions = Array.from(byVersion.entries()).filter(([, s]) => s.complete).map(([v]) => v);
|
||||
const stuck = Array.from(byVersion.entries())
|
||||
.filter(([v, s]) => {
|
||||
if (!s.partial || s.complete) return false;
|
||||
const supersededBy = completedVersions.find(cv => compareVersions(cv, v) >= 0);
|
||||
return supersededBy === undefined;
|
||||
})
|
||||
.map(([v]) => v);
|
||||
const wedged: string[] = [];
|
||||
for (const v of stuck) {
|
||||
const partialCount = completed.filter(e => e.version === v && e.status === 'partial').length;
|
||||
if (partialCount >= 3) wedged.push(v);
|
||||
}
|
||||
if (wedged.length > 0) {
|
||||
const cmd = wedged.map(v => `gbrain apply-migrations --force-retry ${v}`).join(' && ');
|
||||
checks.push({
|
||||
name: 'minions_migration',
|
||||
status: 'fail',
|
||||
message: `WEDGED MIGRATION(s) on brain host: ${wedged.join(', ')}. Run on the host: ${cmd}`,
|
||||
});
|
||||
} else if (stuck.length > 0) {
|
||||
checks.push({
|
||||
name: 'minions_migration',
|
||||
status: 'fail',
|
||||
message: `MINIONS HALF-INSTALLED on brain host: ${stuck.join(', ')}. Run on the host: gbrain apply-migrations --yes`,
|
||||
});
|
||||
}
|
||||
} catch {
|
||||
// Best-effort. A broken JSONL on the brain server should not stop the
|
||||
// remote doctor.
|
||||
}
|
||||
|
||||
// 4. Sync failures (file-plane state, not in-DB; see src/core/sync.ts).
|
||||
// Read the JSONL file directly at the canonical path; cheap and engine-agnostic.
|
||||
try {
|
||||
@@ -224,6 +269,48 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
checks.push({ name: 'sync_failures', status: 'ok', message: 'No failures recorded' });
|
||||
}
|
||||
|
||||
// 4b. Multi-source drift (v0.31.8 — D8 + D14). Same shape as the local
|
||||
// doctor's check at the same name. Runs server-side; the result is
|
||||
// returned to the thin-client over MCP.
|
||||
try {
|
||||
const { findMisroutedPages } = await import('../core/multi-source-drift.ts');
|
||||
const sources = await engine.executeRaw<{ id: string; local_path: string | null }>(
|
||||
`SELECT id, local_path FROM sources`,
|
||||
);
|
||||
const nonDefaultWithPath = sources.filter(s => s.id !== 'default' && s.local_path);
|
||||
if (sources.length > 1 && nonDefaultWithPath.length > 0) {
|
||||
const result = await findMisroutedPages(
|
||||
engine,
|
||||
nonDefaultWithPath.map(s => ({ id: s.id, local_path: s.local_path as string })),
|
||||
);
|
||||
if (result.walk_truncated) {
|
||||
checks.push({
|
||||
name: 'multi_source_drift',
|
||||
status: 'warn',
|
||||
message: 'Multi-source drift check skipped — FS walk hit limit/timeout on the brain server.',
|
||||
});
|
||||
} else if (result.count > 0) {
|
||||
const sampleStr = result.sample.map(s => `${s.slug} (intended=${s.intended_source})`).join(', ');
|
||||
checks.push({
|
||||
name: 'multi_source_drift',
|
||||
status: 'warn',
|
||||
message:
|
||||
`${result.count} page slug(s) appear at 'default' but NOT at the intended source ` +
|
||||
`(e.g., ${sampleStr}). Likely pre-v0.30.3 misroutes OR an incomplete initial sync. ` +
|
||||
`Verify on the brain host: \`gbrain sources status\` then \`gbrain sync --source <id> --full\`.`,
|
||||
});
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'multi_source_drift',
|
||||
status: 'ok',
|
||||
message: 'No cross-source slug drift detected.',
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Best-effort, like the rest of doctorReportRemote.
|
||||
}
|
||||
|
||||
// 5. Queue health (Postgres-only). PGLite has no minion_jobs in the same
|
||||
// shape; skip the check there with an informational message.
|
||||
if (engine.kind === 'postgres') {
|
||||
@@ -395,7 +482,36 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
return supersededBy === undefined;
|
||||
})
|
||||
.map(([v]) => v);
|
||||
if (stuck.length > 0) {
|
||||
|
||||
// v0.31.8 (D19): detect 3-consecutive-partials shape (the apply-migrations
|
||||
// wedge condition). The `stuck` filter above already excludes
|
||||
// forward-progress-superseded versions, so we only count actual unresolved
|
||||
// partials per version. A version with >=3 trailing partials needs
|
||||
// `gbrain apply-migrations --force-retry <v>` once before plain --yes
|
||||
// will succeed (the 3-consecutive-partials guard in apply-migrations.ts
|
||||
// is still active). Without this hint, operators wedged on v0.29.1 (and
|
||||
// any future migration that hits the same guard) get "run --yes" advice
|
||||
// that won't unstick them.
|
||||
const wedged: string[] = [];
|
||||
for (const v of stuck) {
|
||||
const partialCount = completed.filter(
|
||||
e => e.version === v && e.status === 'partial',
|
||||
).length;
|
||||
if (partialCount >= 3) wedged.push(v);
|
||||
}
|
||||
|
||||
if (wedged.length > 0) {
|
||||
// The wedged set is a STRICT subset of the stuck set, so a wedged
|
||||
// version is also stuck. Surface the force-retry hint instead of the
|
||||
// generic --yes hint; chained with `&&` when multiple versions are
|
||||
// wedged so the operator can copy-paste a single line.
|
||||
const cmd = wedged.map(v => `gbrain apply-migrations --force-retry ${v}`).join(' && ');
|
||||
checks.push({
|
||||
name: 'minions_migration',
|
||||
status: 'fail',
|
||||
message: `WEDGED MIGRATION(s): ${wedged.join(', ')} (>=3 consecutive partials). Run: ${cmd}`,
|
||||
});
|
||||
} else if (stuck.length > 0) {
|
||||
checks.push({
|
||||
name: 'minions_migration',
|
||||
status: 'fail',
|
||||
@@ -532,6 +648,59 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
// Best-effort. A broken JSONL should not stop doctor.
|
||||
}
|
||||
|
||||
// 3b-multi-source. Multi-source drift (v0.31.8 — D8 + D17 + OV12 + OV13).
|
||||
// Pre-v0.30.3 putPage misrouted multi-source writes to (default, slug).
|
||||
// For each non-default source with local_path set, walk the FS and surface
|
||||
// slugs that exist at default but NOT at the intended source. Only runs
|
||||
// on multi-source brains (sources count > 1). Single-source brains skip.
|
||||
// Engine is nullable in runDoctor (--fast / DB-down skip the DB phase);
|
||||
// bail silently here when engine is null since the check needs DB access.
|
||||
if (engine !== null) try {
|
||||
const { findMisroutedPages } = await import('../core/multi-source-drift.ts');
|
||||
const sources = await engine!.executeRaw<{ id: string; local_path: string | null }>(
|
||||
`SELECT id, local_path FROM sources`,
|
||||
);
|
||||
const nonDefaultWithPath = sources.filter(s => s.id !== 'default' && s.local_path);
|
||||
if (sources.length > 1 && nonDefaultWithPath.length > 0) {
|
||||
const result = await findMisroutedPages(
|
||||
engine!,
|
||||
nonDefaultWithPath.map(s => ({ id: s.id, local_path: s.local_path as string })),
|
||||
);
|
||||
if (result.walk_truncated) {
|
||||
checks.push({
|
||||
name: 'multi_source_drift',
|
||||
status: 'warn',
|
||||
message:
|
||||
`Multi-source drift check skipped — FS walk hit limit/timeout. ` +
|
||||
`Re-run on a quieter brain or shorter walk via GBRAIN_DRIFT_LIMIT/GBRAIN_DRIFT_TIMEOUT_MS.`,
|
||||
});
|
||||
} else if (result.count > 0) {
|
||||
const sampleStr = result.sample.map(s => `${s.slug} (intended=${s.intended_source})`).join(', ');
|
||||
checks.push({
|
||||
name: 'multi_source_drift',
|
||||
status: 'warn',
|
||||
message:
|
||||
`${result.count} page slug(s) appear at 'default' but NOT at the intended source ` +
|
||||
`(e.g., ${sampleStr}). Two possible causes: (1) pre-v0.30.3 putPage misroutes; ` +
|
||||
`(2) source X never completed initial sync and the default page is unrelated. ` +
|
||||
`Verify with 'gbrain sources status', then either re-sync with ` +
|
||||
`'gbrain sync --source <id> --full' or 'gbrain delete <slug>' if the default-source ` +
|
||||
`row is the misroute. (A 'gbrain sources rehome' cleanup command is tracked for v0.32.0.)`,
|
||||
});
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'multi_source_drift',
|
||||
status: 'ok',
|
||||
message: 'No cross-source slug drift detected.',
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Best-effort. A broken sources table or unreadable local_path should
|
||||
// not stop doctor. The walk itself catches per-directory errors; this
|
||||
// outer try covers the executeRaw path.
|
||||
}
|
||||
|
||||
// 3c. Orphan clone temp dirs (v0.28 P1). `gbrain sources add --url` clones
|
||||
// into $GBRAIN_HOME/clones/.tmp/<id>-<rand>/ and renames atomically; if the
|
||||
// process is SIGKILL'd between clone-finish and rename, the temp dir
|
||||
|
||||
@@ -84,6 +84,16 @@ const DEFAULT_CHARS_PER_TOKEN = 4;
|
||||
/** Default safety factor when a recipe omits it. */
|
||||
const DEFAULT_SAFETY_FACTOR = 0.8;
|
||||
|
||||
/**
|
||||
* v0.31.8 (D2 + D10): hard ceiling on Voyage response size, sized as
|
||||
* "unambiguously not a real Voyage response" rather than tight against
|
||||
* typical batches. voyage-3-large × 16K embeddings ≈ 200 MB raw (3072
|
||||
* dims × 4 bytes × 16K), which fits within this cap. Anything larger is
|
||||
* unambiguously not legitimate. Layer 1 (Content-Length pre-check) and
|
||||
* Layer 2 (per-embedding base64 cap) both compare against this constant.
|
||||
*/
|
||||
const MAX_VOYAGE_RESPONSE_BYTES = 256 * 1024 * 1024;
|
||||
|
||||
/** Configure the gateway. Called by cli.ts#connectEngine. Clears cached models. */
|
||||
export function configureGateway(config: AIGatewayConfig): void {
|
||||
_config = {
|
||||
@@ -317,6 +327,30 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit)
|
||||
const ct = resp.headers.get('content-type') ?? '';
|
||||
if (!ct.toLowerCase().includes('application/json')) return resp;
|
||||
|
||||
// v0.31.8 (D2 + D10): Layer 1 — Content-Length pre-check BEFORE the
|
||||
// body is parsed. The pre-fix code did `await resp.clone().json()`
|
||||
// first, which fully parses arbitrary-size JSON into JS heap before
|
||||
// any size check could fire. A compromised/malicious Voyage endpoint
|
||||
// could OOM the worker on a single response. The 256 MB cap is sized
|
||||
// as "unambiguously not a real Voyage response" — voyage-3-large at
|
||||
// 3072 dims × 4 bytes × 16K embeddings (the plausible upper bound on
|
||||
// realistic load) decodes to ~200 MB raw and fits. Anything bigger
|
||||
// is unambiguously not legitimate.
|
||||
//
|
||||
// When Content-Length is missing (chunked transfer encoding), we
|
||||
// proceed and rely on Layer 2 (per-embedding base64 length check)
|
||||
// for OOM defense.
|
||||
const contentLengthHeader = resp.headers.get('content-length');
|
||||
if (contentLengthHeader) {
|
||||
const len = parseInt(contentLengthHeader, 10);
|
||||
if (Number.isFinite(len) && len > MAX_VOYAGE_RESPONSE_BYTES) {
|
||||
throw new Error(
|
||||
`Voyage response Content-Length=${len} exceeds ${MAX_VOYAGE_RESPONSE_BYTES} bytes — ` +
|
||||
`likely compromised endpoint or misconfiguration`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// INBOUND: rewrite response so the AI SDK's Zod schema validates.
|
||||
// Voyage diverges from OpenAI in two places that break the parser:
|
||||
// - `embedding` is a base64 string (SDK schema expects `number[]`)
|
||||
@@ -328,6 +362,18 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit)
|
||||
if (Array.isArray(json.data)) {
|
||||
for (const item of json.data) {
|
||||
if (item && typeof item.embedding === 'string') {
|
||||
// v0.31.8 (D10 Layer 2): per-embedding cap. Catches the rare
|
||||
// case where Layer 1 was skipped (no Content-Length on chunked
|
||||
// encoding) AND a single embedding string is unreasonably large.
|
||||
// Estimate decoded size as 0.75 × base64 length (the canonical
|
||||
// base64 → bytes ratio).
|
||||
const estDecoded = Math.ceil(item.embedding.length * 0.75);
|
||||
if (estDecoded > MAX_VOYAGE_RESPONSE_BYTES) {
|
||||
throw new Error(
|
||||
`Voyage embedding base64 exceeds ${MAX_VOYAGE_RESPONSE_BYTES} bytes ` +
|
||||
`(estimated ${estDecoded} bytes from ${item.embedding.length} base64 chars)`,
|
||||
);
|
||||
}
|
||||
// Voyage returns Float32 little-endian base64.
|
||||
const bytes = Buffer.from(item.embedding, 'base64');
|
||||
const floats = new Float32Array(
|
||||
|
||||
+44
-7
@@ -490,8 +490,13 @@ export interface BrainEngine {
|
||||
* Returns the slug of every page in the brain. Used by batch commands as a
|
||||
* mutation-immune iteration source (alternative to listPages OFFSET pagination,
|
||||
* which is unstable when ordering by updated_at and writes are happening).
|
||||
*
|
||||
* v0.31.8 (D12): `opts.sourceId` scopes the result to a single source
|
||||
* (used by the source-aware reconcileLinks path so wikilink resolution
|
||||
* doesn't span unrelated sources). When omitted, returns the union of
|
||||
* slugs across every source (pre-v0.31.8 behavior).
|
||||
*/
|
||||
getAllSlugs(): Promise<Set<string>>;
|
||||
getAllSlugs(opts?: { sourceId?: string }): Promise<Set<string>>;
|
||||
|
||||
// Search
|
||||
searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]>;
|
||||
@@ -579,8 +584,19 @@ export interface BrainEngine {
|
||||
linkSource?: string,
|
||||
opts?: { fromSourceId?: string; toSourceId?: string },
|
||||
): Promise<void>;
|
||||
getLinks(slug: string): Promise<Link[]>;
|
||||
getBacklinks(slug: string): Promise<Link[]>;
|
||||
/**
|
||||
* v0.31.8 (D12 + D16): `opts.sourceId` source-scopes the from-page lookup.
|
||||
* When omitted, the read returns links from every same-slug page across
|
||||
* sources (pre-v0.31.8 behavior; preserved via two-branch query in both
|
||||
* engines). When set, the from-page filter becomes
|
||||
* `WHERE f.slug = $1 AND f.source_id = $X`.
|
||||
*/
|
||||
getLinks(slug: string, opts?: { sourceId?: string }): Promise<Link[]>;
|
||||
/**
|
||||
* v0.31.8 (D12 + D16): same `opts.sourceId` semantics as `getLinks`,
|
||||
* applied to the to-page side of the join.
|
||||
*/
|
||||
getBacklinks(slug: string, opts?: { sourceId?: string }): Promise<Link[]>;
|
||||
/**
|
||||
* Fuzzy-match a display name to a page slug using pg_trgm similarity.
|
||||
* Zero embedding cost, zero LLM cost — designed for the v0.13 resolver used
|
||||
@@ -695,8 +711,19 @@ export interface BrainEngine {
|
||||
getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]>;
|
||||
|
||||
// Raw data
|
||||
putRawData(slug: string, source: string, data: object): Promise<void>;
|
||||
getRawData(slug: string, source?: string): Promise<RawData[]>;
|
||||
/**
|
||||
* v0.31.8 (D21): `opts.sourceId` source-scopes the page-id lookup. When
|
||||
* omitted, the write targets the bare slug (pre-v0.31.8 behavior); the
|
||||
* Postgres 21000 hazard for multi-source brains exists on this path.
|
||||
* Multi-source callers MUST pass sourceId to land on the intended row.
|
||||
*/
|
||||
putRawData(slug: string, source: string, data: object, opts?: { sourceId?: string }): Promise<void>;
|
||||
/**
|
||||
* v0.31.8 (D21): `opts.sourceId` source-scopes the page-id lookup. Without
|
||||
* it, multi-source brains return raw_data rows from every same-slug page
|
||||
* (preserved via two-branch query for back-compat).
|
||||
*/
|
||||
getRawData(slug: string, source?: string, opts?: { sourceId?: string }): Promise<RawData[]>;
|
||||
|
||||
// Files (v0.27.1: binary asset metadata + storage_path. Image bytes never
|
||||
// enter the DB; storage_path references a path inside the brain repo or an
|
||||
@@ -908,8 +935,18 @@ export interface BrainEngine {
|
||||
* first when the slug exists across multiple sources.
|
||||
*/
|
||||
createVersion(slug: string, opts?: { sourceId?: string }): Promise<PageVersion>;
|
||||
getVersions(slug: string): Promise<PageVersion[]>;
|
||||
revertToVersion(slug: string, versionId: number): Promise<void>;
|
||||
/**
|
||||
* v0.31.8 (D12 + D16): `opts.sourceId` source-scopes the page-id lookup.
|
||||
* When omitted, returns versions for every same-slug page across sources
|
||||
* (pre-v0.31.8 behavior; preserved via two-branch query).
|
||||
*/
|
||||
getVersions(slug: string, opts?: { sourceId?: string }): Promise<PageVersion[]>;
|
||||
/**
|
||||
* v0.31.8 (D12): `opts.sourceId` source-scopes both the version lookup
|
||||
* and the page revert. Without it, multi-source brains can revert the
|
||||
* wrong row when the slug exists in 2+ sources.
|
||||
*/
|
||||
revertToVersion(slug: string, versionId: number, opts?: { sourceId?: string }): Promise<void>;
|
||||
|
||||
// Stats + health
|
||||
getStats(): Promise<BrainStats>;
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
/**
|
||||
* Multi-source drift detection (v0.31.8 — D8 + D17 + OV12 + OV13).
|
||||
*
|
||||
* Pre-v0.30.3 putPage misrouted multi-source writes from intended source X
|
||||
* to (default, slug). The fixwave fixed forward-going writes but explicitly
|
||||
* deferred backfilling the misrouted rows. This module surfaces evidence of
|
||||
* misroute to operators via `gbrain doctor`.
|
||||
*
|
||||
* Heuristic (codex OV12 — softened from "is misrouted" to "appears misrouted"):
|
||||
* a non-default source X is configured with `local_path`, AND the filesystem
|
||||
* at `local_path` contains a markdown file whose slug exists at (default,
|
||||
* slug) in the DB but is missing from (X, slug). Two possible causes:
|
||||
* 1. Pre-v0.30.3 putPage misroute (the case this check was designed for).
|
||||
* 2. Source X never completed initial sync, and the default page is
|
||||
* unrelated content that happens to share the slug.
|
||||
* The doctor warning surfaces evidence; the operator decides which cause
|
||||
* applies and runs `gbrain sync --source X --full` or `gbrain delete <slug>`
|
||||
* accordingly.
|
||||
*
|
||||
* Implementation notes:
|
||||
* - FS walk handles `.md` AND `.mdx` (codex OV13: matches `src/core/sync.ts`
|
||||
* which treats both as markdown).
|
||||
* - Batched single-query DB lookup (D17): collect all candidate slugs from
|
||||
* the FS walk into one array, then run ONE SELECT against pages with a
|
||||
* VALUES clause. NOT a per-file loop (which would be 20K round trips on
|
||||
* a 10K-file source).
|
||||
* - Time + size bounds: cap the walk at 10K files OR 5s. Bail with a "check
|
||||
* skipped, walk too large" status instead of letting doctor hang.
|
||||
* - Wrapper try/catch around the walk per OV13: ENOENT/EACCES on local_path
|
||||
* yields zero files, NOT a thrown crash that takes down the whole doctor
|
||||
* run.
|
||||
*/
|
||||
|
||||
import { readdirSync, lstatSync, statSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import { pathToSlug } from './sync.ts';
|
||||
|
||||
export interface SourceWithPath {
|
||||
id: string;
|
||||
local_path: string;
|
||||
}
|
||||
|
||||
export interface MisroutedSample {
|
||||
slug: string;
|
||||
intended_source: string;
|
||||
local_path: string;
|
||||
}
|
||||
|
||||
export interface MisroutedResult {
|
||||
/** True when the FS walk hit the limit/timeout and the result is partial. */
|
||||
walk_truncated: boolean;
|
||||
/** Per-source breakdown: slugs that appear at (default, slug) but NOT at (X, slug). */
|
||||
count: number;
|
||||
sample: MisroutedSample[];
|
||||
}
|
||||
|
||||
const DEFAULT_FILE_LIMIT = 10_000;
|
||||
const DEFAULT_TIMEOUT_MS = 5_000;
|
||||
const SAMPLE_LIMIT = 5;
|
||||
|
||||
/**
|
||||
* Walk a directory tree for `.md` + `.mdx` files. Skips dotfiles (`.git`),
|
||||
* `_*.md` files (the existing extract.ts convention), and silently swallows
|
||||
* read errors on individual entries. Returns relative paths from `root`.
|
||||
*
|
||||
* Bounded by `limit` (max files) and `deadlineMs` (epoch ms). Returns early
|
||||
* with `truncated=true` if either bound is hit. The root-not-readable case
|
||||
* surfaces as `truncated=false, files=[]` (caller treats as "no candidates").
|
||||
*/
|
||||
function walkMarkdownAndMdxFiles(
|
||||
root: string,
|
||||
limit: number,
|
||||
deadlineMs: number,
|
||||
): { files: { relPath: string }[]; truncated: boolean } {
|
||||
const files: { relPath: string }[] = [];
|
||||
let truncated = false;
|
||||
function walk(d: string): void {
|
||||
if (truncated) return;
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = readdirSync(d);
|
||||
} catch {
|
||||
// Unreadable directory; skip without crashing the whole walk.
|
||||
return;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
if (truncated) return;
|
||||
if (entry.startsWith('.')) continue;
|
||||
const full = join(d, entry);
|
||||
let isDir = false;
|
||||
try {
|
||||
isDir = lstatSync(full).isDirectory();
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (isDir) {
|
||||
walk(full);
|
||||
continue;
|
||||
}
|
||||
const isMd = entry.endsWith('.md') || entry.endsWith('.mdx');
|
||||
if (!isMd) continue;
|
||||
if (entry.startsWith('_')) continue; // matches extract.ts convention
|
||||
files.push({ relPath: relative(root, full) });
|
||||
if (files.length >= limit) {
|
||||
truncated = true;
|
||||
return;
|
||||
}
|
||||
// Time check is cheap; do it on every push so a slow filesystem can't
|
||||
// run unbounded.
|
||||
if (Date.now() >= deadlineMs) {
|
||||
truncated = true;
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
// Wrap the top-level walk in try/catch so a missing/unreadable root
|
||||
// doesn't bubble up to doctor (codex OV13 — pre-fix the readdirSync at
|
||||
// the root would throw and crash the whole doctor run).
|
||||
try {
|
||||
statSync(root); // probe readable; throws ENOENT/EACCES if not
|
||||
walk(root);
|
||||
} catch {
|
||||
// local_path is unreadable; return zero files, NOT truncated. Caller
|
||||
// surfaces this as "ok with note" rather than an error.
|
||||
}
|
||||
return { files, truncated };
|
||||
}
|
||||
|
||||
/**
|
||||
* For a list of slugs, query DB for existence at (default, slug) AND at
|
||||
* (sourceId, slug) in ONE batched query. Returns a Map<slug, Set<source_id>>.
|
||||
*
|
||||
* Engine-agnostic: uses executeRaw with a VALUES clause. PGLite + Postgres
|
||||
* both support the shape.
|
||||
*/
|
||||
async function batchProbeExistence(
|
||||
engine: BrainEngine,
|
||||
slugs: string[],
|
||||
sourceId: string,
|
||||
): Promise<Map<string, Set<string>>> {
|
||||
if (slugs.length === 0) return new Map();
|
||||
// Build a positional VALUES clause: ($1::text), ($2), ($3), ...
|
||||
const valuePlaceholders = slugs.map((_, i) => `($${i + 1}::text)`).join(', ');
|
||||
const sourceParamIdx = slugs.length + 1;
|
||||
const sql = `
|
||||
WITH candidates(slug) AS (VALUES ${valuePlaceholders})
|
||||
SELECT c.slug, p.source_id
|
||||
FROM candidates c
|
||||
LEFT JOIN pages p
|
||||
ON p.slug = c.slug AND p.deleted_at IS NULL
|
||||
AND p.source_id IN ('default', $${sourceParamIdx}::text)
|
||||
ORDER BY c.slug, p.source_id
|
||||
`;
|
||||
const rows = await engine.executeRaw<{ slug: string; source_id: string | null }>(
|
||||
sql,
|
||||
[...slugs, sourceId],
|
||||
);
|
||||
const map = new Map<string, Set<string>>();
|
||||
for (const r of rows) {
|
||||
if (!map.has(r.slug)) map.set(r.slug, new Set());
|
||||
if (r.source_id != null) map.get(r.slug)!.add(r.source_id);
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
/**
|
||||
* Find pages that appear misrouted from intended source X to source 'default'.
|
||||
* For each non-default source with a configured local_path, walk the
|
||||
* filesystem and cross-check against the DB.
|
||||
*
|
||||
* @returns aggregated MisroutedResult across all checked sources. The sample
|
||||
* array is bounded at 5 entries so the doctor message stays scannable.
|
||||
*/
|
||||
export async function findMisroutedPages(
|
||||
engine: BrainEngine,
|
||||
sources: SourceWithPath[],
|
||||
opts: { limit?: number; timeoutMs?: number } = {},
|
||||
): Promise<MisroutedResult> {
|
||||
const limit = opts.limit ?? DEFAULT_FILE_LIMIT;
|
||||
const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
||||
const deadlineMs = Date.now() + timeoutMs;
|
||||
|
||||
let totalCount = 0;
|
||||
let walkTruncated = false;
|
||||
const sample: MisroutedSample[] = [];
|
||||
|
||||
for (const src of sources) {
|
||||
if (src.id === 'default') continue;
|
||||
if (!src.local_path) continue;
|
||||
if (Date.now() >= deadlineMs) {
|
||||
walkTruncated = true;
|
||||
break;
|
||||
}
|
||||
const { files, truncated } = walkMarkdownAndMdxFiles(src.local_path, limit, deadlineMs);
|
||||
if (truncated) walkTruncated = true;
|
||||
if (files.length === 0) continue;
|
||||
|
||||
// Convert FS paths to canonical slugs (lowercased, extension stripped).
|
||||
const slugs = Array.from(new Set(files.map(f => pathToSlug(f.relPath))));
|
||||
const existenceMap = await batchProbeExistence(engine, slugs, src.id);
|
||||
|
||||
for (const slug of slugs) {
|
||||
const present = existenceMap.get(slug);
|
||||
if (!present) continue; // missing both — uningested, not misroute
|
||||
const hasDefault = present.has('default');
|
||||
const hasSource = present.has(src.id);
|
||||
// The misroute heuristic: present at default, missing from intended source.
|
||||
if (hasDefault && !hasSource) {
|
||||
totalCount++;
|
||||
if (sample.length < SAMPLE_LIMIT) {
|
||||
sample.push({ slug, intended_source: src.id, local_path: src.local_path });
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { walk_truncated: walkTruncated, count: totalCount, sample };
|
||||
}
|
||||
+102
-28
@@ -376,14 +376,20 @@ const get_page: Operation = {
|
||||
const slug = p.slug as string;
|
||||
const fuzzy = (p.fuzzy as boolean) || false;
|
||||
const includeDeleted = (p.include_deleted as boolean) === true;
|
||||
// v0.31.8 (D20): thread ctx.sourceId through read-side ops. Only pass
|
||||
// sourceId when it's set on ctx — when unset (local CLI default chain
|
||||
// resolves to no source), the engine two-branch query falls through to
|
||||
// the cross-source view, preserving pre-v0.31.8 behavior. MCP callers
|
||||
// (stdio + HTTP) populate ctx.sourceId via the transport layer.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
|
||||
let page = await ctx.engine.getPage(slug, { includeDeleted });
|
||||
let page = await ctx.engine.getPage(slug, { includeDeleted, ...sourceOpts });
|
||||
let resolved_slug: string | undefined;
|
||||
|
||||
if (!page && fuzzy) {
|
||||
const candidates = await ctx.engine.resolveSlugs(slug);
|
||||
if (candidates.length === 1) {
|
||||
page = await ctx.engine.getPage(candidates[0], { includeDeleted });
|
||||
page = await ctx.engine.getPage(candidates[0], { includeDeleted, ...sourceOpts });
|
||||
resolved_slug = candidates[0];
|
||||
} else if (candidates.length > 1) {
|
||||
return { error: 'ambiguous_slug', candidates };
|
||||
@@ -394,7 +400,7 @@ const get_page: Operation = {
|
||||
throw new OperationError('page_not_found', `Page not found: ${slug}`, includeDeleted ? 'Check the slug or use fuzzy: true' : 'Page may be soft-deleted; pass include_deleted: true to verify');
|
||||
}
|
||||
|
||||
const tags = await ctx.engine.getTags(page.slug);
|
||||
const tags = await ctx.engine.getTags(page.slug, sourceOpts);
|
||||
// Privacy boundary for the per-token takes-holder allow-list (v0.28.6).
|
||||
// takes_list / takes_search / think.gather filter rows by holder at the
|
||||
// SQL layer, but takes are also rendered as a markdown table inside the
|
||||
@@ -464,7 +470,15 @@ const put_page: Operation = {
|
||||
// so Gemini / Ollama / Voyage brains don't silently drop embeddings (Codex C2).
|
||||
const { isAvailable } = await import('./ai/gateway.ts');
|
||||
const noEmbed = !isAvailable('embedding');
|
||||
const result = await importFromContent(ctx.engine, slug, p.content as string, { noEmbed });
|
||||
// v0.31.8 (D7 / codex OV-1): thread ctx.sourceId so put_page on a
|
||||
// multi-source brain lands in the intended source instead of the
|
||||
// default-source clobber path. importFromContent already accepts
|
||||
// opts.sourceId (PR #707/#757 engine work); previously the op handler
|
||||
// just didn't pass it.
|
||||
const result = await importFromContent(ctx.engine, slug, p.content as string, {
|
||||
noEmbed,
|
||||
...(ctx.sourceId ? { sourceId: ctx.sourceId } : {}),
|
||||
});
|
||||
|
||||
// Auto-link post-hook: runs AFTER importFromContent (which is its own
|
||||
// transaction). Runs even on status='skipped' so reconciliation catches drift
|
||||
@@ -499,7 +513,7 @@ const put_page: Operation = {
|
||||
try {
|
||||
const enabled = await isAutoLinkEnabled(ctx.engine);
|
||||
if (enabled) {
|
||||
autoLinks = await runAutoLink(ctx.engine, slug, result.parsedPage);
|
||||
autoLinks = await runAutoLink(ctx.engine, slug, result.parsedPage, ctx.sourceId ? { sourceId: ctx.sourceId } : undefined);
|
||||
}
|
||||
} catch (e) {
|
||||
autoLinks = { error: e instanceof Error ? e.message : String(e) };
|
||||
@@ -628,8 +642,23 @@ async function runAutoLink(
|
||||
engine: BrainEngine,
|
||||
slug: string,
|
||||
parsed: { type: PageType; compiled_truth: string; timeline: string; frontmatter: Record<string, unknown> },
|
||||
opts?: { sourceId?: string },
|
||||
): Promise<{ created: number; removed: number; errors: number; unresolved: UnresolvedFrontmatterRef[] }> {
|
||||
const fullContent = parsed.compiled_truth + '\n' + parsed.timeline;
|
||||
// v0.31.8 (codex OV-2): thread sourceId through every read + write inside
|
||||
// reconcileLinks. Without this the FS walker reads cross-source links/slugs
|
||||
// but writes scoped to one source — phantom stale-deletions and duplicate
|
||||
// inserts. opts.sourceId is set when caller knows the source (put_page from
|
||||
// a multi-source-aware handler); when omitted, every read returns the
|
||||
// pre-v0.31.8 cross-source view (back-compat for any existing caller).
|
||||
const sourceOpts = opts?.sourceId ? { sourceId: opts.sourceId } : {};
|
||||
const linkSourceOpts = opts?.sourceId
|
||||
? { fromSourceId: opts.sourceId, toSourceId: opts.sourceId, originSourceId: opts.sourceId }
|
||||
: {};
|
||||
const removeSourceOpts = opts?.sourceId
|
||||
? { fromSourceId: opts.sourceId, toSourceId: opts.sourceId }
|
||||
: {};
|
||||
|
||||
// Live-mode resolver: per-put throwaway cache, pg_trgm + optional search.
|
||||
const resolver = makeResolver(engine, { mode: 'live' });
|
||||
const { candidates, unresolved } = await extractPageLinks(
|
||||
@@ -638,7 +667,9 @@ async function runAutoLink(
|
||||
|
||||
// Resolve which targets exist (skip refs to non-existent pages to avoid FK
|
||||
// violation churn in addLink). One getAllSlugs call upfront, O(1) lookup.
|
||||
const allSlugs = await engine.getAllSlugs();
|
||||
// v0.31.8 (D12): scoped to the source when opts.sourceId is set so wikilink
|
||||
// resolution doesn't span unrelated sources.
|
||||
const allSlugs = await engine.getAllSlugs(sourceOpts);
|
||||
const valid = candidates.filter(c =>
|
||||
allSlugs.has(c.targetSlug) && (!c.fromSlug || allSlugs.has(c.fromSlug))
|
||||
);
|
||||
@@ -670,10 +701,10 @@ async function runAutoLink(
|
||||
} catch {
|
||||
// engine doesn't support advisory locks — fall through
|
||||
}
|
||||
const existingOut = await tx.getLinks(slug);
|
||||
const existingOut = await tx.getLinks(slug, sourceOpts);
|
||||
// Incoming: we only look at frontmatter edges WE authored (origin_slug=slug).
|
||||
// Non-frontmatter and other-page frontmatter edges survive untouched.
|
||||
const existingInRaw = await tx.getBacklinks(slug);
|
||||
const existingInRaw = await tx.getBacklinks(slug, sourceOpts);
|
||||
const existingIn = existingInRaw.filter(
|
||||
l => l.link_source === 'frontmatter' && l.origin_slug === slug,
|
||||
);
|
||||
@@ -700,6 +731,7 @@ async function runAutoLink(
|
||||
await tx.addLink(
|
||||
slug, c.targetSlug, c.context, c.linkType,
|
||||
c.linkSource, c.originSlug, c.originField,
|
||||
linkSourceOpts,
|
||||
);
|
||||
const existKey = `${c.targetSlug}\u0000${c.linkType}\u0000${c.linkSource ?? 'markdown'}`;
|
||||
const exists = reconcilableOut.some(l =>
|
||||
@@ -717,6 +749,7 @@ async function runAutoLink(
|
||||
await tx.addLink(
|
||||
c.fromSlug!, c.targetSlug, c.context, c.linkType,
|
||||
'frontmatter', c.originSlug, c.originField,
|
||||
linkSourceOpts,
|
||||
);
|
||||
const existKey = `${c.fromSlug}\u0000${c.linkType}`;
|
||||
const exists = existingIn.some(l =>
|
||||
@@ -733,7 +766,7 @@ async function runAutoLink(
|
||||
const key = `${l.to_slug}\u0000${l.link_type}\u0000${l.link_source ?? 'markdown'}`;
|
||||
if (!outKeys.has(key)) {
|
||||
try {
|
||||
await tx.removeLink(slug, l.to_slug, l.link_type, l.link_source ?? undefined);
|
||||
await tx.removeLink(slug, l.to_slug, l.link_type, l.link_source ?? undefined, removeSourceOpts);
|
||||
removed++;
|
||||
} catch {
|
||||
errors++;
|
||||
@@ -746,7 +779,7 @@ async function runAutoLink(
|
||||
const key = `${l.from_slug}\u0000${l.link_type}`;
|
||||
if (!incKeys.has(key)) {
|
||||
try {
|
||||
await tx.removeLink(l.from_slug, slug, l.link_type, 'frontmatter');
|
||||
await tx.removeLink(l.from_slug, slug, l.link_type, 'frontmatter', removeSourceOpts);
|
||||
removed++;
|
||||
} catch {
|
||||
errors++;
|
||||
@@ -771,15 +804,18 @@ const delete_page: Operation = {
|
||||
handler: async (ctx, p) => {
|
||||
const slug = p.slug as string;
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'soft_delete_page', slug };
|
||||
// v0.31.8 (D7): thread ctx.sourceId so multi-source brains soft-delete the
|
||||
// intended row instead of always targeting (default, slug).
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
// v0.26.5: rewired from hard-delete to soft-delete. The hard-delete primitive
|
||||
// (engine.deletePage) is now reserved for purgeDeletedPages and explicit
|
||||
// tests. softDeletePage returns null when the slug is unknown OR already
|
||||
// soft-deleted (idempotent-as-null) — preserve that as a clean no-op shape.
|
||||
const result = await ctx.engine.softDeletePage(slug);
|
||||
const result = await ctx.engine.softDeletePage(slug, sourceOpts);
|
||||
if (result === null) {
|
||||
// Distinguish "not found" from "already soft-deleted" so the agent gets a
|
||||
// clear signal. Probe once with include_deleted to disambiguate.
|
||||
const existing = await ctx.engine.getPage(slug, { includeDeleted: true });
|
||||
const existing = await ctx.engine.getPage(slug, { includeDeleted: true, ...sourceOpts });
|
||||
if (!existing) {
|
||||
throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug.');
|
||||
}
|
||||
@@ -801,10 +837,12 @@ const restore_page: Operation = {
|
||||
handler: async (ctx, p) => {
|
||||
const slug = p.slug as string;
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'restore_page', slug };
|
||||
const ok = await ctx.engine.restorePage(slug);
|
||||
// v0.31.8 (D7): thread ctx.sourceId.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
const ok = await ctx.engine.restorePage(slug, sourceOpts);
|
||||
if (!ok) {
|
||||
// Distinguish "not found" from "already active" (idempotent-as-false).
|
||||
const existing = await ctx.engine.getPage(slug, { includeDeleted: true });
|
||||
const existing = await ctx.engine.getPage(slug, { includeDeleted: true, ...sourceOpts });
|
||||
if (!existing) {
|
||||
throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug.');
|
||||
}
|
||||
@@ -1244,7 +1282,9 @@ const add_tag: Operation = {
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'add_tag', slug: p.slug, tag: p.tag };
|
||||
await ctx.engine.addTag(p.slug as string, p.tag as string);
|
||||
// v0.31.8 (D7): thread ctx.sourceId.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
await ctx.engine.addTag(p.slug as string, p.tag as string, sourceOpts);
|
||||
return { status: 'ok' };
|
||||
},
|
||||
cliHints: { name: 'tag', positional: ['slug', 'tag'] },
|
||||
@@ -1261,7 +1301,8 @@ const remove_tag: Operation = {
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'remove_tag', slug: p.slug, tag: p.tag };
|
||||
await ctx.engine.removeTag(p.slug as string, p.tag as string);
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
await ctx.engine.removeTag(p.slug as string, p.tag as string, sourceOpts);
|
||||
return { status: 'ok' };
|
||||
},
|
||||
cliHints: { name: 'untag', positional: ['slug', 'tag'] },
|
||||
@@ -1274,7 +1315,9 @@ const get_tags: Operation = {
|
||||
slug: { type: 'string', required: true },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
return ctx.engine.getTags(p.slug as string);
|
||||
// v0.31.8 (D20): thread ctx.sourceId for read-side ops on multi-source brains.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
return ctx.engine.getTags(p.slug as string, sourceOpts);
|
||||
},
|
||||
scope: 'read',
|
||||
cliHints: { name: 'tags', positional: ['slug'] },
|
||||
@@ -1295,9 +1338,17 @@ const add_link: Operation = {
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'add_link', from: p.from, to: p.to };
|
||||
// v0.31.8 (D7): single ctx.sourceId scopes both endpoints + origin. Cross-
|
||||
// source link creation is out of scope for this wave; use the engine API
|
||||
// directly for that edge case.
|
||||
const linkOpts = ctx.sourceId
|
||||
? { fromSourceId: ctx.sourceId, toSourceId: ctx.sourceId, originSourceId: ctx.sourceId }
|
||||
: undefined;
|
||||
await ctx.engine.addLink(
|
||||
p.from as string, p.to as string,
|
||||
(p.context as string) || '', (p.link_type as string) || '',
|
||||
undefined, undefined, undefined,
|
||||
linkOpts,
|
||||
);
|
||||
return { status: 'ok' };
|
||||
},
|
||||
@@ -1315,7 +1366,10 @@ const remove_link: Operation = {
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'remove_link', from: p.from, to: p.to };
|
||||
await ctx.engine.removeLink(p.from as string, p.to as string);
|
||||
const linkOpts = ctx.sourceId
|
||||
? { fromSourceId: ctx.sourceId, toSourceId: ctx.sourceId }
|
||||
: undefined;
|
||||
await ctx.engine.removeLink(p.from as string, p.to as string, undefined, undefined, linkOpts);
|
||||
return { status: 'ok' };
|
||||
},
|
||||
cliHints: { name: 'unlink', positional: ['from', 'to'] },
|
||||
@@ -1328,7 +1382,10 @@ const get_links: Operation = {
|
||||
slug: { type: 'string', required: true },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
return ctx.engine.getLinks(p.slug as string);
|
||||
// v0.31.8 (D16): thread ctx.sourceId. When unset, engine falls through
|
||||
// to cross-source view (back-compat).
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
return ctx.engine.getLinks(p.slug as string, sourceOpts);
|
||||
},
|
||||
scope: 'read',
|
||||
};
|
||||
@@ -1340,7 +1397,8 @@ const get_backlinks: Operation = {
|
||||
slug: { type: 'string', required: true },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
return ctx.engine.getBacklinks(p.slug as string);
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
return ctx.engine.getBacklinks(p.slug as string, sourceOpts);
|
||||
},
|
||||
scope: 'read',
|
||||
cliHints: { name: 'backlinks', positional: ['slug'] },
|
||||
@@ -1417,12 +1475,14 @@ const add_timeline_entry: Operation = {
|
||||
if (Number.isNaN(parsed.getTime()) || parsed.toISOString().slice(0, 10) !== date) {
|
||||
throw new Error(`Invalid calendar date "${date}"`);
|
||||
}
|
||||
// v0.31.8 (D7): thread ctx.sourceId.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
await ctx.engine.addTimelineEntry(p.slug as string, {
|
||||
date,
|
||||
source: (p.source as string) || '',
|
||||
summary: p.summary as string,
|
||||
detail: (p.detail as string) || '',
|
||||
});
|
||||
}, sourceOpts);
|
||||
return { status: 'ok' };
|
||||
},
|
||||
cliHints: { name: 'timeline-add', positional: ['slug', 'date', 'summary'] },
|
||||
@@ -1435,7 +1495,9 @@ const get_timeline: Operation = {
|
||||
slug: { type: 'string', required: true },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
return ctx.engine.getTimeline(p.slug as string);
|
||||
// v0.31.8 (D20): thread ctx.sourceId.
|
||||
const sourceId = ctx.sourceId;
|
||||
return ctx.engine.getTimeline(p.slug as string, sourceId ? { sourceId } : undefined);
|
||||
},
|
||||
scope: 'read',
|
||||
cliHints: { name: 'timeline', positional: ['slug'] },
|
||||
@@ -1532,7 +1594,9 @@ const get_versions: Operation = {
|
||||
slug: { type: 'string', required: true },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
const versions = await ctx.engine.getVersions(p.slug as string);
|
||||
// v0.31.8 (D20): thread ctx.sourceId.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
const versions = await ctx.engine.getVersions(p.slug as string, sourceOpts);
|
||||
// Same takes-allow-list privacy boundary as get_page. Snapshots persist
|
||||
// historical compiled_truth verbatim, including the takes fence, so
|
||||
// a remote token bypassing get_page via /history would re-introduce
|
||||
@@ -1555,8 +1619,12 @@ const revert_version: Operation = {
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'revert_version', slug: p.slug, version_id: p.version_id };
|
||||
await ctx.engine.createVersion(p.slug as string);
|
||||
await ctx.engine.revertToVersion(p.slug as string, p.version_id as number);
|
||||
// v0.31.8 (D7): thread ctx.sourceId so multi-source brains revert the
|
||||
// intended page row instead of whichever same-slug row Postgres returns
|
||||
// first.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
await ctx.engine.createVersion(p.slug as string, sourceOpts);
|
||||
await ctx.engine.revertToVersion(p.slug as string, p.version_id as number, sourceOpts);
|
||||
return { status: 'reverted' };
|
||||
},
|
||||
cliHints: { name: 'revert', positional: ['slug', 'version_id'] },
|
||||
@@ -1604,7 +1672,9 @@ const put_raw_data: Operation = {
|
||||
scope: 'write',
|
||||
handler: async (ctx, p) => {
|
||||
if (ctx.dryRun) return { dry_run: true, action: 'put_raw_data', slug: p.slug, source: p.source };
|
||||
await ctx.engine.putRawData(p.slug as string, p.source as string, p.data as object);
|
||||
// v0.31.8 (D7 + D21): thread ctx.sourceId.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
await ctx.engine.putRawData(p.slug as string, p.source as string, p.data as object, sourceOpts);
|
||||
return { status: 'ok' };
|
||||
},
|
||||
};
|
||||
@@ -1617,7 +1687,9 @@ const get_raw_data: Operation = {
|
||||
source: { type: 'string', description: 'Filter by source' },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
return ctx.engine.getRawData(p.slug as string, p.source as string | undefined);
|
||||
// v0.31.8 (D20 + D21): thread ctx.sourceId.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
return ctx.engine.getRawData(p.slug as string, p.source as string | undefined, sourceOpts);
|
||||
},
|
||||
scope: 'read',
|
||||
};
|
||||
@@ -1643,7 +1715,9 @@ const get_chunks: Operation = {
|
||||
slug: { type: 'string', required: true },
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
return ctx.engine.getChunks(p.slug as string);
|
||||
// v0.31.8 (D20): thread ctx.sourceId.
|
||||
const sourceOpts = ctx.sourceId ? { sourceId: ctx.sourceId } : {};
|
||||
return ctx.engine.getChunks(p.slug as string, sourceOpts);
|
||||
},
|
||||
scope: 'read',
|
||||
};
|
||||
|
||||
+151
-48
@@ -656,7 +656,19 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return (rows as Record<string, unknown>[]).map(rowToPage);
|
||||
}
|
||||
|
||||
async getAllSlugs(): Promise<Set<string>> {
|
||||
async getAllSlugs(opts?: { sourceId?: string }): Promise<Set<string>> {
|
||||
// v0.31.8 (D12): when opts.sourceId is set, return only that source's
|
||||
// slugs (used by reconcileLinks so wikilink resolution doesn't span
|
||||
// unrelated sources). Without opts, returns the union across sources
|
||||
// (pre-v0.31.8 behavior — preserved for callers that still expect the
|
||||
// brain-wide slug index, e.g. extract.ts's link resolver).
|
||||
if (opts?.sourceId) {
|
||||
const { rows } = await this.db.query(
|
||||
'SELECT slug FROM pages WHERE source_id = $1',
|
||||
[opts.sourceId]
|
||||
);
|
||||
return new Set((rows as { slug: string }[]).map(r => r.slug));
|
||||
}
|
||||
const { rows } = await this.db.query('SELECT slug FROM pages');
|
||||
return new Set((rows as { slug: string }[]).map(r => r.slug));
|
||||
}
|
||||
@@ -1233,7 +1245,26 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
}
|
||||
|
||||
async getLinks(slug: string): Promise<Link[]> {
|
||||
async getLinks(slug: string, opts?: { sourceId?: string }): Promise<Link[]> {
|
||||
// v0.31.8 (D16): two-branch query. Without opts.sourceId, no source filter
|
||||
// (preserves pre-v0.31.8 cross-source semantics for back-link validators
|
||||
// and read-side op handlers that haven't threaded sourceId yet). With
|
||||
// opts.sourceId, scope to that source — used by reconcileLinks and any
|
||||
// ctx.sourceId-aware read op (D20).
|
||||
if (opts?.sourceId) {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE f.slug = $1 AND f.source_id = $2`,
|
||||
[slug, opts.sourceId]
|
||||
);
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
@@ -1248,7 +1279,22 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
|
||||
async getBacklinks(slug: string): Promise<Link[]> {
|
||||
async getBacklinks(slug: string, opts?: { sourceId?: string }): Promise<Link[]> {
|
||||
// v0.31.8 (D16): two-branch query. See getLinks() comment.
|
||||
if (opts?.sourceId) {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE t.slug = $1 AND t.source_id = $2`,
|
||||
[slug, opts.sourceId]
|
||||
);
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
@@ -1616,40 +1662,59 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
|
||||
// v0.31.8 (D16): build WHERE clause dynamically so opts.sourceId composes
|
||||
// cleanly with the existing after/before filters. Without sourceId, no
|
||||
// source filter applies (preserves pre-v0.31.8 cross-source semantics).
|
||||
const limit = opts?.limit || 100;
|
||||
|
||||
let result;
|
||||
if (opts?.after && opts?.before) {
|
||||
result = await this.db.query(
|
||||
`SELECT te.* FROM timeline_entries te
|
||||
JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = $1 AND te.date >= $2::date AND te.date <= $3::date
|
||||
ORDER BY te.date DESC LIMIT $4`,
|
||||
[slug, opts.after, opts.before, limit]
|
||||
);
|
||||
} else if (opts?.after) {
|
||||
result = await this.db.query(
|
||||
`SELECT te.* FROM timeline_entries te
|
||||
JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = $1 AND te.date >= $2::date
|
||||
ORDER BY te.date DESC LIMIT $3`,
|
||||
[slug, opts.after, limit]
|
||||
);
|
||||
} else {
|
||||
result = await this.db.query(
|
||||
`SELECT te.* FROM timeline_entries te
|
||||
JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = $1
|
||||
ORDER BY te.date DESC LIMIT $2`,
|
||||
[slug, limit]
|
||||
);
|
||||
const where: string[] = ['p.slug = $1'];
|
||||
const params: unknown[] = [slug];
|
||||
if (opts?.after) {
|
||||
params.push(opts.after);
|
||||
where.push(`te.date >= $${params.length}::date`);
|
||||
}
|
||||
|
||||
if (opts?.before) {
|
||||
params.push(opts.before);
|
||||
where.push(`te.date <= $${params.length}::date`);
|
||||
}
|
||||
if (opts?.sourceId) {
|
||||
params.push(opts.sourceId);
|
||||
where.push(`p.source_id = $${params.length}`);
|
||||
}
|
||||
params.push(limit);
|
||||
const result = await this.db.query(
|
||||
`SELECT te.* FROM timeline_entries te
|
||||
JOIN pages p ON p.id = te.page_id
|
||||
WHERE ${where.join(' AND ')}
|
||||
ORDER BY te.date DESC LIMIT $${params.length}`,
|
||||
params
|
||||
);
|
||||
return result.rows as unknown as TimelineEntry[];
|
||||
}
|
||||
|
||||
// Raw data
|
||||
async putRawData(slug: string, source: string, data: object): Promise<void> {
|
||||
async putRawData(
|
||||
slug: string,
|
||||
source: string,
|
||||
data: object,
|
||||
opts?: { sourceId?: string },
|
||||
): Promise<void> {
|
||||
// v0.31.8 (D21): two-branch INSERT-SELECT. Without opts.sourceId, the
|
||||
// page-id lookup matches every same-slug page (pre-v0.31.8 behavior; can
|
||||
// still trip Postgres 21000 on multi-source brains — caller's choice).
|
||||
// With opts.sourceId, the lookup is source-scoped so the right row
|
||||
// gets the raw_data attached.
|
||||
if (opts?.sourceId) {
|
||||
await this.db.query(
|
||||
`INSERT INTO raw_data (page_id, source, data)
|
||||
SELECT id, $2, $3::jsonb
|
||||
FROM pages WHERE slug = $1 AND source_id = $4
|
||||
ON CONFLICT (page_id, source) DO UPDATE SET
|
||||
data = EXCLUDED.data,
|
||||
fetched_at = now()`,
|
||||
[slug, source, JSON.stringify(data), opts.sourceId]
|
||||
);
|
||||
return;
|
||||
}
|
||||
await this.db.query(
|
||||
`INSERT INTO raw_data (page_id, source, data)
|
||||
SELECT id, $2, $3::jsonb
|
||||
@@ -1661,23 +1726,29 @@ export class PGLiteEngine implements BrainEngine {
|
||||
);
|
||||
}
|
||||
|
||||
async getRawData(slug: string, source?: string): Promise<RawData[]> {
|
||||
let result;
|
||||
async getRawData(
|
||||
slug: string,
|
||||
source?: string,
|
||||
opts?: { sourceId?: string },
|
||||
): Promise<RawData[]> {
|
||||
// v0.31.8 (D21): build WHERE clause dynamically. Without opts.sourceId,
|
||||
// no source filter (preserves pre-v0.31.8 cross-source read).
|
||||
const where: string[] = ['p.slug = $1'];
|
||||
const params: unknown[] = [slug];
|
||||
if (source) {
|
||||
result = await this.db.query(
|
||||
`SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
||||
JOIN pages p ON p.id = rd.page_id
|
||||
WHERE p.slug = $1 AND rd.source = $2`,
|
||||
[slug, source]
|
||||
);
|
||||
} else {
|
||||
result = await this.db.query(
|
||||
`SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
||||
JOIN pages p ON p.id = rd.page_id
|
||||
WHERE p.slug = $1`,
|
||||
[slug]
|
||||
);
|
||||
params.push(source);
|
||||
where.push(`rd.source = $${params.length}`);
|
||||
}
|
||||
if (opts?.sourceId) {
|
||||
params.push(opts.sourceId);
|
||||
where.push(`p.source_id = $${params.length}`);
|
||||
}
|
||||
const result = await this.db.query(
|
||||
`SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
||||
JOIN pages p ON p.id = rd.page_id
|
||||
WHERE ${where.join(' AND ')}`,
|
||||
params
|
||||
);
|
||||
return result.rows as unknown as RawData[];
|
||||
}
|
||||
|
||||
@@ -2423,7 +2494,19 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return rows[0] as unknown as PageVersion;
|
||||
}
|
||||
|
||||
async getVersions(slug: string): Promise<PageVersion[]> {
|
||||
async getVersions(slug: string, opts?: { sourceId?: string }): Promise<PageVersion[]> {
|
||||
// v0.31.8 (D16): two-branch. Without opts.sourceId, joins return versions
|
||||
// for every same-slug page (preserves pre-v0.31.8 cross-source view).
|
||||
if (opts?.sourceId) {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT pv.* FROM page_versions pv
|
||||
JOIN pages p ON p.id = pv.page_id
|
||||
WHERE p.slug = $1 AND p.source_id = $2
|
||||
ORDER BY pv.snapshot_at DESC`,
|
||||
[slug, opts.sourceId]
|
||||
);
|
||||
return rows as unknown as PageVersion[];
|
||||
}
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT pv.* FROM page_versions pv
|
||||
JOIN pages p ON p.id = pv.page_id
|
||||
@@ -2434,7 +2517,27 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return rows as unknown as PageVersion[];
|
||||
}
|
||||
|
||||
async revertToVersion(slug: string, versionId: number): Promise<void> {
|
||||
async revertToVersion(
|
||||
slug: string,
|
||||
versionId: number,
|
||||
opts?: { sourceId?: string },
|
||||
): Promise<void> {
|
||||
// v0.31.8 (D12): when opts.sourceId is set, scope BOTH the page lookup
|
||||
// and the version row reference. Without it, multi-source brains can
|
||||
// revert the wrong same-slug page (the one Postgres returns first).
|
||||
if (opts?.sourceId) {
|
||||
await this.db.query(
|
||||
`UPDATE pages SET
|
||||
compiled_truth = pv.compiled_truth,
|
||||
frontmatter = pv.frontmatter,
|
||||
updated_at = now()
|
||||
FROM page_versions pv
|
||||
WHERE pages.slug = $1 AND pages.source_id = $3
|
||||
AND pv.id = $2 AND pv.page_id = pages.id`,
|
||||
[slug, versionId, opts.sourceId]
|
||||
);
|
||||
return;
|
||||
}
|
||||
await this.db.query(
|
||||
`UPDATE pages SET
|
||||
compiled_truth = pv.compiled_truth,
|
||||
|
||||
+153
-34
@@ -700,8 +700,13 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows.map(rowToPage);
|
||||
}
|
||||
|
||||
async getAllSlugs(): Promise<Set<string>> {
|
||||
async getAllSlugs(opts?: { sourceId?: string }): Promise<Set<string>> {
|
||||
const sql = this.sql;
|
||||
// v0.31.8 (D12): two-branch. See pglite-engine.ts:getAllSlugs for context.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await sql`SELECT slug FROM pages WHERE source_id = ${opts.sourceId}`;
|
||||
return new Set(rows.map((r) => r.slug as string));
|
||||
}
|
||||
const rows = await sql`SELECT slug FROM pages`;
|
||||
return new Set(rows.map((r) => r.slug as string));
|
||||
}
|
||||
@@ -1374,8 +1379,24 @@ export class PostgresEngine implements BrainEngine {
|
||||
}
|
||||
}
|
||||
|
||||
async getLinks(slug: string): Promise<Link[]> {
|
||||
async getLinks(slug: string, opts?: { sourceId?: string }): Promise<Link[]> {
|
||||
const sql = this.sql;
|
||||
// v0.31.8 (D16): two-branch query. Without opts.sourceId, no source filter
|
||||
// (preserves pre-v0.31.8 cross-source semantics). With opts.sourceId,
|
||||
// scope the from-page lookup. See pglite-engine.ts:getLinks for context.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE f.slug = ${slug} AND f.source_id = ${opts.sourceId}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
@@ -1389,8 +1410,22 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
|
||||
async getBacklinks(slug: string): Promise<Link[]> {
|
||||
async getBacklinks(slug: string, opts?: { sourceId?: string }): Promise<Link[]> {
|
||||
const sql = this.sql;
|
||||
// v0.31.8 (D16): two-branch query, mirrors getLinks above.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
o.slug as origin_slug, l.origin_field
|
||||
FROM links l
|
||||
JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
LEFT JOIN pages o ON o.id = l.origin_page_id
|
||||
WHERE t.slug = ${slug} AND t.source_id = ${opts.sourceId}
|
||||
`;
|
||||
return rows as unknown as Link[];
|
||||
}
|
||||
const rows = await sql`
|
||||
SELECT f.slug as from_slug, t.slug as to_slug,
|
||||
l.link_type, l.context, l.link_source,
|
||||
@@ -1764,37 +1799,80 @@ export class PostgresEngine implements BrainEngine {
|
||||
async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
|
||||
const sql = this.sql;
|
||||
const limit = opts?.limit || 100;
|
||||
|
||||
// v0.31.8 (D16): branch on every combination of (after, before, sourceId).
|
||||
// 8 cases is too many — use an explicit branch on sourceId, then nested
|
||||
// branches on after/before. Mirrors pglite-engine but stays in postgres.js
|
||||
// template-literal idiom (which doesn't compose fragment WHERE chains
|
||||
// cleanly).
|
||||
const sourceId = opts?.sourceId;
|
||||
let rows;
|
||||
if (opts?.after && opts?.before) {
|
||||
rows = await sql`
|
||||
SELECT te.* FROM timeline_entries te
|
||||
JOIN pages p ON p.id = te.page_id
|
||||
if (sourceId) {
|
||||
if (opts?.after && opts?.before) {
|
||||
rows = await sql`SELECT te.* FROM timeline_entries te JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = ${slug} AND p.source_id = ${sourceId}
|
||||
AND te.date >= ${opts.after}::date AND te.date <= ${opts.before}::date
|
||||
ORDER BY te.date DESC LIMIT ${limit}`;
|
||||
} else if (opts?.after) {
|
||||
rows = await sql`SELECT te.* FROM timeline_entries te JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = ${slug} AND p.source_id = ${sourceId}
|
||||
AND te.date >= ${opts.after}::date
|
||||
ORDER BY te.date DESC LIMIT ${limit}`;
|
||||
} else if (opts?.before) {
|
||||
rows = await sql`SELECT te.* FROM timeline_entries te JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = ${slug} AND p.source_id = ${sourceId}
|
||||
AND te.date <= ${opts.before}::date
|
||||
ORDER BY te.date DESC LIMIT ${limit}`;
|
||||
} else {
|
||||
rows = await sql`SELECT te.* FROM timeline_entries te JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = ${slug} AND p.source_id = ${sourceId}
|
||||
ORDER BY te.date DESC LIMIT ${limit}`;
|
||||
}
|
||||
} else if (opts?.after && opts?.before) {
|
||||
rows = await sql`SELECT te.* FROM timeline_entries te JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = ${slug} AND te.date >= ${opts.after}::date AND te.date <= ${opts.before}::date
|
||||
ORDER BY te.date DESC LIMIT ${limit}
|
||||
`;
|
||||
ORDER BY te.date DESC LIMIT ${limit}`;
|
||||
} else if (opts?.after) {
|
||||
rows = await sql`
|
||||
SELECT te.* FROM timeline_entries te
|
||||
JOIN pages p ON p.id = te.page_id
|
||||
rows = await sql`SELECT te.* FROM timeline_entries te JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = ${slug} AND te.date >= ${opts.after}::date
|
||||
ORDER BY te.date DESC LIMIT ${limit}
|
||||
`;
|
||||
ORDER BY te.date DESC LIMIT ${limit}`;
|
||||
} else if (opts?.before) {
|
||||
rows = await sql`SELECT te.* FROM timeline_entries te JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = ${slug} AND te.date <= ${opts.before}::date
|
||||
ORDER BY te.date DESC LIMIT ${limit}`;
|
||||
} else {
|
||||
rows = await sql`
|
||||
SELECT te.* FROM timeline_entries te
|
||||
JOIN pages p ON p.id = te.page_id
|
||||
rows = await sql`SELECT te.* FROM timeline_entries te JOIN pages p ON p.id = te.page_id
|
||||
WHERE p.slug = ${slug}
|
||||
ORDER BY te.date DESC LIMIT ${limit}
|
||||
`;
|
||||
ORDER BY te.date DESC LIMIT ${limit}`;
|
||||
}
|
||||
|
||||
return rows as unknown as TimelineEntry[];
|
||||
}
|
||||
|
||||
// Raw data
|
||||
async putRawData(slug: string, source: string, data: object): Promise<void> {
|
||||
async putRawData(
|
||||
slug: string,
|
||||
source: string,
|
||||
data: object,
|
||||
opts?: { sourceId?: string },
|
||||
): Promise<void> {
|
||||
const sql = this.sql;
|
||||
// v0.31.8 (D21): two-branch INSERT-SELECT. Without opts.sourceId, the
|
||||
// page-id lookup matches every same-slug page (pre-v0.31.8 behavior).
|
||||
// With opts.sourceId, the lookup is source-scoped.
|
||||
if (opts?.sourceId) {
|
||||
const result = await sql`
|
||||
INSERT INTO raw_data (page_id, source, data)
|
||||
SELECT id, ${source}, ${sql.json(data as Parameters<typeof sql.json>[0])}
|
||||
FROM pages WHERE slug = ${slug} AND source_id = ${opts.sourceId}
|
||||
ON CONFLICT (page_id, source) DO UPDATE SET
|
||||
data = EXCLUDED.data,
|
||||
fetched_at = now()
|
||||
RETURNING id
|
||||
`;
|
||||
if (result.length === 0) {
|
||||
throw new Error(`putRawData failed: page "${slug}" (source=${opts.sourceId}) not found`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
const result = await sql`
|
||||
INSERT INTO raw_data (page_id, source, data)
|
||||
SELECT id, ${source}, ${sql.json(data as Parameters<typeof sql.json>[0])}
|
||||
@@ -1807,21 +1885,33 @@ export class PostgresEngine implements BrainEngine {
|
||||
if (result.length === 0) throw new Error(`putRawData failed: page "${slug}" not found`);
|
||||
}
|
||||
|
||||
async getRawData(slug: string, source?: string): Promise<RawData[]> {
|
||||
async getRawData(
|
||||
slug: string,
|
||||
source?: string,
|
||||
opts?: { sourceId?: string },
|
||||
): Promise<RawData[]> {
|
||||
const sql = this.sql;
|
||||
// v0.31.8 (D21): four-branch shape on (source provided, sourceId provided).
|
||||
// Postgres.js template-literal style doesn't compose fragments cleanly so
|
||||
// we enumerate.
|
||||
const sourceId = opts?.sourceId;
|
||||
let rows;
|
||||
if (source) {
|
||||
rows = await sql`
|
||||
SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
||||
if (source && sourceId) {
|
||||
rows = await sql`SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
||||
JOIN pages p ON p.id = rd.page_id
|
||||
WHERE p.slug = ${slug} AND rd.source = ${source}
|
||||
`;
|
||||
WHERE p.slug = ${slug} AND rd.source = ${source} AND p.source_id = ${sourceId}`;
|
||||
} else if (source) {
|
||||
rows = await sql`SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
||||
JOIN pages p ON p.id = rd.page_id
|
||||
WHERE p.slug = ${slug} AND rd.source = ${source}`;
|
||||
} else if (sourceId) {
|
||||
rows = await sql`SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
||||
JOIN pages p ON p.id = rd.page_id
|
||||
WHERE p.slug = ${slug} AND p.source_id = ${sourceId}`;
|
||||
} else {
|
||||
rows = await sql`
|
||||
SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
||||
rows = await sql`SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
||||
JOIN pages p ON p.id = rd.page_id
|
||||
WHERE p.slug = ${slug}
|
||||
`;
|
||||
WHERE p.slug = ${slug}`;
|
||||
}
|
||||
return rows as unknown as RawData[];
|
||||
}
|
||||
@@ -2519,8 +2609,18 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows[0] as unknown as PageVersion;
|
||||
}
|
||||
|
||||
async getVersions(slug: string): Promise<PageVersion[]> {
|
||||
async getVersions(slug: string, opts?: { sourceId?: string }): Promise<PageVersion[]> {
|
||||
const sql = this.sql;
|
||||
// v0.31.8 (D16): two-branch.
|
||||
if (opts?.sourceId) {
|
||||
const rows = await sql`
|
||||
SELECT pv.* FROM page_versions pv
|
||||
JOIN pages p ON p.id = pv.page_id
|
||||
WHERE p.slug = ${slug} AND p.source_id = ${opts.sourceId}
|
||||
ORDER BY pv.snapshot_at DESC
|
||||
`;
|
||||
return rows as unknown as PageVersion[];
|
||||
}
|
||||
const rows = await sql`
|
||||
SELECT pv.* FROM page_versions pv
|
||||
JOIN pages p ON p.id = pv.page_id
|
||||
@@ -2530,8 +2630,27 @@ export class PostgresEngine implements BrainEngine {
|
||||
return rows as unknown as PageVersion[];
|
||||
}
|
||||
|
||||
async revertToVersion(slug: string, versionId: number): Promise<void> {
|
||||
async revertToVersion(
|
||||
slug: string,
|
||||
versionId: number,
|
||||
opts?: { sourceId?: string },
|
||||
): Promise<void> {
|
||||
const sql = this.sql;
|
||||
// v0.31.8 (D12): two-branch. With opts.sourceId, scope BOTH the page lookup
|
||||
// AND the version reference. Without it, multi-source brains can revert
|
||||
// the wrong same-slug page.
|
||||
if (opts?.sourceId) {
|
||||
await sql`
|
||||
UPDATE pages SET
|
||||
compiled_truth = pv.compiled_truth,
|
||||
frontmatter = pv.frontmatter,
|
||||
updated_at = now()
|
||||
FROM page_versions pv
|
||||
WHERE pages.slug = ${slug} AND pages.source_id = ${opts.sourceId}
|
||||
AND pv.id = ${versionId} AND pv.page_id = pages.id
|
||||
`;
|
||||
return;
|
||||
}
|
||||
await sql`
|
||||
UPDATE pages SET
|
||||
compiled_truth = pv.compiled_truth,
|
||||
|
||||
@@ -567,6 +567,12 @@ export interface TimelineOpts {
|
||||
limit?: number;
|
||||
after?: string;
|
||||
before?: string;
|
||||
/**
|
||||
* v0.31.8: when set, scope the page-id lookup to this source. When omitted,
|
||||
* the read returns timeline entries for every same-slug page across sources
|
||||
* (pre-v0.31.8 behavior; preserved by the two-branch query in both engines).
|
||||
*/
|
||||
sourceId?: string;
|
||||
}
|
||||
|
||||
// Raw data
|
||||
|
||||
@@ -72,10 +72,14 @@ export async function startMcpServer(engine: BrainEngine) {
|
||||
}
|
||||
|
||||
// Backward compat: used by `gbrain call` command (trusted local path).
|
||||
// v0.31.8 (D22): accept opts.sourceId so `gbrain call --source X <op> <json>`
|
||||
// can scope the op handler to that source. resolveSourceId() in call.ts is
|
||||
// the upstream resolver; this layer just passes the resolved id through.
|
||||
export async function handleToolCall(
|
||||
engine: BrainEngine,
|
||||
tool: string,
|
||||
params: Record<string, unknown>,
|
||||
opts?: { sourceId?: string },
|
||||
): Promise<unknown> {
|
||||
const op = operations.find(o => o.name === tool);
|
||||
if (!op) throw new Error(`Unknown tool: ${tool}`);
|
||||
@@ -86,6 +90,7 @@ export async function handleToolCall(
|
||||
const ctx = buildOperationContext(engine, params, {
|
||||
remote: false,
|
||||
logger: { info: console.log, warn: console.warn, error: console.error },
|
||||
...(opts?.sourceId ? { sourceId: opts.sourceId } : {}),
|
||||
});
|
||||
|
||||
return op.handler(ctx, params);
|
||||
|
||||
@@ -360,3 +360,68 @@ describe('doctor command', () => {
|
||||
expect(result.message).toContain('Could not check takes weight grid');
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// v0.31.8 D19 — wedge migration force-retry hint.
|
||||
//
|
||||
// The pre-v0.31.8 minions_migration check emitted a generic
|
||||
// `gbrain apply-migrations --yes` hint regardless of how partial the
|
||||
// migration was. Operators wedged on v0.29.1 (3 consecutive partials)
|
||||
// needed `--force-retry <v>` first because the apply-migrations runner's
|
||||
// 3-consecutive-partials guard rejected plain --yes. The v0.31.8 fix
|
||||
// extends the existing block in place: detect the wedge condition,
|
||||
// emit the force-retry hint when matched, fall back to the plain --yes
|
||||
// hint when the partial count is < 3.
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
describe('v0.31.8 — wedge migration force-retry hint (D19)', () => {
|
||||
test('local doctor source contains wedge detection alongside the existing stuck path', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
|
||||
// The existing forward-progress override stays intact. Both branches
|
||||
// must be present and live next to each other; replacing the override
|
||||
// with statusForVersion() would re-open stale wedge alerts (codex OV11).
|
||||
expect(source).toContain('Forward-progress override');
|
||||
expect(source).toContain('partialCount >= 3');
|
||||
// Both branches must coexist. Wedged path builds the command list with
|
||||
// --force-retry; partial path falls back to plain --yes. Order varies
|
||||
// between the local + remote doctor blocks, so just assert presence.
|
||||
expect(source).toContain('WEDGED MIGRATION(s)');
|
||||
expect(source).toContain('MINIONS HALF-INSTALLED');
|
||||
expect(source).toContain('--force-retry');
|
||||
expect(source).toMatch(/MINIONS HALF-INSTALLED[\s\S]{0,400}--yes/);
|
||||
});
|
||||
|
||||
test('wedge detection is local to doctor — no statusForVersion import (D19 anti-regression)', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
|
||||
// D19 explicitly chose to extend the existing block in place rather than
|
||||
// import statusForVersion, because statusForVersion is per-version only
|
||||
// and doesn't encode the cross-version forward-progress override. If a
|
||||
// future refactor re-introduces the import this regression guard
|
||||
// catches it.
|
||||
expect(source).not.toMatch(/import\s*\{\s*statusForVersion\s*\}/);
|
||||
expect(source).not.toMatch(/from\s*['"]\.\/apply-migrations\.ts['"]/);
|
||||
});
|
||||
|
||||
test('multiple wedged versions chain force-retry calls with &&', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
|
||||
// The local doctor block uses `.join(' && ')` so multiple wedged
|
||||
// versions render as a single copy-pasteable command line. Match BOTH
|
||||
// engine.ts blocks (local doctor + remote doctor) — the regex finds
|
||||
// either occurrence.
|
||||
expect(source).toMatch(/wedged\.map\(v\s*=>\s*`gbrain apply-migrations --force-retry [^`]+`\)\.join\(' && '\)/);
|
||||
});
|
||||
|
||||
test('remote doctor (doctorReportRemote) also emits the force-retry hint (D14)', async () => {
|
||||
const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text();
|
||||
// Check that the wedge detection is duplicated in the remote doctor
|
||||
// path so thin-client operators see it. Find the doctorReportRemote
|
||||
// function span and verify the wedge-hint code lives inside it.
|
||||
const remoteStart = source.indexOf('export async function doctorReportRemote(');
|
||||
expect(remoteStart).toBeGreaterThan(0);
|
||||
const remoteEnd = source.indexOf('\nexport async function runDoctor(', remoteStart);
|
||||
expect(remoteEnd).toBeGreaterThan(remoteStart);
|
||||
const remoteBlock = source.slice(remoteStart, remoteEnd);
|
||||
expect(remoteBlock).toContain('--force-retry');
|
||||
expect(remoteBlock).toContain('partialCount >= 3');
|
||||
expect(remoteBlock).toMatch(/WEDGED MIGRATION\(s\) on brain host/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
/**
|
||||
* v0.31.8 — multi_source_drift doctor check (D8 + D14 + D17 + OV12 + OV13).
|
||||
*
|
||||
* Heuristic: a non-default source X with local_path set, where the FS at
|
||||
* local_path contains a markdown file whose slug exists at (default, slug)
|
||||
* in DB but is missing from (X, slug). Surfaces evidence of pre-v0.30.3
|
||||
* putPage misroutes OR an incomplete initial sync.
|
||||
*
|
||||
* Test cases (5):
|
||||
* 1. Single-source brain → check skipped (no row in checks output).
|
||||
* 2. Multi-source brain, no misroutes → status `ok`.
|
||||
* 3. Multi-source brain, 2 misrouted slugs → status `warn` with sample.
|
||||
* 4. Multi-source brain, healthy same-slug-across-sources (file at X has
|
||||
* DB row at X AND default has its own legitimate slug) → ok (NOT a
|
||||
* false positive).
|
||||
* 5. FS walk hits limit → status `warn 'check skipped, walk too large'`.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { mkdirSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runSources } from '../src/commands/sources.ts';
|
||||
import { findMisroutedPages } from '../src/core/multi-source-drift.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
const TMP_ROOTS: string[] = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({ type: 'pglite' } as never);
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
for (const dir of TMP_ROOTS) {
|
||||
try { rmSync(dir, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||
}
|
||||
});
|
||||
|
||||
function makeTmpRoot(label: string): string {
|
||||
const dir = join(tmpdir(), `gbrain-drift-test-${label}-${Date.now()}-${Math.random().toString(36).slice(2)}`);
|
||||
mkdirSync(dir, { recursive: true });
|
||||
TMP_ROOTS.push(dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
function seedFile(root: string, relPath: string, content = 'placeholder\n'): void {
|
||||
const full = join(root, relPath);
|
||||
mkdirSync(join(full, '..'), { recursive: true });
|
||||
writeFileSync(full, content);
|
||||
}
|
||||
|
||||
describe('findMisroutedPages — heuristic correctness', () => {
|
||||
test('case 1: no non-default sources → returns empty result (caller skips check)', async () => {
|
||||
// Findfn is called by doctor only when at least one non-default source
|
||||
// with local_path exists; passing an empty array is the equivalent.
|
||||
const result = await findMisroutedPages(engine, []);
|
||||
expect(result.count).toBe(0);
|
||||
expect(result.sample).toEqual([]);
|
||||
expect(result.walk_truncated).toBe(false);
|
||||
});
|
||||
|
||||
test('case 2: multi-source brain, no misroutes → count=0', async () => {
|
||||
const root = makeTmpRoot('case2');
|
||||
seedFile(root, 'people/alice.md');
|
||||
seedFile(root, 'people/bob.md');
|
||||
|
||||
// Register the source via runSources, then update local_path directly.
|
||||
await runSources(engine, ['add', 'src-case2', '--no-federated']);
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET local_path = $1 WHERE id = $2`,
|
||||
[root, 'src-case2'],
|
||||
);
|
||||
// Both slugs land in (src-case2, *), NOT in (default, *). Healthy.
|
||||
await engine.putPage('people/alice', { type: 'person', title: 'Alice', compiled_truth: '.' }, { sourceId: 'src-case2' });
|
||||
await engine.putPage('people/bob', { type: 'person', title: 'Bob', compiled_truth: '.' }, { sourceId: 'src-case2' });
|
||||
|
||||
const result = await findMisroutedPages(engine, [{ id: 'src-case2', local_path: root }]);
|
||||
expect(result.count).toBe(0);
|
||||
expect(result.sample).toEqual([]);
|
||||
});
|
||||
|
||||
test('case 3: multi-source brain, 2 misrouted slugs → warn with sample', async () => {
|
||||
const root = makeTmpRoot('case3');
|
||||
seedFile(root, 'people/charlie.md');
|
||||
seedFile(root, 'people/dana.md');
|
||||
|
||||
await runSources(engine, ['add', 'src-case3', '--no-federated']);
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET local_path = $1 WHERE id = $2`,
|
||||
[root, 'src-case3'],
|
||||
);
|
||||
// Both slugs land in (default, *) — the misroute shape.
|
||||
await engine.putPage('people/charlie', { type: 'person', title: 'Charlie', compiled_truth: '.' });
|
||||
await engine.putPage('people/dana', { type: 'person', title: 'Dana', compiled_truth: '.' });
|
||||
// src-case3 has neither.
|
||||
|
||||
const result = await findMisroutedPages(engine, [{ id: 'src-case3', local_path: root }]);
|
||||
expect(result.count).toBe(2);
|
||||
expect(result.sample.length).toBe(2);
|
||||
const slugs = result.sample.map(s => s.slug).sort();
|
||||
expect(slugs).toEqual(['people/charlie', 'people/dana']);
|
||||
for (const s of result.sample) {
|
||||
expect(s.intended_source).toBe('src-case3');
|
||||
expect(s.local_path).toBe(root);
|
||||
}
|
||||
});
|
||||
|
||||
test('case 4: healthy same-slug-across-sources is NOT a false positive (OV4 redesign)', async () => {
|
||||
const root = makeTmpRoot('case4');
|
||||
seedFile(root, 'topics/widget.md');
|
||||
|
||||
await runSources(engine, ['add', 'src-case4', '--no-federated']);
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET local_path = $1 WHERE id = $2`,
|
||||
[root, 'src-case4'],
|
||||
);
|
||||
// Page exists at BOTH sources — the v0.18.0 supported state. The FS file
|
||||
// at src-case4 has a row at (src-case4, ...) AND default has its own.
|
||||
await engine.putPage('topics/widget', { type: 'concept', title: 'Default widget', compiled_truth: '.' });
|
||||
await engine.putPage('topics/widget', { type: 'concept', title: 'Src widget', compiled_truth: '.' }, { sourceId: 'src-case4' });
|
||||
|
||||
const result = await findMisroutedPages(engine, [{ id: 'src-case4', local_path: root }]);
|
||||
// Heuristic requires "(default, slug) AND NOT (X, slug)". Since both
|
||||
// exist, it's NOT misroute. Count must be 0 — this is the codex OV4 fix
|
||||
// case, the original "same-slug-across-sources = corruption" heuristic
|
||||
// would have false-positived here.
|
||||
expect(result.count).toBe(0);
|
||||
expect(result.sample).toEqual([]);
|
||||
});
|
||||
|
||||
test('case 5: FS walk hits limit → walk_truncated=true', async () => {
|
||||
const root = makeTmpRoot('case5');
|
||||
// Seed 12 files with a limit of 5 to force truncation.
|
||||
for (let i = 0; i < 12; i++) {
|
||||
seedFile(root, `topics/file-${i}.md`);
|
||||
}
|
||||
|
||||
const result = await findMisroutedPages(engine, [{ id: 'src-case5-fake', local_path: root }], {
|
||||
limit: 5,
|
||||
timeoutMs: 5000,
|
||||
});
|
||||
expect(result.walk_truncated).toBe(true);
|
||||
});
|
||||
|
||||
test('case 6 (OV13): unreadable local_path does NOT crash; returns empty', async () => {
|
||||
const result = await findMisroutedPages(engine, [
|
||||
{ id: 'src-fake', local_path: '/nonexistent/path/that/does/not/exist' },
|
||||
]);
|
||||
// Walk silently returns zero files; count=0, NOT throw.
|
||||
expect(result.count).toBe(0);
|
||||
expect(result.walk_truncated).toBe(false);
|
||||
});
|
||||
|
||||
test('case 7 (OV13): .mdx files are walked alongside .md', async () => {
|
||||
const root = makeTmpRoot('case7');
|
||||
seedFile(root, 'topics/mdx-page.mdx');
|
||||
|
||||
await runSources(engine, ['add', 'src-case7', '--no-federated']);
|
||||
await engine.executeRaw(
|
||||
`UPDATE sources SET local_path = $1 WHERE id = $2`,
|
||||
[root, 'src-case7'],
|
||||
);
|
||||
// Misroute the slug into default.
|
||||
await engine.putPage('topics/mdx-page', { type: 'concept', title: 'mdx', compiled_truth: '.' });
|
||||
|
||||
const result = await findMisroutedPages(engine, [{ id: 'src-case7', local_path: root }]);
|
||||
expect(result.count).toBe(1);
|
||||
expect(result.sample[0].slug).toBe('topics/mdx-page');
|
||||
});
|
||||
});
|
||||
@@ -464,3 +464,171 @@ describe('deletePage + updateSlug source-scoping (Data R2 CRITICAL + HIGH fix)',
|
||||
expect(rows[0].slug).toBe(REN_TO_2);
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// v0.31.8 — op-handler-layer threading (D7 + D11 + D16 + D20 + D21 + D22)
|
||||
//
|
||||
// The pre-v0.31.8 op handlers in src/core/operations.ts called engine methods
|
||||
// bare-slug, ignoring ctx.sourceId. Result: a remote MCP token whose
|
||||
// ctx.sourceId='X' calling put_page / add_tag / get_links / etc. silently
|
||||
// landed on source 'default'. This block drives the actual op handlers
|
||||
// (operations.ts) — not just the engine surface — through a mock
|
||||
// OperationContext carrying sourceId='X' and asserts only the X-source row
|
||||
// is mutated/read.
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
import { operations } from '../src/core/operations.ts';
|
||||
import type { OperationContext } from '../src/core/operations.ts';
|
||||
|
||||
function makeCtx(eng: PGLiteEngine, overrides: Partial<OperationContext> = {}): OperationContext {
|
||||
return {
|
||||
engine: eng as unknown as OperationContext['engine'],
|
||||
config: { engine: 'pglite' } as never,
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {} },
|
||||
dryRun: false,
|
||||
remote: false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function getOp(name: string) {
|
||||
const op = operations.find(o => o.name === name);
|
||||
if (!op) throw new Error(`op not registered: ${name}`);
|
||||
return op;
|
||||
}
|
||||
|
||||
describe('v0.31.8 op-handler ctx.sourceId threading', () => {
|
||||
// Two-source fixture seeded fresh for this block. Use unique slugs so the
|
||||
// earlier engine-layer suite's leftover state doesn't pollute assertions.
|
||||
const TAG_SLUG = 'topics/op-tag-target';
|
||||
|
||||
beforeAll(async () => {
|
||||
// Page exists at BOTH sources (the v0.18.0 supported state).
|
||||
await engine.putPage(TAG_SLUG, {
|
||||
type: 'concept', title: 'Default tag target', compiled_truth: '.',
|
||||
});
|
||||
await engine.putPage(TAG_SLUG, {
|
||||
type: 'concept', title: 'Testsrc tag target', compiled_truth: '.',
|
||||
}, { sourceId: 'testsrc' });
|
||||
});
|
||||
|
||||
test('add_tag handler with ctx.sourceId=testsrc tags only the testsrc row', async () => {
|
||||
const op = getOp('add_tag');
|
||||
await op.handler(makeCtx(engine, { sourceId: 'testsrc' }), { slug: TAG_SLUG, tag: 'op-handler-test-1' });
|
||||
|
||||
const rows = await engine.executeRaw<{ source_id: string }>(
|
||||
`SELECT p.source_id FROM tags t JOIN pages p ON p.id = t.page_id
|
||||
WHERE p.slug = $1 AND t.tag = $2`,
|
||||
[TAG_SLUG, 'op-handler-test-1'],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].source_id).toBe('testsrc');
|
||||
});
|
||||
|
||||
test('add_tag handler without ctx.sourceId tags the default row (back-compat)', async () => {
|
||||
const op = getOp('add_tag');
|
||||
await op.handler(makeCtx(engine), { slug: TAG_SLUG, tag: 'op-handler-test-2' });
|
||||
|
||||
const rows = await engine.executeRaw<{ source_id: string }>(
|
||||
`SELECT p.source_id FROM tags t JOIN pages p ON p.id = t.page_id
|
||||
WHERE p.slug = $1 AND t.tag = $2`,
|
||||
[TAG_SLUG, 'op-handler-test-2'],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].source_id).toBe('default');
|
||||
});
|
||||
|
||||
test('get_tags handler with ctx.sourceId=testsrc returns only testsrc tags', async () => {
|
||||
// Both rows should now have one tag each from the two tests above.
|
||||
const op = getOp('get_tags');
|
||||
const tags = await op.handler(makeCtx(engine, { sourceId: 'testsrc' }), { slug: TAG_SLUG }) as string[];
|
||||
expect(tags).toContain('op-handler-test-1');
|
||||
expect(tags).not.toContain('op-handler-test-2');
|
||||
});
|
||||
|
||||
test('get_tags handler without ctx.sourceId returns default-source tags (back-compat)', async () => {
|
||||
const op = getOp('get_tags');
|
||||
const tags = await op.handler(makeCtx(engine), { slug: TAG_SLUG }) as string[];
|
||||
// getTags is a v0.18.0-era source-aware method: it already defaults to
|
||||
// source='default' when opts is omitted (see engine.ts addTag/removeTag/
|
||||
// getTags comment). It does NOT use the D16 two-branch pattern — the
|
||||
// pre-v0.31.8 behavior for tags on multi-source brains was always
|
||||
// "scoped to default unless told otherwise." So with ctx.sourceId unset,
|
||||
// only the default-source tag surfaces. (D16 two-branch applies to
|
||||
// getLinks/getBacklinks/getTimeline/getRawData/getVersions/getAllSlugs/
|
||||
// revertToVersion — the methods that pre-D12 had no source filter at all.)
|
||||
expect(tags).toContain('op-handler-test-2');
|
||||
expect(tags).not.toContain('op-handler-test-1');
|
||||
});
|
||||
|
||||
test('add_link handler with ctx.sourceId scopes both endpoints', async () => {
|
||||
// Seed a target page at both sources so addLink's INTERSECT pre-check passes.
|
||||
const TARGET = 'topics/op-link-target';
|
||||
await engine.putPage(TARGET, { type: 'concept', title: 'Default target', compiled_truth: '.' });
|
||||
await engine.putPage(TARGET, { type: 'concept', title: 'Testsrc target', compiled_truth: '.' }, { sourceId: 'testsrc' });
|
||||
|
||||
const op = getOp('add_link');
|
||||
await op.handler(makeCtx(engine, { sourceId: 'testsrc' }), {
|
||||
from: TAG_SLUG, to: TARGET, link_type: 'mentions', context: 'op-test',
|
||||
});
|
||||
|
||||
const rows = await engine.executeRaw<{ from_source: string; to_source: string }>(
|
||||
`SELECT f.source_id AS from_source, t.source_id AS to_source
|
||||
FROM links l JOIN pages f ON f.id = l.from_page_id
|
||||
JOIN pages t ON t.id = l.to_page_id
|
||||
WHERE f.slug = $1 AND t.slug = $2 AND l.link_type = 'mentions'`,
|
||||
[TAG_SLUG, TARGET],
|
||||
);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].from_source).toBe('testsrc');
|
||||
expect(rows[0].to_source).toBe('testsrc');
|
||||
});
|
||||
|
||||
test('get_links handler scopes to ctx.sourceId; back-compat cross-source view preserved (D16)', async () => {
|
||||
const op = getOp('get_links');
|
||||
const scoped = await op.handler(makeCtx(engine, { sourceId: 'testsrc' }), { slug: TAG_SLUG }) as Array<{ to_slug: string }>;
|
||||
const cross = await op.handler(makeCtx(engine), { slug: TAG_SLUG }) as Array<{ to_slug: string }>;
|
||||
// testsrc has the link from add_link test above. default has none.
|
||||
expect(scoped.length).toBeGreaterThanOrEqual(1);
|
||||
// Cross-source view sees at least the same edges (and would see default's
|
||||
// if we'd seeded any). Under the two-branch back-compat path, this is the
|
||||
// pre-v0.31.8 semantic — no source filter on the engine join.
|
||||
expect(cross.length).toBeGreaterThanOrEqual(scoped.length);
|
||||
});
|
||||
|
||||
test('delete_page handler scopes to ctx.sourceId (soft-delete only the testsrc row)', async () => {
|
||||
// Use a fresh slug so we don't impact other tests in this describe block.
|
||||
const DEL_SLUG = 'topics/op-delete-target';
|
||||
await engine.putPage(DEL_SLUG, { type: 'concept', title: 'Default', compiled_truth: '.' });
|
||||
await engine.putPage(DEL_SLUG, { type: 'concept', title: 'Testsrc', compiled_truth: '.' }, { sourceId: 'testsrc' });
|
||||
|
||||
const op = getOp('delete_page');
|
||||
await op.handler(makeCtx(engine, { sourceId: 'testsrc' }), { slug: DEL_SLUG });
|
||||
|
||||
const rows = await engine.executeRaw<{ source_id: string; deleted_at: string | null }>(
|
||||
`SELECT source_id, deleted_at FROM pages WHERE slug = $1 ORDER BY source_id`,
|
||||
[DEL_SLUG],
|
||||
);
|
||||
expect(rows.length).toBe(2);
|
||||
const def = rows.find(r => r.source_id === 'default')!;
|
||||
const tst = rows.find(r => r.source_id === 'testsrc')!;
|
||||
expect(def.deleted_at).toBeNull(); // default row untouched
|
||||
expect(tst.deleted_at).not.toBeNull(); // testsrc row soft-deleted
|
||||
});
|
||||
|
||||
test('put_raw_data handler threads ctx.sourceId (D21)', async () => {
|
||||
const op = getOp('put_raw_data');
|
||||
await op.handler(makeCtx(engine, { sourceId: 'testsrc' }), {
|
||||
slug: TAG_SLUG, source: 'unit-test', data: { variant: 'testsrc' },
|
||||
});
|
||||
|
||||
// Read via the engine to assert which source row got the raw_data.
|
||||
const rd = await engine.getRawData(TAG_SLUG, 'unit-test', { sourceId: 'testsrc' });
|
||||
expect(rd.length).toBe(1);
|
||||
expect((rd[0].data as { variant: string }).variant).toBe('testsrc');
|
||||
|
||||
// Default-source raw_data should be untouched.
|
||||
const defRd = await engine.getRawData(TAG_SLUG, 'unit-test', { sourceId: 'default' });
|
||||
expect(defRd.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
/**
|
||||
* v0.31.8 — voyage adapter Content-Length pre-check + per-item cap (D2 + D10).
|
||||
*
|
||||
* Pre-fix: voyageCompatFetch in src/core/ai/gateway.ts called
|
||||
* `await resp.clone().json()` BEFORE iterating embeddings. A malicious or
|
||||
* compromised Voyage response of arbitrary size was fully parsed into the
|
||||
* JS heap before any cap could fire. The fix adds two layers:
|
||||
*
|
||||
* Layer 1 (PRIMARY): Content-Length header pre-check, fires BEFORE
|
||||
* `resp.clone().json()` so the JSON.parse OOM vector is gated.
|
||||
* Layer 2 (defense-in-depth): per-embedding base64 length check inside
|
||||
* the iteration; catches the rare case where Layer 1 was skipped
|
||||
* because the response uses chunked encoding (no Content-Length).
|
||||
*
|
||||
* The cap is 256 MB — sized as "unambiguously not legitimate" (the
|
||||
* realistic upper bound is voyage-3-large × 16K embeddings ≈ 200 MB raw).
|
||||
*
|
||||
* Tests are structural source-string assertions (matches the doctor.test.ts
|
||||
* style) because voyageCompatFetch is a closed-over helper inside the
|
||||
* recipe instantiation path and isn't exported. Behavioral coverage of the
|
||||
* voyage adapter lives in the existing E2E tests that hit a live Voyage
|
||||
* endpoint; these regression guards pin the SHAPE so a silent revert
|
||||
* (e.g. moving the cap below the resp.clone().json() call) fails loudly.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
|
||||
describe('v0.31.8 — voyage Content-Length pre-check + per-item cap', () => {
|
||||
test('MAX_VOYAGE_RESPONSE_BYTES constant is declared at 256 MB (D2 sizing)', async () => {
|
||||
const source = await Bun.file(new URL('../src/core/ai/gateway.ts', import.meta.url)).text();
|
||||
// 256 * 1024 * 1024 == 268435456. Either form is acceptable; assert
|
||||
// both so a silent re-tighten to 64 MB or 128 MB fails this test.
|
||||
expect(source).toContain('MAX_VOYAGE_RESPONSE_BYTES');
|
||||
expect(source).toMatch(/MAX_VOYAGE_RESPONSE_BYTES\s*=\s*256\s*\*\s*1024\s*\*\s*1024/);
|
||||
});
|
||||
|
||||
test('Layer 1: Content-Length pre-check fires BEFORE resp.clone().json() (D10 OOM defense)', async () => {
|
||||
const source = await Bun.file(new URL('../src/core/ai/gateway.ts', import.meta.url)).text();
|
||||
// Anchor relative to the post-fetch handler block. The function declaration
|
||||
// contains an OUTBOUND request body section earlier; we want to verify
|
||||
// the INBOUND ordering (everything after `await fetch(input, init)`).
|
||||
const fetchCallIdx = source.indexOf('const resp = await fetch(input, init);');
|
||||
expect(fetchCallIdx).toBeGreaterThan(0);
|
||||
const inboundBlock = source.slice(fetchCallIdx, fetchCallIdx + 6000);
|
||||
|
||||
// Pre-check string. Use the actual code shape from gateway.ts so the test
|
||||
// doesn't pin to comment text.
|
||||
const preCheckIdx = inboundBlock.indexOf("resp.headers.get('content-length')");
|
||||
// Use the full lvalue assignment so the match doesn't accidentally hit
|
||||
// comment text that mentions `await resp.clone().json()` for context.
|
||||
const jsonParseIdx = inboundBlock.indexOf('const json: any = await resp.clone().json()');
|
||||
expect(preCheckIdx).toBeGreaterThan(0);
|
||||
expect(jsonParseIdx).toBeGreaterThan(0);
|
||||
// The pre-check MUST appear before the JSON parse — otherwise the OOM
|
||||
// defense is theatrical (the original codex OV8 finding). This is the
|
||||
// critical structural invariant for D10.
|
||||
expect(preCheckIdx).toBeLessThan(jsonParseIdx);
|
||||
});
|
||||
|
||||
test('Layer 1 throws on Content-Length over the cap (not silent return)', async () => {
|
||||
const source = await Bun.file(new URL('../src/core/ai/gateway.ts', import.meta.url)).text();
|
||||
// The cap check must use `throw new Error(...)` so the embed flow's
|
||||
// retry/backoff sees a failure, NOT a silent pass-through.
|
||||
expect(source).toMatch(/exceeds[^`]*MAX_VOYAGE_RESPONSE_BYTES[^`]*bytes/);
|
||||
expect(source).toMatch(/throw new Error\([\s\S]{0,200}Voyage response Content-Length/);
|
||||
});
|
||||
|
||||
test('Layer 2: per-embedding base64 cap fires inside the json.data iteration', async () => {
|
||||
const source = await Bun.file(new URL('../src/core/ai/gateway.ts', import.meta.url)).text();
|
||||
// Defense-in-depth: even when Content-Length header is missing
|
||||
// (chunked encoding), each embedding string is bounded.
|
||||
expect(source).toMatch(/item\.embedding\.length\s*\*\s*0\.75/);
|
||||
expect(source).toMatch(/throw new Error\([\s\S]{0,200}Voyage embedding base64/);
|
||||
});
|
||||
|
||||
test('comment thread documents both layers + the cap-sizing decision', async () => {
|
||||
const source = await Bun.file(new URL('../src/core/ai/gateway.ts', import.meta.url)).text();
|
||||
// Anti-regression: the comment thread is part of the contract. If a
|
||||
// refactor strips them, future maintainers won't know why the order
|
||||
// matters or why the cap is 256 MB.
|
||||
expect(source).toContain('Layer 1');
|
||||
expect(source).toContain('Layer 2');
|
||||
expect(source).toMatch(/16K embeddings/);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user