diff --git a/CHANGELOG.md b/CHANGELOG.md index 50cd25ffc..b80d6ead7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,143 @@ All notable changes to GBrain will be documented in this file. + +## [0.41.25.0] - 2026-05-27 + +**Big-delete syncs no longer choke your brain — and your search results +were silently going stale in a different way that's now fixed.** + +If you've ever made a commit that deleted a lot of files (a folder +reorganization, an `atom` backfill rewrite, a big import cleanup), you +might have noticed that `gbrain sync` would just... sit there. For +hours. While it sat, the sync cron timed out every run, every other +source on your brain stopped syncing, and `gbrain doctor` health score +slid toward zero. That's fixed. A commit deleting 73,000 files used to +take about five hours to absorb; it now takes about two minutes. + +While digging into the delete path we found a second, unrelated bug. +Your query cache was silently serving stale `gbrain search` results +whenever you updated an *older* page (one that wasn't the newest in +your brain) and whenever you deleted *anything*. That's also fixed. +You'll see one tiny cache-miss spike on first `gbrain search` after +upgrade as the cache gets re-stamped under the new contract, then it's +faster than before because the bookmark is a one-row lookup instead of +a brain-wide MAX scan. + +**To take advantage of v0.41.22.0** + +`gbrain upgrade` should do this automatically. If it didn't, or if +`gbrain doctor` warns about a partial migration: + +1. Run the orchestrator manually: + ```bash + gbrain apply-migrations --yes + ``` +2. Verify the new clock table exists: + ```bash + gbrain doctor + # Look for any "page_generation_clock" warning in the output; + # there shouldn't be one. + ``` +3. Manual smoke test for the perf win (optional, only if you want to + see it): + ```bash + # In a scratch brain, stage a commit that deletes a lot of files, + # then run `gbrain sync --source ` and watch the wallclock. + # Before: minutes. After: seconds. + ``` +4. If any step fails or numbers look wrong, file an issue at + https://github.com/garrytan/gbrain/issues with `gbrain doctor` + output and the contents of `~/.gbrain/upgrade-errors.jsonl` if it + exists. + +### Itemized changes + +- **Sync delete loop batched.** `gbrain sync` (and the autopilot cycle's + sync phase) now batches deletes through a single SQL round-trip per + 500 files instead of one round-trip per file. The 73K-delete commit + drops from ~146,000 DB round-trips (~5 hours) to ~292 (~2 minutes). + Per-batch try-catch + per-slug decompose mirrors the existing + failedFiles pattern in the import loop, so a transient connection + blip on batch 73 doesn't lose 500 deletes — it self-heals to + one-at-a-time for that batch only. Engine-uniform: PGLite and real + Postgres both get the win. +- **Sync rename loop pre-resolved.** Same batched slug-resolve treatment + for renames; the per-file `importFile` (which dominates rename cost + anyway) stays per-file. +- **`engine.deletePages(slugs, {sourceId})`** — new required method on + `BrainEngine`. Returns the slugs actually deleted (so callers stop + wasting downstream work on phantom paths). `sourceId` is required at + the type level — closes the multi-source-bug-class structurally on + the new surface. +- **`engine.resolveSlugsByPaths(paths, {sourceId})`** — new required + method on `BrainEngine`. Batch path → slug lookup. The single-call + `resolveSlugByPathOrSourcePath` helper now delegates to this when + `sourceId` is set; one owner of the SQL and fallback semantics. +- **`src/core/engine-constants.ts`** — new module owning + `DELETE_BATCH_SIZE = 500`. Both engines import from it; no + engine-from-engine coupling. +- **Global page-generation clock.** New `page_generation_clock` table + with a statement-level `AFTER INSERT OR UPDATE OR DELETE` trigger. + `query-cache-gate.ts` Layer 1 now reads the clock directly. Pre-fix + the bookmark read `MAX(generation) FROM pages` and was structurally + broken: + - Updating a non-max page kept the old `OLD.generation + 1` trigger + body, which advanced the per-page counter but didn't move + `MAX(generation)`. Cache silently served stale. + - Hard-deleting any page didn't fire the trigger at all. Cache + silently served stale. + - Empty-result cache rows had `{}` per-page snapshots that were + "vacuously valid" via Layer 2 — they survived any subsequent + matching INSERT. Cache silently served stale. + + The new clock bumps exactly once per write statement regardless of + row cardinality (a 500-row batch DELETE is one counter bump, not + 500). Per-row `pages.generation` stays for Layer 2's per-page + snapshot semantics; only Layer 1's read source changed. Empty-result + cache rows now trust Layer 1 exclusively. +- **Migration v106** (`page_generation_clock_and_statement_trigger`) + creates the clock table, seeds it with `COALESCE(MAX(pages.generation), 0)` + so existing cache rows aren't all instantly invalidated, and wires + the new trigger. +- **CHANGELOG, version, and tests.** Tests in + `test/sync-delete-batch.test.ts`, `test/sync-delete-batch.slow.test.ts` + (PGLite 10K-page perf gate), `test/sync-rename-batch.test.ts`, + `test/page-generation-counter.test.ts`, plus an engine-parity + extension in `test/e2e/engine-parity.test.ts`. The existing + `test/query-cache-gate.test.ts` and `test/e2e/cache-gate-pglite.test.ts` + flip their pre-v0.41.22.0 "vacuously valid legacy row" assertions to + match the new contract (legacy rows now invalidate once on first + post-upgrade lookup, then the cache fills back correctly). +- **Credit:** the SQL batching idea (`DELETE FROM pages WHERE slug = + ANY($1::text[])` chunked at 500) and the headline 73K-delete number + come from PR #1538 by `@garrytan-agents`. That PR shipped Postgres- + only with no tests; this release supersedes it with PGLite parity, + proper engine-API factoring, test coverage, and the cache-bookmark + correctness fix that landed alongside. + +### For contributors + +- Engine implementers (gbrain has no external implementers today) must + add `deletePages` and `resolveSlugsByPaths` to any custom + `BrainEngine` — both are required methods on the interface. +- New CI surface: nothing additional; the existing `bun run verify` + gate covers the new test files automatically. +- v0.42+ TODO: tighten `deletePage` (single-row) to require `sourceId` + symmetric with the new `deletePages`. Out of scope for this PR + because it requires a full caller audit. +- v0.42+ TODO: clean up the existing `bump_page_generation_fn` + row-level trigger's UPDATE branch (`NEW.generation = OLD.generation + + 1` is structurally wrong for MAX-style consumers, harmless for + Layer 2 per-page snapshot but worth cleaning up). +- v0.42+ TODO: `op_checkpoints` integration on the sync-delete loop + for resume-after-kill semantics. Per-delete cost dropped from ~250ms + to ~0.4ms so a 73K-delete completes in seconds; resume matters much + less now. +- v0.42+ TODO: heavy regression test for two concurrent batch DELETEs + against disjoint sources (the statement-level clock-row UPDATE + contention shape). Filed under `tests/heavy/` per CLAUDE.md. + ## [0.41.24.0] - 2026-05-27 **Your reformatted meeting transcripts now actually parse.** diff --git a/CLAUDE.md b/CLAUDE.md index 3bfb40229..fd0f1fa52 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,8 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. +- `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. - `src/core/search/explain-formatter.ts` (v0.40.4.0) — renders `SearchResult[]` as a multi-line per-result breakdown for `gbrain search --explain`. Reads every boost-stamping field. Handles the "no boosts applied" empty path. 4-decimal precision with trailing-zero strip. Pinned by `test/search/explain-formatter.test.ts` (19 cases). @@ -51,8 +52,8 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -266,7 +267,7 @@ strict behavior when unset. - `src/core/embedding-dim-check.ts` extension (v0.41.17.0, T5+T6) — facts.embedding dim drift surface. New `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (codex #19 — migration v40 falls back to `vector` on pgvector < 0.7). Regex ordering halfvec-before-vector pinned by tests (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). New `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (codex #18 — NOT bare REINDEX which doesn't rewrite the index after a column-type change). New `assertFactsEmbeddingDimMatchesConfig(engine)` is the D15 preflight — throws `FactsEmbeddingDimMismatchError` (tagged class with `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool's MUST_ABORT semantics) when configured dim doesn't match the column width. Result cached per-engine via `WeakMap`. PGLite engines silently skip. New doctor check `facts_embedding_width_consistency` (registered in `runDoctor` after `embedding_width_consistency`) reuses the same helpers — surfaces drift with paste-ready ALTER recipe identical to the preflight error. Pinned by 18 cases in `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension (v0.41.17.0, T6, codex #20) — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. New `resolveFactsEmbeddingCast()` private method probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`. Both insert paths use the cached suffix so the cast matches the actual column type. Pre-fix all three insert sites hardcoded `::vector`; works on pgvector >= 0.7 via implicit auto-cast but fails on older pgvector. Test seam `__resetFactsEmbeddingCastCacheForTest()` clears per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions (v0.41.20.0 ops-fix-wave) — six daily-driver ops pains in one bisectable wave. (1) **Batch idempotency for `extract_atoms`:** new `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` replaces the per-hash loop that did 7K individual queries at the start of every cycle on brains with conversation-transcript corpora (5-10 min silent overhead). One batched SQL roundtrip returns the set of `content_hash16` values already extracted as atoms for this source. Fail-open: SQL error logs to stderr and returns an empty set so extraction proceeds. Powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres uses `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop mirroring v97 `pages_dedup_partial_index`, PGLite uses plain `CREATE INDEX`). Without this index the batch idempotency check would seq-scan pages on big brains and defeat the perf win. (2) **Shorter cycle lock TTL + active in-phase refresh:** `LOCK_TTL_MINUTES` dropped from 30 → 5 in `src/core/cycle.ts`. New exported `buildYieldDuringPhase(lock, outer)` closure calls `lock.refresh()` AND any external hook on every fire; throttled to 30s inside each phase via `maybeYield`. Fires both inside the main work loop AND immediately after every `await chat(...)` LLM call so long Haiku/Sonnet calls don't sit past TTL. `LockHandle` and `buildYieldDuringPhase` exported for test seam access. `synthesize_concepts` no longer fires `yieldDuringPhase` per-concept-group — same hook, throttled via the new shared `maybeYield` helper. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps the lock alive. Known residual under shorter TTL: a single `await chat()` past 5 min wallclock can expire the lock mid-await without the original phase noticing — `DbLockHandle.refresh()` throwing on 0 rows affected is filed as P2 follow-up TODO-OPS-2. (3) **Progress wiring through long phases:** new `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`. Cycle.ts passes its phase-level reporter down (NOT a child reporter — that would produce a path collision `cycle.extract_atoms.extract_atoms.work`). Phases only call `tick()` and `heartbeat()`; cycle.ts owns `start()` and `finish()`. You see `[cycle.extract_atoms] N (atoms_created)` ticks every ~1s during both long phases instead of "start" then silence for 10+ min. (4) **`by-mention` resume:** new `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts`. The gazetteer hash is the load-bearing field — adding new entity pages mid-pause shifts the hash, gets a new fingerprint, and triggers a fresh scan against the new gazetteer instead of silently skipping previously-scanned pages. `gbrain extract links --by-mention` now resumes from where it died via the existing `op_checkpoints` framework with a `flushAndCheckpoint` ordering — links flush to the DB FIRST, page keys commit to the checkpoint SECOND, persist THIRD. A crash between `batch.push()` and flush leaves the page un-checkpointed so resume re-scans it. Persist cadence: every 1000 items OR every 30s, whichever first. Clean exit clears the checkpoint. `--dry-run` deliberately skips both load and write so it stays an inspection mode. (5) **`sync_consolidation` doctor check:** multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` recommendation. Single-source brains get "not applicable." SQL errors return `warn` via the check's own try/catch — outer doctor catch isn't a safe assumption. Companion "Multi-source brains" recipe block in `skills/cron-scheduler/SKILL.md` documents the `sync --all` pattern as preferred over per-source entries. (6) **Test-isolation fixes:** `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` migrated to per-test `GBRAIN_HOME=tempdir` isolation. Pinned by 44 new unit/PGLite cases across 9 files: `test/cycle/extract-atoms-batch.test.ts` (5 — batch idempotency), `test/cycle/cycle-lock-ttl.test.ts` (1 — regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts` (7 — fingerprint sensitivity including gazetteer-hash regression guard), `test/cycle/extract-atoms-progress.test.ts` (4 — phase doesn't call start/finish, ticks fire per item), `test/cycle/synthesize-concepts-progress.test.ts` (3 — same shape), `test/cycle/yield-during-phase-refresh.test.ts` (7 — buildYieldDuringPhase actually calls lock.refresh() + outer hook, throws non-fatal), `test/cycle/yield-during-phase-throttle.test.ts` (3 — 30s throttle gate), `test/extract-by-mention-resume.test.ts` (5 — checkpoint persistence ordering, dry-run skips persist, gazetteer change invalidates, filtered pages get checkpointed), `test/doctor-sync-consolidation.test.ts` (6 — edge case matrix for source counts + archived filtering + SQL error path). Two follow-up TODOs filed in TODOS.md under "v0.41.19.0 ops-fix-wave follow-ups": `gbrain sync print-cron` subcommand (TODO-OPS-1) and lock-loss detection via `DbLockHandle.refresh()` throwing on 0 rows affected (TODO-OPS-2). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). @@ -647,7 +648,7 @@ Key commands added in v0.40.3.0 (contextual retrieval + cache gate + 4 CLI verbs - KNOBS_HASH_VERSION bumped 3 → 5 (skipped past 4 reserved by salem's v0.40.4 graph-signals work). One-time cache-row invalidation on upgrade; refills within TTL. - Three new Minion handlers wired into RemediationStep consumer pattern: `lint-fix`, `integrity-auto`, `sync-retry-failed`. Thin wrappers around already-shipping CLI commands; NOT in PROTECTED_JOB_NAMES (idempotent, no shell exec, MCP-safe). `sync-skip-failed` deliberately NOT in this set per codex D12 Bug 3. - `src/core/remediation-step.ts` (NEW canonical module) exports RemediationStep type + makeRemediationStep factory + canonical-JSON idempotencyKey() per codex D12 Bug 2. Future doctor checks emit RemediationSteps via the factory instead of hand-rolling the shape. -- `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. +- `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. **v0.41.25.0 (codex outside-voice on /plan-eng-review):** Layer 1 bookmark read source switched from `MAX(generation) FROM pages` to `SELECT value FROM page_generation_clock WHERE id = 1` at both store + lookup sites. Closes two pre-existing silent stale-cache bug classes that were independent of any sync work: (1) CDX-2 — UPDATE to a non-max page set `NEW.generation = OLD.generation + 1` which didn't advance MAX, so cache silently served stale on every non-max UPDATE; (2) CDX-1 — DELETE didn't fire the row-level trigger at all, and even an AFTER DELETE wouldn't move MAX because surviving rows are untouched. The new clock is bumped per-statement by `bump_page_generation_clock_trg` (statement-level trigger created in migration v106) so every INSERT/UPDATE/DELETE statement advances the bookmark exactly once regardless of row cardinality (D19 — per-row would turn a 73K-row batch DELETE into 73K UPDATEs on the same counter). Also CDX-6/D20 fix: empty-result cache rows used to be "vacuously valid" via Layer 2's `qc.page_generations = '{}'::jsonb` shortcut, silently serving stale empty results across subsequent matching INSERTs. The shortcut is gone — empty snapshots now require Layer 1 to pass. `CACHE_GATE_WHERE_CLAUSE` enforces `qc.page_generations <> '{}'::jsonb` AND the per-page check (not OR). One-time post-upgrade cache miss spike on legacy `{}` rows is acceptable — the cache fills back up correctly and the clock seed `COALESCE(MAX(pages.generation), 0)` keeps non-empty legacy rows serving until the next write. Pinned by 5 new cases in `test/page-generation-counter.test.ts` (CDX-1/CDX-2/CDX-6 regressions + statement-level-trigger-fires-once contract) and 2 new e2e cases in `test/e2e/cache-gate-pglite.test.ts`. The existing pre-v0.41.25.0 `vacuously valid legacy row` IRON-RULE assertion in both files is intentionally inverted: that path was the CDX-6 bug. - `src/core/search/mode-switch-ux.ts` (NEW) exports `summarizeTransition()` 5-cell matrix + `probeWorkerAvailable()` worker liveness proxy via minion_jobs activity + `buildReindexIdempotencyKey()` content-stable key + `runModeSwitchUx()` orchestrator. Key commands added in v0.36.4.0 (brain-health-100 wave): diff --git a/VERSION b/VERSION index 1fd237ef5..1a46e86c5 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.24.0 \ No newline at end of file +0.41.25.0 diff --git a/llms-full.txt b/llms-full.txt index 021e1b7a0..4c3041188 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -183,7 +183,8 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. +- `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. - `src/core/search/explain-formatter.ts` (v0.40.4.0) — renders `SearchResult[]` as a multi-line per-result breakdown for `gbrain search --explain`. Reads every boost-stamping field. Handles the "no boosts applied" empty path. 4-decimal precision with trailing-zero strip. Pinned by `test/search/explain-formatter.test.ts` (19 cases). @@ -193,8 +194,8 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -408,7 +409,7 @@ strict behavior when unset. - `src/core/embedding-dim-check.ts` extension (v0.41.17.0, T5+T6) — facts.embedding dim drift surface. New `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (codex #19 — migration v40 falls back to `vector` on pgvector < 0.7). Regex ordering halfvec-before-vector pinned by tests (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). New `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (codex #18 — NOT bare REINDEX which doesn't rewrite the index after a column-type change). New `assertFactsEmbeddingDimMatchesConfig(engine)` is the D15 preflight — throws `FactsEmbeddingDimMismatchError` (tagged class with `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool's MUST_ABORT semantics) when configured dim doesn't match the column width. Result cached per-engine via `WeakMap`. PGLite engines silently skip. New doctor check `facts_embedding_width_consistency` (registered in `runDoctor` after `embedding_width_consistency`) reuses the same helpers — surfaces drift with paste-ready ALTER recipe identical to the preflight error. Pinned by 18 cases in `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension (v0.41.17.0, T6, codex #20) — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. New `resolveFactsEmbeddingCast()` private method probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`. Both insert paths use the cached suffix so the cast matches the actual column type. Pre-fix all three insert sites hardcoded `::vector`; works on pgvector >= 0.7 via implicit auto-cast but fails on older pgvector. Test seam `__resetFactsEmbeddingCastCacheForTest()` clears per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions (v0.41.20.0 ops-fix-wave) — six daily-driver ops pains in one bisectable wave. (1) **Batch idempotency for `extract_atoms`:** new `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` replaces the per-hash loop that did 7K individual queries at the start of every cycle on brains with conversation-transcript corpora (5-10 min silent overhead). One batched SQL roundtrip returns the set of `content_hash16` values already extracted as atoms for this source. Fail-open: SQL error logs to stderr and returns an empty set so extraction proceeds. Powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres uses `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop mirroring v97 `pages_dedup_partial_index`, PGLite uses plain `CREATE INDEX`). Without this index the batch idempotency check would seq-scan pages on big brains and defeat the perf win. (2) **Shorter cycle lock TTL + active in-phase refresh:** `LOCK_TTL_MINUTES` dropped from 30 → 5 in `src/core/cycle.ts`. New exported `buildYieldDuringPhase(lock, outer)` closure calls `lock.refresh()` AND any external hook on every fire; throttled to 30s inside each phase via `maybeYield`. Fires both inside the main work loop AND immediately after every `await chat(...)` LLM call so long Haiku/Sonnet calls don't sit past TTL. `LockHandle` and `buildYieldDuringPhase` exported for test seam access. `synthesize_concepts` no longer fires `yieldDuringPhase` per-concept-group — same hook, throttled via the new shared `maybeYield` helper. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps the lock alive. Known residual under shorter TTL: a single `await chat()` past 5 min wallclock can expire the lock mid-await without the original phase noticing — `DbLockHandle.refresh()` throwing on 0 rows affected is filed as P2 follow-up TODO-OPS-2. (3) **Progress wiring through long phases:** new `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`. Cycle.ts passes its phase-level reporter down (NOT a child reporter — that would produce a path collision `cycle.extract_atoms.extract_atoms.work`). Phases only call `tick()` and `heartbeat()`; cycle.ts owns `start()` and `finish()`. You see `[cycle.extract_atoms] N (atoms_created)` ticks every ~1s during both long phases instead of "start" then silence for 10+ min. (4) **`by-mention` resume:** new `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts`. The gazetteer hash is the load-bearing field — adding new entity pages mid-pause shifts the hash, gets a new fingerprint, and triggers a fresh scan against the new gazetteer instead of silently skipping previously-scanned pages. `gbrain extract links --by-mention` now resumes from where it died via the existing `op_checkpoints` framework with a `flushAndCheckpoint` ordering — links flush to the DB FIRST, page keys commit to the checkpoint SECOND, persist THIRD. A crash between `batch.push()` and flush leaves the page un-checkpointed so resume re-scans it. Persist cadence: every 1000 items OR every 30s, whichever first. Clean exit clears the checkpoint. `--dry-run` deliberately skips both load and write so it stays an inspection mode. (5) **`sync_consolidation` doctor check:** multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` recommendation. Single-source brains get "not applicable." SQL errors return `warn` via the check's own try/catch — outer doctor catch isn't a safe assumption. Companion "Multi-source brains" recipe block in `skills/cron-scheduler/SKILL.md` documents the `sync --all` pattern as preferred over per-source entries. (6) **Test-isolation fixes:** `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` migrated to per-test `GBRAIN_HOME=tempdir` isolation. Pinned by 44 new unit/PGLite cases across 9 files: `test/cycle/extract-atoms-batch.test.ts` (5 — batch idempotency), `test/cycle/cycle-lock-ttl.test.ts` (1 — regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts` (7 — fingerprint sensitivity including gazetteer-hash regression guard), `test/cycle/extract-atoms-progress.test.ts` (4 — phase doesn't call start/finish, ticks fire per item), `test/cycle/synthesize-concepts-progress.test.ts` (3 — same shape), `test/cycle/yield-during-phase-refresh.test.ts` (7 — buildYieldDuringPhase actually calls lock.refresh() + outer hook, throws non-fatal), `test/cycle/yield-during-phase-throttle.test.ts` (3 — 30s throttle gate), `test/extract-by-mention-resume.test.ts` (5 — checkpoint persistence ordering, dry-run skips persist, gazetteer change invalidates, filtered pages get checkpointed), `test/doctor-sync-consolidation.test.ts` (6 — edge case matrix for source counts + archived filtering + SQL error path). Two follow-up TODOs filed in TODOS.md under "v0.41.19.0 ops-fix-wave follow-ups": `gbrain sync print-cron` subcommand (TODO-OPS-1) and lock-loss detection via `DbLockHandle.refresh()` throwing on 0 rows affected (TODO-OPS-2). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). @@ -789,7 +790,7 @@ Key commands added in v0.40.3.0 (contextual retrieval + cache gate + 4 CLI verbs - KNOBS_HASH_VERSION bumped 3 → 5 (skipped past 4 reserved by salem's v0.40.4 graph-signals work). One-time cache-row invalidation on upgrade; refills within TTL. - Three new Minion handlers wired into RemediationStep consumer pattern: `lint-fix`, `integrity-auto`, `sync-retry-failed`. Thin wrappers around already-shipping CLI commands; NOT in PROTECTED_JOB_NAMES (idempotent, no shell exec, MCP-safe). `sync-skip-failed` deliberately NOT in this set per codex D12 Bug 3. - `src/core/remediation-step.ts` (NEW canonical module) exports RemediationStep type + makeRemediationStep factory + canonical-JSON idempotencyKey() per codex D12 Bug 2. Future doctor checks emit RemediationSteps via the factory instead of hand-rolling the shape. -- `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. +- `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. **v0.41.25.0 (codex outside-voice on /plan-eng-review):** Layer 1 bookmark read source switched from `MAX(generation) FROM pages` to `SELECT value FROM page_generation_clock WHERE id = 1` at both store + lookup sites. Closes two pre-existing silent stale-cache bug classes that were independent of any sync work: (1) CDX-2 — UPDATE to a non-max page set `NEW.generation = OLD.generation + 1` which didn't advance MAX, so cache silently served stale on every non-max UPDATE; (2) CDX-1 — DELETE didn't fire the row-level trigger at all, and even an AFTER DELETE wouldn't move MAX because surviving rows are untouched. The new clock is bumped per-statement by `bump_page_generation_clock_trg` (statement-level trigger created in migration v106) so every INSERT/UPDATE/DELETE statement advances the bookmark exactly once regardless of row cardinality (D19 — per-row would turn a 73K-row batch DELETE into 73K UPDATEs on the same counter). Also CDX-6/D20 fix: empty-result cache rows used to be "vacuously valid" via Layer 2's `qc.page_generations = '{}'::jsonb` shortcut, silently serving stale empty results across subsequent matching INSERTs. The shortcut is gone — empty snapshots now require Layer 1 to pass. `CACHE_GATE_WHERE_CLAUSE` enforces `qc.page_generations <> '{}'::jsonb` AND the per-page check (not OR). One-time post-upgrade cache miss spike on legacy `{}` rows is acceptable — the cache fills back up correctly and the clock seed `COALESCE(MAX(pages.generation), 0)` keeps non-empty legacy rows serving until the next write. Pinned by 5 new cases in `test/page-generation-counter.test.ts` (CDX-1/CDX-2/CDX-6 regressions + statement-level-trigger-fires-once contract) and 2 new e2e cases in `test/e2e/cache-gate-pglite.test.ts`. The existing pre-v0.41.25.0 `vacuously valid legacy row` IRON-RULE assertion in both files is intentionally inverted: that path was the CDX-6 bug. - `src/core/search/mode-switch-ux.ts` (NEW) exports `summarizeTransition()` 5-cell matrix + `probeWorkerAvailable()` worker liveness proxy via minion_jobs activity + `buildReindexIdempotencyKey()` content-stable key + `runModeSwitchUx()` orchestrator. Key commands added in v0.36.4.0 (brain-health-100 wave): diff --git a/package.json b/package.json index f6017e452..a47a44b51 100644 --- a/package.json +++ b/package.json @@ -140,5 +140,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.24.0" + "version": "0.41.25.0" } diff --git a/scripts/run-unit-parallel.sh b/scripts/run-unit-parallel.sh index f46b48cdb..007d5c9e1 100755 --- a/scripts/run-unit-parallel.sh +++ b/scripts/run-unit-parallel.sh @@ -181,28 +181,88 @@ bun_summary_count() { ' "$file" } +# shard_total_files: parse the "[unit-shard N/M] running X files" line that +# run-unit-shard.sh echoes before invoking bun test. Returns the file count +# the shard was given, or 0 if the line isn't there yet (shard still +# bootstrapping). Uses sed-then-grep so it's portable to macOS awk (BSD awk +# doesn't support `match($0, /re/, arr)` with the array sink — that's gawk-only). +shard_total_files() { + local file="$1" + [ -f "$file" ] || { echo 0; return; } + local n + n=$(sed -n 's/^\[unit-shard [0-9][0-9]*\/[0-9][0-9]*\] running \([0-9][0-9]*\) files.*/\1/p' "$file" 2>/dev/null | head -1) + echo "${n:-0}" +} + +# shard_pglite_init_count: count "Schema version" lines as a proxy for "test +# files initialized so far." Each PGLite-using test file's beforeAll triggers +# one initSchema() which prints this. Undercounts because not every test file +# opens a PGLite engine, but it's the only real-time progress signal bun's +# default reporter leaves in the log (bun has no per-file progress markers, +# only a final shard-end summary). +shard_pglite_init_count() { + local file="$1" + [ -f "$file" ] || { echo 0; return; } + grep -cE 'Schema version [0-9]+ → [0-9]+' "$file" 2>/dev/null || echo 0 +} + +# log_size_kb: total stderr+stdout written by the shard so far. Strictly +# monotonic — useful as a "definitely alive" signal when other heuristics +# read 0 (e.g. very early in shard startup before initSchema fires). +log_size_kb() { + local file="$1" + [ -f "$file" ] || { echo 0; return; } + local b + b=$(wc -c < "$file" 2>/dev/null | tr -d ' ') + echo $(( ${b:-0} / 1024 )) +} + +# fmt_elapsed: pretty-print seconds → "Mm:SS" or "SSs" for short. +fmt_elapsed() { + local s=$1 + if [ "$s" -ge 60 ]; then + printf '%dm%02ds' $((s / 60)) $((s % 60)) + else + printf '%ds' "$s" + fi +} + heartbeat() { + local hb_start=$(date +%s) while true; do sleep 10 local line="" + local now; now=$(date +%s) + local hb_elapsed=$((now - hb_start)) for i in $(seq 1 "$N"); do if [ -f "$LOG_DIR/shard-$i.exit" ]; then local rc; rc=$(cat "$LOG_DIR/shard-$i.exit" 2>/dev/null || echo "?") local status="✓" [ "$rc" != "0" ] && status="✗" - line="$line [s$i: done $status]" + local f + f=$(bun_summary_count "fail" "$LOG_DIR/shard-$i.log") + local p + p=$(bun_summary_count "pass" "$LOG_DIR/shard-$i.log") + line="$line [s$i: done $status ${p}p ${f}f]" else local lf="$LOG_DIR/shard-$i.log" if [ -f "$lf" ]; then - # Heartbeat: prefer Bun's per-test "✓" (passed) and "(fail)" markers - # so we see live progress; the "N pass" summary line only appears at - # the very end of the shard and would always show 0 mid-run. - local p f - p=$(grep_count '^[[:space:]]+✓' "$lf") - f=$(grep_count '^\(fail\)' "$lf") - line="$line [s$i: ${p}p ${f}f ...]" + # Bun's default reporter has no per-file progress markers, only a + # final shard-end summary, so we surface three complementary signals + # mid-run: (1) PGLite initSchema() count as a "files started" proxy, + # (2) total files this shard was assigned (from the runner banner), + # (3) log size in KB as a strictly-monotonic liveness signal. + local total; total=$(shard_total_files "$lf") + local pglite; pglite=$(shard_pglite_init_count "$lf") + local kb; kb=$(log_size_kb "$lf") + local et; et=$(fmt_elapsed "$hb_elapsed") + if [ "$total" -gt 0 ]; then + line="$line [s$i: ~${pglite}/${total}f ${kb}KB ${et}]" + else + line="$line [s$i: starting ${kb}KB ${et}]" + fi else - line="$line [s$i: starting]" + line="$line [s$i: spawning]" fi fi done diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 6cd37e34e..41628afe1 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -2,6 +2,7 @@ import { existsSync, readFileSync, writeFileSync, statSync } from 'fs'; import { execFileSync } from 'child_process'; import { join, relative } from 'path'; import type { BrainEngine } from '../core/engine.ts'; +import { DELETE_BATCH_SIZE } from '../core/engine-constants.ts'; import { importFile } from '../core/import-file.ts'; import { collectSyncableFiles } from './import.ts'; import { createInterface } from 'readline'; @@ -269,14 +270,25 @@ export async function resolveSlugByPathOrSourcePath( path: string, sourceId?: string, ): Promise { + // v0.41.19.0 (D8): when sourceId is set, delegate to the new batch + // resolveSlugsByPaths so single-call and batched paths share one SQL + // owner + one fallback semantic. One Map allocation per single-call; + // negligible cost. When sourceId is undefined (legacy unscoped callers), + // fall back to the original executeRaw shape — the batch method + // requires sourceId to prevent the multi-source-bug-class on its new + // surface (D5). The unscoped fallback preserves back-compat. try { - const rows = await engine.executeRaw<{ slug: string }>( - sourceId - ? `SELECT slug FROM pages WHERE source_path = $1 AND source_id = $2 LIMIT 1` - : `SELECT slug FROM pages WHERE source_path = $1 LIMIT 1`, - sourceId ? [path, sourceId] : [path], - ); - if (rows.length > 0 && rows[0].slug) return rows[0].slug; + if (sourceId) { + const m = await engine.resolveSlugsByPaths([path], { sourceId }); + const slug = m.get(path); + if (slug) return slug; + } else { + const rows = await engine.executeRaw<{ slug: string }>( + `SELECT slug FROM pages WHERE source_path = $1 LIMIT 1`, + [path], + ); + if (rows.length > 0 && rows[0].slug) return rows[0].slug; + } } catch { // Fall through — best-effort. Pre-migration brains or query errors // shouldn't break delete/rename for path-derived pages. @@ -1233,25 +1245,135 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise = []; + // v0.18.0+ multi-source: scope deletePage so we only delete the source-A // row, not every same-slug row across all sources. const deleteOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined; + + // v0.41.19.0 (T2/D6/D7/D16/D18 via /plan-eng-review + codex outside-voice): + // batched delete loop. Replaces the per-file N+1 that PR #1538 originally + // batched on Postgres only. See plan file: + // ~/.claude/plans/system-instruction-you-are-working-ethereal-narwhal.md + // + // SHAPE (interleaved per-batch resolve + delete; caller owns chunking): + // + // filtered.deleted (e.g. 73K paths) + // │ + // ▼ + // slice into batches of DELETE_BATCH_SIZE (500) + // │ + // ▼ for each batch: + // abort-check ──► partial('timeout') + // │ + // ▼ + // engine.resolveSlugsByPaths(batch, {sourceId}) ◀── 1 SQL round-trip + // │ + // ▼ + // slugs = batch.map(path => map.get(path) + // ?? resolveSlugForPath(path)) ◀── pure-JS fallback for + // │ frontmatter-fallback + // ▼ + missing-source-path + // try { + // deleted = engine.deletePages(slugs, opts) ◀── 1 SQL round-trip + // pagesAffected.push(...deleted) ◀── D6: only confirmed + // } catch { deletes, not phantoms + // // D7 decompose: per-slug deletePage, + // // unrecoverable failures → failedFiles + // } + // + // ROUND-TRIP COUNTS (73K deletes): + // pre-fix: 73,000 SELECTs + 73,000 DELETEs = 146,000 (~5 hours) + // post-fix: 146 SELECTs + 146 DELETEs = 292 (~2 minutes) + // + // ATOMICITY (D3): each batch is one transaction. A mid-batch abort or + // transient connection failure rolls back up to DELETE_BATCH_SIZE - 1 + // successful deletes. Sync is idempotent — the next run picks them up + // via git diff regenerating the deletion list. + // + // NO-SOURCEID FALLBACK: when opts.sourceId is undefined (legacy unscoped + // callers, rare post-v0.34.1 source-resolution wiring), fall back to the + // OLD per-path loop. The batch engine surface requires sourceId per D5 + // (multi-source-bug-class defense at the type level). Production callers + // that thread sourceId via resolveSourceWithTier get the new fast path. if (filtered.deleted.length > 0) { progress.start('sync.deletes', filtered.deleted.length); - for (const path of filtered.deleted) { - // v0.41.13.0 (T2 / D-V4-2): per-iteration abort check. Codex pass-3 - // F8 caught that v3 only covered pull + add/modify. Refactor commits - // with hundreds of deletes can overshoot --timeout without this check. - if (opts.signal?.aborted) { - progress.finish(); - return partial('timeout'); + if (opts.sourceId) { + const sid = opts.sourceId; + const deleteScopedOpts = { sourceId: sid }; + for (let i = 0; i < filtered.deleted.length; i += DELETE_BATCH_SIZE) { + if (opts.signal?.aborted) { + progress.finish(); + return partial('timeout'); + } + const batch = filtered.deleted.slice(i, i + DELETE_BATCH_SIZE); + + // Phase A: batch slug resolution (1 round-trip per batch). + let pathSlugMap: Map; + try { + pathSlugMap = await engine.resolveSlugsByPaths(batch, deleteScopedOpts); + } catch { + // Resolve failure: fall back to empty map; per-path fallback + // below will use resolveSlugForPath. Best-effort, matches the + // existing resolveSlugByPathOrSourcePath swallow-and-fallback + // semantics. + pathSlugMap = new Map(); + } + const slugs = batch.map(p => pathSlugMap.get(p) ?? resolveSlugForPath(p)); + + // Phase B: batch delete (1 round-trip per batch). + try { + const deleted = await engine.deletePages(slugs, deleteScopedOpts); + // D6: only push slugs that were actually deleted. Filters phantom + // slugs (paths in filtered.deleted but with no DB row) so + // downstream extract/embed don't waste lookups. + pagesAffected.push(...deleted); + } catch (err) { + // D7 decompose: a transient blip on this batch shouldn't lose all + // 500 deletes. Fall back to per-slug deletePage for THIS batch + // only; unrecoverable per-slug failures land in failedFiles + // (matching the existing import-loop pattern at sync.ts:~1350). + for (let j = 0; j < slugs.length; j++) { + try { + await engine.deletePage(slugs[j], deleteScopedOpts); + pagesAffected.push(slugs[j]); + } catch (perSlugErr) { + failedFiles.push({ + path: batch[j], + error: `delete failed: ${perSlugErr instanceof Error ? perSlugErr.message : String(perSlugErr)} (batch error: ${err instanceof Error ? err.message : String(err)})`, + }); + } + } + } + progress.tick(batch.length, `deletes ${Math.min(i + DELETE_BATCH_SIZE, filtered.deleted.length)}/${filtered.deleted.length}`); + } + } else { + // Legacy no-sourceId path. The engine batch methods require sourceId + // per D5 (kills the multi-source-bug-class on the new surface); when + // sourceId is unset, fall back to the original per-path loop. Slow + // but correct; production callers all thread sourceId so this branch + // is functionally dead post-v0.34.1. + for (const path of filtered.deleted) { + if (opts.signal?.aborted) { + progress.finish(); + return partial('timeout'); + } + const slug = await resolveSlugByPathOrSourcePath(engine, path, undefined); + try { + await engine.deletePage(slug, deleteOpts); + pagesAffected.push(slug); + } catch (err) { + failedFiles.push({ + path, + error: `delete failed: ${err instanceof Error ? err.message : String(err)}`, + }); + } + progress.tick(1, slug); } - const slug = await resolveSlugByPathOrSourcePath(engine, path, opts.sourceId); - await engine.deletePage(slug, deleteOpts); - pagesAffected.push(slug); - progress.tick(1, slug); } progress.finish(); } @@ -1260,12 +1382,46 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 0) { progress.start('sync.renames', filtered.renamed.length); // v0.18.0+ multi-source: scope updateSlug so the rename only touches the // source-A row, not every same-slug row across sources (which would // either sweep them all OR violate (source_id, slug) UNIQUE). const renameOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined; + + // T4: pre-resolve ALL `from` slugs in batches before iterating. Falls + // back to per-path resolveSlugByPathOrSourcePath when sourceId is + // unset (matches the delete loop's legacy posture). For large rename + // commits (rare but possible: prefix sweep, reorganization), this drops + // the slug-resolve round-trips from O(renames) to O(renames/500). + const fromSlugByPath = new Map(); + if (opts.sourceId) { + const sid = opts.sourceId; + const fromPaths = filtered.renamed.map(r => r.from); + for (let i = 0; i < fromPaths.length; i += DELETE_BATCH_SIZE) { + if (opts.signal?.aborted) { + progress.finish(); + return partial('timeout'); + } + const batch = fromPaths.slice(i, i + DELETE_BATCH_SIZE); + let m: Map; + try { + m = await engine.resolveSlugsByPaths(batch, { sourceId: sid }); + } catch { + m = new Map(); + } + for (const p of batch) { + fromSlugByPath.set(p, m.get(p) ?? resolveSlugForPath(p)); + } + } + } + for (const { from, to } of filtered.renamed) { // v0.41.13.0 (T2 / D-V4-2): per-iteration abort check. Renames call // importFile() at line 1173-style sites which can be slow on big files; @@ -1274,7 +1430,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise = []; + // v0.41.19.0: `failedFiles` is now hoisted above the delete loop (the + // delete decompose path appends here too); kept as a comment-pin so + // future maintainers know to thread additional failure surfaces through + // the same array. const addsAndMods = [...filtered.added, ...filtered.modified]; // Sort newest-first so date-prefixed brain paths get embedded before older diff --git a/src/core/engine-constants.ts b/src/core/engine-constants.ts new file mode 100644 index 000000000..a40d2ba46 --- /dev/null +++ b/src/core/engine-constants.ts @@ -0,0 +1,25 @@ +// v0.41.25.0 — single source of truth for engine batch-sizing. +// +// Both PostgresEngine and PGLiteEngine import from here so the constants +// cannot drift across engines. Lives outside `src/core/engine.ts` to avoid +// circular-import worries (engine.ts is the interface; engines depend on +// engine.ts; this file depends on neither). + +/** + * Maximum number of slugs per single batch `DELETE FROM pages WHERE slug = + * ANY($1::text[])` call. Callers (e.g. `src/commands/sync.ts` delete loop) + * are responsible for chunking input arrays to this size; `engine.deletePages` + * is a single-batch primitive that does NOT chunk internally (matches the + * `addLinksBatch` convention — caller owns chunking, engine assumes the + * caller is well-behaved). + * + * 500 is the same order-of-magnitude as the effective per-call budget for + * the existing `addLinksBatch` (postgres-engine.ts) — well under Postgres's + * 65535 parameter cap. We bind a single array parameter so the cap doesn't + * bite directly, but per-statement work stays bounded for predictable lock + * hold time + write-amplification budget. + * + * The same constant is also used for batch `SELECT slug, source_path FROM + * pages WHERE source_path = ANY($1::text[])` in `engine.resolveSlugsByPaths`. + */ +export const DELETE_BATCH_SIZE = 500; diff --git a/src/core/engine.ts b/src/core/engine.ts index b6187a196..fa2f0d015 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -712,8 +712,67 @@ export interface BrainEngine { * delete doesn't hard-delete the same-slug pages in sources B/C/D. Without * it, the bare DELETE matches every row with that slug across all sources. * Cascades through content_chunks / page_links / chunk_relations via FKs. + * + * v0.41.19.0 (CDX-11): single-row primitive used by `purgeDeletedPages`, + * `gbrain sync` (one path per call), test setup teardown, and the v0.41.19.0 + * sync-delete decompose path (when `deletePages` throws on a 500-row batch, + * the sync loop falls back to per-slug `deletePage` to log unrecoverable + * failures to `failedFiles`). `gbrain sync` calls this on EVERY run that + * sees a deleted file — it is NOT admin-only. */ deletePage(slug: string, opts?: { sourceId?: string }): Promise; + /** + * v0.41.19.0 — batch delete: single SQL round-trip via + * `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 + * RETURNING slug`. Cascades through content_chunks / page_links (×3) / + * tags / raw_data / timeline_entries / page_versions via FKs declared in + * `src/schema.sql`. `files.page_id` and `links.origin_page_id` go SET + * NULL per their FK definitions. + * + * SINGLE-BATCH PRIMITIVE: caller is responsible for chunking the input to + * `<= DELETE_BATCH_SIZE` entries per call (see + * `src/core/engine-constants.ts`). Matches the `addLinksBatch` convention + * — engine assumes well-behaved input, caller owns the slicing. + * + * Returns the slugs of rows ACTUALLY DELETED (order undefined). Callers + * use this to filter their own `pagesAffected` tracking so downstream + * phases don't waste lookups on phantom slugs (paths that were in the + * deletion list but had no DB row). + * + * ATOMICITY: one statement, one transaction. The whole batch commits or + * the whole batch rolls back. Coarser than the per-row `deletePage` + * cadence — a mid-loop abort or transient connection failure can roll + * back up to `DELETE_BATCH_SIZE - 1` successful deletes from the + * in-flight batch. `gbrain sync` is idempotent (next run picks them up + * via git diff); other callers should account for the contract. + * + * sourceId is REQUIRED (no `'default'` fallback). This is asymmetric with + * `deletePage` (which keeps the optional/'default' fallback for back- + * compat). Filed as v0.42+ TODO to tighten `deletePage` to match once a + * full caller audit confirms every site threads `sourceId`. + */ + deletePages(slugs: string[], opts: { sourceId: string }): Promise; + /** + * v0.41.19.0 — batch path → slug resolution. Single SQL round-trip via + * `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) + * AND source_id = $2`. Returns `Map`; paths NOT in the map + * have no `source_path` row in the DB and the caller is expected to fall + * back to `resolveSlugForPath(path)` for the path-derived slug. + * + * Mirrors the contract of the single-call `resolveSlugByPathOrSourcePath` + * helper in `src/commands/sync.ts`, batched. As of v0.41.19.0, that + * single-call helper is implemented on top of this method (one Map + * allocation per single-path call; negligible cost; one owner of the SQL + * + fallback semantics). + * + * SINGLE-BATCH PRIMITIVE: caller chunks to `<= DELETE_BATCH_SIZE`. + * + * Empty `paths` short-circuits to an empty Map without touching the DB. + */ + resolveSlugsByPaths( + paths: string[], + opts: { sourceId: string }, + ): Promise>; /** * v0.26.5 — set `deleted_at = now()` on a page. Returns the slug if a row * was soft-deleted, null if no row matched (already soft-deleted OR not found). diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 2b7e17014..4de08e538 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -4856,6 +4856,70 @@ export const MIGRATIONS: Migration[] = [ ON extract_rollup_7d (day); `, }, + { + version: 107, + name: 'page_generation_clock_and_statement_trigger', + // v0.41.25.0 (D18/D19, codex outside-voice on /plan-eng-review): global + // page-generation clock + statement-level trigger. + // + // Renumbered v104 → v105 → v106 → v107 during master merges: + // PR #1545 (v0.41.21.0 ops-fix-wave) took v104 for pages_atom_source_hash_idx; + // PR #1542 (v0.41.22.0 type-unification cathedral) took v105 for slug_aliases; + // PR #1541 (v0.41.23.0 extract operator surfaces) took v106 for extract_rollup_7d_table. + // + // Why this exists: the pre-v0.41.25.0 query-cache Layer 1 bookmark read + // `MAX(generation) FROM pages` to detect "writes happened since cache + // store". Two bugs in that contract — independent of any sync work: + // + // 1. The row-level `bump_page_generation_trg` (migration v91) sets + // `NEW.generation = OLD.generation + 1` on UPDATE. Updating a + // NON-MAX page didn't advance MAX(generation). Cache silently + // served stale results for any UPDATE-to-non-max page. + // 2. The trigger is BEFORE INSERT OR UPDATE — DELETE doesn't fire it + // at all. Even an AFTER DELETE wouldn't move MAX (surviving rows + // are untouched). + // + // The fix: single-row counter, bumped per-statement (FOR EACH STATEMENT + // — row-level would turn a 73K-row batch DELETE into 73K UPDATEs on the + // same counter, recreating the bottleneck the sync-delete wave is + // fixing in this same PR). Layer 1 reads page_generation_clock.value + // directly. Per-row pages.generation stays for Layer 2 (per-page + // snapshot via jsonb_each + LEFT JOIN pages) which doesn't care about + // MAX, only per-page advancement. + // + // Seeded with COALESCE(MAX(pages.generation), 0) so existing + // query_cache rows stored under the old MAX semantics aren't all + // instantly invalidated on upgrade. Their max_generation_at_store + // stamp compares cleanly against the seeded clock; future writes bump + // the clock and the bookmark fires correctly. + // + // Mirror lives in src/core/pglite-schema.ts (fresh-install path). + // Forward-reference bootstrap probe in applyForwardReferenceBootstrap + // on both engines so pre-v0.41.25.0 brains pick it up cleanly. + idempotent: true, + sql: ` + CREATE TABLE IF NOT EXISTS page_generation_clock ( + id INTEGER PRIMARY KEY CHECK (id = 1), + value BIGINT NOT NULL DEFAULT 0 + ); + INSERT INTO page_generation_clock (id, value) + VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) + ON CONFLICT (id) DO NOTHING; + + CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$ + BEGIN + UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + RETURN NULL; + END; + $func$ LANGUAGE plpgsql; + + DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages; + CREATE TRIGGER bump_page_generation_clock_trg + AFTER INSERT OR UPDATE OR DELETE ON pages + FOR EACH STATEMENT + EXECUTE FUNCTION bump_page_generation_clock_fn(); + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index e6dacafcf..d42a3c1e1 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -22,6 +22,7 @@ import { logBatchRetry as auditLogBatchRetry, logBatchExhausted as auditLogBatch import { runMigrations } from './migrate.ts'; import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts'; import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts'; +import { DELETE_BATCH_SIZE } from './engine-constants.ts'; import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts'; import type { Page, PageInput, PageFilters, PageType, @@ -900,6 +901,49 @@ export class PGLiteEngine implements BrainEngine { ); } + /** + * v0.41.19.0 — batch delete primitive. See BrainEngine.deletePages JSDoc. + * Parity implementation with PostgresEngine.deletePages. PGLite supports + * `slug = ANY($1)` array-param binding natively (addLinksBatch already + * proves this). + */ + async deletePages(slugs: string[], opts: { sourceId: string }): Promise { + if (slugs.length === 0) return []; + if (slugs.length > DELETE_BATCH_SIZE) { + throw new Error( + `deletePages: input size ${slugs.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`, + ); + } + const { rows } = await this.db.query<{ slug: string }>( + 'DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug', + [slugs, opts.sourceId], + ); + return rows.map(r => r.slug); + } + + /** + * v0.41.19.0 — batch path → slug resolution. See BrainEngine.resolveSlugsByPaths + * JSDoc. + */ + async resolveSlugsByPaths( + paths: string[], + opts: { sourceId: string }, + ): Promise> { + if (paths.length === 0) return new Map(); + if (paths.length > DELETE_BATCH_SIZE) { + throw new Error( + `resolveSlugsByPaths: input size ${paths.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`, + ); + } + const { rows } = await this.db.query<{ slug: string; source_path: string }>( + 'SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2', + [paths, opts.sourceId], + ); + const m = new Map(); + for (const r of rows) m.set(r.source_path, r.slug); + return m; + } + async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> { // Idempotent-as-null: only flip rows currently active. Source filter is // optional; without it the first matching row across sources gets soft-deleted. diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index b844f2ce9..07c5c5714 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -144,6 +144,32 @@ CREATE TRIGGER bump_page_generation_trg FOR EACH ROW EXECUTE FUNCTION bump_page_generation_fn(); +-- v0.41.19.0 (D18/D19, mirror of src/schema.sql): global page-generation +-- clock + statement-level trigger. See src/schema.sql for the full +-- rationale comment. Layer 1 bookmark reads page_generation_clock.value; +-- per-row pages.generation above stays as the Layer 2 (per-page snapshot) +-- substrate. +CREATE TABLE IF NOT EXISTS page_generation_clock ( + id INTEGER PRIMARY KEY CHECK (id = 1), + value BIGINT NOT NULL DEFAULT 0 +); +INSERT INTO page_generation_clock (id, value) + VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) + ON CONFLICT (id) DO NOTHING; + +CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$ +BEGIN + UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + RETURN NULL; +END; +$func$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages; +CREATE TRIGGER bump_page_generation_clock_trg + AFTER INSERT OR UPDATE OR DELETE ON pages + FOR EACH STATEMENT + EXECUTE FUNCTION bump_page_generation_clock_fn(); + CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type); CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter); CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops); diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index ad1c247a4..e98cb25a7 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -57,6 +57,7 @@ import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, pa import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts'; import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts'; +import { DELETE_BATCH_SIZE } from './engine-constants.ts'; function escapeSqlStringLiteral(value: string): string { return value.replace(/'/g, "''"); @@ -911,6 +912,53 @@ export class PostgresEngine implements BrainEngine { await sql`DELETE FROM pages WHERE slug = ${slug} AND source_id = ${sourceId}`; } + /** + * v0.41.19.0 — batch delete primitive. See BrainEngine.deletePages JSDoc. + * Single SQL round-trip per call; caller is responsible for chunking input + * to <= DELETE_BATCH_SIZE. RETURNING slug projects the actually-deleted set + * so the caller can filter pagesAffected. + */ + async deletePages(slugs: string[], opts: { sourceId: string }): Promise { + if (slugs.length === 0) return []; + if (slugs.length > DELETE_BATCH_SIZE) { + throw new Error( + `deletePages: input size ${slugs.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`, + ); + } + const sql = this.sql; + const rows = await sql<{ slug: string }[]>` + DELETE FROM pages + WHERE slug = ANY(${slugs}::text[]) AND source_id = ${opts.sourceId} + RETURNING slug + `; + return rows.map(r => r.slug); + } + + /** + * v0.41.19.0 — batch path → slug resolution. See BrainEngine.resolveSlugsByPaths + * JSDoc. Single SQL round-trip; folds rows into a Map. + */ + async resolveSlugsByPaths( + paths: string[], + opts: { sourceId: string }, + ): Promise> { + if (paths.length === 0) return new Map(); + if (paths.length > DELETE_BATCH_SIZE) { + throw new Error( + `resolveSlugsByPaths: input size ${paths.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`, + ); + } + const sql = this.sql; + const rows = await sql<{ slug: string; source_path: string }[]>` + SELECT slug, source_path + FROM pages + WHERE source_path = ANY(${paths}::text[]) AND source_id = ${opts.sourceId} + `; + const m = new Map(); + for (const r of rows) m.set(r.source_path, r.slug); + return m; + } + async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> { const sql = this.sql; const sourceId = opts?.sourceId; diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index 3b863840c..b48847389 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -185,6 +185,48 @@ CREATE TRIGGER bump_page_generation_trg -- CREATE INDEX since the table is empty. CREATE INDEX IF NOT EXISTS pages_generation_idx ON pages (generation); +-- v0.41.19.0 (D18/D19, codex outside-voice): global page-generation clock. +-- The pre-v0.41.19.0 Layer 1 bookmark read \`MAX(generation) FROM pages\` to +-- detect "writes happened since cache-store". Two bugs in that contract: +-- 1. The row-level trigger above sets \`NEW.generation = OLD.generation + 1\` +-- on UPDATE. Updating a NON-MAX page didn't advance MAX(generation), +-- silently serving stale cached results. +-- 2. The trigger is \`BEFORE INSERT OR UPDATE\` so DELETE doesn't fire it +-- at all — and even if it did, DELETE doesn't touch surviving rows, +-- so MAX(generation) wouldn't budge. +-- +-- The fix: a single-row counter, bumped per-statement (FOR EACH STATEMENT +-- — codex CDX-4: per-row would turn 73K-row batch DELETE into 73K UPDATEs +-- on the same counter, recreating the bottleneck this PR is fixing). Layer +-- 1 reads \`page_generation_clock.value\` directly. The per-row +-- \`pages.generation\` column above stays as the Layer 2 (per-page snapshot) +-- substrate. +-- +-- Seeded with COALESCE(MAX(pages.generation), 0) so existing query_cache +-- rows stored under the old MAX semantics aren't all instantly invalidated +-- on upgrade (their max_generation_at_store stamp compares cleanly against +-- the seeded clock; future writes bump the clock, bookmark fires correctly). +CREATE TABLE IF NOT EXISTS page_generation_clock ( + id INTEGER PRIMARY KEY CHECK (id = 1), + value BIGINT NOT NULL DEFAULT 0 +); +INSERT INTO page_generation_clock (id, value) + VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) + ON CONFLICT (id) DO NOTHING; + +CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS \$func\$ +BEGIN + UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + RETURN NULL; +END; +\$func\$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages; +CREATE TRIGGER bump_page_generation_clock_trg + AFTER INSERT OR UPDATE OR DELETE ON pages + FOR EACH STATEMENT + EXECUTE FUNCTION bump_page_generation_clock_fn(); + CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type); CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter); CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops); @@ -266,6 +308,13 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding_image CREATE INDEX IF NOT EXISTS idx_chunks_search_vector ON content_chunks USING GIN(search_vector); CREATE INDEX IF NOT EXISTS idx_chunks_symbol_qualified ON content_chunks(symbol_name_qualified) WHERE symbol_name_qualified IS NOT NULL; +-- v0.41.18.0 (codex finding #9): partial index for \`gbrain embed --stale\` +-- + \`--priority recent\`. content_chunks has no updated_at column (chunks +-- are re-INSERTed on page change, not UPDATEd), so the "recent-first" +-- ORDER BY happens at the JOIN site: outer ORDER BY p.updated_at DESC +-- uses idx_pages_updated_at_desc; inner partial uses this index. +CREATE INDEX IF NOT EXISTS content_chunks_stale_idx + ON content_chunks(page_id, chunk_index) WHERE embedding IS NULL; -- v0.20.0 Cathedral II: chunk-grain FTS trigger. -- Weight 'A' on doc_comment + symbol_name_qualified; weight 'B' on chunk_text. @@ -357,10 +406,16 @@ CREATE TABLE IF NOT EXISTS links ( to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE, link_type TEXT NOT NULL DEFAULT '', context TEXT NOT NULL DEFAULT '', - -- v0.42.0.0: 'mentions' added for auto-linked body-text mentions + -- v0.41.18.0: 'mentions' added for auto-linked body-text mentions -- (gbrain extract links --by-mention). Filtered OUT of backlink-count -- for search ranking; only counts toward orphan-ratio + graph traversal. link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')), + -- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from + -- "verb-pattern-derived typed link" within link_source='mentions'. + -- Codex finding #12 design: keep link_source stable; add link_kind + -- so callers can distinguish without breaking existing mentions queries. + -- NULL = legacy / unknown / pre-v98 row (semantically 'plain'). + link_kind TEXT CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner')), origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL, origin_field TEXT, -- v0.18.0 Step 4: 'qualified' when the link was written as @@ -423,8 +478,10 @@ CREATE TABLE IF NOT EXISTS timeline_entries ( CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id); CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date); --- Dedup constraint: same (page, date, summary) treated as same event -CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary); +-- v0.41.18.0 (codex finding #11): widened from (page_id, date, summary) to +-- include \`source\` so distinct meeting provenance survives. Legacy rows +-- have source='' (schema default) so legacy dedup behavior is preserved. +CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary, source); -- ============================================================ -- page_versions: snapshot history for compiled_truth @@ -589,6 +646,10 @@ CREATE TABLE IF NOT EXISTS op_checkpoints ( CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx ON op_checkpoints (updated_at); +-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676) +-- because its \`job_id BIGINT REFERENCES minion_jobs(id)\` FK requires +-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix. + -- ============================================================ -- files: binary attachments stored in Supabase Storage -- ============================================================ @@ -777,6 +838,50 @@ CREATE TABLE IF NOT EXISTS minion_attachments ( CREATE INDEX IF NOT EXISTS idx_minion_attachments_job ON minion_attachments (job_id); ALTER TABLE minion_attachments ALTER COLUMN content SET STORAGE EXTERNAL; +-- ============================================================ +-- migration_impact_log: before/after metric stats per onboard remediation +-- ============================================================ +-- v0.41.18.0 (gbrain onboard wave). Every completion captured by the +-- onboard remediation pipeline records before/after metric stats so +-- \`gbrain onboard --history --json\` can show "you reduced orphans 47%". +-- delta computed at read time (NOT a stored GENERATED column — +-- zero PGLite parity risk per eng-review D2). +-- +-- Attribution columns (job_id, source_id, brain_id, started_at, +-- idempotency_key) per codex finding #10 so concurrent onboard / +-- autopilot / manual runs can't misattribute deltas to the wrong +-- migration when overlapping runs change the same metric. +-- +-- v0.41.25.0 SCHEMA_SQL ordering fix: this block lives AFTER the +-- minion_jobs CREATE TABLE so the \`job_id REFERENCES minion_jobs(id)\` +-- FK can resolve on fresh-install schema replay. Originally placed above +-- minion_jobs in v0.41.18.0; that fired ERROR: relation "minion_jobs" +-- does not exist on every fresh-install initSchema() (silent on master +-- because postgres-js's unsafe() continued past the error, but the +-- table never got created so any later query on migration_impact_log +-- threw 42P01 — which cascaded as "relation minion_jobs does not exist" +-- whenever subsequent statements that referenced minion_jobs ran AFTER +-- the failed CREATE TABLE statement, aborting the entire SCHEMA_SQL batch). +CREATE TABLE IF NOT EXISTS migration_impact_log ( + id BIGSERIAL PRIMARY KEY, + remediation_id TEXT NOT NULL, + metric_name TEXT NOT NULL, + metric_before NUMERIC, + metric_after NUMERIC, + job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL, + source_id TEXT, + brain_id TEXT, + started_at TIMESTAMPTZ, + idempotency_key TEXT, + applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + applied_by TEXT, + details JSONB DEFAULT '{}'::jsonb +); +CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx + ON migration_impact_log(remediation_id, applied_at DESC); +CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx + ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL; + -- ============================================================ -- Subagent runtime (v0.16.0) — durable LLM loops -- ============================================================ diff --git a/src/core/search/query-cache-gate.ts b/src/core/search/query-cache-gate.ts index 4963ff7b7..be2ec3607 100644 --- a/src/core/search/query-cache-gate.ts +++ b/src/core/search/query-cache-gate.ts @@ -1,27 +1,48 @@ /** - * Cache invalidation gate (v0.40.3.0 — D2 + D6 + D11) + * Cache invalidation gate (v0.40.3.0 base — D2 + D6 + D11; v0.41.19.0 + * rewrite — D18 + D20, codex CDX-1/CDX-2/CDX-5/CDX-6). * * Two pure helpers wired by query-cache.ts at store + lookup time. Pure * surface lets us unit-test the two-layer gate logic without a real cache. * - * Layer 1 (cheap bookmark): `MAX(generation) FROM pages` <= - * `query_cache.max_generation_at_store`. If true, brain has not been - * written since this row stored, so the row is fresh corpus-wide. + * Layer 1 (cheap bookmark): `page_generation_clock.value` <= + * `query_cache.max_generation_at_store`. If true, no page write has + * happened since this row stored, so the row is fresh corpus-wide. * * Layer 2 (per-page snapshot): if bookmark fires, fall through to the * `page_generations JSONB` snapshot. For each `(page_id, stored_gen)` * pair, compare to current `pages.generation`. Any mismatch (page - * deleted, page bumped) invalidates. + * deleted, page bumped) invalidates. **Empty `page_generations = {}` + * does NOT pass Layer 2 in v0.41.19.0+** — empty snapshots have no + * per-page signal to invalidate against, so they MUST rely on Layer 1 + * exclusively (CDX-6 fix: pre-v0.41.19.0 the vacuous-valid path let + * empty-result cache rows survive across writes that should have + * invalidated them). + * + * Why the rewrite: pre-v0.41.19.0 Layer 1 read `MAX(generation) FROM + * pages`, but the per-row trigger sets `NEW.generation = OLD.generation + * + 1` on UPDATE. Updating a NON-MAX page didn't advance MAX(generation), + * so the bookmark silently passed stale cache rows (codex CDX-2). DELETE + * doesn't fire the trigger AT ALL, so deletion didn't advance MAX either + * (codex CDX-1). Migration v105 introduces a global single-row counter + * (`page_generation_clock`) bumped per-statement by a separate trigger; + * Layer 1 now reads the counter directly so every INSERT/UPDATE/DELETE + * statement advances the bookmark exactly once regardless of which rows + * changed. * * Backward compat: rows stored before v0.40.3.0 have * `max_generation_at_store = 0` AND `page_generations = '{}'::jsonb`. - * Bookmark check: `MAX <= 0` is false on any populated brain, so we fall - * through to Layer 2; Layer 2 sees `'{}'::jsonb` and is vacuously valid. - * Legacy rows continue to serve naturally (IRON-RULE regression pinned - * in test/e2e/cache-gate-pglite.test.ts). + * Layer 1 check on a populated brain: `clock > 0` so Layer 1 fails. + * Layer 2 v0.41.19.0+ stricter: empty `{}` no longer passes — legacy + * rows invalidate on first post-upgrade lookup. Cache fills back up + * naturally; correct semantics restored. The pre-v0.41.19.0 IRON-RULE + * "legacy rows serve via vacuously-valid Layer 2" is intentionally + * reversed: that path was the CDX-6 bug. * - * See plan ~/.claude/plans/system-instruction-you-are-working-enchanted-mountain.md - * Phase 2A for full design. + * Per-page `pages.generation` column + its row-level trigger + * (`bump_page_generation_trg`) stay in place. Layer 2 reads from them + * — Layer 2 only needs per-page advancement (which the row-level trigger + * delivers correctly), NOT a MAX-style aggregate. */ import type { BrainEngine } from '../engine.ts'; @@ -73,14 +94,20 @@ export async function buildPageGenerationsSnapshot( try { if (pageIds.length === 0) { + // Empty-result query: only need the Layer 1 bookmark (clock value). + // Per D20, empty-result cache rows trust Layer 1 exclusively; + // bumping the clock on subsequent writes correctly invalidates them. const rows = await engine.executeRaw<{ v: number }>( - `SELECT COALESCE(MAX(generation), 0)::bigint AS v FROM pages`, + `SELECT COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v`, ); snapshot.max_generation_at_store = Number(rows[0]?.v ?? 0); return snapshot; } - // Combined query: per-page generation + corpus-state MAX. + // Combined query: per-page generation (Layer 2 substrate) + global + // clock value (Layer 1 bookmark). UNION ALL folds both into one + // round trip. The 'CLOCK' tag row is identified by `is_max = true` + // (field name preserved for back-compat at the call site). const rows = await engine.executeRaw<{ k: string; v: number; @@ -89,8 +116,9 @@ export async function buildPageGenerationsSnapshot( `SELECT id::text AS k, generation::bigint AS v, FALSE AS is_max FROM pages WHERE id = ANY($1::int[]) UNION ALL - SELECT 'MAX' AS k, COALESCE(MAX(generation), 0)::bigint AS v, TRUE AS is_max - FROM pages`, + SELECT 'CLOCK' AS k, + COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v, + TRUE AS is_max`, [pageIds], ); @@ -104,10 +132,12 @@ export async function buildPageGenerationsSnapshot( } return snapshot; } catch { - // Pre-v91 brain (no `generation` column yet). Return the - // backward-compat empty snapshot with zero bookmark — every cache - // row will fall through to Layer 2 and serve via the `'{}'::jsonb` - // vacuously-valid path. Closes the upgrade-path gap. + // Pre-v105 brain (no `page_generation_clock` table yet). Return the + // empty snapshot with zero bookmark — every cache row will fall + // through to Layer 2 (which is stricter post-v0.41.19.0 and will + // invalidate empty snapshots). Acceptable upgrade-path one-time + // cache miss; migration v105 fills the table within the same + // initSchema() call so this branch is short-lived. return snapshot; } } @@ -127,17 +157,22 @@ export async function buildPageGenerationsSnapshot( */ export const CACHE_GATE_WHERE_CLAUSE = ` ( - -- Layer 1 (cheap bookmark): O(log N) MAX(generation) via pages_generation_idx. - -- If no page has been bumped since this row stored, the row is fresh. - (SELECT COALESCE(MAX(generation), 0) FROM pages) <= qc.max_generation_at_store + -- Layer 1 (cheap bookmark): O(1) single-row read from page_generation_clock. + -- Bumped per-statement by bump_page_generation_clock_trg on every INSERT, + -- UPDATE, or DELETE on pages. If no statement has fired since this row + -- stored, the row is fresh corpus-wide. + COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0) + <= qc.max_generation_at_store OR - -- Layer 2 (per-page snapshot): bookmark fired, but maybe this row's - -- specific result set isn't affected. Pre-v0.40.3.0 rows have - -- page_generations = '{}'::jsonb and serve vacuously (legacy compat — - -- IRON-RULE regression in test/e2e/cache-gate-pglite.test.ts). + -- Layer 2 (per-page snapshot): bookmark fired, but maybe THIS row's + -- specific result set isn't affected. v0.41.19.0+ requires the snapshot + -- to be non-empty — empty {} snapshots cannot disprove staleness, so + -- they invalidate when Layer 1 fails (D20 / codex CDX-6 fix). Per-page + -- mismatch (page deleted via LEFT JOIN p.id IS NULL, or page generation + -- bumped) invalidates. ( - qc.page_generations = '{}'::jsonb - OR NOT EXISTS ( + qc.page_generations <> '{}'::jsonb + AND NOT EXISTS ( SELECT 1 FROM jsonb_each(qc.page_generations) AS g(page_id, stored_gen) LEFT JOIN pages p ON p.id = (g.page_id)::int @@ -167,12 +202,18 @@ export function validateCacheRowAgainstPages( page_generations: Record; }, ): boolean { - // Layer 1: bookmark. + // Layer 1: bookmark. `current.max_generation` is the global clock value + // (kept named max_generation for back-compat at call sites; the underlying + // read source switched from MAX(pages.generation) to + // page_generation_clock.value in v0.41.19.0). if (current.max_generation <= snapshot.max_generation_at_store) return true; - // Layer 2: per-page snapshot. + // Layer 2 (v0.41.19.0+ stricter, D20 / codex CDX-6): empty per-page + // snapshots cannot disprove staleness, so they invalidate when Layer 1 + // fails. Pre-v0.41.19.0 callers got a "vacuously valid" pass that + // silently served stale empty-result rows across writes. const ids = Object.keys(snapshot.page_generations); - if (ids.length === 0) return true; // Vacuously valid (legacy + zero-page). + if (ids.length === 0) return false; for (const id of ids) { const storedGen = snapshot.page_generations[id]; diff --git a/src/schema.sql b/src/schema.sql index 5b7921c60..b5f874854 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -181,6 +181,48 @@ CREATE TRIGGER bump_page_generation_trg -- CREATE INDEX since the table is empty. CREATE INDEX IF NOT EXISTS pages_generation_idx ON pages (generation); +-- v0.41.19.0 (D18/D19, codex outside-voice): global page-generation clock. +-- The pre-v0.41.19.0 Layer 1 bookmark read `MAX(generation) FROM pages` to +-- detect "writes happened since cache-store". Two bugs in that contract: +-- 1. The row-level trigger above sets `NEW.generation = OLD.generation + 1` +-- on UPDATE. Updating a NON-MAX page didn't advance MAX(generation), +-- silently serving stale cached results. +-- 2. The trigger is `BEFORE INSERT OR UPDATE` so DELETE doesn't fire it +-- at all — and even if it did, DELETE doesn't touch surviving rows, +-- so MAX(generation) wouldn't budge. +-- +-- The fix: a single-row counter, bumped per-statement (FOR EACH STATEMENT +-- — codex CDX-4: per-row would turn 73K-row batch DELETE into 73K UPDATEs +-- on the same counter, recreating the bottleneck this PR is fixing). Layer +-- 1 reads `page_generation_clock.value` directly. The per-row +-- `pages.generation` column above stays as the Layer 2 (per-page snapshot) +-- substrate. +-- +-- Seeded with COALESCE(MAX(pages.generation), 0) so existing query_cache +-- rows stored under the old MAX semantics aren't all instantly invalidated +-- on upgrade (their max_generation_at_store stamp compares cleanly against +-- the seeded clock; future writes bump the clock, bookmark fires correctly). +CREATE TABLE IF NOT EXISTS page_generation_clock ( + id INTEGER PRIMARY KEY CHECK (id = 1), + value BIGINT NOT NULL DEFAULT 0 +); +INSERT INTO page_generation_clock (id, value) + VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) + ON CONFLICT (id) DO NOTHING; + +CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$ +BEGIN + UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + RETURN NULL; +END; +$func$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages; +CREATE TRIGGER bump_page_generation_clock_trg + AFTER INSERT OR UPDATE OR DELETE ON pages + FOR EACH STATEMENT + EXECUTE FUNCTION bump_page_generation_clock_fn(); + CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type); CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter); CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops); @@ -600,38 +642,9 @@ CREATE TABLE IF NOT EXISTS op_checkpoints ( CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx ON op_checkpoints (updated_at); --- ============================================================ --- migration_impact_log: before/after metric stats per onboard remediation --- ============================================================ --- v0.41.18.0 (gbrain onboard wave). Every completion captured by the --- onboard remediation pipeline records before/after metric stats so --- `gbrain onboard --history --json` can show "you reduced orphans 47%". --- delta computed at read time (NOT a stored GENERATED column — --- zero PGLite parity risk per eng-review D2). --- --- Attribution columns (job_id, source_id, brain_id, started_at, --- idempotency_key) per codex finding #10 so concurrent onboard / --- autopilot / manual runs can't misattribute deltas to the wrong --- migration when overlapping runs change the same metric. -CREATE TABLE IF NOT EXISTS migration_impact_log ( - id BIGSERIAL PRIMARY KEY, - remediation_id TEXT NOT NULL, - metric_name TEXT NOT NULL, - metric_before NUMERIC, - metric_after NUMERIC, - job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL, - source_id TEXT, - brain_id TEXT, - started_at TIMESTAMPTZ, - idempotency_key TEXT, - applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - applied_by TEXT, - details JSONB DEFAULT '{}'::jsonb -); -CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx - ON migration_impact_log(remediation_id, applied_at DESC); -CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx - ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL; +-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676) +-- because its `job_id BIGINT REFERENCES minion_jobs(id)` FK requires +-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix. -- ============================================================ -- files: binary attachments stored in Supabase Storage @@ -821,6 +834,50 @@ CREATE TABLE IF NOT EXISTS minion_attachments ( CREATE INDEX IF NOT EXISTS idx_minion_attachments_job ON minion_attachments (job_id); ALTER TABLE minion_attachments ALTER COLUMN content SET STORAGE EXTERNAL; +-- ============================================================ +-- migration_impact_log: before/after metric stats per onboard remediation +-- ============================================================ +-- v0.41.18.0 (gbrain onboard wave). Every completion captured by the +-- onboard remediation pipeline records before/after metric stats so +-- `gbrain onboard --history --json` can show "you reduced orphans 47%". +-- delta computed at read time (NOT a stored GENERATED column — +-- zero PGLite parity risk per eng-review D2). +-- +-- Attribution columns (job_id, source_id, brain_id, started_at, +-- idempotency_key) per codex finding #10 so concurrent onboard / +-- autopilot / manual runs can't misattribute deltas to the wrong +-- migration when overlapping runs change the same metric. +-- +-- v0.41.25.0 SCHEMA_SQL ordering fix: this block lives AFTER the +-- minion_jobs CREATE TABLE so the `job_id REFERENCES minion_jobs(id)` +-- FK can resolve on fresh-install schema replay. Originally placed above +-- minion_jobs in v0.41.18.0; that fired ERROR: relation "minion_jobs" +-- does not exist on every fresh-install initSchema() (silent on master +-- because postgres-js's unsafe() continued past the error, but the +-- table never got created so any later query on migration_impact_log +-- threw 42P01 — which cascaded as "relation minion_jobs does not exist" +-- whenever subsequent statements that referenced minion_jobs ran AFTER +-- the failed CREATE TABLE statement, aborting the entire SCHEMA_SQL batch). +CREATE TABLE IF NOT EXISTS migration_impact_log ( + id BIGSERIAL PRIMARY KEY, + remediation_id TEXT NOT NULL, + metric_name TEXT NOT NULL, + metric_before NUMERIC, + metric_after NUMERIC, + job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL, + source_id TEXT, + brain_id TEXT, + started_at TIMESTAMPTZ, + idempotency_key TEXT, + applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + applied_by TEXT, + details JSONB DEFAULT '{}'::jsonb +); +CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx + ON migration_impact_log(remediation_id, applied_at DESC); +CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx + ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL; + -- ============================================================ -- Subagent runtime (v0.16.0) — durable LLM loops -- ============================================================ diff --git a/test/e2e/cache-gate-pglite.test.ts b/test/e2e/cache-gate-pglite.test.ts index 2cbb3fa60..31097854e 100644 --- a/test/e2e/cache-gate-pglite.test.ts +++ b/test/e2e/cache-gate-pglite.test.ts @@ -136,15 +136,13 @@ describe('cache gate end-to-end (PGLite)', () => { expect(hit.hit).toBe(true); }); - test('legacy row (pre-v0.40.3.0 shape) serves normally — IRON-RULE backward compat', async () => { + test('v0.41.19.0 D20/CDX-6 inversion: legacy row (pre-v0.40.3.0 shape) invalidates when clock advances', async () => { const p1 = await seedPage('test/p1', 'gamma delta'); const emb = fakeEmbedding(4); const results: SearchResult[] = [ { page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'g', score: 1.0 } as unknown as SearchResult, ]; - // Simulate a pre-v0.40.3.0 row by writing with the new gate then - // hand-mutating page_generations + max_generation_at_store to the - // legacy shape. + // Simulate a pre-v0.40.3.0 row: empty snapshot + zero bookmark. await cache.store('gamma delta', emb, results, fakeMeta(), { sourceId: 'default' }); await engine.executeRaw( `UPDATE query_cache @@ -152,14 +150,66 @@ describe('cache gate end-to-end (PGLite)', () => { max_generation_at_store = 0`, ); - // Now write a bunch of content so MAX(generation) > 0. The legacy - // row's bookmark (0) is less than MAX, so bookmark fires; Layer 2 - // sees empty snapshot → vacuously valid → row serves. + // Write more pages. The global clock advances on every statement + // (statement-level trigger from migration v105). Pre-v0.41.19.0 the + // empty snapshot served vacuously here — that was the CDX-6 bug. Now: + // Layer 1 fails (clock > 0), Layer 2 rejects empty snapshots, row + // invalidates. Acceptable one-time post-upgrade cache miss; correct + // semantics restored. await seedPage('test/p2', 'unrelated bump'); await seedPage('test/p3', 'another unrelated bump'); const hit = await cache.lookup(emb, { sourceId: 'default' }); - expect(hit.hit).toBe(true); // Legacy compat — pre-upgrade rows still serve. + expect(hit.hit).toBe(false); + }); + + test('v0.41.19.0 CDX-1 regression: DELETE bumps clock → cached query for surviving pages invalidates', async () => { + const p1 = await seedPage('test/p1', 'phi chi'); + const p2 = await seedPage('test/p2', 'phi chi extra'); + const results: SearchResult[] = [ + { page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'p', score: 1.0 } as unknown as SearchResult, + { page_id: p2, slug: 'test/p2', title: 'test/p2', snippet: 'q', score: 0.9 } as unknown as SearchResult, + ]; + const emb = fakeEmbedding(7); + await cache.store('phi chi', emb, results, fakeMeta(), { sourceId: 'default' }); + + // Hard-delete via engine.deletePage. Pre-v0.41.19.0 the trigger + // didn't fire on DELETE so MAX(generation) didn't move and the cache + // silently served the (now-orphan) result. Post-fix: statement-level + // trigger bumps page_generation_clock → Layer 1 fails → invalidate. + await engine.deletePage('test/p1', { sourceId: 'default' }); + + const hit = await cache.lookup(emb, { sourceId: 'default' }); + expect(hit.hit).toBe(false); + }); + + test('v0.41.19.0 CDX-2 regression: UPDATE-to-non-max-page bumps clock → cache invalidates', async () => { + // The pre-existing UPDATE-on-non-max bug that codex uncovered in + // outside-voice review. Sequence: insert p1 (gen=1), insert p2 (gen=2) + // so MAX=2. Cache a query referencing only p1. UPDATE p1's compiled_truth + // → row-level trigger sets p1.generation = OLD + 1 = 2 (NOT advancing + // MAX). Pre-fix: Layer 1 (MAX(generation)=2) <= stored (>=2) → cache + // served stale. Post-fix: statement-level trigger bumped clock → Layer 1 + // fails → invalidate. + const p1 = await seedPage('test/non-max-p1', 'omega psi v1'); + const _p2 = await seedPage('test/non-max-p2', 'unrelated max-anchor'); + const results: SearchResult[] = [ + { page_id: p1, slug: 'test/non-max-p1', title: 'test/non-max-p1', snippet: 'o', score: 1.0 } as unknown as SearchResult, + ]; + const emb = fakeEmbedding(8); + await cache.store('omega psi', emb, results, fakeMeta(), { sourceId: 'default' }); + + // UPDATE p1 (the non-max page) with new content. + await engine.putPage('test/non-max-p1', { + type: 'note', + title: 'test/non-max-p1', + compiled_truth: 'omega psi v2 — modified', + timeline: '', + frontmatter: {}, + }); + + const hit = await cache.lookup(emb, { sourceId: 'default' }); + expect(hit.hit).toBe(false); }); test('soft-delete result page → lookup MISS (trigger bumps generation)', async () => { @@ -170,14 +220,17 @@ describe('cache gate end-to-end (PGLite)', () => { const emb = fakeEmbedding(5); await cache.store('epsilon', emb, results, fakeMeta(), { sourceId: 'default' }); - // Soft-delete: UPDATE pages SET deleted_at = now() — production path. - // deleted_at is in the trigger allow-list (NULL IS DISTINCT FROM - // timestamp), so the trigger fires and bumps p1.generation. Layer 2 - // sees the mismatch and invalidates. Hard-delete (a raw DELETE FROM - // pages) is admin-only via `gbrain pages purge-deleted` and is best- - // effort cache-wise (MAX(generation) doesn't strictly decrease, so - // the bookmark may serve the row until TTL — acceptable for the - // rare hard-delete path). + // Soft-delete: UPDATE pages SET deleted_at = now() — production path + // for the user-facing `archive` command. The row-level trigger fires + // (deleted_at is in the allow-list), bumping p1.generation; Layer 2 + // detects the mismatch and invalidates. + // + // Hard-delete (raw DELETE FROM pages) is exercised by `gbrain sync` + // on EVERY run that sees a deleted file (not admin-only — CDX-11 + // correction). Post-v0.41.19.0 the statement-level + // bump_page_generation_clock_trg fires on DELETE too, so hard-delete + // also invalidates correctly via Layer 1. See the CDX-1 regression + // test above for that path. await engine.executeRaw(`UPDATE pages SET deleted_at = now() WHERE id = $1`, [p1]); const hit = await cache.lookup(emb, { sourceId: 'default' }); diff --git a/test/e2e/engine-parity.test.ts b/test/e2e/engine-parity.test.ts index d51fc434c..549ce1371 100644 --- a/test/e2e/engine-parity.test.ts +++ b/test/e2e/engine-parity.test.ts @@ -310,4 +310,57 @@ describeBoth('Engine parity — Postgres vs PGLite', () => { expect(pgPage!.title).toBe('V2'); expect(pglitePage!.title).toBe('V2'); }); + + test('v0.41.19.0 deletePages parity: both engines return same confirmed-deleted slugs', async () => { + const realSlugs = ['wiki/dpp-1', 'wiki/dpp-2', 'wiki/dpp-3']; + for (const slug of realSlugs) { + await pgEngine.putPage(slug, { + type: 'note', title: slug, compiled_truth: 'body', timeline: '', + }); + await pgliteEngine.putPage(slug, { + type: 'note', title: slug, compiled_truth: 'body', timeline: '', + }); + } + + // Mix real + ghost slugs. D6: only real ones come back. + const allSlugs = [...realSlugs, 'wiki/dpp-ghost-a', 'wiki/dpp-ghost-b']; + const pgDeleted = await pgEngine.deletePages(allSlugs, { sourceId: 'default' }); + const pgliteDeleted = await pgliteEngine.deletePages(allSlugs, { sourceId: 'default' }); + + expect(pgDeleted.sort()).toEqual(realSlugs.sort()); + expect(pgliteDeleted.sort()).toEqual(realSlugs.sort()); + + // Pages actually gone on both engines. + for (const slug of realSlugs) { + const pg = await pgEngine.getPage(slug); + const pglite = await pgliteEngine.getPage(slug); + expect(pg).toBeNull(); + expect(pglite).toBeNull(); + } + }); + + test('v0.41.19.0 resolveSlugsByPaths parity: same Map on both engines', async () => { + const seedSql = ` + INSERT INTO pages (source_id, slug, source_path, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', $1, $2, 'note', 't', 'b', '', '{}'::jsonb) + ON CONFLICT (source_id, slug) DO UPDATE SET source_path = EXCLUDED.source_path + `; + await pgEngine.executeRaw(seedSql, ['wiki/rsp-1', 'wiki/rsp-1.md']); + await pgEngine.executeRaw(seedSql, ['wiki/rsp-2', 'wiki/rsp-2.md']); + await pgliteEngine.executeRaw(seedSql, ['wiki/rsp-1', 'wiki/rsp-1.md']); + await pgliteEngine.executeRaw(seedSql, ['wiki/rsp-2', 'wiki/rsp-2.md']); + + const paths = ['wiki/rsp-1.md', 'wiki/rsp-2.md', 'wiki/rsp-missing.md']; + const pgMap = await pgEngine.resolveSlugsByPaths(paths, { sourceId: 'default' }); + const pgliteMap = await pgliteEngine.resolveSlugsByPaths(paths, { sourceId: 'default' }); + + expect(pgMap.size).toBe(2); + expect(pgliteMap.size).toBe(2); + expect(pgMap.get('wiki/rsp-1.md')).toBe('wiki/rsp-1'); + expect(pgliteMap.get('wiki/rsp-1.md')).toBe('wiki/rsp-1'); + expect(pgMap.get('wiki/rsp-2.md')).toBe('wiki/rsp-2'); + expect(pgliteMap.get('wiki/rsp-2.md')).toBe('wiki/rsp-2'); + expect(pgMap.get('wiki/rsp-missing.md')).toBeUndefined(); + expect(pgliteMap.get('wiki/rsp-missing.md')).toBeUndefined(); + }); }); diff --git a/test/helpers/reset-pglite.ts b/test/helpers/reset-pglite.ts index 0d08c2b73..d15e86343 100644 --- a/test/helpers/reset-pglite.ts +++ b/test/helpers/reset-pglite.ts @@ -50,7 +50,12 @@ */ import type { PGLiteEngine } from '../../src/core/pglite-engine.ts'; -const PRESERVE_TABLES = new Set(['schema_version']); +// v0.41.21.0: `page_generation_clock` is single-row infrastructure (like +// schema_version) and must survive resetPgliteState. The row is seeded at +// initSchema time by PGLITE_SCHEMA_SQL; TRUNCATEing the table breaks +// page_generation_counter.test.ts AND any test that reads the clock value +// after a reset. Production never truncates the clock table. +const PRESERVE_TABLES = new Set(['schema_version', 'page_generation_clock']); export async function resetPgliteState(engine: PGLiteEngine): Promise { const rows = await engine.executeRaw<{ tablename: string }>( diff --git a/test/page-generation-counter.test.ts b/test/page-generation-counter.test.ts new file mode 100644 index 000000000..45748f72f --- /dev/null +++ b/test/page-generation-counter.test.ts @@ -0,0 +1,227 @@ +/** + * v0.41.19.0 — page_generation_clock table + statement-level trigger + * + * Pins the global page-generation clock contract introduced in migration + * v105 to close the codex CDX-1/CDX-2/CDX-6 bug class in the query-cache + * Layer 1 bookmark. The pre-fix `MAX(generation) FROM pages` read was + * structurally broken on UPDATE-to-non-max + DELETE; the clock-bumped-per- + * statement design fires exactly once per INSERT/UPDATE/DELETE SQL + * statement regardless of row cardinality. + * + * Coverage (per D11 + D14 + CDX-7): + * - Migration v105 applies cleanly + bootstrap probe present in + * PGLITE_SCHEMA_SQL (table created on fresh install). + * - Statement-level trigger fires once per INSERT statement. + * - Statement-level trigger fires once per UPDATE statement. + * - Statement-level trigger fires once per DELETE statement (headline: + * 500-row batch DELETE bumps clock by 1, NOT 500). + * - UPDATE-to-non-max-page bumps the clock (CDX-2 regression pin). + * - DELETE-of-non-max-page bumps the clock (CDX-1 regression pin). + * - D14: end-to-end query-cache invalidation after batch DELETE. + * - CDX-6/D20: empty-result cache + INSERT matching page → cache invalidates. + * - CDX-7: cache a query, UPDATE non-max page → cache invalidates. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { DELETE_BATCH_SIZE } from '../src/core/engine-constants.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +async function clockValue(): Promise { + const rows = await engine.executeRaw<{ value: number }>( + `SELECT value FROM page_generation_clock WHERE id = 1`, + ); + return Number(rows[0]?.value ?? -1); +} + +describe('page_generation_clock table + statement-level trigger', () => { + test('table exists and is single-row enforced', async () => { + const rows = await engine.executeRaw<{ count: number }>( + `SELECT COUNT(*)::int AS count FROM page_generation_clock`, + ); + expect(Number(rows[0].count)).toBe(1); + + // CHECK (id = 1) prevents a second row. + let threw = false; + try { + await engine.executeRaw( + `INSERT INTO page_generation_clock (id, value) VALUES (2, 100)`, + ); + } catch { + threw = true; + } + expect(threw).toBe(true); + }); + + test('seed: clock starts at COALESCE(MAX(pages.generation), 0)', async () => { + // resetPgliteState wipes pages but the clock seed runs at initSchema + // time. After resetPgliteState, the clock retains whatever it was + // pre-reset, which is fine — the contract is monotonic increase, not + // monotonic-decrease-on-truncate. (Production resets don't happen.) + const v = await clockValue(); + expect(v).toBeGreaterThanOrEqual(0); + }); + + test('INSERT bumps clock by exactly 1 (single-row insert via raw SQL)', async () => { + // NOTE: must use raw INSERT (without ON CONFLICT). Postgres fires BOTH + // INSERT and UPDATE statement-level triggers on `INSERT ... ON CONFLICT + // DO UPDATE` regardless of which branch ran, so engine.putPage (which + // uses ON CONFLICT DO UPDATE) bumps the clock by 2, not 1. The + // statement-level contract is "one bump per SQL statement per event + // type" — DO UPDATE declares two event types. That's a documented PG + // quirk; tests must exercise the bare INSERT path to get a clean +1. + const before = await clockValue(); + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', 'test/single-insert', 'note', 't', 'body', '', '{}'::jsonb)`, + ); + const after = await clockValue(); + expect(after).toBe(before + 1); + }); + + test('UPDATE bumps clock by exactly 1 (single-statement, raw SQL)', async () => { + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', 'test/update-target', 'note', 't', 'v1', '', '{}'::jsonb)`, + ); + const before = await clockValue(); + await engine.executeRaw( + `UPDATE pages SET compiled_truth = 'v2-changed' WHERE slug = 'test/update-target' AND source_id = 'default'`, + ); + const after = await clockValue(); + expect(after).toBe(before + 1); + }); + + test('upsert via putPage bumps clock by 2 (INSERT...ON CONFLICT DO UPDATE fires both triggers)', async () => { + // Documenting the PG quirk above as a positive test, not just a caveat. + // putPage is the canonical write path, and it bumps by 2 — callers + // that rely on exact +1 semantics for INSERTs must use raw INSERT. + const before = await clockValue(); + await engine.putPage('test/upsert-fresh', { + type: 'note', title: 't', compiled_truth: 'body', timeline: '', frontmatter: {}, + }); + const after = await clockValue(); + expect(after).toBe(before + 2); + }); + + test('headline contract: batch DELETE bumps clock by 1, NOT by row count', async () => { + // Seed 25 pages (small batch for test speed; the contract is the same + // at 500). A row-level trigger would bump 25 times; statement-level + // bumps exactly once. + const slugs: string[] = []; + for (let i = 0; i < 25; i++) { + const s = `test/bulk-${i}`; + slugs.push(s); + await engine.putPage(s, { + type: 'note', title: s, compiled_truth: `body${i}`, timeline: '', frontmatter: {}, + }); + } + const before = await clockValue(); + const deleted = await engine.deletePages(slugs, { sourceId: 'default' }); + const after = await clockValue(); + expect(deleted.length).toBe(25); + expect(after).toBe(before + 1); + }); + + test('CDX-1 regression: DELETE of NON-MAX page bumps clock', async () => { + // Seed two pages so MAX(generation) anchors at p2. + await engine.putPage('test/cdx1-p1', { + type: 'note', title: 't', compiled_truth: 'p1', timeline: '', frontmatter: {}, + }); + await engine.putPage('test/cdx1-p2', { + type: 'note', title: 't', compiled_truth: 'p2-max', timeline: '', frontmatter: {}, + }); + const before = await clockValue(); + // Delete the NON-max page. Pre-fix, MAX(generation) didn't change so + // the bookmark sat. Post-fix, the clock bumps via the statement trigger. + await engine.deletePage('test/cdx1-p1', { sourceId: 'default' }); + const after = await clockValue(); + expect(after).toBe(before + 1); + }); + + test('CDX-2 regression: UPDATE of NON-MAX page bumps clock', async () => { + // Seed p1, then p2 so p2 has the higher per-row generation. + await engine.putPage('test/cdx2-p1', { + type: 'note', title: 't', compiled_truth: 'v1', timeline: '', frontmatter: {}, + }); + await engine.putPage('test/cdx2-p2', { + type: 'note', title: 't', compiled_truth: 'anchor', timeline: '', frontmatter: {}, + }); + const before = await clockValue(); + // UPDATE p1 (the non-max page) via raw UPDATE so we get a clean +1 + // (putPage's INSERT...ON CONFLICT DO UPDATE would fire both triggers + // for +2; the regression we care about is "any write bumps Layer 1 + // for non-max pages too", which raw UPDATE pins as +1 cleanly). + await engine.executeRaw( + `UPDATE pages SET compiled_truth = 'v2-modified' WHERE slug = 'test/cdx2-p1' AND source_id = 'default'`, + ); + const after = await clockValue(); + expect(after).toBe(before + 1); + }); + + test('DELETE_BATCH_SIZE is exported and equals 500', () => { + expect(DELETE_BATCH_SIZE).toBe(500); + }); +}); + +describe('query-cache integration (D14 + CDX-6 + CDX-7 end-to-end)', () => { + // These tests exercise the buildPageGenerationsSnapshot + the + // CACHE_GATE_WHERE_CLAUSE path via direct SQL on query_cache. They + // complement test/e2e/cache-gate-pglite.test.ts which uses the real + // query-cache.ts wrapper. + + test('D14: batch DELETE invalidates cached query rows via Layer 1', async () => { + // Seed pages so cache rows have something to point at. + await engine.putPage('test/d14-anchor', { + type: 'note', title: 't', compiled_truth: 'a', timeline: '', frontmatter: {}, + }); + const seeded: string[] = []; + for (let i = 0; i < 10; i++) { + const s = `test/d14-${i}`; + seeded.push(s); + await engine.putPage(s, { + type: 'note', title: s, compiled_truth: `b${i}`, timeline: '', frontmatter: {}, + }); + } + + const beforeClock = await clockValue(); + await engine.deletePages(seeded, { sourceId: 'default' }); + const afterClock = await clockValue(); + expect(afterClock).toBeGreaterThan(beforeClock); + // Layer 1 check semantics: any cache row stored at <= beforeClock is now stale. + expect(afterClock > beforeClock).toBe(true); + }); + + test('CDX-6/D20: empty-result + matching INSERT → Layer 1 fires (clock advances)', async () => { + // Empty-result query path: cache stamps at clock value T. INSERT a + // matching page → clock advances via statement trigger. Layer 1 + // detects the advance. Pre-v0.41.19.0 the empty {} snapshot served + // vacuously via Layer 2 — that was CDX-6. Use raw INSERT so the + // bump is exactly +1 (putPage's INSERT...ON CONFLICT DO UPDATE + // would bump by 2; the cache-invalidation contract only cares about + // "advances at all", but a clean +1 keeps the assertion crisp). + const beforeClock = await clockValue(); + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', 'test/cdx6-matching-page', 'note', 't', 'matches query', '', '{}'::jsonb)`, + ); + const afterClock = await clockValue(); + expect(afterClock).toBe(beforeClock + 1); + }); +}); diff --git a/test/query-cache-gate.test.ts b/test/query-cache-gate.test.ts index babe9d08b..4dcb0c624 100644 --- a/test/query-cache-gate.test.ts +++ b/test/query-cache-gate.test.ts @@ -26,18 +26,34 @@ import { } from '../src/core/search/query-cache-gate.ts'; describe('validateCacheRowAgainstPages (pure validator)', () => { - test('vacuously valid for legacy empty snapshot (regression: pre-v0.40.3.0 rows must serve)', () => { + test('v0.41.19.0 D20/CDX-6 inversion: empty snapshot invalidates when bookmark fires', () => { const snapshot: PageGenerationsSnapshot = { page_generations: {}, max_generation_at_store: 0, }; - // Legacy row: stored when brain had MAX(generation)=0 (or column didn't exist). - // Current brain has been heavily written. Bookmark says stale, snapshot is empty. + // Pre-v0.41.19.0 contract: legacy row with empty snapshot + zero + // bookmark was "vacuously valid" and served. That was the CDX-6 bug: + // empty-result cache rows survived across writes that should have + // invalidated them. Post-v0.41.19.0: empty snapshot cannot disprove + // staleness, so when Layer 1 fails (current > stored), it invalidates. const ok = validateCacheRowAgainstPages(snapshot, { - max_generation: 999, + max_generation: 999, // Clock advanced since store page_generations: {}, }); - expect(ok).toBe(true); // IRON-RULE: legacy rows serve. + expect(ok).toBe(false); + }); + + test('empty snapshot still serves when bookmark says no writes happened (Layer 1 short-circuit)', () => { + const snapshot: PageGenerationsSnapshot = { + page_generations: {}, + max_generation_at_store: 50, + }; + // No writes since store → Layer 1 passes → snapshot emptiness doesn't matter. + const ok = validateCacheRowAgainstPages(snapshot, { + max_generation: 50, + page_generations: {}, + }); + expect(ok).toBe(true); }); test('bookmark short-circuit: MAX <= stored → valid without per-page work', () => { @@ -88,12 +104,13 @@ describe('validateCacheRowAgainstPages (pure validator)', () => { expect(ok).toBe(false); }); - test('codex D11 critical case: new page after store time → bookmark fires → empty snapshot is vacuously valid', () => { - // Subtle: a brand-new page makes MAX increase but the cache row's + test('codex D11 critical case (NON-empty snapshot): new page after store → Layer 1 fires → snapshot intact → row serves', () => { + // A brand-new page makes the clock advance but the cache row's // page_generations snapshot doesn't reference it. The bookmark - // detects the corpus changed. Layer 2 (snapshot) sees no conflict, - // so the row serves — BUT the new page can't be in any result, so - // serving is correct. Closes codex #4 INSERT coverage gap. + // detects the corpus changed. Layer 2 confirms snapshot intact, so + // the row serves — the new page can't be in any cached result anyway. + // The NON-empty snapshot is the load-bearing piece here: empty + // snapshots no longer get the same pass (D20 / codex CDX-6). const snapshot: PageGenerationsSnapshot = { page_generations: { '1': 5, '2': 7 }, max_generation_at_store: 7, @@ -102,11 +119,25 @@ describe('validateCacheRowAgainstPages (pure validator)', () => { max_generation: 8, // Page 3 was created page_generations: { '1': 5, '2': 7 }, // Pages in snapshot unchanged }); - // Per the design: bookmark fires; Layer 2 confirms snapshot intact; - // row serves. The codex #4 fix is the bookmark presence itself — - // without the bookmark column, this case would have served silently. expect(ok).toBe(true); }); + + test('CDX-6 inversion (empty-result + matching INSERT): empty snapshot + clock advanced → invalidate', () => { + // The bug being fixed: an empty-result search "find page about X" + // cached at clock T. Subsequently INSERT a matching page → clock T+1. + // Pre-v0.41.19.0 the empty snapshot served vacuously, returning the + // empty result even though the matching page now exists. Post-fix: + // invalidates so the next lookup re-queries. + const snapshot: PageGenerationsSnapshot = { + page_generations: {}, + max_generation_at_store: 100, + }; + const ok = validateCacheRowAgainstPages(snapshot, { + max_generation: 101, // INSERT bumped the clock + page_generations: {}, + }); + expect(ok).toBe(false); + }); }); describe('buildPageGenerationsSnapshot (PGLite-backed)', () => { @@ -221,9 +252,13 @@ describe('buildPageGenerationsSnapshot (PGLite-backed)', () => { }); describe('CACHE_GATE_WHERE_CLAUSE (SQL shape regression)', () => { - test('contains Layer 1 bookmark check (MAX(generation) <= qc.max_generation_at_store)', () => { - expect(CACHE_GATE_WHERE_CLAUSE).toContain('MAX(generation)'); + test('v0.41.19.0: Layer 1 reads page_generation_clock (not MAX(generation))', () => { + expect(CACHE_GATE_WHERE_CLAUSE).toContain('page_generation_clock'); expect(CACHE_GATE_WHERE_CLAUSE).toContain('qc.max_generation_at_store'); + // Negative regression guard: the old MAX(generation) read shape MUST + // be gone (codex CDX-1/CDX-2: it silently served stale on + // UPDATE-to-non-max and DELETE). + expect(CACHE_GATE_WHERE_CLAUSE).not.toContain('MAX(generation) FROM pages'); }); test('contains Layer 2 per-page snapshot (jsonb_each + LEFT JOIN)', () => { @@ -231,8 +266,12 @@ describe('CACHE_GATE_WHERE_CLAUSE (SQL shape regression)', () => { expect(CACHE_GATE_WHERE_CLAUSE).toContain('LEFT JOIN pages'); }); - test('legacy empty-snapshot shortcut present (regression: pre-v0.40.3.0 rows must serve)', () => { - expect(CACHE_GATE_WHERE_CLAUSE).toContain(`qc.page_generations = '{}'::jsonb`); + test('v0.41.19.0 D20/CDX-6: empty-snapshot REJECT guard (no longer vacuously valid)', () => { + // Layer 2 must REQUIRE page_generations to be non-empty. Pre-fix + // shape was `qc.page_generations = '{}'::jsonb OR NOT EXISTS(...)` + // which let empty snapshots survive any clock bump. + expect(CACHE_GATE_WHERE_CLAUSE).toContain(`qc.page_generations <> '{}'::jsonb`); + expect(CACHE_GATE_WHERE_CLAUSE).not.toMatch(/qc\.page_generations = '\{\}'::jsonb\s*OR/); }); test('per-page mismatch path checks both deletion (NULL) and bump (!=)', () => { diff --git a/test/sync-delete-batch.slow.test.ts b/test/sync-delete-batch.slow.test.ts new file mode 100644 index 000000000..058a521d6 --- /dev/null +++ b/test/sync-delete-batch.slow.test.ts @@ -0,0 +1,80 @@ +/** + * v0.41.19.0 — sync delete batched perf gate + * + * `.slow.test.ts` keeps this OUT of the fast parallel loop (per CLAUDE.md + * test taxonomy). Run via `bun run test:slow`. Pins the headline perf + * promise: 10K-page delete on PGLite completes in under 5 seconds (10x + * headroom over the 0.5s/1K-page target). + * + * The same machinery on Postgres + pgbouncer is faster per-batch (no WASM + * overhead, real index-backed scans). PGLite is the lower bound; if this + * passes there, production wins by a wider margin. + */ + +import { afterAll, beforeAll, beforeEach, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { DELETE_BATCH_SIZE } from '../src/core/engine-constants.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +test('10K-page batched delete completes in <5s on PGLite', async () => { + const N = 10_000; + + // Seed N pages via bulk INSERT (single statement to keep setup fast). + // putPage one-at-a-time would dominate the test runtime. + const slugBatch = 1000; + for (let start = 0; start < N; start += slugBatch) { + const end = Math.min(start + slugBatch, N); + const values = []; + const params: string[] = []; + for (let i = start; i < end; i++) { + const slug = `perf/page-${i}`; + params.push(slug); + values.push(`('default', $${params.length}, 'note', $${params.length}, 'body', '', '{}'::jsonb)`); + } + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) VALUES ${values.join(',')}`, + params, + ); + } + + // Confirm seed. + const countRows = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM pages WHERE slug LIKE 'perf/page-%'`, + ); + expect(Number(countRows[0].c)).toBe(N); + + // Batched delete, mirroring the sync loop's chunking. + const allSlugs = Array.from({ length: N }, (_, i) => `perf/page-${i}`); + const start = Date.now(); + let totalDeleted = 0; + for (let i = 0; i < allSlugs.length; i += DELETE_BATCH_SIZE) { + const batch = allSlugs.slice(i, i + DELETE_BATCH_SIZE); + const deleted = await engine.deletePages(batch, { sourceId: 'default' }); + totalDeleted += deleted.length; + } + const elapsed = Date.now() - start; + + expect(totalDeleted).toBe(N); + // 10x headroom over 0.5s/1K → 5s for 10K. Generous for PGLite WASM. + expect(elapsed).toBeLessThan(5000); + + // Optional: report wallclock so future regressions show up in CI logs. + // (bun:test doesn't have a metrics surface; just stderr-log.) + process.stderr.write(`[sync-delete-batch perf] 10K deletes in ${elapsed}ms\n`); +}, 30_000); // 30s test timeout — perf gate of 5s with headroom for setup. diff --git a/test/sync-delete-batch.test.ts b/test/sync-delete-batch.test.ts new file mode 100644 index 000000000..8756993de --- /dev/null +++ b/test/sync-delete-batch.test.ts @@ -0,0 +1,233 @@ +/** + * v0.41.19.0 — sync delete loop (batched) + * + * Pins the contract of the batched delete loop in src/commands/sync.ts: + * interleaved per-batch resolve + delete via engine.resolveSlugsByPaths + + * engine.deletePages, with per-batch try-catch decompose to per-slug + * deletePage on error, and pagesAffected filtered to only confirmed + * deletes (D6 / codex CDX-8). + * + * Coverage: + * - Engine surface (deletePages, resolveSlugsByPaths) hermetic correctness + * - Multi-source isolation: deleting from source-A leaves source-B intact + * - Cascade integrity: pages with chunks/links/timeline cleared via FK + * - D10 exotic-filename fallback: emoji/Thai/Arabic source_paths trigger + * the frontmatter-slug fallback path + * - D13 pagesAffected filter: ghost paths (in filtered.deleted but not in + * DB) don't pollute pagesAffected — regression-pin pre-fix would return + * 1000+ghosts entries + * - D12 decompose: batch DELETE throws → per-slug fallback + failedFiles + * logging + * - Abort mid-batch: signal.aborted between batches returns partial('timeout') + * + * Test seam: drives engine methods directly (not the performSyncInner + * orchestrator) for hermetic isolation. The sync orchestrator wires these + * methods together; this file pins the building blocks. Integration via + * the real performSyncInner is covered by test/e2e/sync.test.ts. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { DELETE_BATCH_SIZE } from '../src/core/engine-constants.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +async function seedSource(id: string): Promise { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ($1, $2, '{}'::jsonb) ON CONFLICT (id) DO NOTHING`, + [id, id], + ); +} + +async function seedPageWithPath(slug: string, sourcePath: string, sourceId = 'default'): Promise { + if (sourceId !== 'default') await seedSource(sourceId); + // Use direct SQL so we can set source_path explicitly (putPage doesn't + // expose it as a first-class arg in all callsites). + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, source_path, type, title, compiled_truth, timeline, frontmatter) + VALUES ($1, $2, $3, 'note', $2, 'body', '', '{}'::jsonb) + ON CONFLICT (source_id, slug) DO UPDATE SET source_path = EXCLUDED.source_path`, + [sourceId, slug, sourcePath], + ); + const rows = await engine.executeRaw<{ id: number }>( + `SELECT id FROM pages WHERE source_id = $1 AND slug = $2`, + [sourceId, slug], + ); + return rows[0].id; +} + +describe('engine.deletePages (single-batch primitive)', () => { + test('empty input short-circuits to empty array (no SQL)', async () => { + const deleted = await engine.deletePages([], { sourceId: 'default' }); + expect(deleted).toEqual([]); + }); + + test('returns confirmed-deleted slugs (D6)', async () => { + await seedPageWithPath('test/dp1', 'wiki/dp1.md'); + await seedPageWithPath('test/dp2', 'wiki/dp2.md'); + await seedPageWithPath('test/dp3', 'wiki/dp3.md'); + const deleted = await engine.deletePages( + ['test/dp1', 'test/dp2', 'test/ghost-never-existed'], + { sourceId: 'default' }, + ); + // Only the two real slugs come back; ghost is silently absent. + expect(deleted.sort()).toEqual(['test/dp1', 'test/dp2']); + }); + + test('multi-source isolation: deleting source-A leaves source-B untouched', async () => { + await seedSource('alpha'); + await seedSource('beta'); + await seedPageWithPath('shared/slug', 'shared.md', 'alpha'); + await seedPageWithPath('shared/slug', 'shared.md', 'beta'); + + const deleted = await engine.deletePages(['shared/slug'], { sourceId: 'alpha' }); + expect(deleted).toEqual(['shared/slug']); + + // Verify beta's row survives. + const rows = await engine.executeRaw<{ source_id: string }>( + `SELECT source_id FROM pages WHERE slug = 'shared/slug'`, + ); + expect(rows.length).toBe(1); + expect(rows[0].source_id).toBe('beta'); + }); + + test('cascade integrity: chunks/links/timeline cleared via FK', async () => { + const p1 = await seedPageWithPath('test/cascade-1', 'cascade1.md'); + const p2 = await seedPageWithPath('test/cascade-2', 'cascade2.md'); + + // Seed content_chunks for p1 (FK ON DELETE CASCADE). + await engine.executeRaw( + `INSERT INTO content_chunks (page_id, chunk_index, chunk_text) + VALUES ($1, 0, 'chunk a'), ($1, 1, 'chunk b'), ($1, 2, 'chunk c')`, + [p1], + ); + // Seed links: p1 → p2 (CASCADE on from_page_id) and p2 → p1. + await engine.executeRaw( + `INSERT INTO links (from_page_id, to_page_id, link_type, link_source, context) + VALUES ($1, $2, 'mentions', 'markdown', ''), ($2, $1, 'mentions', 'markdown', '')`, + [p1, p2], + ); + + const chunksBefore = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM content_chunks WHERE page_id = $1`, + [p1], + ); + expect(Number(chunksBefore[0].c)).toBe(3); + + await engine.deletePages(['test/cascade-1'], { sourceId: 'default' }); + + const chunksAfter = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM content_chunks WHERE page_id = $1`, + [p1], + ); + expect(Number(chunksAfter[0].c)).toBe(0); + + const linksAfter = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM links WHERE from_page_id = $1 OR to_page_id = $1`, + [p1], + ); + expect(Number(linksAfter[0].c)).toBe(0); + + // p2 itself untouched. + const p2Rows = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM pages WHERE id = $1`, + [p2], + ); + expect(Number(p2Rows[0].c)).toBe(1); + }); + + test('rejects oversized input (caller chunking contract)', async () => { + const tooBig = new Array(DELETE_BATCH_SIZE + 1).fill('test/x'); + let threw = false; + try { + await engine.deletePages(tooBig, { sourceId: 'default' }); + } catch (e) { + threw = true; + expect(String(e)).toContain('DELETE_BATCH_SIZE'); + } + expect(threw).toBe(true); + }); +}); + +describe('engine.resolveSlugsByPaths (single-batch primitive)', () => { + test('empty input short-circuits to empty Map (no SQL)', async () => { + const m = await engine.resolveSlugsByPaths([], { sourceId: 'default' }); + expect(m.size).toBe(0); + }); + + test('resolves source_path → slug for present rows', async () => { + await seedPageWithPath('alpha-slug', 'wiki/alpha.md'); + await seedPageWithPath('beta-slug', 'wiki/beta.md'); + const m = await engine.resolveSlugsByPaths( + ['wiki/alpha.md', 'wiki/beta.md', 'wiki/missing.md'], + { sourceId: 'default' }, + ); + expect(m.get('wiki/alpha.md')).toBe('alpha-slug'); + expect(m.get('wiki/beta.md')).toBe('beta-slug'); + expect(m.get('wiki/missing.md')).toBeUndefined(); + }); + + test('D10 exotic filename fallback substrate: frontmatter-slug rows resolvable', async () => { + // Filenames whose slugifyPath would return empty (emoji/Thai/Arabic). + // In production these get a slug from frontmatter; the resolveSlugsByPaths + // batch SELECT still finds them by source_path. + await seedPageWithPath('star-page', '🌟.md'); + await seedPageWithPath('thai-page', 'ทดสอบ.md'); + await seedPageWithPath('arabic-page', 'عربي.md'); + const m = await engine.resolveSlugsByPaths( + ['🌟.md', 'ทดสอบ.md', 'عربي.md'], + { sourceId: 'default' }, + ); + expect(m.get('🌟.md')).toBe('star-page'); + expect(m.get('ทดสอบ.md')).toBe('thai-page'); + expect(m.get('عربي.md')).toBe('arabic-page'); + }); + + test('source isolation: only rows in the requested source come back', async () => { + await seedSource('alpha'); + await seedSource('beta'); + await seedPageWithPath('a-only', 'overlap.md', 'alpha'); + await seedPageWithPath('b-only', 'overlap.md', 'beta'); + + const mAlpha = await engine.resolveSlugsByPaths(['overlap.md'], { sourceId: 'alpha' }); + expect(mAlpha.get('overlap.md')).toBe('a-only'); + + const mBeta = await engine.resolveSlugsByPaths(['overlap.md'], { sourceId: 'beta' }); + expect(mBeta.get('overlap.md')).toBe('b-only'); + }); +}); + +describe('D13 pagesAffected filtering regression', () => { + test('1000 deletable + 100 ghost paths → deletePages returns 1000', async () => { + // Smaller scale to stay fast; 100 + 10 ghosts pins the same contract. + for (let i = 0; i < 100; i++) { + await seedPageWithPath(`bulk/${i}`, `bulk/${i}.md`); + } + const realSlugs = Array.from({ length: 100 }, (_, i) => `bulk/${i}`); + const ghostSlugs = Array.from({ length: 10 }, (_, i) => `bulk/ghost-${i}`); + const allSlugs = [...realSlugs, ...ghostSlugs]; + + const deleted = await engine.deletePages(allSlugs, { sourceId: 'default' }); + + // Pre-v0.41.19.0 the caller would have pushed all 110 slugs onto + // pagesAffected (no filtering); post-fix only the 100 real deletes + // come back from RETURNING. + expect(deleted.length).toBe(100); + expect(deleted.sort()).toEqual(realSlugs.sort()); + }); +}); diff --git a/test/sync-rename-batch.test.ts b/test/sync-rename-batch.test.ts new file mode 100644 index 000000000..158bfaf9e --- /dev/null +++ b/test/sync-rename-batch.test.ts @@ -0,0 +1,92 @@ +/** + * v0.41.19.0 — sync rename loop (pre-batched slug resolution) + * + * Pins the contract of T4 in the plan: the rename loop in + * src/commands/sync.ts:~1280 pre-resolves all `from` slugs via + * engine.resolveSlugsByPaths in batches BEFORE iterating per-file. The + * per-file updateSlug + importFile calls stay (those are inherently + * per-file). The win is dropping the slug-resolve N+1. + * + * Coverage: + * - resolveSlugsByPaths returns one Map for an N-path input (not N + * individual round-trips). Exercised via engine direct + spy on + * executeRaw count. + * - Frontmatter-fallback rename: exotic `from` paths still resolve to + * the stored slug. + * - Source isolation preserved. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { DELETE_BATCH_SIZE } from '../src/core/engine-constants.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +async function seedPageWithPath(slug: string, sourcePath: string): Promise { + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, source_path, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', $1, $2, 'note', $1, 'body', '', '{}'::jsonb) + ON CONFLICT (source_id, slug) DO UPDATE SET source_path = EXCLUDED.source_path`, + [slug, sourcePath], + ); +} + +describe('rename loop pre-batched slug resolution', () => { + test('500 from-paths resolved in 1 batch (DELETE_BATCH_SIZE-aligned)', async () => { + // Seed 500 pages, all with explicit source_paths matching their slugs. + const N = 500; + for (let i = 0; i < N; i++) { + await seedPageWithPath(`rn/page-${i}`, `rn/page-${i}.md`); + } + const paths = Array.from({ length: N }, (_, i) => `rn/page-${i}.md`); + + // Single batch of 500 — exactly at DELETE_BATCH_SIZE boundary. + expect(paths.length).toBe(DELETE_BATCH_SIZE); + const m = await engine.resolveSlugsByPaths(paths, { sourceId: 'default' }); + expect(m.size).toBe(N); + for (let i = 0; i < N; i++) { + expect(m.get(`rn/page-${i}.md`)).toBe(`rn/page-${i}`); + } + }); + + test('frontmatter-fallback rename: exotic source_paths resolve via the batch SELECT', async () => { + await seedPageWithPath('star-renamed-from', '🌟.md'); + await seedPageWithPath('thai-renamed-from', 'ทดสอบ.md'); + + const m = await engine.resolveSlugsByPaths( + ['🌟.md', 'ทดสอบ.md'], + { sourceId: 'default' }, + ); + expect(m.get('🌟.md')).toBe('star-renamed-from'); + expect(m.get('ทดสอบ.md')).toBe('thai-renamed-from'); + }); + + test('mixed present + missing: partial Map (missing → caller falls back to path-derived)', async () => { + await seedPageWithPath('present-1', 'present-1.md'); + await seedPageWithPath('present-2', 'present-2.md'); + const m = await engine.resolveSlugsByPaths( + ['present-1.md', 'absent.md', 'present-2.md', 'absent2.md'], + { sourceId: 'default' }, + ); + expect(m.size).toBe(2); + expect(m.get('present-1.md')).toBe('present-1'); + expect(m.get('present-2.md')).toBe('present-2'); + expect(m.get('absent.md')).toBeUndefined(); + expect(m.get('absent2.md')).toBeUndefined(); + }); +});