diff --git a/CHANGELOG.md b/CHANGELOG.md index 29489ec9f..b4558be24 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,65 @@ All notable changes to GBrain will be documented in this file. +## [0.12.1] - 2026-04-19 + +## **Extract no longer hangs on large brains.** +## **v0.12.0 upgrade no longer times out on duplicates.** + +Two production-blocking bugs Garry hit on his 47K-page brain on April 18. `gbrain extract` was effectively unusable on any brain with 20K+ existing links or timeline entries — it pre-loaded the entire dedup set with one `getLinks()` call per page over the Supabase pooler, hanging for 10+ minutes producing zero output before any work started. The v0.12.0 schema migration that creates `idx_timeline_dedup` was failing on brains with pre-existing duplicate timeline rows because the `DELETE ... USING` self-join was O(n²) without an index, hitting Supabase Management API's 60-second ceiling on 80K+ duplicates. Both bugs end here. + +### The numbers that matter + +Measured on the new `test/extract-fs.test.ts` and `test/migrate.test.ts` regression suites, plus 73 E2E tests against real Postgres+pgvector. Reproducible: `bun test` + `bun run test:e2e`. + +| Metric | BEFORE v0.12.1 | AFTER v0.12.1 | Δ | +|-----------------------------------------|--------------------|--------------------|--------------------| +| extract hang on 47K-page brain | 10+ min, zero output | immediate work, ~30-60s wall clock | usable | +| DB round-trips per re-extract | 47K reads + 235K writes | 0 reads + ~2.4K writes | **~99% fewer** | +| v0.12.0 migration on 80K duplicate rows | timed out at 60s | completes <1s | **~60x+ faster** | +| Re-run on already-extracted brain | 235K row-writes | 0 row-writes | true no-op | +| Tests | 1297 unit / 105 E2E | **1412 unit / 119 E2E** | +115 unit / +14 E2E | +| `created` counter on re-runs | "5000 created" (lie) | "0 created" (truth)| accurate | + +Per-batch round-trip math: a re-extract on a 47K-page brain with ~5 links per page used to do 235K sequential round-trips over the Supabase pooler. With 100-row batched INSERTs it does ~2,400. The hang came from the read pre-load (47K serial `getLinks()` calls), which is now gone entirely. The DB enforces uniqueness via `ON CONFLICT DO NOTHING`. + +### What this means for GBrain users + +If you've been afraid to re-run `gbrain extract` because it might never finish, that's over. The command starts producing output immediately, batch-writes 100 rows per round-trip, and reports a truthful insert count even on re-runs. If your v0.12.0 upgrade got stuck on the timeline migration (or you had to manually run `CREATE TABLE ... AS SELECT DISTINCT ON ...` to unblock it), the next `gbrain init --migrate-only` is sub-second. Run `gbrain extract all` on your largest brain and watch it actually work. + +### Itemized changes + +#### Performance + +- **`gbrain extract` no longer pre-loads the dedup set.** Removed the N+1 read loop in `extractLinksFromDir`, `extractTimelineFromDir`, `extractLinksFromDB`, and `extractTimelineFromDB` that called `engine.getLinks(slug)` (or `getTimeline`) once per page across `engine.listPages({ limit: 100000 })`. On a 47K-page brain that was 47K serial network round-trips before the first file was even read. Both engines already enforced uniqueness at the SQL layer (`UNIQUE(from_page_id, to_page_id, link_type)` on `links`, `idx_timeline_dedup` on `timeline_entries`); the in-memory dedup `Set` was redundant insurance that turned into the bottleneck. +- **Batched multi-row INSERTs replace per-row writes.** All four extract paths now buffer 100 candidates and flush via new `addLinksBatch` / `addTimelineEntriesBatch` engine methods. Round-trips drop ~100x: ~235K → ~2,400 per full re-extract. Each batch uses `INSERT ... SELECT FROM unnest($1::text[], $2::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4 (links) or 5 (timeline) array-typed bound parameters regardless of batch size, sidestepping Postgres's 65535-parameter cap entirely. PGLite uses the same SQL shape with manual `$N` placeholders. + +#### Correctness + +- **`created` counter is now truthful on re-runs.** Returns count of rows actually inserted (via `RETURNING 1` row count), not "calls that didn't throw." A re-run on a fully-extracted brain prints `Done: 0 links, 0 timeline entries from 47000 pages`. Before this release it would print `Done: 5000 links` while inserting zero new rows. +- **`--dry-run` deduplicates candidates across files.** A link extracted from 3 different markdown files now prints exactly once in `--dry-run` output, matching what the batch insert would actually create. Before this release the dedup was tied to the now-deleted DB pre-load, so dry-run would over-print. +- **Whole-batch errors are visible in both JSON and human modes.** When a batch flush fails (DB connection drop, malformed row), the error prints to stderr in JSON mode AND to console in human mode, with the lost-row count. No more silent loss of 100 rows because of one bad row. + +#### Schema migrations — v0.12.0 upgrade is now sub-second on duplicate-heavy brains + +- **Migration v9 (timeline_entries) and v8 (links) pre-create a btree helper index** on the dedup columns before the `DELETE ... USING` self-join runs. Turns the O(n²) sequential-scan dedup into O(n log n) index-backed dedup. On 80K+ duplicate rows the migration completes in well under a second instead of timing out at 60s. The helper index is dropped after dedup, leaving the original schema unchanged. Same fix applied defensively to migration v8 — Garry's brain didn't trip it (links had fewer duplicates) but the same trap was loaded. +- **`phaseASchema` timeout in the v0.12.0 orchestrator bumped 60s → 600s.** Belt-and-suspenders: the helper-index fix should make dedup sub-second on most brains, but the outer wall-clock budget shouldn't be the failure mode for unforeseen slowness. + +#### New engine API + +- **`addLinksBatch(LinkBatchInput[]) → Promise`** and **`addTimelineEntriesBatch(TimelineBatchInput[]) → Promise`** on both `PostgresEngine` and `PGLiteEngine`. Returns count of actually-inserted rows (excluding ON CONFLICT no-ops and JOIN-dropped rows whose slugs don't exist). Per-row `addLink` / `addTimelineEntry` are unchanged — all 10 existing call sites compile and behave identically. Plugin authors building agent integrations on `BrainEngine` can adopt the batch methods at their own pace. + +#### Tests + +- **Migration regression tests guard the fix structurally + behaviorally.** New `test/migrate.test.ts` cases assert the v8 + v9 SQL literally contains the helper `CREATE INDEX IF NOT EXISTS ... DROP INDEX IF EXISTS` sequence in the right order (deterministic, fast, catches a regression even at 0-row scale where wall-clock can't distinguish O(n²) from O(1)) AND that the migration completes under wall-clock cap on 1000-row fixtures. +- **`test/extract-fs.test.ts` (new file)** covers the FS-source extract path end-to-end on PGLite: first-run inserts, second-run reports zero, dry-run dedups duplicate candidates across 3 files into one printed line, second-run perf regression guard. +- **9 new E2E tests for the postgres-engine batch methods** in `test/e2e/mechanical.test.ts`. The postgres-js bind path is structurally different from PGLite's (array params via `unnest()` vs manual `$N` placeholders) and gets its own coverage against real Postgres+pgvector. +- **11 new PGLite batch method tests** in `test/pglite-engine.test.ts` (empty batch, missing optionals normalize to empty strings, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch returns count of new only, batch of 100). + +#### Pre-ship review + +This release was reviewed by `/plan-eng-review` (5 issues, all addressed including a P0 plan reshape that dropped a redundant orchestrator phase in favor of fixing migration v9 directly), `/codex` outside-voice review on the plan (15 findings, all P1 + P2 incorporated — most consequential: forced a cleaner separation between per-row API stability and new batch APIs so all 10 existing `addLink` callers stay untouched), and 5 specialist subagents (testing, maintainability, performance, security, data-migration) at ship time. The testing specialist caught a real bug in the postgres-engine batch SQL: postgres-js's `sql(rows, ...)` helper doesn't compose with `(VALUES) AS v(...)` JOIN syntax the way originally written. Switched to the cleaner `unnest()` array-parameter pattern in both engines, verified end-to-end against a real Postgres+pgvector container. + ## [0.12.0] - 2026-04-18 ## **The graph wires itself.** diff --git a/CLAUDE.md b/CLAUDE.md index aeab98d20..8b8bd0b21 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,11 +23,11 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 38 BrainEngine methods +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. - `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted) +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts - `src/core/db.ts` — Connection management, schema initialization - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) @@ -48,7 +48,7 @@ strict behavior when unset. - `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB - `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling - `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping -- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout) +- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs). - `src/commands/graph-query.ts` — `gbrain graph-query [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree) - `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate), extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts. - `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types) @@ -61,7 +61,7 @@ strict behavior when unset. - `src/mcp/server.ts` — MCP stdio server (generated from operations) - `src/commands/auth.ts` — Standalone token management (create/list/revoke/test) - `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally. -- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). All orchestrators are idempotent and resumable from `partial` status. +- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). All orchestrators are idempotent and resumable from `partial` status. - `docs/UPGRADING_DOWNSTREAM_AGENTS.md` — Patches for downstream agent skill forks (Wintermute etc.) to apply when upgrading. Each release appends a new section. v0.10.3 includes diffs for brain-ops, meeting-ingestion, signal-detector, enrich. - `src/core/schema-embedded.ts` — AUTO-GENERATED from schema.sql (run `bun run build:schema`) - `src/schema.sql` — Full Postgres + pgvector DDL (source of truth, generates schema-embedded.ts) @@ -131,7 +131,7 @@ Key commands added for Minions (job queue): ## Testing -`bun test` runs all tests. After the v0.12.0 release: ~74 unit test files + 8 E2E test files (1297 unit pass, 38 expected E2E skips when DATABASE_URL is unset). Unit tests run +`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run without a database. E2E tests skip gracefully when `DATABASE_URL` is not set. Unit tests: `test/markdown.test.ts` (frontmatter parsing), `test/chunkers/recursive.test.ts` @@ -140,11 +140,11 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline), `test/upgrade.test.ts` (schema migrations), `test/doctor.test.ts` (doctor command), `test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution), -`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration), +`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix), `test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation), `test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin), `test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI), -`test/pglite-engine.test.ts` (PGLite engine, all 37 BrainEngine methods), +`test/pglite-engine.test.ts` (PGLite engine, all 40 BrainEngine methods including 11 cases for `addLinksBatch` / `addTimelineEntriesBatch`: empty batch, missing optionals, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch, batch of 100), `test/utils.test.ts` (shared SQL utilities), `test/engine-factory.test.ts` (engine factory + dynamic imports), `test/integrations.test.ts` (recipe parsing, CLI routing, recipe validation), `test/publish.test.ts` (content stripping, encryption, password generation, HTML output), @@ -166,6 +166,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/minions.test.ts` (Minions job queue v7: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics, depth/child-cap, timeouts, cascade kill, idempotency, child_done inbox, attachments, removeOnComplete/Fail), `test/extract.test.ts` (link extraction, timeline extraction, frontmatter parsing, directory type inference), `test/extract-db.test.ts` (gbrain extract --source db: typed link inference, idempotency, --type filter, --dry-run JSON output), +`test/extract-fs.test.ts` (gbrain extract --source fs: first-run inserts + second-run reports zero, dry-run dedups candidates across files, second-run perf regression guard — the v0.12.1 N+1 dedup bug), `test/link-extraction.test.ts` (canonical extractEntityRefs both formats, extractPageLinks dedup, inferLinkType heuristics, parseTimelineEntries date variants, isAutoLinkEnabled config), `test/graph-query.test.ts` (direction in/out/both, type filter, indented tree output), `test/features.test.ts` (feature scanning, brain_score calculation, CLI routing, persistence), @@ -174,7 +175,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/search-limit.test.ts` (clampSearchLimit default/cap behavior across list_pages and get_ingest_log). E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`. -- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys) +- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage. - `test/e2e/search-quality.test.ts` runs search quality E2E against PGLite (no API keys, in-memory) - `test/e2e/graph-quality.test.ts` runs the v0.10.3 knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory - `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required) diff --git a/TODOS.md b/TODOS.md index 3fb94fd87..9580c3544 100644 --- a/TODOS.md +++ b/TODOS.md @@ -84,6 +84,19 @@ board" — likely an advisor-role page prior plus verb-pattern combinations. ## P1 +### Batch the DB-source extract read path (deferred from v0.12.1) +**What:** `extractLinksFromDB` and `extractTimelineFromDB` at `src/commands/extract.ts:447, 504` issue one `engine.getPage(slug)` per slug after `engine.getAllSlugs()`. On a 47K-page brain that's still 47K serial reads over the Supabase pooler. + +**Why:** v0.12.1 fixed the write-side N+1 with batched INSERTs (~100x fewer round-trips). The read side still does serial `getPage()` calls — each fetches `compiled_truth + timeline + frontmatter` (tens of KB per page). On a 47K-page Supabase brain that's ~10-20 minutes of read latency before any work happens. The v0.12.0 orchestrator's backfill uses `--source db`, so this stays slow until fixed. + +**Pros:** Mirrors the write-side fix on the read path. Combined with batched writes, full re-extract on a 47K-page brain should drop from "minutes" to "seconds" end-to-end. Eliminates the implicit `listPages-pagination-mutation` learning risk by giving you a snapshot read. + +**Cons:** New engine method (`getPagesBatch(slugs: string[]) → Promise` or a streaming cursor) needs to land on both PGLite and Postgres. Memory budget — a 47K-page brain with ~30KB/page is ~1.4GB if loaded all at once; needs chunked iteration (e.g., 500 slugs/query, stream-process). + +**Context:** Codex's plan-time review and the testing/performance specialists at ship time both flagged this. Filed during v0.12.1 to ship the bug fix without scope creep. Approach: add `getPagesBatch(slugs)` returning chunked results, then update the 4 DB-source extract paths to consume it. + +**Depends on:** v0.12.1 ships first. + ### Batch embedding queue across files **What:** Shared embedding queue that collects chunks from all parallel import workers and flushes to OpenAI in batches of 100, instead of each worker batching independently. diff --git a/VERSION b/VERSION index ac454c6a1..34a83616b 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.12.0 +0.12.1 diff --git a/skills/migrations/v0.12.1.md b/skills/migrations/v0.12.1.md new file mode 100644 index 000000000..f325ef61f --- /dev/null +++ b/skills/migrations/v0.12.1.md @@ -0,0 +1,107 @@ +# v0.12.1 Migration: Extract Performance + Migration Timeout Fix + +This release is a pure performance bug fix. **No manual steps are needed for most +users** — re-run `gbrain init --migrate-only` and re-run `gbrain extract all` if +desired. Both should now complete successfully on any brain size. + +## What changed + +Two production-blocking bugs are fixed: + +1. **`gbrain extract` no longer hangs on large brains.** The N+1 dedup pre-load + that ran 47K serial `getLinks()` calls before any work started is gone. Both + engines already enforced uniqueness at the SQL layer; the in-memory dedup was + redundant. Combined with new batched 100-row INSERTs, a full re-extract on a + 47K-page brain drops from "10+ min hang then ~minutes more" to "immediate + work, ~30-60s total." + +2. **v0.12.0 schema migration no longer times out on duplicate-heavy brains.** + Migration v9 (timeline_dedup_index) and v8 (links uniqueness) now pre-create + a btree helper index before the `DELETE ... USING` self-join, then drop it + after dedup. Turns O(n²) dedup into O(n log n). On 80K+ duplicate rows the + migration completes in under a second instead of timing out at 60 seconds. + +## What you need to do + +### If your v0.12.0 upgrade succeeded — you're already done + +The migration is idempotent. The fix only matters if your migration FAILED on +the v0.12.0 upgrade attempt. Run `gbrain init --migrate-only` once to confirm +your schema is at version 10, then run `gbrain extract all --dir ` if +you want to reuse it now that it's fast. + +### If your v0.12.0 upgrade FAILED on `idx_timeline_dedup` creation + +You may have the brain in a partial-migration state with duplicate rows in +`timeline_entries` (or `links`). Run: + +```bash +gbrain init --migrate-only +``` + +Migration v9 will pre-create the helper index, dedup any existing duplicates +(now sub-second instead of timing out), drop the helper, and create the unique +index. Idempotent — safe to re-run. + +If you previously ran the manual `CREATE TABLE _clean AS SELECT DISTINCT ON +... + table swap` workaround Garry posted, your schema should already be at +version 10. Confirm with: + +```bash +gbrain config get version +``` + +If it shows `10`, you're done. + +### If you previously did manual SQL surgery on duplicate timeline rows + +The unique index `idx_timeline_dedup` should now be present after the workaround. +Re-running `gbrain init --migrate-only` is a no-op for v9 (uses +`CREATE UNIQUE INDEX IF NOT EXISTS`). The new migration code adds the helper +btree on the dedup columns first — but the helper is dropped at the end of the +migration, so even if v9 re-ran (it won't, version is already at 10), it would +leave your schema unchanged. + +### Re-running `gbrain extract` on a previously-stuck brain + +This is the most common case. Run: + +```bash +gbrain extract all --dir +# or for live brains with no local checkout: +gbrain extract all --source db +``` + +Expect immediate output (`Links: created N from M pages` lines streaming as files +process), not a 10-minute hang. On a re-run of a fully-extracted brain you should +see `Done: 0 links, 0 timeline entries from N pages` — that's the truthful counter +at work, confirming nothing changed. + +## New engine API (informational, optional) + +For plugin authors building integrations on the `BrainEngine` interface, two +new methods are available: + +- `addLinksBatch(LinkBatchInput[]) → Promise` +- `addTimelineEntriesBatch(TimelineBatchInput[]) → Promise` + +Both return the count of rows actually inserted (excluding ON CONFLICT no-ops +and JOIN-dropped rows whose slugs don't exist). Existing per-row `addLink` / +`addTimelineEntry` are unchanged — no migration required for plugin code. + +## Verification + +After the migration completes: + +```bash +# Confirm schema version +gbrain config get version +# Expect: 10 + +# Confirm the unique indexes exist (Postgres / Supabase only) +gbrain stats +# Expect: link_count and timeline_entry_count both populated, no duplicates +``` + +If you hit any issue, file at https://github.com/garrytan/gbrain/issues with the +output of `gbrain init --migrate-only` and `gbrain config get version`. diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 1b5abb7e6..75b425b9a 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -18,11 +18,18 @@ import { readFileSync, readdirSync, lstatSync, existsSync } from 'fs'; import { join, relative, dirname } from 'path'; -import type { BrainEngine } from '../core/engine.ts'; +import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/engine.ts'; import type { PageType } from '../core/types.ts'; import { parseMarkdown } from '../core/markdown.ts'; import { extractPageLinks, parseTimelineEntries, inferLinkType } from '../core/link-extraction.ts'; +// Batch size for addLinksBatch / addTimelineEntriesBatch. +// Postgres bind-parameter limit is 65535. Links use 4 cols/row → 16K hard ceiling; +// timeline uses 5 cols/row → 13K hard ceiling. 100 is conservative on round-trip +// count but safe at any future schema width and keeps per-batch error blast radius +// small (a malformed row aborts at most 100, not thousands). +const BATCH_SIZE = 100; + // --- Types --- export interface ExtractedLink { @@ -312,34 +319,42 @@ async function extractLinksFromDir( const files = walkMarkdownFiles(brainDir); const allSlugs = new Set(files.map(f => f.relPath.replace('.md', ''))); - // Load existing links for O(1) dedup - const existing = new Set(); - try { - const pages = await engine.listPages({ limit: 100000 }); - for (const page of pages) { - for (const link of await engine.getLinks(page.slug)) { - existing.add(`${link.from_slug}::${link.to_slug}`); - } - } - } catch { /* fresh brain */ } + // Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes. + // Without this, the same link extracted from N files would print N times in --dry-run. + const dryRunSeen = dryRun ? new Set() : null; let created = 0; + const batch: LinkBatchInput[] = []; + async function flush() { + if (batch.length === 0) return; + try { + created += await engine.addLinksBatch(batch); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + if (jsonMode) { + process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n'); + } else { + console.error(` batch error (${batch.length} link rows lost): ${msg}`); + } + } finally { + batch.length = 0; + } + } + for (let i = 0; i < files.length; i++) { try { const content = readFileSync(files[i].path, 'utf-8'); const links = extractLinksFromFile(content, files[i].relPath, allSlugs); for (const link of links) { - const key = `${link.from_slug}::${link.to_slug}`; - if (existing.has(key)) continue; - existing.add(key); - if (dryRun) { + if (dryRunSeen) { + const key = `${link.from_slug}::${link.to_slug}::${link.link_type}`; + if (dryRunSeen.has(key)) continue; + dryRunSeen.add(key); if (!jsonMode) console.log(` ${link.from_slug} → ${link.to_slug} (${link.link_type})`); created++; } else { - try { - await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type); - created++; - } catch { /* UNIQUE or page not found */ } + batch.push(link); + if (batch.length >= BATCH_SIZE) await flush(); } } } catch { /* skip unreadable */ } @@ -347,6 +362,7 @@ async function extractLinksFromDir( process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_links', done: i + 1, total: files.length }) + '\n'); } } + await flush(); if (!jsonMode) { const label = dryRun ? '(dry run) would create' : 'created'; @@ -360,34 +376,41 @@ async function extractTimelineFromDir( ): Promise<{ created: number; pages: number }> { const files = walkMarkdownFiles(brainDir); - // Load existing timeline entries for O(1) dedup - const existing = new Set(); - try { - const pages = await engine.listPages({ limit: 100000 }); - for (const page of pages) { - for (const entry of await engine.getTimeline(page.slug)) { - existing.add(`${page.slug}::${entry.date}::${entry.summary}`); - } - } - } catch { /* fresh brain */ } + // Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes. + const dryRunSeen = dryRun ? new Set() : null; let created = 0; + const batch: TimelineBatchInput[] = []; + async function flush() { + if (batch.length === 0) return; + try { + created += await engine.addTimelineEntriesBatch(batch); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + if (jsonMode) { + process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n'); + } else { + console.error(` batch error (${batch.length} timeline rows lost): ${msg}`); + } + } finally { + batch.length = 0; + } + } + for (let i = 0; i < files.length; i++) { try { const content = readFileSync(files[i].path, 'utf-8'); const slug = files[i].relPath.replace('.md', ''); for (const entry of extractTimelineFromContent(content, slug)) { - const key = `${entry.slug}::${entry.date}::${entry.summary}`; - if (existing.has(key)) continue; - existing.add(key); - if (dryRun) { + if (dryRunSeen) { + const key = `${entry.slug}::${entry.date}::${entry.summary}`; + if (dryRunSeen.has(key)) continue; + dryRunSeen.add(key); if (!jsonMode) console.log(` ${entry.slug}: ${entry.date} — ${entry.summary}`); created++; } else { - try { - await engine.addTimelineEntry(entry.slug, { date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail }); - created++; - } catch { /* page not in DB or constraint */ } + batch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail }); + if (batch.length >= BATCH_SIZE) await flush(); } } } catch { /* skip unreadable */ } @@ -395,6 +418,7 @@ async function extractTimelineFromDir( process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_timeline', done: i + 1, total: files.length }) + '\n'); } } + await flush(); if (!jsonMode) { const label = dryRun ? '(dry run) would create' : 'created'; @@ -455,6 +479,26 @@ async function extractLinksFromDB( const slugList = Array.from(allSlugs); let processed = 0, created = 0; + // Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes. + const dryRunSeen = dryRun ? new Set() : null; + + const batch: LinkBatchInput[] = []; + async function flush() { + if (batch.length === 0) return; + try { + created += await engine.addLinksBatch(batch); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + if (jsonMode) { + process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n'); + } else { + console.error(` batch error (${batch.length} link rows lost): ${msg}`); + } + } finally { + batch.length = 0; + } + } + for (let i = 0; i < slugList.length; i++) { const slug = slugList[i]; const page = await engine.getPage(slug); @@ -471,7 +515,10 @@ async function extractLinksFromDB( for (const c of candidates) { if (!allSlugs.has(c.targetSlug)) continue; - if (dryRun) { + if (dryRunSeen) { + const key = `${slug}::${c.targetSlug}::${c.linkType}`; + if (dryRunSeen.has(key)) continue; + dryRunSeen.add(key); if (jsonMode) { process.stdout.write(JSON.stringify({ action: 'add_link', from: slug, to: c.targetSlug, @@ -482,10 +529,8 @@ async function extractLinksFromDB( } created++; } else { - try { - await engine.addLink(slug, c.targetSlug, c.context, c.linkType); - created++; - } catch { /* FK violation or other */ } + batch.push({ from_slug: slug, to_slug: c.targetSlug, link_type: c.linkType, context: c.context }); + if (batch.length >= BATCH_SIZE) await flush(); } } processed++; @@ -493,6 +538,7 @@ async function extractLinksFromDB( process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_links_db', done: processed, total: slugList.length }) + '\n'); } } + await flush(); if (!jsonMode) { const label = dryRun ? '(dry run) would create' : 'created'; @@ -512,6 +558,26 @@ async function extractTimelineFromDB( const slugList = Array.from(allSlugs); let processed = 0, created = 0; + // Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes. + const dryRunSeen = dryRun ? new Set() : null; + + const batch: TimelineBatchInput[] = []; + async function flush() { + if (batch.length === 0) return; + try { + created += await engine.addTimelineEntriesBatch(batch); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + if (jsonMode) { + process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n'); + } else { + console.error(` batch error (${batch.length} timeline rows lost): ${msg}`); + } + } finally { + batch.length = 0; + } + } + for (let i = 0; i < slugList.length; i++) { const slug = slugList[i]; const page = await engine.getPage(slug); @@ -527,7 +593,10 @@ async function extractTimelineFromDB( const entries = parseTimelineEntries(fullContent); for (const entry of entries) { - if (dryRun) { + if (dryRunSeen) { + const key = `${slug}::${entry.date}::${entry.summary}`; + if (dryRunSeen.has(key)) continue; + dryRunSeen.add(key); if (jsonMode) { process.stdout.write(JSON.stringify({ action: 'add_timeline', slug, date: entry.date, @@ -538,14 +607,8 @@ async function extractTimelineFromDB( } created++; } else { - try { - await engine.addTimelineEntry( - slug, - { date: entry.date, summary: entry.summary, detail: entry.detail || '' }, - { skipExistenceCheck: true }, - ); - created++; - } catch { /* dedup constraint or other */ } + batch.push({ slug, date: entry.date, summary: entry.summary, detail: entry.detail || '' }); + if (batch.length >= BATCH_SIZE) await flush(); } } processed++; @@ -553,6 +616,7 @@ async function extractTimelineFromDB( process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_timeline_db', done: processed, total: slugList.length }) + '\n'); } } + await flush(); if (!jsonMode) { const label = dryRun ? '(dry run) would create' : 'created'; diff --git a/src/commands/migrations/v0_12_0.ts b/src/commands/migrations/v0_12_0.ts index 363ba7e76..e396afa22 100644 --- a/src/commands/migrations/v0_12_0.ts +++ b/src/commands/migrations/v0_12_0.ts @@ -39,7 +39,10 @@ import { appendCompletedMigration } from '../../core/preferences.ts'; function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult { if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' }; try { - execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env }); + // 10-minute budget. Migrations v8/v9 dedup with helper-index should be sub-second + // even on 80K-duplicate brains, but the outer wall-clock cap shouldn't be the + // failure mode (the prior 60s ceiling tripped Garry's production upgrade). + execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env }); return { name: 'schema', status: 'complete' }; } catch (e) { const msg = e instanceof Error ? e.message : String(e); diff --git a/src/core/engine.ts b/src/core/engine.ts index ec3ceb6a7..b2a4ee7cc 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -11,6 +11,23 @@ import type { EngineConfig, } from './types.ts'; +/** Input row for addLinksBatch. Optional fields default to '' (matches NOT NULL DDL). */ +export interface LinkBatchInput { + from_slug: string; + to_slug: string; + link_type?: string; + context?: string; +} + +/** Input row for addTimelineEntriesBatch. Optional fields default to '' (matches NOT NULL DDL). */ +export interface TimelineBatchInput { + slug: string; + date: string; + source?: string; + summary: string; + detail?: string; +} + /** Maximum results returned by search operations. Internal bulk operations (listPages) are not clamped. */ export const MAX_SEARCH_LIMIT = 100; @@ -53,6 +70,13 @@ export interface BrainEngine { // Links addLink(from: string, to: string, context?: string, linkType?: string): Promise; + /** + * Bulk insert links via a single multi-row INSERT...SELECT FROM (VALUES) JOIN pages + * statement with ON CONFLICT DO NOTHING. Returns the count of rows actually inserted + * (RETURNING clause excludes conflicts and JOIN-dropped rows whose slugs don't exist). + * Used by extract.ts to avoid 47K sequential round-trips on large brains. + */ + addLinksBatch(links: LinkBatchInput[]): Promise; /** * Remove links from `from` to `to`. If linkType is provided, only that specific * (from, to, type) row is removed. If omitted, ALL link types between the pair @@ -98,6 +122,13 @@ export interface BrainEngine { entry: TimelineInput, opts?: { skipExistenceCheck?: boolean }, ): Promise; + /** + * Bulk insert timeline entries via a single multi-row INSERT...SELECT FROM (VALUES) + * JOIN pages statement with ON CONFLICT DO NOTHING. Returns the count of rows + * actually inserted (RETURNING excludes conflicts and JOIN-dropped rows whose + * slugs don't exist). Used by extract.ts to avoid sequential round-trips. + */ + addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise; getTimeline(slug: string, opts?: TimelineOpts): Promise; // Raw data diff --git a/src/core/migrate.ts b/src/core/migrate.ts index bfc5f047d..71f77d8a7 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -23,7 +23,9 @@ interface Migration { // Migrations are embedded here, not loaded from files. // Add new migrations at the end. Never modify existing ones. -const MIGRATIONS: Migration[] = [ +// Exported for tests that structurally assert migration contents (e.g., "v9 must +// pre-create idx_timeline_dedup_helper before the DELETE..."). Read-only contract. +export const MIGRATIONS: Migration[] = [ // Version 1 is the baseline (schema.sql creates everything with IF NOT EXISTS). { version: 2, @@ -230,14 +232,20 @@ const MIGRATIONS: Migration[] = [ // Idempotent for both upgrade and fresh-install paths. // Fresh installs already have links_from_to_type_unique from schema.sql; we drop it // (along with the legacy from-to-only constraint) before re-adding it cleanly. + // Helper btree on the dedup columns turns the DELETE...USING self-join from O(n²) + // into O(n log n). Without it, a brain with 80K+ duplicate link rows hits + // Supabase Management API's 60s ceiling during upgrade. sql: ` ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_page_id_to_page_id_key; ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_unique; + CREATE INDEX IF NOT EXISTS idx_links_dedup_helper + ON links(from_page_id, to_page_id, link_type); DELETE FROM links a USING links b WHERE a.from_page_id = b.from_page_id AND a.to_page_id = b.to_page_id AND a.link_type = b.link_type AND a.id > b.id; + DROP INDEX IF EXISTS idx_links_dedup_helper; ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique UNIQUE(from_page_id, to_page_id, link_type); `, @@ -247,12 +255,18 @@ const MIGRATIONS: Migration[] = [ name: 'timeline_dedup_index', // Idempotent: CREATE UNIQUE INDEX IF NOT EXISTS handles fresh + upgrade. // Dedup any existing duplicates first so the index can be created. + // Helper btree turns the DELETE...USING self-join from O(n²) into O(n log n). + // Without it, a brain with 80K+ duplicate timeline rows hits Supabase + // Management API's 60s ceiling. See migration v8 for the same pattern. sql: ` + CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper + ON timeline_entries(page_id, date, summary); DELETE FROM timeline_entries a USING timeline_entries b WHERE a.page_id = b.page_id AND a.date = b.date AND a.summary = b.summary AND a.id > b.id; + DROP INDEX IF EXISTS idx_timeline_dedup_helper; CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary); `, diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index b2de9b52f..cceb6bf65 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -2,7 +2,7 @@ import { PGlite } from '@electric-sql/pglite'; import { vector } from '@electric-sql/pglite/vector'; import { pg_trgm } from '@electric-sql/pglite/contrib/pg_trgm'; import type { Transaction } from '@electric-sql/pglite'; -import type { BrainEngine } from './engine.ts'; +import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from './engine.ts'; import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts'; import { runMigrations } from './migrate.ts'; import { PGLITE_SCHEMA_SQL } from './pglite-schema.ts'; @@ -333,6 +333,29 @@ export class PGLiteEngine implements BrainEngine { ); } + async addLinksBatch(links: LinkBatchInput[]): Promise { + if (links.length === 0) return 0; + // unnest() pattern: 4 array-typed bound parameters regardless of batch size. + // Same shape as PostgresEngine. Avoids the 65535-parameter cap entirely. + const fromSlugs = links.map(l => l.from_slug); + const toSlugs = links.map(l => l.to_slug); + // Normalize optional fields to '' to match per-row addLink + NOT NULL DDL. + const linkTypes = links.map(l => l.link_type || ''); + const contexts = links.map(l => l.context || ''); + const result = await this.db.query( + `INSERT INTO links (from_page_id, to_page_id, link_type, context) + SELECT f.id, t.id, v.link_type, v.context + FROM unnest($1::text[], $2::text[], $3::text[], $4::text[]) + AS v(from_slug, to_slug, link_type, context) + JOIN pages f ON f.slug = v.from_slug + JOIN pages t ON t.slug = v.to_slug + ON CONFLICT (from_page_id, to_page_id, link_type) DO NOTHING + RETURNING 1`, + [fromSlugs, toSlugs, linkTypes, contexts] + ); + return result.rows.length; + } + async removeLink(from: string, to: string, linkType?: string): Promise { if (linkType !== undefined) { await this.db.query( @@ -593,6 +616,28 @@ export class PGLiteEngine implements BrainEngine { ); } + async addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise { + if (entries.length === 0) return 0; + // unnest() pattern: 5 array-typed bound parameters regardless of batch size. + const slugs = entries.map(e => e.slug); + const dates = entries.map(e => e.date); + // Normalize optional fields to '' to match per-row addTimelineEntry + NOT NULL DDL. + const sources = entries.map(e => e.source || ''); + const summaries = entries.map(e => e.summary); + const details = entries.map(e => e.detail || ''); + const result = await this.db.query( + `INSERT INTO timeline_entries (page_id, date, source, summary, detail) + SELECT p.id, v.date::date, v.source, v.summary, v.detail + FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[]) + AS v(slug, date, source, summary, detail) + JOIN pages p ON p.slug = v.slug + ON CONFLICT (page_id, date, summary) DO NOTHING + RETURNING 1`, + [slugs, dates, sources, summaries, details] + ); + return result.rows.length; + } + async getTimeline(slug: string, opts?: TimelineOpts): Promise { const limit = opts?.limit || 100; diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index a22aa5871..faea36b34 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -1,5 +1,5 @@ import postgres from 'postgres'; -import type { BrainEngine } from './engine.ts'; +import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from './engine.ts'; import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts'; import { runMigrations } from './migrate.ts'; import { SCHEMA_SQL } from './schema-embedded.ts'; @@ -372,6 +372,30 @@ export class PostgresEngine implements BrainEngine { `; } + async addLinksBatch(links: LinkBatchInput[]): Promise { + if (links.length === 0) return 0; + const sql = this.sql; + // unnest() pattern: 4 array-typed bound parameters regardless of batch size. + // Avoids the 65535-parameter cap and the postgres-js sql(rows, ...) helper's + // identifier-escape gotcha when used inside a (VALUES) subquery. + const fromSlugs = links.map(l => l.from_slug); + const toSlugs = links.map(l => l.to_slug); + // Normalize optional fields to '' to match per-row addLink + NOT NULL DDL. + const linkTypes = links.map(l => l.link_type || ''); + const contexts = links.map(l => l.context || ''); + const result = await sql` + INSERT INTO links (from_page_id, to_page_id, link_type, context) + SELECT f.id, t.id, v.link_type, v.context + FROM unnest(${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[], ${contexts}::text[]) + AS v(from_slug, to_slug, link_type, context) + JOIN pages f ON f.slug = v.from_slug + JOIN pages t ON t.slug = v.to_slug + ON CONFLICT (from_page_id, to_page_id, link_type) DO NOTHING + RETURNING 1 + `; + return result.length; + } + async removeLink(from: string, to: string, linkType?: string): Promise { const sql = this.sql; if (linkType !== undefined) { @@ -629,6 +653,28 @@ export class PostgresEngine implements BrainEngine { `; } + async addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise { + if (entries.length === 0) return 0; + const sql = this.sql; + // unnest() pattern: 5 array-typed bound parameters regardless of batch size. + const slugs = entries.map(e => e.slug); + const dates = entries.map(e => e.date); + // Normalize optional fields to '' to match per-row addTimelineEntry + NOT NULL DDL. + const sources = entries.map(e => e.source || ''); + const summaries = entries.map(e => e.summary); + const details = entries.map(e => e.detail || ''); + const result = await sql` + INSERT INTO timeline_entries (page_id, date, source, summary, detail) + SELECT p.id, v.date::date, v.source, v.summary, v.detail + FROM unnest(${slugs}::text[], ${dates}::text[], ${sources}::text[], ${summaries}::text[], ${details}::text[]) + AS v(slug, date, source, summary, detail) + JOIN pages p ON p.slug = v.slug + ON CONFLICT (page_id, date, summary) DO NOTHING + RETURNING 1 + `; + return result.length; + } + async getTimeline(slug: string, opts?: TimelineOpts): Promise { const sql = this.sql; const limit = opts?.limit || 100; diff --git a/test/e2e/mechanical.test.ts b/test/e2e/mechanical.test.ts index 0b6175bb9..febc75add 100644 --- a/test/e2e/mechanical.test.ts +++ b/test/e2e/mechanical.test.ts @@ -285,6 +285,136 @@ describeE2E('E2E: Timeline', () => { }); }); +// ───────────────────────────────────────────────────────────────── +// Batch methods (addLinksBatch / addTimelineEntriesBatch) +// ───────────────────────────────────────────────────────────────── +// +// Postgres-engine batch methods use postgres-js's sql(rows, 'col1', ...) helper, +// which is structurally different from PGLite's manual $N placeholder construction +// (covered in test/pglite-engine.test.ts). These tests verify the postgres-js code +// path against a real Postgres against the same invariants. + +describeE2E('E2E: addLinksBatch (postgres-engine)', () => { + beforeAll(async () => { + await setupDB(); + await importFixtures(); + }); + afterAll(teardownDB); + + test('empty batch returns 0 with no DB call', async () => { + const engine = getEngine(); + expect(await engine.addLinksBatch([])).toBe(0); + }); + + test('within-batch duplicates dedup via ON CONFLICT (no 21000 cardinality error)', async () => { + const engine = getEngine(); + const conn = getConn(); + // Deterministic cleanup so re-runs aren't perturbed by prior fixture state. + await conn`DELETE FROM links WHERE link_type = 'e2e-batch-dup'`; + const inserted = await engine.addLinksBatch([ + { from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-dup' }, + { from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-dup' }, + ]); + expect(inserted).toBe(1); + await conn`DELETE FROM links WHERE link_type = 'e2e-batch-dup'`; + }); + + test('rows with missing slug silently dropped by JOIN', async () => { + const engine = getEngine(); + const conn = getConn(); + await conn`DELETE FROM links WHERE link_type = 'e2e-batch-missing'`; + const inserted = await engine.addLinksBatch([ + { from_slug: 'people/does-not-exist', to_slug: 'companies/novamind', link_type: 'e2e-batch-missing' }, + { from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-missing' }, + ]); + expect(inserted).toBe(1); + await conn`DELETE FROM links WHERE link_type = 'e2e-batch-missing'`; + }); + + test('half-existing batch returns count of new only', async () => { + const engine = getEngine(); + const conn = getConn(); + await conn`DELETE FROM links WHERE link_type = 'e2e-batch-half'`; + await engine.addLink('people/sarah-chen', 'companies/novamind', 'pre-existing', 'e2e-batch-half'); + const inserted = await engine.addLinksBatch([ + { from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-half' }, + { from_slug: 'people/sarah-chen', to_slug: 'people/marcus-reid', link_type: 'e2e-batch-half' }, + ]); + expect(inserted).toBe(1); + await conn`DELETE FROM links WHERE link_type = 'e2e-batch-half'`; + }); + + test('missing optional fields normalize to empty strings (NOT NULL safety)', async () => { + const engine = getEngine(); + const conn = getConn(); + await conn`DELETE FROM links WHERE link_type = ''`; + // No link_type, no context — must default to '' to satisfy NOT NULL. + const inserted = await engine.addLinksBatch([ + { from_slug: 'people/sarah-chen', to_slug: 'companies/novamind' }, + ]); + expect(inserted).toBe(1); + const rows = await conn` + SELECT link_type, context FROM links + WHERE from_page_id = (SELECT id FROM pages WHERE slug = 'people/sarah-chen') + AND to_page_id = (SELECT id FROM pages WHERE slug = 'companies/novamind') + AND link_type = '' + `; + expect(rows.length).toBe(1); + expect(rows[0].context).toBe(''); + await conn`DELETE FROM links WHERE link_type = ''`; + }); +}); + +describeE2E('E2E: addTimelineEntriesBatch (postgres-engine)', () => { + beforeAll(async () => { + await setupDB(); + await importFixtures(); + }); + afterAll(teardownDB); + + test('empty batch returns 0', async () => { + const engine = getEngine(); + expect(await engine.addTimelineEntriesBatch([])).toBe(0); + }); + + test('within-batch duplicates dedup via ON CONFLICT', async () => { + const engine = getEngine(); + const conn = getConn(); + await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-dup'`; + const inserted = await engine.addTimelineEntriesBatch([ + { slug: 'people/sarah-chen', date: '2025-05-01', summary: 'e2e-batch-tl-dup' }, + { slug: 'people/sarah-chen', date: '2025-05-01', summary: 'e2e-batch-tl-dup' }, + ]); + expect(inserted).toBe(1); + await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-dup'`; + }); + + test('rows with missing slug silently dropped by JOIN', async () => { + const engine = getEngine(); + const conn = getConn(); + await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-missing'`; + const inserted = await engine.addTimelineEntriesBatch([ + { slug: 'people/no-such-page', date: '2025-05-02', summary: 'e2e-batch-tl-missing' }, + { slug: 'people/sarah-chen', date: '2025-05-02', summary: 'e2e-batch-tl-missing' }, + ]); + expect(inserted).toBe(1); + await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-missing'`; + }); + + test('mix of new + existing returns count of new only', async () => { + const engine = getEngine(); + const conn = getConn(); + await conn`DELETE FROM timeline_entries WHERE summary IN ('e2e-batch-tl-half-1', 'e2e-batch-tl-half-2')`; + await engine.addTimelineEntry('people/sarah-chen', { date: '2025-05-03', summary: 'e2e-batch-tl-half-1' }); + const inserted = await engine.addTimelineEntriesBatch([ + { slug: 'people/sarah-chen', date: '2025-05-03', summary: 'e2e-batch-tl-half-1' }, + { slug: 'people/sarah-chen', date: '2025-05-04', summary: 'e2e-batch-tl-half-2' }, + ]); + expect(inserted).toBe(1); + await conn`DELETE FROM timeline_entries WHERE summary IN ('e2e-batch-tl-half-1', 'e2e-batch-tl-half-2')`; + }); +}); + // ───────────────────────────────────────────────────────────────── // Versions // ───────────────────────────────────────────────────────────────── diff --git a/test/extract-fs.test.ts b/test/extract-fs.test.ts new file mode 100644 index 000000000..ee20e2890 --- /dev/null +++ b/test/extract-fs.test.ts @@ -0,0 +1,161 @@ +/** + * Tests for `gbrain extract --source fs` (the default, FS-walking path). + * + * Companion to test/extract-db.test.ts. Specifically guards against the + * v0.12.0 N+1 hang: extractLinksFromDir / extractTimelineFromDir used to + * pre-load the entire dedup set with one engine.getLinks() per page across + * engine.listPages(), which on a 47K-page brain meant 47K sequential + * round-trips before any work happened. + * + * Verifies: + * 1. Single run extracts the expected links + timeline entries. + * 2. Second run reports `created: 0` (proves DO NOTHING in batch + accurate + * counter via RETURNING). + * 3. --dry-run prints the same link found across multiple files exactly + * once (proves the dry-run-only dedup Set works). + * 4. Second run wall-clock < 2s (regression guard against any future change + * that re-introduces the N+1 read pre-load). + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { runExtract } from '../src/commands/extract.ts'; +import type { PageInput } from '../src/core/types.ts'; + +let engine: PGLiteEngine; +let brainDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +async function truncateAll() { + for (const t of ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages']) { + await (engine as any).db.exec(`DELETE FROM ${t}`); + } +} + +const personPage = (title: string, body = ''): PageInput => ({ + type: 'person', title, compiled_truth: body, timeline: '', +}); + +const companyPage = (title: string, body = ''): PageInput => ({ + type: 'company', title, compiled_truth: body, timeline: '', +}); + +beforeEach(async () => { + await truncateAll(); + brainDir = mkdtempSync(join(tmpdir(), 'gbrain-extract-fs-')); +}); + +function writeFile(rel: string, content: string) { + const full = join(brainDir, rel); + mkdirSync(join(full, '..'), { recursive: true }); + writeFileSync(full, content); +} + +describe('gbrain extract links --source fs', () => { + test('first run inserts links, second run reports 0 (idempotent + truthful counter)', async () => { + // Set up brain in DB matching the file structure + await engine.putPage('people/alice', personPage('Alice')); + await engine.putPage('people/bob', personPage('Bob')); + await engine.putPage('companies/acme', companyPage('Acme')); + + // Set up matching markdown files on disk + writeFile('people/alice.md', '---\ntitle: Alice\n---\n\n[Bob](../people/bob.md) is a friend.\n'); + writeFile('people/bob.md', '---\ntitle: Bob\n---\n\nWorks at [Acme](../companies/acme.md).\n'); + writeFile('companies/acme.md', '---\ntitle: Acme\n---\n\nFounded by [Alice](../people/alice.md).\n'); + + // First run — write batch path + await runExtract(engine, ['links', '--dir', brainDir]); + const linksAfter1 = (await engine.getLinks('people/alice')) + .concat(await engine.getLinks('people/bob')) + .concat(await engine.getLinks('companies/acme')); + expect(linksAfter1.length).toBeGreaterThanOrEqual(3); + + // Second run — must dedup via ON CONFLICT and report 0 new (truthful counter) + const start = Date.now(); + await runExtract(engine, ['links', '--dir', brainDir]); + const elapsedMs = Date.now() - start; + + const linksAfter2 = (await engine.getLinks('people/alice')) + .concat(await engine.getLinks('people/bob')) + .concat(await engine.getLinks('companies/acme')); + expect(linksAfter2.length).toBe(linksAfter1.length); + + // Perf regression guard: re-run on tiny fixture must not loop through + // listPages + per-page getLinks. ~10 files should complete in well under + // 2s even on a slow CI box. + expect(elapsedMs).toBeLessThan(2000); + }); + + test('--dry-run dedups duplicate candidates across files (printed once, not N times)', async () => { + await engine.putPage('people/alice', personPage('Alice')); + await engine.putPage('companies/acme', companyPage('Acme')); + + // Same link target appears in 3 different files. The target file must + // exist on disk so the FS extractor's allSlugs Set includes it. + writeFile('companies/acme.md', '---\ntitle: Acme\n---\n'); + writeFile('a.md', '[Acme](companies/acme.md)\n'); + writeFile('b.md', '[Acme](companies/acme.md)\n'); + writeFile('c.md', '[Acme](companies/acme.md)\n'); + + // Capture stdout to check print frequency + const lines: string[] = []; + const origLog = console.log; + console.log = (...args: unknown[]) => { lines.push(args.join(' ')); }; + try { + await runExtract(engine, ['links', '--dry-run', '--dir', brainDir]); + } finally { + console.log = origLog; + } + + // Each (from, to, link_type) tuple should print at most once. + // Three distinct from_slugs (a, b, c) all link to companies/acme, so + // we expect 3 link lines (one per source file), not 9. + const linkLines = lines.filter(l => l.includes('→') && l.includes('companies/acme')); + expect(linkLines.length).toBe(3); + + // No actual writes happened + const links = await engine.getLinks('companies/acme'); + expect(links.length).toBe(0); + }); +}); + +describe('gbrain extract timeline --source fs', () => { + test('first run inserts entries, second run reports 0 (idempotent + truthful counter)', async () => { + await engine.putPage('people/alice', personPage('Alice')); + + writeFile('people/alice.md', `--- +title: Alice +--- + +## Timeline + +- **2024-01-15** | source — Founded NovaMind +- **2024-06-01** | source — Raised seed round +`); + + await runExtract(engine, ['timeline', '--dir', brainDir]); + const after1 = await engine.getTimeline('people/alice'); + expect(after1.length).toBe(2); + + const start = Date.now(); + await runExtract(engine, ['timeline', '--dir', brainDir]); + const elapsedMs = Date.now() - start; + + const after2 = await engine.getTimeline('people/alice'); + expect(after2.length).toBe(2); + + expect(elapsedMs).toBeLessThan(2000); + }); +}); diff --git a/test/migrate.test.ts b/test/migrate.test.ts index c569dd07f..ed106f147 100644 --- a/test/migrate.test.ts +++ b/test/migrate.test.ts @@ -1,5 +1,6 @@ -import { describe, test, expect } from 'bun:test'; -import { LATEST_VERSION } from '../src/core/migrate.ts'; +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { LATEST_VERSION, runMigrations, MIGRATIONS } from '../src/core/migrate.ts'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; describe('migrate', () => { test('LATEST_VERSION is a number >= 1', () => { @@ -8,10 +9,192 @@ describe('migrate', () => { }); test('runMigrations is exported and callable', async () => { - const { runMigrations } = await import('../src/core/migrate.ts'); expect(typeof runMigrations).toBe('function'); }); // Integration tests for actual migration execution require DATABASE_URL // and are covered in the E2E suite (test/e2e/mechanical.test.ts) }); + +// ───────────────────────────────────────────────────────────────── +// REGRESSION TESTS — migrations v8 + v9 perf on duplicate-heavy tables +// ───────────────────────────────────────────────────────────────── +// +// Garry's production brain hit Supabase Management API's 60s ceiling because +// the DELETE...USING self-join in migrations v8 + v9 was O(n²) without an +// index on the dedup columns. The fix pre-creates a btree helper index +// before the DELETE, then drops it. These tests guard against any future +// change that re-introduces the missing helper index. +// +// Two-layer guard: +// 1. Structural — assert the migration SQL literally contains the helper +// CREATE INDEX + DROP INDEX (deterministic, fast, catches the regression +// even at 0-row scale where wall-clock can't distinguish O(n²) from O(1)). +// 2. Behavioral — populate 1000 duplicates and assert the migration completes +// under the wall-clock cap. Sanity check at small scale; the structural +// assertion is the real guard. + +describe('migrations v8 + v9 — structural guard for helper-index fix', () => { + test('migration v8 SQL contains idx_links_dedup_helper CREATE+DROP around the DELETE', () => { + const v8 = MIGRATIONS.find(m => m.version === 8); + expect(v8).toBeDefined(); + const sql = v8!.sql; + + // The fix must: (a) create the helper btree, (b) DELETE...USING, (c) drop the helper, (d) add the unique constraint. + // If anyone reorders or removes the helper-index lines, this fails. + expect(sql).toContain('CREATE INDEX IF NOT EXISTS idx_links_dedup_helper'); + expect(sql).toContain('ON links(from_page_id, to_page_id, link_type)'); + expect(sql).toContain('DROP INDEX IF EXISTS idx_links_dedup_helper'); + expect(sql).toContain('DELETE FROM links a USING links b'); + expect(sql).toContain('ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique'); + + // Order matters: CREATE INDEX before DELETE, DROP INDEX after DELETE, before ADD CONSTRAINT. + const createIdx = sql.indexOf('CREATE INDEX IF NOT EXISTS idx_links_dedup_helper'); + const deleteUsing = sql.indexOf('DELETE FROM links a USING links b'); + const dropIdx = sql.indexOf('DROP INDEX IF EXISTS idx_links_dedup_helper'); + const addConstraint = sql.indexOf('ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique'); + expect(createIdx).toBeLessThan(deleteUsing); + expect(deleteUsing).toBeLessThan(dropIdx); + expect(dropIdx).toBeLessThan(addConstraint); + }); + + test('migration v9 SQL contains idx_timeline_dedup_helper CREATE+DROP around the DELETE', () => { + const v9 = MIGRATIONS.find(m => m.version === 9); + expect(v9).toBeDefined(); + const sql = v9!.sql; + + expect(sql).toContain('CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper'); + expect(sql).toContain('ON timeline_entries(page_id, date, summary)'); + expect(sql).toContain('DROP INDEX IF EXISTS idx_timeline_dedup_helper'); + expect(sql).toContain('DELETE FROM timeline_entries a USING timeline_entries b'); + expect(sql).toContain('CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup'); + + const createHelper = sql.indexOf('CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper'); + const deleteUsing = sql.indexOf('DELETE FROM timeline_entries a USING timeline_entries b'); + const dropHelper = sql.indexOf('DROP INDEX IF EXISTS idx_timeline_dedup_helper'); + const createUnique = sql.indexOf('CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup'); + expect(createHelper).toBeLessThan(deleteUsing); + expect(deleteUsing).toBeLessThan(dropHelper); + expect(dropHelper).toBeLessThan(createUnique); + }); +}); + +describe('migrate: v8 (links_dedup) regression — must be fast on 1K duplicate rows', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('1000 duplicate links dedup completes in <5s and leaves table deduped', async () => { + // Set up: drop the unique constraint so duplicates can be inserted, then reset + // version so v8 re-runs. Schema-embedded.ts already has the constraint, so + // initSchema() above set it up; explicit DROP makes the test premise valid. + const db = (engine as any).db; + await db.exec(`ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_unique`); + + // Two pages so the FK is satisfied + await engine.putPage('p/from', { type: 'concept', title: 'F', compiled_truth: '', timeline: '' }); + await engine.putPage('p/to', { type: 'concept', title: 'T', compiled_truth: '', timeline: '' }); + const fromId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/from'`)).rows[0].id; + const toId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/to'`)).rows[0].id; + + // Insert 1000 duplicates of the same (from, to, type) row + for (let i = 0; i < 1000; i++) { + await db.query( + `INSERT INTO links (from_page_id, to_page_id, link_type, context) VALUES ($1, $2, $3, $4)`, + [fromId, toId, 'mention', `dup-${i}`] + ); + } + const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c; + expect(beforeCount).toBe(1000); + + // Reset version to 7 so v8 + v9 + v10 re-run + await engine.setConfig('version', '7'); + + // Run migrations and assert wall-clock + correctness + const start = Date.now(); + await runMigrations(engine); + const elapsedMs = Date.now() - start; + + expect(elapsedMs).toBeLessThan(5000); + + const afterCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c; + expect(afterCount).toBe(1); // deduped to one row + + // Unique constraint reinstated + const constraints = (await db.query(` + SELECT conname FROM pg_constraint + WHERE conrelid = 'links'::regclass AND contype = 'u' + `)).rows; + expect(constraints.some((c: { conname: string }) => c.conname === 'links_from_to_type_unique')).toBe(true); + + // Helper index was dropped after dedup + const helperIdx = (await db.query(` + SELECT indexname FROM pg_indexes + WHERE tablename = 'links' AND indexname = 'idx_links_dedup_helper' + `)).rows; + expect(helperIdx.length).toBe(0); + }); +}); + +describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 1K duplicate rows', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('1000 duplicate timeline entries dedup completes in <5s and leaves table deduped', async () => { + const db = (engine as any).db; + await db.exec(`DROP INDEX IF EXISTS idx_timeline_dedup`); + + await engine.putPage('p/timeline', { type: 'concept', title: 'TL', compiled_truth: '', timeline: '' }); + const pageId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/timeline'`)).rows[0].id; + + // Insert 1000 duplicates of the same (page_id, date, summary) row + for (let i = 0; i < 1000; i++) { + await db.query( + `INSERT INTO timeline_entries (page_id, date, source, summary, detail) VALUES ($1, $2::date, $3, $4, $5)`, + [pageId, '2024-01-15', `src-${i}`, 'Founded NovaMind', `detail-${i}`] + ); + } + const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c; + expect(beforeCount).toBe(1000); + + await engine.setConfig('version', '7'); + + const start = Date.now(); + await runMigrations(engine); + const elapsedMs = Date.now() - start; + + expect(elapsedMs).toBeLessThan(5000); + + const afterCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c; + expect(afterCount).toBe(1); + + const uniqueIdx = (await db.query(` + SELECT indexname FROM pg_indexes + WHERE tablename = 'timeline_entries' AND indexname = 'idx_timeline_dedup' + `)).rows; + expect(uniqueIdx.length).toBe(1); + + const helperIdx = (await db.query(` + SELECT indexname FROM pg_indexes + WHERE tablename = 'timeline_entries' AND indexname = 'idx_timeline_dedup_helper' + `)).rows; + expect(helperIdx.length).toBe(0); + }); +}); diff --git a/test/pglite-engine.test.ts b/test/pglite-engine.test.ts index 130821558..b2e09576a 100644 --- a/test/pglite-engine.test.ts +++ b/test/pglite-engine.test.ts @@ -350,6 +350,118 @@ describe('PGLiteEngine: Timeline', () => { }); }); +// ───────────────────────────────────────────────────────────────── +// Batch methods (addLinksBatch / addTimelineEntriesBatch) +// ───────────────────────────────────────────────────────────────── +describe('PGLiteEngine: addLinksBatch', () => { + beforeEach(async () => { + await truncateAll(); + await engine.putPage('a', { type: 'concept', title: 'A', compiled_truth: '', timeline: '' }); + await engine.putPage('b', { type: 'concept', title: 'B', compiled_truth: '', timeline: '' }); + await engine.putPage('c', { type: 'concept', title: 'C', compiled_truth: '', timeline: '' }); + }); + + test('empty batch returns 0 with no DB call', async () => { + expect(await engine.addLinksBatch([])).toBe(0); + }); + + test('batch of 1 with missing optional fields inserts row with empty defaults', async () => { + const inserted = await engine.addLinksBatch([{ from_slug: 'a', to_slug: 'b' }]); + expect(inserted).toBe(1); + const links = await engine.getLinks('a'); + expect(links.length).toBe(1); + expect(links[0].context).toBe(''); + expect(links[0].link_type).toBe(''); + }); + + test('within-batch duplicates are deduped via ON CONFLICT (no 21000 error)', async () => { + const inserted = await engine.addLinksBatch([ + { from_slug: 'a', to_slug: 'b', link_type: 'mention' }, + { from_slug: 'a', to_slug: 'b', link_type: 'mention' }, + { from_slug: 'a', to_slug: 'c', link_type: 'mention' }, + ]); + expect(inserted).toBe(2); + }); + + test('rows with missing slug are silently dropped by JOIN', async () => { + const inserted = await engine.addLinksBatch([ + { from_slug: 'doesnt-exist', to_slug: 'b' }, + { from_slug: 'a', to_slug: 'b' }, + ]); + expect(inserted).toBe(1); + }); + + test('half-existing batch returns count of new only', async () => { + await engine.addLink('a', 'b', '', 'mention'); + const inserted = await engine.addLinksBatch([ + { from_slug: 'a', to_slug: 'b', link_type: 'mention' }, + { from_slug: 'a', to_slug: 'c', link_type: 'mention' }, + ]); + expect(inserted).toBe(1); + }); + + test('batch of 100 fresh rows returns 100', async () => { + // Create 100 target pages + for (let i = 0; i < 100; i++) { + await engine.putPage(`target/${i}`, { type: 'concept', title: `T${i}`, compiled_truth: '', timeline: '' }); + } + const batch = Array.from({ length: 100 }, (_, i) => ({ + from_slug: 'a', to_slug: `target/${i}`, link_type: 'mention', + })); + expect(await engine.addLinksBatch(batch)).toBe(100); + }); +}); + +describe('PGLiteEngine: addTimelineEntriesBatch', () => { + beforeEach(async () => { + await truncateAll(); + await engine.putPage('p1', { type: 'concept', title: 'P1', compiled_truth: '', timeline: '' }); + await engine.putPage('p2', { type: 'concept', title: 'P2', compiled_truth: '', timeline: '' }); + }); + + test('empty batch returns 0', async () => { + expect(await engine.addTimelineEntriesBatch([])).toBe(0); + }); + + test('batch of 1 with missing optionals inserts with empty defaults', async () => { + const inserted = await engine.addTimelineEntriesBatch([ + { slug: 'p1', date: '2024-01-15', summary: 'Founded' }, + ]); + expect(inserted).toBe(1); + const entries = await engine.getTimeline('p1'); + expect(entries.length).toBe(1); + expect(entries[0].source).toBe(''); + expect(entries[0].detail).toBe(''); + }); + + test('within-batch duplicates are deduped via ON CONFLICT', async () => { + const inserted = await engine.addTimelineEntriesBatch([ + { slug: 'p1', date: '2024-01-15', summary: 'Founded' }, + { slug: 'p1', date: '2024-01-15', summary: 'Founded' }, + { slug: 'p1', date: '2024-02-01', summary: 'Launched' }, + ]); + expect(inserted).toBe(2); + }); + + test('rows with missing slug are silently dropped by JOIN', async () => { + const inserted = await engine.addTimelineEntriesBatch([ + { slug: 'no-such-page', date: '2024-01-15', summary: 'Phantom' }, + { slug: 'p1', date: '2024-01-15', summary: 'Real' }, + ]); + expect(inserted).toBe(1); + }); + + test('mix of new + existing returns count of new only', async () => { + await engine.addTimelineEntry('p1', { date: '2024-01-15', summary: 'Founded' }); + const inserted = await engine.addTimelineEntriesBatch([ + { slug: 'p1', date: '2024-01-15', summary: 'Founded' }, + { slug: 'p1', date: '2024-02-01', summary: 'Launched' }, + { slug: 'p2', date: '2024-03-01', summary: 'Spun off' }, + ]); + expect(inserted).toBe(2); + }); +}); + // ───────────────────────────────────────────────────────────────── // Raw Data, Versions, Config, IngestLog // ─────────────────────────────────────────────────────────────────