mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
fix(extract+migrate): kill N+1 hang + v0.12.0 migration timeout (v0.12.1) (#198)
* feat(engine): add addLinksBatch + addTimelineEntriesBatch via unnest()
Multi-row INSERT...SELECT FROM unnest() JOIN pages ON CONFLICT DO NOTHING
RETURNING 1. 4 array-typed bound parameters (links) or 5 (timeline)
regardless of batch size, sidesteps Postgres's 65535-parameter cap.
Returns count of rows actually inserted (excluding ON CONFLICT no-ops
and JOIN-dropped rows whose slugs don't exist).
Per-row addLink / addTimelineEntry signatures and SQL behavior unchanged.
All 10 existing call sites compile and behave identically.
Tests: 11 PGLite cases (empty batch, missing optionals, within-batch dedup,
JOIN drops missing slug, half-existing batch, batch of 100) + 9 E2E
postgres-engine cases against real Postgres+pgvector.
* fix(migrate): pre-create btree helper in v8 + v9 dedup; bump phaseASchema timeout
Production bug: v0.12.0 schema migration timed out at Supabase Management API's
60s ceiling on brains with 80K+ duplicate timeline rows. The DELETE...USING
self-join was O(n²) without an index on the dedup columns.
Fix: pre-create idx_links_dedup_helper / idx_timeline_dedup_helper on the
dedup columns BEFORE the DELETE, drop after. Turns O(n²) into O(n log n).
On 80K+ rows the migration completes in <1s instead of timing out.
Also bumps the v0.12.0 orchestrator's phaseASchema timeout 60s -> 600s as
belt-and-suspenders for unforeseen slowness.
Exports MIGRATIONS for structural test assertions.
Tests: 2 structural assertions (helper-index DDL must appear in v8/v9 SQL
in the right order — catches regression even at 0-row scale) + 2 behavioral
regression tests (1000-row dedup completes <5s).
* perf(extract): kill N+1 dedup pre-load; switch to batched writes
Production bug: gbrain extract hung 10+ minutes producing zero output on
47K-page brains. The pre-load loop called engine.getLinks(slug) (or
getTimeline) once per page across engine.listPages({limit: 100000}) — 47K
serial round-trips over the Supabase pooler before the first file was read.
Both engines already enforced uniqueness at the SQL layer
(UNIQUE(from, to, link_type) on links, idx_timeline_dedup on timeline_entries).
The in-memory dedup Set was redundant insurance that became the bottleneck.
Fix: delete the pre-load entirely. Buffer 100 candidates per file walk,
flush via engine.addLinksBatch / engine.addTimelineEntriesBatch. ~99% fewer
DB round-trips per re-extract.
Also fixes counter accuracy: 'created' now counts rows actually inserted
(via batch RETURNING 1 row count). Re-run on a fully-extracted brain
prints 'Done: 0 links' instead of lying.
Dry-run mode keeps a per-run dedup Set so duplicate candidates from N
markdown files print exactly once, not N times.
Batch errors are visible in BOTH json and human modes — silent loss of
100 rows is worse than per-row error visibility.
Tests: extract-fs.test.ts (idempotency + truthful counter + dry-run dedup
+ perf regression guard <2s).
* chore: bump version + changelog (v0.12.1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: update CLAUDE.md for v0.12.1 (batch engine API, test counts)
Reflect what shipped in v0.12.1:
- New engine methods addLinksBatch + addTimelineEntriesBatch (PGLite via
unnest() + manual $N, postgres-engine via INSERT...SELECT FROM
unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING).
- extract.ts no longer pre-loads dedup set; candidates are buffered 100
at a time and flushed via the new batch methods.
- v0.12.0 orchestrator phaseASchema timeout bumped 60s to 600s.
- Test counts 1297 unit / 105 E2E to 1412 unit / 119 E2E.
- New test/extract-fs.test.ts covers the N+1 regression guard.
- BrainEngine method count 37/38 to 40.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
81b3f7afac
commit
699db50a3d
@@ -2,6 +2,65 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.12.1] - 2026-04-19
|
||||
|
||||
## **Extract no longer hangs on large brains.**
|
||||
## **v0.12.0 upgrade no longer times out on duplicates.**
|
||||
|
||||
Two production-blocking bugs Garry hit on his 47K-page brain on April 18. `gbrain extract` was effectively unusable on any brain with 20K+ existing links or timeline entries — it pre-loaded the entire dedup set with one `getLinks()` call per page over the Supabase pooler, hanging for 10+ minutes producing zero output before any work started. The v0.12.0 schema migration that creates `idx_timeline_dedup` was failing on brains with pre-existing duplicate timeline rows because the `DELETE ... USING` self-join was O(n²) without an index, hitting Supabase Management API's 60-second ceiling on 80K+ duplicates. Both bugs end here.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured on the new `test/extract-fs.test.ts` and `test/migrate.test.ts` regression suites, plus 73 E2E tests against real Postgres+pgvector. Reproducible: `bun test` + `bun run test:e2e`.
|
||||
|
||||
| Metric | BEFORE v0.12.1 | AFTER v0.12.1 | Δ |
|
||||
|-----------------------------------------|--------------------|--------------------|--------------------|
|
||||
| extract hang on 47K-page brain | 10+ min, zero output | immediate work, ~30-60s wall clock | usable |
|
||||
| DB round-trips per re-extract | 47K reads + 235K writes | 0 reads + ~2.4K writes | **~99% fewer** |
|
||||
| v0.12.0 migration on 80K duplicate rows | timed out at 60s | completes <1s | **~60x+ faster** |
|
||||
| Re-run on already-extracted brain | 235K row-writes | 0 row-writes | true no-op |
|
||||
| Tests | 1297 unit / 105 E2E | **1412 unit / 119 E2E** | +115 unit / +14 E2E |
|
||||
| `created` counter on re-runs | "5000 created" (lie) | "0 created" (truth)| accurate |
|
||||
|
||||
Per-batch round-trip math: a re-extract on a 47K-page brain with ~5 links per page used to do 235K sequential round-trips over the Supabase pooler. With 100-row batched INSERTs it does ~2,400. The hang came from the read pre-load (47K serial `getLinks()` calls), which is now gone entirely. The DB enforces uniqueness via `ON CONFLICT DO NOTHING`.
|
||||
|
||||
### What this means for GBrain users
|
||||
|
||||
If you've been afraid to re-run `gbrain extract` because it might never finish, that's over. The command starts producing output immediately, batch-writes 100 rows per round-trip, and reports a truthful insert count even on re-runs. If your v0.12.0 upgrade got stuck on the timeline migration (or you had to manually run `CREATE TABLE ... AS SELECT DISTINCT ON ...` to unblock it), the next `gbrain init --migrate-only` is sub-second. Run `gbrain extract all` on your largest brain and watch it actually work.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Performance
|
||||
|
||||
- **`gbrain extract` no longer pre-loads the dedup set.** Removed the N+1 read loop in `extractLinksFromDir`, `extractTimelineFromDir`, `extractLinksFromDB`, and `extractTimelineFromDB` that called `engine.getLinks(slug)` (or `getTimeline`) once per page across `engine.listPages({ limit: 100000 })`. On a 47K-page brain that was 47K serial network round-trips before the first file was even read. Both engines already enforced uniqueness at the SQL layer (`UNIQUE(from_page_id, to_page_id, link_type)` on `links`, `idx_timeline_dedup` on `timeline_entries`); the in-memory dedup `Set` was redundant insurance that turned into the bottleneck.
|
||||
- **Batched multi-row INSERTs replace per-row writes.** All four extract paths now buffer 100 candidates and flush via new `addLinksBatch` / `addTimelineEntriesBatch` engine methods. Round-trips drop ~100x: ~235K → ~2,400 per full re-extract. Each batch uses `INSERT ... SELECT FROM unnest($1::text[], $2::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4 (links) or 5 (timeline) array-typed bound parameters regardless of batch size, sidestepping Postgres's 65535-parameter cap entirely. PGLite uses the same SQL shape with manual `$N` placeholders.
|
||||
|
||||
#### Correctness
|
||||
|
||||
- **`created` counter is now truthful on re-runs.** Returns count of rows actually inserted (via `RETURNING 1` row count), not "calls that didn't throw." A re-run on a fully-extracted brain prints `Done: 0 links, 0 timeline entries from 47000 pages`. Before this release it would print `Done: 5000 links` while inserting zero new rows.
|
||||
- **`--dry-run` deduplicates candidates across files.** A link extracted from 3 different markdown files now prints exactly once in `--dry-run` output, matching what the batch insert would actually create. Before this release the dedup was tied to the now-deleted DB pre-load, so dry-run would over-print.
|
||||
- **Whole-batch errors are visible in both JSON and human modes.** When a batch flush fails (DB connection drop, malformed row), the error prints to stderr in JSON mode AND to console in human mode, with the lost-row count. No more silent loss of 100 rows because of one bad row.
|
||||
|
||||
#### Schema migrations — v0.12.0 upgrade is now sub-second on duplicate-heavy brains
|
||||
|
||||
- **Migration v9 (timeline_entries) and v8 (links) pre-create a btree helper index** on the dedup columns before the `DELETE ... USING` self-join runs. Turns the O(n²) sequential-scan dedup into O(n log n) index-backed dedup. On 80K+ duplicate rows the migration completes in well under a second instead of timing out at 60s. The helper index is dropped after dedup, leaving the original schema unchanged. Same fix applied defensively to migration v8 — Garry's brain didn't trip it (links had fewer duplicates) but the same trap was loaded.
|
||||
- **`phaseASchema` timeout in the v0.12.0 orchestrator bumped 60s → 600s.** Belt-and-suspenders: the helper-index fix should make dedup sub-second on most brains, but the outer wall-clock budget shouldn't be the failure mode for unforeseen slowness.
|
||||
|
||||
#### New engine API
|
||||
|
||||
- **`addLinksBatch(LinkBatchInput[]) → Promise<number>`** and **`addTimelineEntriesBatch(TimelineBatchInput[]) → Promise<number>`** on both `PostgresEngine` and `PGLiteEngine`. Returns count of actually-inserted rows (excluding ON CONFLICT no-ops and JOIN-dropped rows whose slugs don't exist). Per-row `addLink` / `addTimelineEntry` are unchanged — all 10 existing call sites compile and behave identically. Plugin authors building agent integrations on `BrainEngine` can adopt the batch methods at their own pace.
|
||||
|
||||
#### Tests
|
||||
|
||||
- **Migration regression tests guard the fix structurally + behaviorally.** New `test/migrate.test.ts` cases assert the v8 + v9 SQL literally contains the helper `CREATE INDEX IF NOT EXISTS ... DROP INDEX IF EXISTS` sequence in the right order (deterministic, fast, catches a regression even at 0-row scale where wall-clock can't distinguish O(n²) from O(1)) AND that the migration completes under wall-clock cap on 1000-row fixtures.
|
||||
- **`test/extract-fs.test.ts` (new file)** covers the FS-source extract path end-to-end on PGLite: first-run inserts, second-run reports zero, dry-run dedups duplicate candidates across 3 files into one printed line, second-run perf regression guard.
|
||||
- **9 new E2E tests for the postgres-engine batch methods** in `test/e2e/mechanical.test.ts`. The postgres-js bind path is structurally different from PGLite's (array params via `unnest()` vs manual `$N` placeholders) and gets its own coverage against real Postgres+pgvector.
|
||||
- **11 new PGLite batch method tests** in `test/pglite-engine.test.ts` (empty batch, missing optionals normalize to empty strings, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch returns count of new only, batch of 100).
|
||||
|
||||
#### Pre-ship review
|
||||
|
||||
This release was reviewed by `/plan-eng-review` (5 issues, all addressed including a P0 plan reshape that dropped a redundant orchestrator phase in favor of fixing migration v9 directly), `/codex` outside-voice review on the plan (15 findings, all P1 + P2 incorporated — most consequential: forced a cleaner separation between per-row API stability and new batch APIs so all 10 existing `addLink` callers stay untouched), and 5 specialist subagents (testing, maintainability, performance, security, data-migration) at ship time. The testing specialist caught a real bug in the postgres-engine batch SQL: postgres-js's `sql(rows, ...)` helper doesn't compose with `(VALUES) AS v(...)` JOIN syntax the way originally written. Switched to the cleaner `unnest()` array-parameter pattern in both engines, verified end-to-end against a real Postgres+pgvector container.
|
||||
|
||||
## [0.12.0] - 2026-04-18
|
||||
|
||||
## **The graph wires itself.**
|
||||
|
||||
@@ -23,11 +23,11 @@ strict behavior when unset.
|
||||
## Key files
|
||||
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`).
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 38 BrainEngine methods
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders.
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts
|
||||
- `src/core/db.ts` — Connection management, schema initialization
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
|
||||
@@ -48,7 +48,7 @@ strict behavior when unset.
|
||||
- `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB
|
||||
- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling
|
||||
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout)
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs).
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate), extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types)
|
||||
@@ -61,7 +61,7 @@ strict behavior when unset.
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
|
||||
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
|
||||
- `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). All orchestrators are idempotent and resumable from `partial` status.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). All orchestrators are idempotent and resumable from `partial` status.
|
||||
- `docs/UPGRADING_DOWNSTREAM_AGENTS.md` — Patches for downstream agent skill forks (Wintermute etc.) to apply when upgrading. Each release appends a new section. v0.10.3 includes diffs for brain-ops, meeting-ingestion, signal-detector, enrich.
|
||||
- `src/core/schema-embedded.ts` — AUTO-GENERATED from schema.sql (run `bun run build:schema`)
|
||||
- `src/schema.sql` — Full Postgres + pgvector DDL (source of truth, generates schema-embedded.ts)
|
||||
@@ -131,7 +131,7 @@ Key commands added for Minions (job queue):
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests. After the v0.12.0 release: ~74 unit test files + 8 E2E test files (1297 unit pass, 38 expected E2E skips when DATABASE_URL is unset). Unit tests run
|
||||
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
|
||||
without a database. E2E tests skip gracefully when `DATABASE_URL` is not set.
|
||||
|
||||
Unit tests: `test/markdown.test.ts` (frontmatter parsing), `test/chunkers/recursive.test.ts`
|
||||
@@ -140,11 +140,11 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline),
|
||||
`test/upgrade.test.ts` (schema migrations), `test/doctor.test.ts` (doctor command),
|
||||
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix),
|
||||
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
|
||||
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
|
||||
`test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI),
|
||||
`test/pglite-engine.test.ts` (PGLite engine, all 37 BrainEngine methods),
|
||||
`test/pglite-engine.test.ts` (PGLite engine, all 40 BrainEngine methods including 11 cases for `addLinksBatch` / `addTimelineEntriesBatch`: empty batch, missing optionals, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch, batch of 100),
|
||||
`test/utils.test.ts` (shared SQL utilities), `test/engine-factory.test.ts` (engine factory + dynamic imports),
|
||||
`test/integrations.test.ts` (recipe parsing, CLI routing, recipe validation),
|
||||
`test/publish.test.ts` (content stripping, encryption, password generation, HTML output),
|
||||
@@ -166,6 +166,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/minions.test.ts` (Minions job queue v7: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics, depth/child-cap, timeouts, cascade kill, idempotency, child_done inbox, attachments, removeOnComplete/Fail),
|
||||
`test/extract.test.ts` (link extraction, timeline extraction, frontmatter parsing, directory type inference),
|
||||
`test/extract-db.test.ts` (gbrain extract --source db: typed link inference, idempotency, --type filter, --dry-run JSON output),
|
||||
`test/extract-fs.test.ts` (gbrain extract --source fs: first-run inserts + second-run reports zero, dry-run dedups candidates across files, second-run perf regression guard — the v0.12.1 N+1 dedup bug),
|
||||
`test/link-extraction.test.ts` (canonical extractEntityRefs both formats, extractPageLinks dedup, inferLinkType heuristics, parseTimelineEntries date variants, isAutoLinkEnabled config),
|
||||
`test/graph-query.test.ts` (direction in/out/both, type filter, indented tree output),
|
||||
`test/features.test.ts` (feature scanning, brain_score calculation, CLI routing, persistence),
|
||||
@@ -174,7 +175,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/search-limit.test.ts` (clampSearchLimit default/cap behavior across list_pages and get_ingest_log).
|
||||
|
||||
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys)
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
|
||||
- `test/e2e/search-quality.test.ts` runs search quality E2E against PGLite (no API keys, in-memory)
|
||||
- `test/e2e/graph-quality.test.ts` runs the v0.10.3 knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
|
||||
@@ -84,6 +84,19 @@ board" — likely an advisor-role page prior plus verb-pattern combinations.
|
||||
|
||||
## P1
|
||||
|
||||
### Batch the DB-source extract read path (deferred from v0.12.1)
|
||||
**What:** `extractLinksFromDB` and `extractTimelineFromDB` at `src/commands/extract.ts:447, 504` issue one `engine.getPage(slug)` per slug after `engine.getAllSlugs()`. On a 47K-page brain that's still 47K serial reads over the Supabase pooler.
|
||||
|
||||
**Why:** v0.12.1 fixed the write-side N+1 with batched INSERTs (~100x fewer round-trips). The read side still does serial `getPage()` calls — each fetches `compiled_truth + timeline + frontmatter` (tens of KB per page). On a 47K-page Supabase brain that's ~10-20 minutes of read latency before any work happens. The v0.12.0 orchestrator's backfill uses `--source db`, so this stays slow until fixed.
|
||||
|
||||
**Pros:** Mirrors the write-side fix on the read path. Combined with batched writes, full re-extract on a 47K-page brain should drop from "minutes" to "seconds" end-to-end. Eliminates the implicit `listPages-pagination-mutation` learning risk by giving you a snapshot read.
|
||||
|
||||
**Cons:** New engine method (`getPagesBatch(slugs: string[]) → Promise<Page[]>` or a streaming cursor) needs to land on both PGLite and Postgres. Memory budget — a 47K-page brain with ~30KB/page is ~1.4GB if loaded all at once; needs chunked iteration (e.g., 500 slugs/query, stream-process).
|
||||
|
||||
**Context:** Codex's plan-time review and the testing/performance specialists at ship time both flagged this. Filed during v0.12.1 to ship the bug fix without scope creep. Approach: add `getPagesBatch(slugs)` returning chunked results, then update the 4 DB-source extract paths to consume it.
|
||||
|
||||
**Depends on:** v0.12.1 ships first.
|
||||
|
||||
### Batch embedding queue across files
|
||||
**What:** Shared embedding queue that collects chunks from all parallel import workers and flushes to OpenAI in batches of 100, instead of each worker batching independently.
|
||||
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
# v0.12.1 Migration: Extract Performance + Migration Timeout Fix
|
||||
|
||||
This release is a pure performance bug fix. **No manual steps are needed for most
|
||||
users** — re-run `gbrain init --migrate-only` and re-run `gbrain extract all` if
|
||||
desired. Both should now complete successfully on any brain size.
|
||||
|
||||
## What changed
|
||||
|
||||
Two production-blocking bugs are fixed:
|
||||
|
||||
1. **`gbrain extract` no longer hangs on large brains.** The N+1 dedup pre-load
|
||||
that ran 47K serial `getLinks()` calls before any work started is gone. Both
|
||||
engines already enforced uniqueness at the SQL layer; the in-memory dedup was
|
||||
redundant. Combined with new batched 100-row INSERTs, a full re-extract on a
|
||||
47K-page brain drops from "10+ min hang then ~minutes more" to "immediate
|
||||
work, ~30-60s total."
|
||||
|
||||
2. **v0.12.0 schema migration no longer times out on duplicate-heavy brains.**
|
||||
Migration v9 (timeline_dedup_index) and v8 (links uniqueness) now pre-create
|
||||
a btree helper index before the `DELETE ... USING` self-join, then drop it
|
||||
after dedup. Turns O(n²) dedup into O(n log n). On 80K+ duplicate rows the
|
||||
migration completes in under a second instead of timing out at 60 seconds.
|
||||
|
||||
## What you need to do
|
||||
|
||||
### If your v0.12.0 upgrade succeeded — you're already done
|
||||
|
||||
The migration is idempotent. The fix only matters if your migration FAILED on
|
||||
the v0.12.0 upgrade attempt. Run `gbrain init --migrate-only` once to confirm
|
||||
your schema is at version 10, then run `gbrain extract all --dir <brain>` if
|
||||
you want to reuse it now that it's fast.
|
||||
|
||||
### If your v0.12.0 upgrade FAILED on `idx_timeline_dedup` creation
|
||||
|
||||
You may have the brain in a partial-migration state with duplicate rows in
|
||||
`timeline_entries` (or `links`). Run:
|
||||
|
||||
```bash
|
||||
gbrain init --migrate-only
|
||||
```
|
||||
|
||||
Migration v9 will pre-create the helper index, dedup any existing duplicates
|
||||
(now sub-second instead of timing out), drop the helper, and create the unique
|
||||
index. Idempotent — safe to re-run.
|
||||
|
||||
If you previously ran the manual `CREATE TABLE _clean AS SELECT DISTINCT ON
|
||||
... + table swap` workaround Garry posted, your schema should already be at
|
||||
version 10. Confirm with:
|
||||
|
||||
```bash
|
||||
gbrain config get version
|
||||
```
|
||||
|
||||
If it shows `10`, you're done.
|
||||
|
||||
### If you previously did manual SQL surgery on duplicate timeline rows
|
||||
|
||||
The unique index `idx_timeline_dedup` should now be present after the workaround.
|
||||
Re-running `gbrain init --migrate-only` is a no-op for v9 (uses
|
||||
`CREATE UNIQUE INDEX IF NOT EXISTS`). The new migration code adds the helper
|
||||
btree on the dedup columns first — but the helper is dropped at the end of the
|
||||
migration, so even if v9 re-ran (it won't, version is already at 10), it would
|
||||
leave your schema unchanged.
|
||||
|
||||
### Re-running `gbrain extract` on a previously-stuck brain
|
||||
|
||||
This is the most common case. Run:
|
||||
|
||||
```bash
|
||||
gbrain extract all --dir <brain-dir>
|
||||
# or for live brains with no local checkout:
|
||||
gbrain extract all --source db
|
||||
```
|
||||
|
||||
Expect immediate output (`Links: created N from M pages` lines streaming as files
|
||||
process), not a 10-minute hang. On a re-run of a fully-extracted brain you should
|
||||
see `Done: 0 links, 0 timeline entries from N pages` — that's the truthful counter
|
||||
at work, confirming nothing changed.
|
||||
|
||||
## New engine API (informational, optional)
|
||||
|
||||
For plugin authors building integrations on the `BrainEngine` interface, two
|
||||
new methods are available:
|
||||
|
||||
- `addLinksBatch(LinkBatchInput[]) → Promise<number>`
|
||||
- `addTimelineEntriesBatch(TimelineBatchInput[]) → Promise<number>`
|
||||
|
||||
Both return the count of rows actually inserted (excluding ON CONFLICT no-ops
|
||||
and JOIN-dropped rows whose slugs don't exist). Existing per-row `addLink` /
|
||||
`addTimelineEntry` are unchanged — no migration required for plugin code.
|
||||
|
||||
## Verification
|
||||
|
||||
After the migration completes:
|
||||
|
||||
```bash
|
||||
# Confirm schema version
|
||||
gbrain config get version
|
||||
# Expect: 10
|
||||
|
||||
# Confirm the unique indexes exist (Postgres / Supabase only)
|
||||
gbrain stats
|
||||
# Expect: link_count and timeline_entry_count both populated, no duplicates
|
||||
```
|
||||
|
||||
If you hit any issue, file at https://github.com/garrytan/gbrain/issues with the
|
||||
output of `gbrain init --migrate-only` and `gbrain config get version`.
|
||||
+115
-51
@@ -18,11 +18,18 @@
|
||||
|
||||
import { readFileSync, readdirSync, lstatSync, existsSync } from 'fs';
|
||||
import { join, relative, dirname } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/engine.ts';
|
||||
import type { PageType } from '../core/types.ts';
|
||||
import { parseMarkdown } from '../core/markdown.ts';
|
||||
import { extractPageLinks, parseTimelineEntries, inferLinkType } from '../core/link-extraction.ts';
|
||||
|
||||
// Batch size for addLinksBatch / addTimelineEntriesBatch.
|
||||
// Postgres bind-parameter limit is 65535. Links use 4 cols/row → 16K hard ceiling;
|
||||
// timeline uses 5 cols/row → 13K hard ceiling. 100 is conservative on round-trip
|
||||
// count but safe at any future schema width and keeps per-batch error blast radius
|
||||
// small (a malformed row aborts at most 100, not thousands).
|
||||
const BATCH_SIZE = 100;
|
||||
|
||||
// --- Types ---
|
||||
|
||||
export interface ExtractedLink {
|
||||
@@ -312,34 +319,42 @@ async function extractLinksFromDir(
|
||||
const files = walkMarkdownFiles(brainDir);
|
||||
const allSlugs = new Set(files.map(f => f.relPath.replace('.md', '')));
|
||||
|
||||
// Load existing links for O(1) dedup
|
||||
const existing = new Set<string>();
|
||||
try {
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
for (const page of pages) {
|
||||
for (const link of await engine.getLinks(page.slug)) {
|
||||
existing.add(`${link.from_slug}::${link.to_slug}`);
|
||||
}
|
||||
}
|
||||
} catch { /* fresh brain */ }
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
// Without this, the same link extracted from N files would print N times in --dry-run.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
let created = 0;
|
||||
const batch: LinkBatchInput[] = [];
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
try {
|
||||
created += await engine.addLinksBatch(batch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n');
|
||||
} else {
|
||||
console.error(` batch error (${batch.length} link rows lost): ${msg}`);
|
||||
}
|
||||
} finally {
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
try {
|
||||
const content = readFileSync(files[i].path, 'utf-8');
|
||||
const links = extractLinksFromFile(content, files[i].relPath, allSlugs);
|
||||
for (const link of links) {
|
||||
const key = `${link.from_slug}::${link.to_slug}`;
|
||||
if (existing.has(key)) continue;
|
||||
existing.add(key);
|
||||
if (dryRun) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${link.from_slug}::${link.to_slug}::${link.link_type}`;
|
||||
if (dryRunSeen.has(key)) continue;
|
||||
dryRunSeen.add(key);
|
||||
if (!jsonMode) console.log(` ${link.from_slug} → ${link.to_slug} (${link.link_type})`);
|
||||
created++;
|
||||
} else {
|
||||
try {
|
||||
await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type);
|
||||
created++;
|
||||
} catch { /* UNIQUE or page not found */ }
|
||||
batch.push(link);
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
} catch { /* skip unreadable */ }
|
||||
@@ -347,6 +362,7 @@ async function extractLinksFromDir(
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_links', done: i + 1, total: files.length }) + '\n');
|
||||
}
|
||||
}
|
||||
await flush();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
@@ -360,34 +376,41 @@ async function extractTimelineFromDir(
|
||||
): Promise<{ created: number; pages: number }> {
|
||||
const files = walkMarkdownFiles(brainDir);
|
||||
|
||||
// Load existing timeline entries for O(1) dedup
|
||||
const existing = new Set<string>();
|
||||
try {
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
for (const page of pages) {
|
||||
for (const entry of await engine.getTimeline(page.slug)) {
|
||||
existing.add(`${page.slug}::${entry.date}::${entry.summary}`);
|
||||
}
|
||||
}
|
||||
} catch { /* fresh brain */ }
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
let created = 0;
|
||||
const batch: TimelineBatchInput[] = [];
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
try {
|
||||
created += await engine.addTimelineEntriesBatch(batch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n');
|
||||
} else {
|
||||
console.error(` batch error (${batch.length} timeline rows lost): ${msg}`);
|
||||
}
|
||||
} finally {
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
try {
|
||||
const content = readFileSync(files[i].path, 'utf-8');
|
||||
const slug = files[i].relPath.replace('.md', '');
|
||||
for (const entry of extractTimelineFromContent(content, slug)) {
|
||||
const key = `${entry.slug}::${entry.date}::${entry.summary}`;
|
||||
if (existing.has(key)) continue;
|
||||
existing.add(key);
|
||||
if (dryRun) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${entry.slug}::${entry.date}::${entry.summary}`;
|
||||
if (dryRunSeen.has(key)) continue;
|
||||
dryRunSeen.add(key);
|
||||
if (!jsonMode) console.log(` ${entry.slug}: ${entry.date} — ${entry.summary}`);
|
||||
created++;
|
||||
} else {
|
||||
try {
|
||||
await engine.addTimelineEntry(entry.slug, { date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
|
||||
created++;
|
||||
} catch { /* page not in DB or constraint */ }
|
||||
batch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
} catch { /* skip unreadable */ }
|
||||
@@ -395,6 +418,7 @@ async function extractTimelineFromDir(
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_timeline', done: i + 1, total: files.length }) + '\n');
|
||||
}
|
||||
}
|
||||
await flush();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
@@ -455,6 +479,26 @@ async function extractLinksFromDB(
|
||||
const slugList = Array.from(allSlugs);
|
||||
let processed = 0, created = 0;
|
||||
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
const batch: LinkBatchInput[] = [];
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
try {
|
||||
created += await engine.addLinksBatch(batch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n');
|
||||
} else {
|
||||
console.error(` batch error (${batch.length} link rows lost): ${msg}`);
|
||||
}
|
||||
} finally {
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < slugList.length; i++) {
|
||||
const slug = slugList[i];
|
||||
const page = await engine.getPage(slug);
|
||||
@@ -471,7 +515,10 @@ async function extractLinksFromDB(
|
||||
|
||||
for (const c of candidates) {
|
||||
if (!allSlugs.has(c.targetSlug)) continue;
|
||||
if (dryRun) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${slug}::${c.targetSlug}::${c.linkType}`;
|
||||
if (dryRunSeen.has(key)) continue;
|
||||
dryRunSeen.add(key);
|
||||
if (jsonMode) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
action: 'add_link', from: slug, to: c.targetSlug,
|
||||
@@ -482,10 +529,8 @@ async function extractLinksFromDB(
|
||||
}
|
||||
created++;
|
||||
} else {
|
||||
try {
|
||||
await engine.addLink(slug, c.targetSlug, c.context, c.linkType);
|
||||
created++;
|
||||
} catch { /* FK violation or other */ }
|
||||
batch.push({ from_slug: slug, to_slug: c.targetSlug, link_type: c.linkType, context: c.context });
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
processed++;
|
||||
@@ -493,6 +538,7 @@ async function extractLinksFromDB(
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_links_db', done: processed, total: slugList.length }) + '\n');
|
||||
}
|
||||
}
|
||||
await flush();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
@@ -512,6 +558,26 @@ async function extractTimelineFromDB(
|
||||
const slugList = Array.from(allSlugs);
|
||||
let processed = 0, created = 0;
|
||||
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
const batch: TimelineBatchInput[] = [];
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
try {
|
||||
created += await engine.addTimelineEntriesBatch(batch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n');
|
||||
} else {
|
||||
console.error(` batch error (${batch.length} timeline rows lost): ${msg}`);
|
||||
}
|
||||
} finally {
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < slugList.length; i++) {
|
||||
const slug = slugList[i];
|
||||
const page = await engine.getPage(slug);
|
||||
@@ -527,7 +593,10 @@ async function extractTimelineFromDB(
|
||||
const entries = parseTimelineEntries(fullContent);
|
||||
|
||||
for (const entry of entries) {
|
||||
if (dryRun) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${slug}::${entry.date}::${entry.summary}`;
|
||||
if (dryRunSeen.has(key)) continue;
|
||||
dryRunSeen.add(key);
|
||||
if (jsonMode) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
action: 'add_timeline', slug, date: entry.date,
|
||||
@@ -538,14 +607,8 @@ async function extractTimelineFromDB(
|
||||
}
|
||||
created++;
|
||||
} else {
|
||||
try {
|
||||
await engine.addTimelineEntry(
|
||||
slug,
|
||||
{ date: entry.date, summary: entry.summary, detail: entry.detail || '' },
|
||||
{ skipExistenceCheck: true },
|
||||
);
|
||||
created++;
|
||||
} catch { /* dedup constraint or other */ }
|
||||
batch.push({ slug, date: entry.date, summary: entry.summary, detail: entry.detail || '' });
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
processed++;
|
||||
@@ -553,6 +616,7 @@ async function extractTimelineFromDB(
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_timeline_db', done: processed, total: slugList.length }) + '\n');
|
||||
}
|
||||
}
|
||||
await flush();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
|
||||
@@ -39,7 +39,10 @@ import { appendCompletedMigration } from '../../core/preferences.ts';
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
// 10-minute budget. Migrations v8/v9 dedup with helper-index should be sub-second
|
||||
// even on 80K-duplicate brains, but the outer wall-clock cap shouldn't be the
|
||||
// failure mode (the prior 60s ceiling tripped Garry's production upgrade).
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
|
||||
@@ -11,6 +11,23 @@ import type {
|
||||
EngineConfig,
|
||||
} from './types.ts';
|
||||
|
||||
/** Input row for addLinksBatch. Optional fields default to '' (matches NOT NULL DDL). */
|
||||
export interface LinkBatchInput {
|
||||
from_slug: string;
|
||||
to_slug: string;
|
||||
link_type?: string;
|
||||
context?: string;
|
||||
}
|
||||
|
||||
/** Input row for addTimelineEntriesBatch. Optional fields default to '' (matches NOT NULL DDL). */
|
||||
export interface TimelineBatchInput {
|
||||
slug: string;
|
||||
date: string;
|
||||
source?: string;
|
||||
summary: string;
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
/** Maximum results returned by search operations. Internal bulk operations (listPages) are not clamped. */
|
||||
export const MAX_SEARCH_LIMIT = 100;
|
||||
|
||||
@@ -53,6 +70,13 @@ export interface BrainEngine {
|
||||
|
||||
// Links
|
||||
addLink(from: string, to: string, context?: string, linkType?: string): Promise<void>;
|
||||
/**
|
||||
* Bulk insert links via a single multi-row INSERT...SELECT FROM (VALUES) JOIN pages
|
||||
* statement with ON CONFLICT DO NOTHING. Returns the count of rows actually inserted
|
||||
* (RETURNING clause excludes conflicts and JOIN-dropped rows whose slugs don't exist).
|
||||
* Used by extract.ts to avoid 47K sequential round-trips on large brains.
|
||||
*/
|
||||
addLinksBatch(links: LinkBatchInput[]): Promise<number>;
|
||||
/**
|
||||
* Remove links from `from` to `to`. If linkType is provided, only that specific
|
||||
* (from, to, type) row is removed. If omitted, ALL link types between the pair
|
||||
@@ -98,6 +122,13 @@ export interface BrainEngine {
|
||||
entry: TimelineInput,
|
||||
opts?: { skipExistenceCheck?: boolean },
|
||||
): Promise<void>;
|
||||
/**
|
||||
* Bulk insert timeline entries via a single multi-row INSERT...SELECT FROM (VALUES)
|
||||
* JOIN pages statement with ON CONFLICT DO NOTHING. Returns the count of rows
|
||||
* actually inserted (RETURNING excludes conflicts and JOIN-dropped rows whose
|
||||
* slugs don't exist). Used by extract.ts to avoid sequential round-trips.
|
||||
*/
|
||||
addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise<number>;
|
||||
getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]>;
|
||||
|
||||
// Raw data
|
||||
|
||||
+15
-1
@@ -23,7 +23,9 @@ interface Migration {
|
||||
|
||||
// Migrations are embedded here, not loaded from files.
|
||||
// Add new migrations at the end. Never modify existing ones.
|
||||
const MIGRATIONS: Migration[] = [
|
||||
// Exported for tests that structurally assert migration contents (e.g., "v9 must
|
||||
// pre-create idx_timeline_dedup_helper before the DELETE..."). Read-only contract.
|
||||
export const MIGRATIONS: Migration[] = [
|
||||
// Version 1 is the baseline (schema.sql creates everything with IF NOT EXISTS).
|
||||
{
|
||||
version: 2,
|
||||
@@ -230,14 +232,20 @@ const MIGRATIONS: Migration[] = [
|
||||
// Idempotent for both upgrade and fresh-install paths.
|
||||
// Fresh installs already have links_from_to_type_unique from schema.sql; we drop it
|
||||
// (along with the legacy from-to-only constraint) before re-adding it cleanly.
|
||||
// Helper btree on the dedup columns turns the DELETE...USING self-join from O(n²)
|
||||
// into O(n log n). Without it, a brain with 80K+ duplicate link rows hits
|
||||
// Supabase Management API's 60s ceiling during upgrade.
|
||||
sql: `
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_page_id_to_page_id_key;
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_unique;
|
||||
CREATE INDEX IF NOT EXISTS idx_links_dedup_helper
|
||||
ON links(from_page_id, to_page_id, link_type);
|
||||
DELETE FROM links a USING links b
|
||||
WHERE a.from_page_id = b.from_page_id
|
||||
AND a.to_page_id = b.to_page_id
|
||||
AND a.link_type = b.link_type
|
||||
AND a.id > b.id;
|
||||
DROP INDEX IF EXISTS idx_links_dedup_helper;
|
||||
ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique
|
||||
UNIQUE(from_page_id, to_page_id, link_type);
|
||||
`,
|
||||
@@ -247,12 +255,18 @@ const MIGRATIONS: Migration[] = [
|
||||
name: 'timeline_dedup_index',
|
||||
// Idempotent: CREATE UNIQUE INDEX IF NOT EXISTS handles fresh + upgrade.
|
||||
// Dedup any existing duplicates first so the index can be created.
|
||||
// Helper btree turns the DELETE...USING self-join from O(n²) into O(n log n).
|
||||
// Without it, a brain with 80K+ duplicate timeline rows hits Supabase
|
||||
// Management API's 60s ceiling. See migration v8 for the same pattern.
|
||||
sql: `
|
||||
CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper
|
||||
ON timeline_entries(page_id, date, summary);
|
||||
DELETE FROM timeline_entries a USING timeline_entries b
|
||||
WHERE a.page_id = b.page_id
|
||||
AND a.date = b.date
|
||||
AND a.summary = b.summary
|
||||
AND a.id > b.id;
|
||||
DROP INDEX IF EXISTS idx_timeline_dedup_helper;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup
|
||||
ON timeline_entries(page_id, date, summary);
|
||||
`,
|
||||
|
||||
@@ -2,7 +2,7 @@ import { PGlite } from '@electric-sql/pglite';
|
||||
import { vector } from '@electric-sql/pglite/vector';
|
||||
import { pg_trgm } from '@electric-sql/pglite/contrib/pg_trgm';
|
||||
import type { Transaction } from '@electric-sql/pglite';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from './engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
||||
import { runMigrations } from './migrate.ts';
|
||||
import { PGLITE_SCHEMA_SQL } from './pglite-schema.ts';
|
||||
@@ -333,6 +333,29 @@ export class PGLiteEngine implements BrainEngine {
|
||||
);
|
||||
}
|
||||
|
||||
async addLinksBatch(links: LinkBatchInput[]): Promise<number> {
|
||||
if (links.length === 0) return 0;
|
||||
// unnest() pattern: 4 array-typed bound parameters regardless of batch size.
|
||||
// Same shape as PostgresEngine. Avoids the 65535-parameter cap entirely.
|
||||
const fromSlugs = links.map(l => l.from_slug);
|
||||
const toSlugs = links.map(l => l.to_slug);
|
||||
// Normalize optional fields to '' to match per-row addLink + NOT NULL DDL.
|
||||
const linkTypes = links.map(l => l.link_type || '');
|
||||
const contexts = links.map(l => l.context || '');
|
||||
const result = await this.db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context)
|
||||
SELECT f.id, t.id, v.link_type, v.context
|
||||
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[])
|
||||
AS v(from_slug, to_slug, link_type, context)
|
||||
JOIN pages f ON f.slug = v.from_slug
|
||||
JOIN pages t ON t.slug = v.to_slug
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type) DO NOTHING
|
||||
RETURNING 1`,
|
||||
[fromSlugs, toSlugs, linkTypes, contexts]
|
||||
);
|
||||
return result.rows.length;
|
||||
}
|
||||
|
||||
async removeLink(from: string, to: string, linkType?: string): Promise<void> {
|
||||
if (linkType !== undefined) {
|
||||
await this.db.query(
|
||||
@@ -593,6 +616,28 @@ export class PGLiteEngine implements BrainEngine {
|
||||
);
|
||||
}
|
||||
|
||||
async addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise<number> {
|
||||
if (entries.length === 0) return 0;
|
||||
// unnest() pattern: 5 array-typed bound parameters regardless of batch size.
|
||||
const slugs = entries.map(e => e.slug);
|
||||
const dates = entries.map(e => e.date);
|
||||
// Normalize optional fields to '' to match per-row addTimelineEntry + NOT NULL DDL.
|
||||
const sources = entries.map(e => e.source || '');
|
||||
const summaries = entries.map(e => e.summary);
|
||||
const details = entries.map(e => e.detail || '');
|
||||
const result = await this.db.query(
|
||||
`INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
||||
SELECT p.id, v.date::date, v.source, v.summary, v.detail
|
||||
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[])
|
||||
AS v(slug, date, source, summary, detail)
|
||||
JOIN pages p ON p.slug = v.slug
|
||||
ON CONFLICT (page_id, date, summary) DO NOTHING
|
||||
RETURNING 1`,
|
||||
[slugs, dates, sources, summaries, details]
|
||||
);
|
||||
return result.rows.length;
|
||||
}
|
||||
|
||||
async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
|
||||
const limit = opts?.limit || 100;
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import postgres from 'postgres';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from './engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
||||
import { runMigrations } from './migrate.ts';
|
||||
import { SCHEMA_SQL } from './schema-embedded.ts';
|
||||
@@ -372,6 +372,30 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
}
|
||||
|
||||
async addLinksBatch(links: LinkBatchInput[]): Promise<number> {
|
||||
if (links.length === 0) return 0;
|
||||
const sql = this.sql;
|
||||
// unnest() pattern: 4 array-typed bound parameters regardless of batch size.
|
||||
// Avoids the 65535-parameter cap and the postgres-js sql(rows, ...) helper's
|
||||
// identifier-escape gotcha when used inside a (VALUES) subquery.
|
||||
const fromSlugs = links.map(l => l.from_slug);
|
||||
const toSlugs = links.map(l => l.to_slug);
|
||||
// Normalize optional fields to '' to match per-row addLink + NOT NULL DDL.
|
||||
const linkTypes = links.map(l => l.link_type || '');
|
||||
const contexts = links.map(l => l.context || '');
|
||||
const result = await sql`
|
||||
INSERT INTO links (from_page_id, to_page_id, link_type, context)
|
||||
SELECT f.id, t.id, v.link_type, v.context
|
||||
FROM unnest(${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[], ${contexts}::text[])
|
||||
AS v(from_slug, to_slug, link_type, context)
|
||||
JOIN pages f ON f.slug = v.from_slug
|
||||
JOIN pages t ON t.slug = v.to_slug
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type) DO NOTHING
|
||||
RETURNING 1
|
||||
`;
|
||||
return result.length;
|
||||
}
|
||||
|
||||
async removeLink(from: string, to: string, linkType?: string): Promise<void> {
|
||||
const sql = this.sql;
|
||||
if (linkType !== undefined) {
|
||||
@@ -629,6 +653,28 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
}
|
||||
|
||||
async addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise<number> {
|
||||
if (entries.length === 0) return 0;
|
||||
const sql = this.sql;
|
||||
// unnest() pattern: 5 array-typed bound parameters regardless of batch size.
|
||||
const slugs = entries.map(e => e.slug);
|
||||
const dates = entries.map(e => e.date);
|
||||
// Normalize optional fields to '' to match per-row addTimelineEntry + NOT NULL DDL.
|
||||
const sources = entries.map(e => e.source || '');
|
||||
const summaries = entries.map(e => e.summary);
|
||||
const details = entries.map(e => e.detail || '');
|
||||
const result = await sql`
|
||||
INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
||||
SELECT p.id, v.date::date, v.source, v.summary, v.detail
|
||||
FROM unnest(${slugs}::text[], ${dates}::text[], ${sources}::text[], ${summaries}::text[], ${details}::text[])
|
||||
AS v(slug, date, source, summary, detail)
|
||||
JOIN pages p ON p.slug = v.slug
|
||||
ON CONFLICT (page_id, date, summary) DO NOTHING
|
||||
RETURNING 1
|
||||
`;
|
||||
return result.length;
|
||||
}
|
||||
|
||||
async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
|
||||
const sql = this.sql;
|
||||
const limit = opts?.limit || 100;
|
||||
|
||||
@@ -285,6 +285,136 @@ describeE2E('E2E: Timeline', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Batch methods (addLinksBatch / addTimelineEntriesBatch)
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Postgres-engine batch methods use postgres-js's sql(rows, 'col1', ...) helper,
|
||||
// which is structurally different from PGLite's manual $N placeholder construction
|
||||
// (covered in test/pglite-engine.test.ts). These tests verify the postgres-js code
|
||||
// path against a real Postgres against the same invariants.
|
||||
|
||||
describeE2E('E2E: addLinksBatch (postgres-engine)', () => {
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
await importFixtures();
|
||||
});
|
||||
afterAll(teardownDB);
|
||||
|
||||
test('empty batch returns 0 with no DB call', async () => {
|
||||
const engine = getEngine();
|
||||
expect(await engine.addLinksBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('within-batch duplicates dedup via ON CONFLICT (no 21000 cardinality error)', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
// Deterministic cleanup so re-runs aren't perturbed by prior fixture state.
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-dup'`;
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-dup' },
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-dup' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-dup'`;
|
||||
});
|
||||
|
||||
test('rows with missing slug silently dropped by JOIN', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-missing'`;
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/does-not-exist', to_slug: 'companies/novamind', link_type: 'e2e-batch-missing' },
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-missing' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-missing'`;
|
||||
});
|
||||
|
||||
test('half-existing batch returns count of new only', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-half'`;
|
||||
await engine.addLink('people/sarah-chen', 'companies/novamind', 'pre-existing', 'e2e-batch-half');
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-half' },
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'people/marcus-reid', link_type: 'e2e-batch-half' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-half'`;
|
||||
});
|
||||
|
||||
test('missing optional fields normalize to empty strings (NOT NULL safety)', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM links WHERE link_type = ''`;
|
||||
// No link_type, no context — must default to '' to satisfy NOT NULL.
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
const rows = await conn`
|
||||
SELECT link_type, context FROM links
|
||||
WHERE from_page_id = (SELECT id FROM pages WHERE slug = 'people/sarah-chen')
|
||||
AND to_page_id = (SELECT id FROM pages WHERE slug = 'companies/novamind')
|
||||
AND link_type = ''
|
||||
`;
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].context).toBe('');
|
||||
await conn`DELETE FROM links WHERE link_type = ''`;
|
||||
});
|
||||
});
|
||||
|
||||
describeE2E('E2E: addTimelineEntriesBatch (postgres-engine)', () => {
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
await importFixtures();
|
||||
});
|
||||
afterAll(teardownDB);
|
||||
|
||||
test('empty batch returns 0', async () => {
|
||||
const engine = getEngine();
|
||||
expect(await engine.addTimelineEntriesBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('within-batch duplicates dedup via ON CONFLICT', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-dup'`;
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-01', summary: 'e2e-batch-tl-dup' },
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-01', summary: 'e2e-batch-tl-dup' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-dup'`;
|
||||
});
|
||||
|
||||
test('rows with missing slug silently dropped by JOIN', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-missing'`;
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'people/no-such-page', date: '2025-05-02', summary: 'e2e-batch-tl-missing' },
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-02', summary: 'e2e-batch-tl-missing' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-missing'`;
|
||||
});
|
||||
|
||||
test('mix of new + existing returns count of new only', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM timeline_entries WHERE summary IN ('e2e-batch-tl-half-1', 'e2e-batch-tl-half-2')`;
|
||||
await engine.addTimelineEntry('people/sarah-chen', { date: '2025-05-03', summary: 'e2e-batch-tl-half-1' });
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-03', summary: 'e2e-batch-tl-half-1' },
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-04', summary: 'e2e-batch-tl-half-2' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM timeline_entries WHERE summary IN ('e2e-batch-tl-half-1', 'e2e-batch-tl-half-2')`;
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Versions
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
/**
|
||||
* Tests for `gbrain extract --source fs` (the default, FS-walking path).
|
||||
*
|
||||
* Companion to test/extract-db.test.ts. Specifically guards against the
|
||||
* v0.12.0 N+1 hang: extractLinksFromDir / extractTimelineFromDir used to
|
||||
* pre-load the entire dedup set with one engine.getLinks() per page across
|
||||
* engine.listPages(), which on a 47K-page brain meant 47K sequential
|
||||
* round-trips before any work happened.
|
||||
*
|
||||
* Verifies:
|
||||
* 1. Single run extracts the expected links + timeline entries.
|
||||
* 2. Second run reports `created: 0` (proves DO NOTHING in batch + accurate
|
||||
* counter via RETURNING).
|
||||
* 3. --dry-run prints the same link found across multiple files exactly
|
||||
* once (proves the dry-run-only dedup Set works).
|
||||
* 4. Second run wall-clock < 2s (regression guard against any future change
|
||||
* that re-introduces the N+1 read pre-load).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runExtract } from '../src/commands/extract.ts';
|
||||
import type { PageInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let brainDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function truncateAll() {
|
||||
for (const t of ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages']) {
|
||||
await (engine as any).db.exec(`DELETE FROM ${t}`);
|
||||
}
|
||||
}
|
||||
|
||||
const personPage = (title: string, body = ''): PageInput => ({
|
||||
type: 'person', title, compiled_truth: body, timeline: '',
|
||||
});
|
||||
|
||||
const companyPage = (title: string, body = ''): PageInput => ({
|
||||
type: 'company', title, compiled_truth: body, timeline: '',
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-extract-fs-'));
|
||||
});
|
||||
|
||||
function writeFile(rel: string, content: string) {
|
||||
const full = join(brainDir, rel);
|
||||
mkdirSync(join(full, '..'), { recursive: true });
|
||||
writeFileSync(full, content);
|
||||
}
|
||||
|
||||
describe('gbrain extract links --source fs', () => {
|
||||
test('first run inserts links, second run reports 0 (idempotent + truthful counter)', async () => {
|
||||
// Set up brain in DB matching the file structure
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
await engine.putPage('people/bob', personPage('Bob'));
|
||||
await engine.putPage('companies/acme', companyPage('Acme'));
|
||||
|
||||
// Set up matching markdown files on disk
|
||||
writeFile('people/alice.md', '---\ntitle: Alice\n---\n\n[Bob](../people/bob.md) is a friend.\n');
|
||||
writeFile('people/bob.md', '---\ntitle: Bob\n---\n\nWorks at [Acme](../companies/acme.md).\n');
|
||||
writeFile('companies/acme.md', '---\ntitle: Acme\n---\n\nFounded by [Alice](../people/alice.md).\n');
|
||||
|
||||
// First run — write batch path
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
const linksAfter1 = (await engine.getLinks('people/alice'))
|
||||
.concat(await engine.getLinks('people/bob'))
|
||||
.concat(await engine.getLinks('companies/acme'));
|
||||
expect(linksAfter1.length).toBeGreaterThanOrEqual(3);
|
||||
|
||||
// Second run — must dedup via ON CONFLICT and report 0 new (truthful counter)
|
||||
const start = Date.now();
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
const linksAfter2 = (await engine.getLinks('people/alice'))
|
||||
.concat(await engine.getLinks('people/bob'))
|
||||
.concat(await engine.getLinks('companies/acme'));
|
||||
expect(linksAfter2.length).toBe(linksAfter1.length);
|
||||
|
||||
// Perf regression guard: re-run on tiny fixture must not loop through
|
||||
// listPages + per-page getLinks. ~10 files should complete in well under
|
||||
// 2s even on a slow CI box.
|
||||
expect(elapsedMs).toBeLessThan(2000);
|
||||
});
|
||||
|
||||
test('--dry-run dedups duplicate candidates across files (printed once, not N times)', async () => {
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
await engine.putPage('companies/acme', companyPage('Acme'));
|
||||
|
||||
// Same link target appears in 3 different files. The target file must
|
||||
// exist on disk so the FS extractor's allSlugs Set includes it.
|
||||
writeFile('companies/acme.md', '---\ntitle: Acme\n---\n');
|
||||
writeFile('a.md', '[Acme](companies/acme.md)\n');
|
||||
writeFile('b.md', '[Acme](companies/acme.md)\n');
|
||||
writeFile('c.md', '[Acme](companies/acme.md)\n');
|
||||
|
||||
// Capture stdout to check print frequency
|
||||
const lines: string[] = [];
|
||||
const origLog = console.log;
|
||||
console.log = (...args: unknown[]) => { lines.push(args.join(' ')); };
|
||||
try {
|
||||
await runExtract(engine, ['links', '--dry-run', '--dir', brainDir]);
|
||||
} finally {
|
||||
console.log = origLog;
|
||||
}
|
||||
|
||||
// Each (from, to, link_type) tuple should print at most once.
|
||||
// Three distinct from_slugs (a, b, c) all link to companies/acme, so
|
||||
// we expect 3 link lines (one per source file), not 9.
|
||||
const linkLines = lines.filter(l => l.includes('→') && l.includes('companies/acme'));
|
||||
expect(linkLines.length).toBe(3);
|
||||
|
||||
// No actual writes happened
|
||||
const links = await engine.getLinks('companies/acme');
|
||||
expect(links.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain extract timeline --source fs', () => {
|
||||
test('first run inserts entries, second run reports 0 (idempotent + truthful counter)', async () => {
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
|
||||
writeFile('people/alice.md', `---
|
||||
title: Alice
|
||||
---
|
||||
|
||||
## Timeline
|
||||
|
||||
- **2024-01-15** | source — Founded NovaMind
|
||||
- **2024-06-01** | source — Raised seed round
|
||||
`);
|
||||
|
||||
await runExtract(engine, ['timeline', '--dir', brainDir]);
|
||||
const after1 = await engine.getTimeline('people/alice');
|
||||
expect(after1.length).toBe(2);
|
||||
|
||||
const start = Date.now();
|
||||
await runExtract(engine, ['timeline', '--dir', brainDir]);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
const after2 = await engine.getTimeline('people/alice');
|
||||
expect(after2.length).toBe(2);
|
||||
|
||||
expect(elapsedMs).toBeLessThan(2000);
|
||||
});
|
||||
});
|
||||
+186
-3
@@ -1,5 +1,6 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { LATEST_VERSION } from '../src/core/migrate.ts';
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { LATEST_VERSION, runMigrations, MIGRATIONS } from '../src/core/migrate.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
describe('migrate', () => {
|
||||
test('LATEST_VERSION is a number >= 1', () => {
|
||||
@@ -8,10 +9,192 @@ describe('migrate', () => {
|
||||
});
|
||||
|
||||
test('runMigrations is exported and callable', async () => {
|
||||
const { runMigrations } = await import('../src/core/migrate.ts');
|
||||
expect(typeof runMigrations).toBe('function');
|
||||
});
|
||||
|
||||
// Integration tests for actual migration execution require DATABASE_URL
|
||||
// and are covered in the E2E suite (test/e2e/mechanical.test.ts)
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// REGRESSION TESTS — migrations v8 + v9 perf on duplicate-heavy tables
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Garry's production brain hit Supabase Management API's 60s ceiling because
|
||||
// the DELETE...USING self-join in migrations v8 + v9 was O(n²) without an
|
||||
// index on the dedup columns. The fix pre-creates a btree helper index
|
||||
// before the DELETE, then drops it. These tests guard against any future
|
||||
// change that re-introduces the missing helper index.
|
||||
//
|
||||
// Two-layer guard:
|
||||
// 1. Structural — assert the migration SQL literally contains the helper
|
||||
// CREATE INDEX + DROP INDEX (deterministic, fast, catches the regression
|
||||
// even at 0-row scale where wall-clock can't distinguish O(n²) from O(1)).
|
||||
// 2. Behavioral — populate 1000 duplicates and assert the migration completes
|
||||
// under the wall-clock cap. Sanity check at small scale; the structural
|
||||
// assertion is the real guard.
|
||||
|
||||
describe('migrations v8 + v9 — structural guard for helper-index fix', () => {
|
||||
test('migration v8 SQL contains idx_links_dedup_helper CREATE+DROP around the DELETE', () => {
|
||||
const v8 = MIGRATIONS.find(m => m.version === 8);
|
||||
expect(v8).toBeDefined();
|
||||
const sql = v8!.sql;
|
||||
|
||||
// The fix must: (a) create the helper btree, (b) DELETE...USING, (c) drop the helper, (d) add the unique constraint.
|
||||
// If anyone reorders or removes the helper-index lines, this fails.
|
||||
expect(sql).toContain('CREATE INDEX IF NOT EXISTS idx_links_dedup_helper');
|
||||
expect(sql).toContain('ON links(from_page_id, to_page_id, link_type)');
|
||||
expect(sql).toContain('DROP INDEX IF EXISTS idx_links_dedup_helper');
|
||||
expect(sql).toContain('DELETE FROM links a USING links b');
|
||||
expect(sql).toContain('ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique');
|
||||
|
||||
// Order matters: CREATE INDEX before DELETE, DROP INDEX after DELETE, before ADD CONSTRAINT.
|
||||
const createIdx = sql.indexOf('CREATE INDEX IF NOT EXISTS idx_links_dedup_helper');
|
||||
const deleteUsing = sql.indexOf('DELETE FROM links a USING links b');
|
||||
const dropIdx = sql.indexOf('DROP INDEX IF EXISTS idx_links_dedup_helper');
|
||||
const addConstraint = sql.indexOf('ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique');
|
||||
expect(createIdx).toBeLessThan(deleteUsing);
|
||||
expect(deleteUsing).toBeLessThan(dropIdx);
|
||||
expect(dropIdx).toBeLessThan(addConstraint);
|
||||
});
|
||||
|
||||
test('migration v9 SQL contains idx_timeline_dedup_helper CREATE+DROP around the DELETE', () => {
|
||||
const v9 = MIGRATIONS.find(m => m.version === 9);
|
||||
expect(v9).toBeDefined();
|
||||
const sql = v9!.sql;
|
||||
|
||||
expect(sql).toContain('CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper');
|
||||
expect(sql).toContain('ON timeline_entries(page_id, date, summary)');
|
||||
expect(sql).toContain('DROP INDEX IF EXISTS idx_timeline_dedup_helper');
|
||||
expect(sql).toContain('DELETE FROM timeline_entries a USING timeline_entries b');
|
||||
expect(sql).toContain('CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup');
|
||||
|
||||
const createHelper = sql.indexOf('CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper');
|
||||
const deleteUsing = sql.indexOf('DELETE FROM timeline_entries a USING timeline_entries b');
|
||||
const dropHelper = sql.indexOf('DROP INDEX IF EXISTS idx_timeline_dedup_helper');
|
||||
const createUnique = sql.indexOf('CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup');
|
||||
expect(createHelper).toBeLessThan(deleteUsing);
|
||||
expect(deleteUsing).toBeLessThan(dropHelper);
|
||||
expect(dropHelper).toBeLessThan(createUnique);
|
||||
});
|
||||
});
|
||||
|
||||
describe('migrate: v8 (links_dedup) regression — must be fast on 1K duplicate rows', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('1000 duplicate links dedup completes in <5s and leaves table deduped', async () => {
|
||||
// Set up: drop the unique constraint so duplicates can be inserted, then reset
|
||||
// version so v8 re-runs. Schema-embedded.ts already has the constraint, so
|
||||
// initSchema() above set it up; explicit DROP makes the test premise valid.
|
||||
const db = (engine as any).db;
|
||||
await db.exec(`ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_unique`);
|
||||
|
||||
// Two pages so the FK is satisfied
|
||||
await engine.putPage('p/from', { type: 'concept', title: 'F', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('p/to', { type: 'concept', title: 'T', compiled_truth: '', timeline: '' });
|
||||
const fromId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/from'`)).rows[0].id;
|
||||
const toId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/to'`)).rows[0].id;
|
||||
|
||||
// Insert 1000 duplicates of the same (from, to, type) row
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
await db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context) VALUES ($1, $2, $3, $4)`,
|
||||
[fromId, toId, 'mention', `dup-${i}`]
|
||||
);
|
||||
}
|
||||
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c;
|
||||
expect(beforeCount).toBe(1000);
|
||||
|
||||
// Reset version to 7 so v8 + v9 + v10 re-run
|
||||
await engine.setConfig('version', '7');
|
||||
|
||||
// Run migrations and assert wall-clock + correctness
|
||||
const start = Date.now();
|
||||
await runMigrations(engine);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
expect(elapsedMs).toBeLessThan(5000);
|
||||
|
||||
const afterCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c;
|
||||
expect(afterCount).toBe(1); // deduped to one row
|
||||
|
||||
// Unique constraint reinstated
|
||||
const constraints = (await db.query(`
|
||||
SELECT conname FROM pg_constraint
|
||||
WHERE conrelid = 'links'::regclass AND contype = 'u'
|
||||
`)).rows;
|
||||
expect(constraints.some((c: { conname: string }) => c.conname === 'links_from_to_type_unique')).toBe(true);
|
||||
|
||||
// Helper index was dropped after dedup
|
||||
const helperIdx = (await db.query(`
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'links' AND indexname = 'idx_links_dedup_helper'
|
||||
`)).rows;
|
||||
expect(helperIdx.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 1K duplicate rows', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('1000 duplicate timeline entries dedup completes in <5s and leaves table deduped', async () => {
|
||||
const db = (engine as any).db;
|
||||
await db.exec(`DROP INDEX IF EXISTS idx_timeline_dedup`);
|
||||
|
||||
await engine.putPage('p/timeline', { type: 'concept', title: 'TL', compiled_truth: '', timeline: '' });
|
||||
const pageId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/timeline'`)).rows[0].id;
|
||||
|
||||
// Insert 1000 duplicates of the same (page_id, date, summary) row
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
await db.query(
|
||||
`INSERT INTO timeline_entries (page_id, date, source, summary, detail) VALUES ($1, $2::date, $3, $4, $5)`,
|
||||
[pageId, '2024-01-15', `src-${i}`, 'Founded NovaMind', `detail-${i}`]
|
||||
);
|
||||
}
|
||||
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c;
|
||||
expect(beforeCount).toBe(1000);
|
||||
|
||||
await engine.setConfig('version', '7');
|
||||
|
||||
const start = Date.now();
|
||||
await runMigrations(engine);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
expect(elapsedMs).toBeLessThan(5000);
|
||||
|
||||
const afterCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c;
|
||||
expect(afterCount).toBe(1);
|
||||
|
||||
const uniqueIdx = (await db.query(`
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'timeline_entries' AND indexname = 'idx_timeline_dedup'
|
||||
`)).rows;
|
||||
expect(uniqueIdx.length).toBe(1);
|
||||
|
||||
const helperIdx = (await db.query(`
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'timeline_entries' AND indexname = 'idx_timeline_dedup_helper'
|
||||
`)).rows;
|
||||
expect(helperIdx.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -350,6 +350,118 @@ describe('PGLiteEngine: Timeline', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Batch methods (addLinksBatch / addTimelineEntriesBatch)
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
describe('PGLiteEngine: addLinksBatch', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
await engine.putPage('a', { type: 'concept', title: 'A', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('b', { type: 'concept', title: 'B', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('c', { type: 'concept', title: 'C', compiled_truth: '', timeline: '' });
|
||||
});
|
||||
|
||||
test('empty batch returns 0 with no DB call', async () => {
|
||||
expect(await engine.addLinksBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('batch of 1 with missing optional fields inserts row with empty defaults', async () => {
|
||||
const inserted = await engine.addLinksBatch([{ from_slug: 'a', to_slug: 'b' }]);
|
||||
expect(inserted).toBe(1);
|
||||
const links = await engine.getLinks('a');
|
||||
expect(links.length).toBe(1);
|
||||
expect(links[0].context).toBe('');
|
||||
expect(links[0].link_type).toBe('');
|
||||
});
|
||||
|
||||
test('within-batch duplicates are deduped via ON CONFLICT (no 21000 error)', async () => {
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'a', to_slug: 'b', link_type: 'mention' },
|
||||
{ from_slug: 'a', to_slug: 'b', link_type: 'mention' },
|
||||
{ from_slug: 'a', to_slug: 'c', link_type: 'mention' },
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
});
|
||||
|
||||
test('rows with missing slug are silently dropped by JOIN', async () => {
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'doesnt-exist', to_slug: 'b' },
|
||||
{ from_slug: 'a', to_slug: 'b' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
|
||||
test('half-existing batch returns count of new only', async () => {
|
||||
await engine.addLink('a', 'b', '', 'mention');
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'a', to_slug: 'b', link_type: 'mention' },
|
||||
{ from_slug: 'a', to_slug: 'c', link_type: 'mention' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
|
||||
test('batch of 100 fresh rows returns 100', async () => {
|
||||
// Create 100 target pages
|
||||
for (let i = 0; i < 100; i++) {
|
||||
await engine.putPage(`target/${i}`, { type: 'concept', title: `T${i}`, compiled_truth: '', timeline: '' });
|
||||
}
|
||||
const batch = Array.from({ length: 100 }, (_, i) => ({
|
||||
from_slug: 'a', to_slug: `target/${i}`, link_type: 'mention',
|
||||
}));
|
||||
expect(await engine.addLinksBatch(batch)).toBe(100);
|
||||
});
|
||||
});
|
||||
|
||||
describe('PGLiteEngine: addTimelineEntriesBatch', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
await engine.putPage('p1', { type: 'concept', title: 'P1', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('p2', { type: 'concept', title: 'P2', compiled_truth: '', timeline: '' });
|
||||
});
|
||||
|
||||
test('empty batch returns 0', async () => {
|
||||
expect(await engine.addTimelineEntriesBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('batch of 1 with missing optionals inserts with empty defaults', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
const entries = await engine.getTimeline('p1');
|
||||
expect(entries.length).toBe(1);
|
||||
expect(entries[0].source).toBe('');
|
||||
expect(entries[0].detail).toBe('');
|
||||
});
|
||||
|
||||
test('within-batch duplicates are deduped via ON CONFLICT', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
{ slug: 'p1', date: '2024-02-01', summary: 'Launched' },
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
});
|
||||
|
||||
test('rows with missing slug are silently dropped by JOIN', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'no-such-page', date: '2024-01-15', summary: 'Phantom' },
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Real' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
|
||||
test('mix of new + existing returns count of new only', async () => {
|
||||
await engine.addTimelineEntry('p1', { date: '2024-01-15', summary: 'Founded' });
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
{ slug: 'p1', date: '2024-02-01', summary: 'Launched' },
|
||||
{ slug: 'p2', date: '2024-03-01', summary: 'Spun off' },
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Raw Data, Versions, Config, IngestLog
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user