mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
Merge remote-tracking branch 'origin/master' into garrytan/gbrain-evals
# Conflicts: # package.json
This commit is contained in:
Garry Tan
+134
@@ -2,6 +2,140 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.12.2] - 2026-04-19
|
||||
|
||||
## **Postgres frontmatter queries actually work now.**
|
||||
## **Wiki articles stop disappearing when you import them.**
|
||||
|
||||
This is a data-correctness hotfix for the `v0.12.0`-and-earlier Postgres-backed brains. If you run gbrain on Postgres or Supabase, you've been losing data without knowing it. PGLite users were unaffected. Upgrade auto-repairs your existing rows. Lands on top of v0.12.1 (extract N+1 fix + migration timeout fix) — pull `gbrain upgrade` and you get both.
|
||||
|
||||
### What was broken
|
||||
|
||||
**Frontmatter columns were silently stored as quoted strings, not JSON.** Every `put_page` wrote `frontmatter` to Postgres via `${JSON.stringify(value)}::jsonb` — postgres.js v3 stringified again on the wire, so the column ended up holding `"\"{\\\"author\\\":\\\"garry\\\"}\""` instead of `{"author":"garry"}`. Every `frontmatter->>'key'` query returned NULL. GIN indexes on JSONB were inert. Same bug on `raw_data.data`, `ingest_log.pages_updated`, `files.metadata`, and `page_versions.frontmatter`. PGLite hid this entirely (different driver path) — which is exactly why it slipped past the existing test suite.
|
||||
|
||||
**Wiki articles got truncated by 83% on import.** `splitBody` treated *any* standalone `---` line in body content as a timeline separator. Discovered by @knee5 migrating a 1,991-article wiki where a 23,887-byte article landed in the DB as 593 bytes (4,856 of 6,680 wikilinks lost).
|
||||
|
||||
**`/wiki/` subdirectories silently typed as `concept`.** Articles under `/wiki/analysis/`, `/wiki/guides/`, `/wiki/hardware/`, `/wiki/architecture/`, and `/writing/` defaulted to `type='concept'` — type-filtered queries lost everything in those buckets.
|
||||
|
||||
**pgvector embeddings sometimes returned as strings → NaN search scores.** Discovered by @leonardsellem on Supabase, where `getEmbeddingsByChunkIds` returned `"[0.1,0.2,…]"` instead of `Float32Array`, producing `[NaN]` query scores.
|
||||
|
||||
### What you can do now that you couldn't before
|
||||
|
||||
- **`frontmatter->>'author'` returns `garry`, not NULL.** GIN indexes work. Postgres queries by frontmatter key actually retrieve pages.
|
||||
- **Wiki articles round-trip intact.** Markdown horizontal rules in body text are horizontal rules, not timeline separators.
|
||||
- **Recover already-truncated pages with `gbrain sync --full`.** Re-import from your source-of-truth markdown rebuilds `compiled_truth` correctly.
|
||||
- **Search scores stop going `NaN` on Supabase.** Cosine rescoring sees real `Float32Array` embeddings.
|
||||
- **Type-filtered queries find your wiki articles.** `/wiki/analysis/` becomes type `analysis`, `/writing/` becomes `writing`, etc.
|
||||
|
||||
### How to upgrade
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
The `v0.12.2` orchestrator runs automatically: applies any schema changes, then `gbrain repair-jsonb` rewrites every double-encoded row in place using `jsonb_typeof = 'string'` as the guard. Idempotent — re-running is a no-op. PGLite engines short-circuit cleanly. Batches well on large brains.
|
||||
|
||||
If you want to recover pages that were truncated by the splitBody bug:
|
||||
|
||||
```bash
|
||||
gbrain sync --full
|
||||
```
|
||||
|
||||
That re-imports every page from disk, so the new `splitBody` rebuilds the full `compiled_truth` correctly.
|
||||
|
||||
### What's new under the hood
|
||||
|
||||
- **`gbrain repair-jsonb`** — standalone command for the JSONB fix. Run it manually if needed; the migration runs it automatically. `--dry-run` shows what would be repaired without touching data. `--json` for scripting.
|
||||
- **CI grep guard** at `scripts/check-jsonb-pattern.sh` — fails the build if anyone reintroduces the `${JSON.stringify(x)}::jsonb` interpolation pattern. Wired into `bun test` so it runs on every CI invocation.
|
||||
- **New E2E regression test** at `test/e2e/postgres-jsonb.test.ts` — round-trips all four JSONB write sites against real Postgres and asserts `jsonb_typeof = 'object'` plus `->>` returns the expected scalar. The test that should have caught the original bug.
|
||||
- **Wikilink extraction** — `[[page]]` and `[[page|Display Text]]` syntaxes now extracted alongside standard `[text](page.md)` markdown links. Includes ancestor-search resolution for wiki KBs where authors omit one or more leading `../`.
|
||||
|
||||
### Migration scope
|
||||
|
||||
The repair touches five JSONB columns:
|
||||
- `pages.frontmatter`
|
||||
- `raw_data.data`
|
||||
- `ingest_log.pages_updated`
|
||||
- `files.metadata`
|
||||
- `page_versions.frontmatter` (downstream of `pages.frontmatter` via INSERT...SELECT)
|
||||
|
||||
Other JSONB columns in the schema (`minion_jobs.{data,result,progress,stacktrace}`, `minion_inbox.payload`) were always written via the parameterized `$N::jsonb` form so they were never affected.
|
||||
|
||||
### Behavior changes (read this if you upgrade)
|
||||
|
||||
`splitBody` now requires an explicit sentinel for timeline content. Recognized markers (in priority order):
|
||||
1. `<!-- timeline -->` (preferred — what `serializeMarkdown` emits)
|
||||
2. `--- timeline ---` (decorated separator)
|
||||
3. `---` directly before `## Timeline` or `## History` heading (backward-compat fallback)
|
||||
|
||||
If you intentionally used a plain `---` to mark your timeline section in source markdown, add `<!-- timeline -->` above it manually. The fallback covers the common case (`---` followed by `## Timeline`).
|
||||
|
||||
### Attribution
|
||||
|
||||
Built from community PRs #187 (@knee5) and #175 (@leonardsellem). The original PRs reported the bugs and proposed the fixes; this release re-implements them on top of the v0.12.0 knowledge graph release with expanded migration scope, schema audit (all 5 affected columns vs the 3 originally reported), engine-aware behavior, CI grep guard, and an E2E regression test that should have caught this in the first place. Codex outside-voice review during planning surfaced the missed `page_versions.frontmatter` propagation path and the noisy-truncated-diagnostic anti-pattern that was dropped from this scope. Thanks for finding the bugs and providing the recovery path — both PRs left work to do but the foundation was right.
|
||||
|
||||
Co-Authored-By: @knee5 (PR #187 — splitBody, inferType wiki, JSONB triple-fix)
|
||||
Co-Authored-By: @leonardsellem (PR #175 — parseEmbedding, getEmbeddingsByChunkIds fix)
|
||||
|
||||
## [0.12.1] - 2026-04-19
|
||||
|
||||
## **Extract no longer hangs on large brains.**
|
||||
## **v0.12.0 upgrade no longer times out on duplicates.**
|
||||
|
||||
Two production-blocking bugs Garry hit on his 47K-page brain on April 18. `gbrain extract` was effectively unusable on any brain with 20K+ existing links or timeline entries — it pre-loaded the entire dedup set with one `getLinks()` call per page over the Supabase pooler, hanging for 10+ minutes producing zero output before any work started. The v0.12.0 schema migration that creates `idx_timeline_dedup` was failing on brains with pre-existing duplicate timeline rows because the `DELETE ... USING` self-join was O(n²) without an index, hitting Supabase Management API's 60-second ceiling on 80K+ duplicates. Both bugs end here.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Measured on the new `test/extract-fs.test.ts` and `test/migrate.test.ts` regression suites, plus 73 E2E tests against real Postgres+pgvector. Reproducible: `bun test` + `bun run test:e2e`.
|
||||
|
||||
| Metric | BEFORE v0.12.1 | AFTER v0.12.1 | Δ |
|
||||
|-----------------------------------------|--------------------|--------------------|--------------------|
|
||||
| extract hang on 47K-page brain | 10+ min, zero output | immediate work, ~30-60s wall clock | usable |
|
||||
| DB round-trips per re-extract | 47K reads + 235K writes | 0 reads + ~2.4K writes | **~99% fewer** |
|
||||
| v0.12.0 migration on 80K duplicate rows | timed out at 60s | completes <1s | **~60x+ faster** |
|
||||
| Re-run on already-extracted brain | 235K row-writes | 0 row-writes | true no-op |
|
||||
| Tests | 1297 unit / 105 E2E | **1412 unit / 119 E2E** | +115 unit / +14 E2E |
|
||||
| `created` counter on re-runs | "5000 created" (lie) | "0 created" (truth)| accurate |
|
||||
|
||||
Per-batch round-trip math: a re-extract on a 47K-page brain with ~5 links per page used to do 235K sequential round-trips over the Supabase pooler. With 100-row batched INSERTs it does ~2,400. The hang came from the read pre-load (47K serial `getLinks()` calls), which is now gone entirely. The DB enforces uniqueness via `ON CONFLICT DO NOTHING`.
|
||||
|
||||
### What this means for GBrain users
|
||||
|
||||
If you've been afraid to re-run `gbrain extract` because it might never finish, that's over. The command starts producing output immediately, batch-writes 100 rows per round-trip, and reports a truthful insert count even on re-runs. If your v0.12.0 upgrade got stuck on the timeline migration (or you had to manually run `CREATE TABLE ... AS SELECT DISTINCT ON ...` to unblock it), the next `gbrain init --migrate-only` is sub-second. Run `gbrain extract all` on your largest brain and watch it actually work.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Performance
|
||||
|
||||
- **`gbrain extract` no longer pre-loads the dedup set.** Removed the N+1 read loop in `extractLinksFromDir`, `extractTimelineFromDir`, `extractLinksFromDB`, and `extractTimelineFromDB` that called `engine.getLinks(slug)` (or `getTimeline`) once per page across `engine.listPages({ limit: 100000 })`. On a 47K-page brain that was 47K serial network round-trips before the first file was even read. Both engines already enforced uniqueness at the SQL layer (`UNIQUE(from_page_id, to_page_id, link_type)` on `links`, `idx_timeline_dedup` on `timeline_entries`); the in-memory dedup `Set` was redundant insurance that turned into the bottleneck.
|
||||
- **Batched multi-row INSERTs replace per-row writes.** All four extract paths now buffer 100 candidates and flush via new `addLinksBatch` / `addTimelineEntriesBatch` engine methods. Round-trips drop ~100x: ~235K → ~2,400 per full re-extract. Each batch uses `INSERT ... SELECT FROM unnest($1::text[], $2::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4 (links) or 5 (timeline) array-typed bound parameters regardless of batch size, sidestepping Postgres's 65535-parameter cap entirely. PGLite uses the same SQL shape with manual `$N` placeholders.
|
||||
|
||||
#### Correctness
|
||||
|
||||
- **`created` counter is now truthful on re-runs.** Returns count of rows actually inserted (via `RETURNING 1` row count), not "calls that didn't throw." A re-run on a fully-extracted brain prints `Done: 0 links, 0 timeline entries from 47000 pages`. Before this release it would print `Done: 5000 links` while inserting zero new rows.
|
||||
- **`--dry-run` deduplicates candidates across files.** A link extracted from 3 different markdown files now prints exactly once in `--dry-run` output, matching what the batch insert would actually create. Before this release the dedup was tied to the now-deleted DB pre-load, so dry-run would over-print.
|
||||
- **Whole-batch errors are visible in both JSON and human modes.** When a batch flush fails (DB connection drop, malformed row), the error prints to stderr in JSON mode AND to console in human mode, with the lost-row count. No more silent loss of 100 rows because of one bad row.
|
||||
|
||||
#### Schema migrations — v0.12.0 upgrade is now sub-second on duplicate-heavy brains
|
||||
|
||||
- **Migration v9 (timeline_entries) and v8 (links) pre-create a btree helper index** on the dedup columns before the `DELETE ... USING` self-join runs. Turns the O(n²) sequential-scan dedup into O(n log n) index-backed dedup. On 80K+ duplicate rows the migration completes in well under a second instead of timing out at 60s. The helper index is dropped after dedup, leaving the original schema unchanged. Same fix applied defensively to migration v8 — Garry's brain didn't trip it (links had fewer duplicates) but the same trap was loaded.
|
||||
- **`phaseASchema` timeout in the v0.12.0 orchestrator bumped 60s → 600s.** Belt-and-suspenders: the helper-index fix should make dedup sub-second on most brains, but the outer wall-clock budget shouldn't be the failure mode for unforeseen slowness.
|
||||
|
||||
#### New engine API
|
||||
|
||||
- **`addLinksBatch(LinkBatchInput[]) → Promise<number>`** and **`addTimelineEntriesBatch(TimelineBatchInput[]) → Promise<number>`** on both `PostgresEngine` and `PGLiteEngine`. Returns count of actually-inserted rows (excluding ON CONFLICT no-ops and JOIN-dropped rows whose slugs don't exist). Per-row `addLink` / `addTimelineEntry` are unchanged — all 10 existing call sites compile and behave identically. Plugin authors building agent integrations on `BrainEngine` can adopt the batch methods at their own pace.
|
||||
|
||||
#### Tests
|
||||
|
||||
- **Migration regression tests guard the fix structurally + behaviorally.** New `test/migrate.test.ts` cases assert the v8 + v9 SQL literally contains the helper `CREATE INDEX IF NOT EXISTS ... DROP INDEX IF EXISTS` sequence in the right order (deterministic, fast, catches a regression even at 0-row scale where wall-clock can't distinguish O(n²) from O(1)) AND that the migration completes under wall-clock cap on 1000-row fixtures.
|
||||
- **`test/extract-fs.test.ts` (new file)** covers the FS-source extract path end-to-end on PGLite: first-run inserts, second-run reports zero, dry-run dedups duplicate candidates across 3 files into one printed line, second-run perf regression guard.
|
||||
- **9 new E2E tests for the postgres-engine batch methods** in `test/e2e/mechanical.test.ts`. The postgres-js bind path is structurally different from PGLite's (array params via `unnest()` vs manual `$N` placeholders) and gets its own coverage against real Postgres+pgvector.
|
||||
- **11 new PGLite batch method tests** in `test/pglite-engine.test.ts` (empty batch, missing optionals normalize to empty strings, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch returns count of new only, batch of 100).
|
||||
|
||||
#### Pre-ship review
|
||||
|
||||
This release was reviewed by `/plan-eng-review` (5 issues, all addressed including a P0 plan reshape that dropped a redundant orchestrator phase in favor of fixing migration v9 directly), `/codex` outside-voice review on the plan (15 findings, all P1 + P2 incorporated — most consequential: forced a cleaner separation between per-row API stability and new batch APIs so all 10 existing `addLink` callers stay untouched), and 5 specialist subagents (testing, maintainability, performance, security, data-migration) at ship time. The testing specialist caught a real bug in the postgres-engine batch SQL: postgres-js's `sql(rows, ...)` helper doesn't compose with `(VALUES) AS v(...)` JOIN syntax the way originally written. Switched to the cleaner `unnest()` array-parameter pattern in both engines, verified end-to-end against a real Postgres+pgvector container.
|
||||
|
||||
## [0.12.0] - 2026-04-18
|
||||
|
||||
## **The graph wires itself.**
|
||||
|
||||
@@ -23,11 +23,11 @@ strict behavior when unset.
|
||||
## Key files
|
||||
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`. `OperationContext.remote` flags untrusted callers.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`).
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 38 BrainEngine methods
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders.
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts
|
||||
- `src/core/db.ts` — Connection management, schema initialization
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
|
||||
@@ -48,7 +48,7 @@ strict behavior when unset.
|
||||
- `src/core/transcription.ts` — Audio transcription: Groq Whisper (default), OpenAI fallback, ffmpeg segmentation for >25MB
|
||||
- `src/core/enrichment-service.ts` — Global enrichment service: entity slug generation, tier auto-escalation, batch throttling
|
||||
- `src/core/data-research.ts` — Recipe validation, field extraction (MRR/ARR regex), dedup, tracker parsing, HTML stripping
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout)
|
||||
- `src/commands/extract.ts` — `gbrain extract links|timeline|all [--source fs|db]`: batch link/timeline extraction. fs walks markdown files, db walks pages from the engine (mutation-immune snapshot iteration; use this for live brains with no local checkout). As of v0.12.1 there is no in-memory dedup pre-load — candidates are buffered 100 at a time and flushed via `addLinksBatch` / `addTimelineEntriesBatch`; `ON CONFLICT DO NOTHING` enforces uniqueness at the DB layer, and the `created` counter returns real rows inserted (truthful on re-runs).
|
||||
- `src/commands/graph-query.ts` — `gbrain graph-query <slug> [--type T] [--depth N] [--direction in|out|both]`: typed-edge relationship traversal (renders indented tree)
|
||||
- `src/core/link-extraction.ts` — shared library for the v0.12.0 graph layer. extractEntityRefs (canonical, replaces backlinks.ts duplicate), extractPageLinks, inferLinkType heuristics (attended/works_at/invested_in/founded/advises/source/mentions), parseTimelineEntries, isAutoLinkEnabled config helper. Used by extract.ts, operations.ts auto-link post-hook, and backlinks.ts.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types)
|
||||
@@ -61,7 +61,10 @@ strict behavior when unset.
|
||||
- `src/mcp/server.ts` — MCP stdio server (generated from operations)
|
||||
- `src/commands/auth.ts` — Standalone token management (create/list/revoke/test)
|
||||
- `src/commands/upgrade.ts` — Self-update CLI. `runPostUpgrade()` enumerates migrations from the TS registry (src/commands/migrations/index.ts) and tail-calls `runApplyMigrations(['--yes', '--non-interactive'])` so the mechanical side of every outstanding migration runs unconditionally.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). All orchestrators are idempotent and resumable from `partial` status.
|
||||
- `src/commands/migrations/` — TS migration registry (compiled into the binary; no filesystem walk of `skills/migrations/*.md` needed at runtime). `index.ts` lists migrations in semver order. `v0_11_0.ts` = Minions adoption orchestrator (8 phases). `v0_12_0.ts` = Knowledge Graph auto-wire orchestrator (5 phases: schema → config check → backfill links → backfill timeline → verify). `phaseASchema` has a 600s timeout (bumped from 60s in v0.12.1 for duplicate-heavy brains). `v0_12_2.ts` = JSONB double-encode repair orchestrator (4 phases: schema → repair-jsonb → verify → record). All orchestrators are idempotent and resumable from `partial` status.
|
||||
- `src/commands/repair-jsonb.ts` — `gbrain repair-jsonb [--dry-run] [--json]`: rewrites `jsonb_typeof='string'` rows in place across 5 affected columns (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter). Fixes v0.12.0 double-encode bug on Postgres; PGLite no-ops. Idempotent.
|
||||
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus the existing people/companies/deals/etc heuristics).
|
||||
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces the `${JSON.stringify(x)}::jsonb` interpolation pattern (which postgres.js v3 double-encodes). Wired into `bun test`.
|
||||
- `docs/UPGRADING_DOWNSTREAM_AGENTS.md` — Patches for downstream agent skill forks (Wintermute etc.) to apply when upgrading. Each release appends a new section. v0.10.3 includes diffs for brain-ops, meeting-ingestion, signal-detector, enrich.
|
||||
- `src/core/schema-embedded.ts` — AUTO-GENERATED from schema.sql (run `bun run build:schema`)
|
||||
- `src/schema.sql` — Full Postgres + pgvector DDL (source of truth, generates schema-embedded.ts)
|
||||
@@ -129,9 +132,12 @@ Key commands added for Minions (job queue):
|
||||
- `gbrain jobs stats` — job health dashboard
|
||||
- `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only)
|
||||
|
||||
Key commands added in v0.12.2:
|
||||
- `gbrain repair-jsonb [--dry-run] [--json]` — repair double-encoded JSONB rows left over from v0.12.0-and-earlier Postgres writes. Idempotent; PGLite no-ops. The `v0_12_2` migration runs this automatically on `gbrain upgrade`.
|
||||
|
||||
## Testing
|
||||
|
||||
`bun test` runs all tests. After the v0.12.0 release: ~74 unit test files + 8 E2E test files (1297 unit pass, 38 expected E2E skips when DATABASE_URL is unset). Unit tests run
|
||||
`bun test` runs all tests. After the v0.12.1 release: ~75 unit test files + 8 E2E test files (1412 unit pass, 119 E2E when `DATABASE_URL` is set — skip gracefully otherwise). Unit tests run
|
||||
without a database. E2E tests skip gracefully when `DATABASE_URL` is not set.
|
||||
|
||||
Unit tests: `test/markdown.test.ts` (frontmatter parsing), `test/chunkers/recursive.test.ts`
|
||||
@@ -140,11 +146,11 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/files.test.ts` (MIME/hash), `test/import-file.test.ts` (import pipeline),
|
||||
`test/upgrade.test.ts` (schema migrations), `test/doctor.test.ts` (doctor command),
|
||||
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix),
|
||||
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
|
||||
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
|
||||
`test/yaml-lite.test.ts` (YAML parsing), `test/check-update.test.ts` (version check + update CLI),
|
||||
`test/pglite-engine.test.ts` (PGLite engine, all 37 BrainEngine methods),
|
||||
`test/pglite-engine.test.ts` (PGLite engine, all 40 BrainEngine methods including 11 cases for `addLinksBatch` / `addTimelineEntriesBatch`: empty batch, missing optionals, within-batch dedup via ON CONFLICT, missing-slug rows dropped by JOIN, half-existing batch, batch of 100),
|
||||
`test/utils.test.ts` (shared SQL utilities), `test/engine-factory.test.ts` (engine factory + dynamic imports),
|
||||
`test/integrations.test.ts` (recipe parsing, CLI routing, recipe validation),
|
||||
`test/publish.test.ts` (content stripping, encryption, password generation, HTML output),
|
||||
@@ -166,17 +172,22 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/minions.test.ts` (Minions job queue v7: CRUD, state machine, backoff, stall detection, dependencies, worker lifecycle, lock management, claim mechanics, depth/child-cap, timeouts, cascade kill, idempotency, child_done inbox, attachments, removeOnComplete/Fail),
|
||||
`test/extract.test.ts` (link extraction, timeline extraction, frontmatter parsing, directory type inference),
|
||||
`test/extract-db.test.ts` (gbrain extract --source db: typed link inference, idempotency, --type filter, --dry-run JSON output),
|
||||
`test/extract-fs.test.ts` (gbrain extract --source fs: first-run inserts + second-run reports zero, dry-run dedups candidates across files, second-run perf regression guard — the v0.12.1 N+1 dedup bug),
|
||||
`test/link-extraction.test.ts` (canonical extractEntityRefs both formats, extractPageLinks dedup, inferLinkType heuristics, parseTimelineEntries date variants, isAutoLinkEnabled config),
|
||||
`test/graph-query.test.ts` (direction in/out/both, type filter, indented tree output),
|
||||
`test/features.test.ts` (feature scanning, brain_score calculation, CLI routing, persistence),
|
||||
`test/file-upload-security.test.ts` (symlink traversal, cwd confinement, slug + filename allowlists, remote vs local trust),
|
||||
`test/query-sanitization.test.ts` (prompt-injection stripping, output sanitization, structural boundary),
|
||||
`test/search-limit.test.ts` (clampSearchLimit default/cap behavior across list_pages and get_ingest_log).
|
||||
`test/search-limit.test.ts` (clampSearchLimit default/cap behavior across list_pages and get_ingest_log),
|
||||
`test/repair-jsonb.test.ts` (v0.12.2 JSONB repair: TARGETS list, idempotency, engine-awareness),
|
||||
`test/migrations-v0_12_2.test.ts` (v0.12.2 orchestrator phases: schema → repair → verify → record),
|
||||
`test/markdown.test.ts` (splitBody sentinel precedence, horizontal-rule preservation, inferType wiki subtypes).
|
||||
|
||||
E2E tests (`test/e2e/`): Run against real Postgres+pgvector. Require `DATABASE_URL`.
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys)
|
||||
- `bun run test:e2e` runs Tier 1 (mechanical, all operations, no API keys). Includes 9 dedicated cases for the postgres-engine `addLinksBatch` / `addTimelineEntriesBatch` bind path — postgres-js's `unnest()` binding is structurally different from PGLite's and gets its own coverage.
|
||||
- `test/e2e/search-quality.test.ts` runs search quality E2E against PGLite (no API keys, in-memory)
|
||||
- `test/e2e/graph-quality.test.ts` runs the v0.10.3 knowledge graph pipeline (auto-link via put_page, reconciliation, traversePaths) against PGLite in-memory
|
||||
- `test/e2e/postgres-jsonb.test.ts` — v0.12.2 regression test. Round-trips all 5 JSONB write sites (pages.frontmatter, raw_data.data, ingest_log.pages_updated, files.metadata, page_versions.frontmatter) against real Postgres and asserts `jsonb_typeof='object'` plus `->>'key'` returns the expected scalar. The test that should have caught the original double-encode bug.
|
||||
- `test/e2e/upgrade.test.ts` runs check-update E2E against real GitHub API (network required)
|
||||
- Tier 2 (`skills.test.ts`) requires OpenClaw + API keys, runs nightly in CI
|
||||
- If `.env.testing` doesn't exist in this directory, check sibling worktrees for one:
|
||||
|
||||
@@ -127,7 +127,7 @@ Verify: `gbrain integrations doctor` (after at least one is configured)
|
||||
|
||||
## Step 9: Verify
|
||||
|
||||
Read `docs/GBRAIN_VERIFY.md` and run all 6 verification checks. Check #4 (live sync
|
||||
Read `docs/GBRAIN_VERIFY.md` and run all 7 verification checks. Check #4 (live sync
|
||||
actually works) is the most important.
|
||||
|
||||
## Upgrade
|
||||
@@ -145,3 +145,10 @@ this is how features ship in the binary but stay dormant in the user's brain.
|
||||
For v0.12.0+ specifically: if your brain was created before v0.12.0, run
|
||||
`gbrain extract links --source db && gbrain extract timeline --source db` to
|
||||
backfill the new graph layer (see Step 4.5 above).
|
||||
|
||||
For v0.12.2+ specifically: if your brain is Postgres- or Supabase-backed and
|
||||
predates v0.12.2, the `v0_12_2` migration runs `gbrain repair-jsonb`
|
||||
automatically during `gbrain post-upgrade` to fix the double-encoded JSONB
|
||||
columns. PGLite brains no-op. If wiki-style imports were truncated by the old
|
||||
`splitBody` bug, run `gbrain sync --full` after upgrading to rebuild
|
||||
`compiled_truth` from source markdown.
|
||||
|
||||
@@ -536,6 +536,7 @@ ADMIN
|
||||
gbrain integrations Integration recipe dashboard
|
||||
gbrain check-backlinks check|fix Back-link enforcement
|
||||
gbrain lint [--fix] LLM artifact detection
|
||||
gbrain repair-jsonb [--dry-run] Repair v0.12.0 double-encoded JSONB (Postgres)
|
||||
gbrain transcribe <audio> Transcribe audio (Groq Whisper)
|
||||
gbrain research init <name> Scaffold a data-research recipe
|
||||
gbrain research list Show available recipes
|
||||
|
||||
@@ -84,6 +84,19 @@ board" — likely an advisor-role page prior plus verb-pattern combinations.
|
||||
|
||||
## P1
|
||||
|
||||
### Batch the DB-source extract read path (deferred from v0.12.1)
|
||||
**What:** `extractLinksFromDB` and `extractTimelineFromDB` at `src/commands/extract.ts:447, 504` issue one `engine.getPage(slug)` per slug after `engine.getAllSlugs()`. On a 47K-page brain that's still 47K serial reads over the Supabase pooler.
|
||||
|
||||
**Why:** v0.12.1 fixed the write-side N+1 with batched INSERTs (~100x fewer round-trips). The read side still does serial `getPage()` calls — each fetches `compiled_truth + timeline + frontmatter` (tens of KB per page). On a 47K-page Supabase brain that's ~10-20 minutes of read latency before any work happens. The v0.12.0 orchestrator's backfill uses `--source db`, so this stays slow until fixed.
|
||||
|
||||
**Pros:** Mirrors the write-side fix on the read path. Combined with batched writes, full re-extract on a 47K-page brain should drop from "minutes" to "seconds" end-to-end. Eliminates the implicit `listPages-pagination-mutation` learning risk by giving you a snapshot read.
|
||||
|
||||
**Cons:** New engine method (`getPagesBatch(slugs: string[]) → Promise<Page[]>` or a streaming cursor) needs to land on both PGLite and Postgres. Memory budget — a 47K-page brain with ~30KB/page is ~1.4GB if loaded all at once; needs chunked iteration (e.g., 500 slugs/query, stream-process).
|
||||
|
||||
**Context:** Codex's plan-time review and the testing/performance specialists at ship time both flagged this. Filed during v0.12.1 to ship the bug fix without scope creep. Approach: add `getPagesBatch(slugs)` returning chunked results, then update the 4 DB-source extract paths to consume it.
|
||||
|
||||
**Depends on:** v0.12.1 ships first.
|
||||
|
||||
### Batch embedding queue across files
|
||||
**What:** Shared embedding queue that collects chunks from all parallel import workers and flushes to OpenAI in batches of 100, instead of each worker batching independently.
|
||||
|
||||
|
||||
+41
-1
@@ -224,6 +224,43 @@ heuristics won't find them — file an issue with a sample page.
|
||||
|
||||
---
|
||||
|
||||
## 8. JSONB Frontmatter Integrity (v0.12.2)
|
||||
|
||||
Postgres-backed brains created before v0.12.2 had double-encoded JSONB columns
|
||||
(`frontmatter->>'key'` returned NULL, GIN indexes were inert). `gbrain upgrade`
|
||||
runs `gbrain repair-jsonb` automatically via the `v0_12_2` orchestrator.
|
||||
Verify the repair succeeded.
|
||||
|
||||
**Command:**
|
||||
|
||||
```bash
|
||||
gbrain repair-jsonb --dry-run --json
|
||||
```
|
||||
|
||||
**Expected:** `totalRepaired: 0` across all 5 columns (`pages.frontmatter`,
|
||||
`raw_data.data`, `ingest_log.pages_updated`, `files.metadata`,
|
||||
`page_versions.frontmatter`). A zero count means every row is properly-typed
|
||||
JSON objects, not string-encoded JSON.
|
||||
|
||||
**If the count is > 0:** The repair didn't run or was interrupted. Re-run
|
||||
without `--dry-run`:
|
||||
|
||||
```bash
|
||||
gbrain repair-jsonb
|
||||
```
|
||||
|
||||
Idempotent. PGLite brains always report 0 (unaffected by the original bug).
|
||||
|
||||
**Bonus check** — frontmatter-keyed queries actually resolve:
|
||||
|
||||
```bash
|
||||
gbrain call list_pages '{"frontmatterKey": "type", "frontmatterValue": "person"}'
|
||||
```
|
||||
|
||||
If this returns rows on a brain with person pages, the JSONB path is healthy.
|
||||
|
||||
---
|
||||
|
||||
## Quick Verification (all checks in one pass)
|
||||
|
||||
```bash
|
||||
@@ -247,7 +284,10 @@ gbrain check-update --json
|
||||
|
||||
# 7. Knowledge graph populated (links + timeline > 0)
|
||||
gbrain stats | grep -E 'links|timeline'
|
||||
|
||||
# 8. JSONB integrity (v0.12.2 — Postgres only, PGLite always 0)
|
||||
gbrain repair-jsonb --dry-run --json
|
||||
```
|
||||
|
||||
If all seven return successfully, the installation is healthy. For the full
|
||||
If all eight return successfully, the installation is healthy. For the full
|
||||
end-to-end sync test (4c), push a real change and verify it appears in search.
|
||||
|
||||
@@ -177,6 +177,74 @@ Timeline entries still need explicit `gbrain timeline-add` calls.
|
||||
```
|
||||
Should return an indented tree of typed edges.
|
||||
|
||||
---
|
||||
|
||||
## v0.12.2 hotfix (data-correctness, no skill edits)
|
||||
|
||||
v0.12.2 is a Postgres data-correctness hotfix. No forked skill files need to
|
||||
change — the skill contracts are unchanged. But you DO need to run the migration,
|
||||
and you should know about one behavior change in markdown parsing.
|
||||
|
||||
### 1. Run the migration (Postgres-backed brains)
|
||||
|
||||
```bash
|
||||
gbrain upgrade
|
||||
```
|
||||
|
||||
The `v0_12_2` orchestrator runs `gbrain repair-jsonb` automatically. It rewrites
|
||||
rows where `jsonb_typeof = 'string'` across `pages.frontmatter`, `raw_data.data`,
|
||||
`ingest_log.pages_updated`, `files.metadata`, and `page_versions.frontmatter`.
|
||||
Idempotent, safe to re-run. PGLite brains no-op cleanly.
|
||||
|
||||
Verify after upgrade:
|
||||
|
||||
```bash
|
||||
gbrain repair-jsonb --dry-run --json # expect totalRepaired: 0
|
||||
```
|
||||
|
||||
### 2. Recover any truncated wiki articles
|
||||
|
||||
If your brain imported wiki-style markdown before v0.12.2, some pages were
|
||||
silently truncated (any standalone `---` in body content was treated as a
|
||||
timeline separator). Re-import from source:
|
||||
|
||||
```bash
|
||||
gbrain sync --full
|
||||
```
|
||||
|
||||
The new `splitBody` rebuilds `compiled_truth` correctly.
|
||||
|
||||
### 3. Know the splitBody contract going forward
|
||||
|
||||
`splitBody` now requires an explicit timeline sentinel. Recognized markers
|
||||
(priority order):
|
||||
|
||||
1. `<!-- timeline -->` (preferred — what `serializeMarkdown` emits)
|
||||
2. `--- timeline ---` (decorated separator)
|
||||
3. `---` directly before `## Timeline` or `## History` heading (backward-compat)
|
||||
|
||||
A bare `---` in body text is now a markdown horizontal rule, not a timeline
|
||||
separator. If your agent writes pages with a bare `---` delimiter, migrate to
|
||||
`<!-- timeline -->` — the `serializeMarkdown` helper already does this.
|
||||
|
||||
### 4. Wiki subtypes now auto-typed
|
||||
|
||||
`inferType` now auto-detects five additional directory patterns as their own
|
||||
page types (previously they all defaulted to `concept`):
|
||||
|
||||
| Path pattern | New type |
|
||||
|------------------------|----------------|
|
||||
| `/wiki/analysis/` | `analysis` |
|
||||
| `/wiki/guides/` | `guide` |
|
||||
| `/wiki/hardware/` | `hardware` |
|
||||
| `/wiki/architecture/` | `architecture` |
|
||||
| `/writing/` | `writing` |
|
||||
|
||||
If your skills or queries filter by `type=concept` and expect wiki content in
|
||||
that bucket, update them to include the new types.
|
||||
|
||||
---
|
||||
|
||||
## Future versions
|
||||
|
||||
When gbrain ships a new version, this doc will be updated with the diffs for that
|
||||
|
||||
+3
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.12.0",
|
||||
"version": "0.12.2",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
@@ -20,9 +20,10 @@
|
||||
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
|
||||
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
||||
"build:schema": "bash scripts/build-schema.sh",
|
||||
"test": "bun test",
|
||||
"test": "scripts/check-jsonb-pattern.sh && bun test",
|
||||
"test:e2e": "bun test test/e2e/",
|
||||
"test:eval": "bun test eval/runner/queries/validator.test.ts eval/runner/adapters/ eval/generators/world-html.test.ts",
|
||||
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
||||
"eval:run": "bun eval/runner/multi-adapter.ts",
|
||||
"eval:run:dev": "BRAINBENCH_N=1 bun eval/runner/multi-adapter.ts",
|
||||
"eval:world:view": "bun eval/cli/world-view.ts",
|
||||
|
||||
Executable
+32
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: fail if any source file uses the buggy `${JSON.stringify(x)}::jsonb`
|
||||
# template-string pattern instead of postgres.js's `sql.json(x)`.
|
||||
#
|
||||
# This is best-effort static analysis. It catches the common copy-paste form
|
||||
# that caused the v0.12.0 silent-data-loss bug (JSONB columns stored as
|
||||
# string literals on Postgres while PGLite hid the bug). Multi-line and
|
||||
# helper-wrapped variants are NOT caught here — those are covered by
|
||||
# test/e2e/postgres-jsonb.test.ts which round-trips actual writes through
|
||||
# real Postgres and asserts `frontmatter->>'k'` returns objects, not strings.
|
||||
#
|
||||
# Usage: scripts/check-jsonb-pattern.sh
|
||||
# Exit: 0 when no matches, 1 when matches found.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# Match the interpolated form: ${JSON.stringify(...)}::jsonb
|
||||
# Using grep -P for Perl-compatible regex (lookahead-free pattern is enough here).
|
||||
PATTERN='\$\{JSON\.stringify\([^)]*\)\}::jsonb'
|
||||
|
||||
if grep -rEn "$PATTERN" src/ 2>/dev/null; then
|
||||
echo
|
||||
echo "ERROR: Found JSON.stringify(...)::jsonb pattern in src/."
|
||||
echo " postgres.js v3 stringifies again, producing JSONB string literals."
|
||||
echo " Use sql.json(x) instead. See feedback_postgres_jsonb_double_encode.md."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: no JSON.stringify(x)::jsonb interpolation pattern in src/"
|
||||
+11
-4
@@ -50,10 +50,17 @@ Universal migration from any wiki, note tool, or brain system into GBrain.
|
||||
## Obsidian Migration
|
||||
|
||||
1. Import the vault directory into gbrain (Obsidian vaults are markdown directories)
|
||||
2. Convert `[[wikilinks]]` to gbrain links:
|
||||
- Read each page from gbrain
|
||||
- For each `[[Name]]` found, resolve to a slug and create a link in gbrain
|
||||
- `[[Name|alias]]` uses the alias for context
|
||||
2. Wire the graph with native wikilink support (v0.12.1+):
|
||||
|
||||
```bash
|
||||
gbrain extract links --source db --dry-run | head -20 # preview
|
||||
gbrain extract links --source db # commit
|
||||
```
|
||||
|
||||
`extract links` natively parses `[[relative/path]]` and `[[relative/path|Display Text]]`
|
||||
alongside standard `[text](page.md)` markdown syntax. Ancestor-search resolution handles
|
||||
wiki KBs where authors omit one or more leading `../` prefixes. The `.md` suffix is
|
||||
inferred automatically for wikilinks.
|
||||
|
||||
Obsidian-specific:
|
||||
- Tags (`#tag`) become gbrain tags
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
# v0.12.1 Migration: Extract Performance + Migration Timeout Fix
|
||||
|
||||
This release is a pure performance bug fix. **No manual steps are needed for most
|
||||
users** — re-run `gbrain init --migrate-only` and re-run `gbrain extract all` if
|
||||
desired. Both should now complete successfully on any brain size.
|
||||
|
||||
## What changed
|
||||
|
||||
Two production-blocking bugs are fixed:
|
||||
|
||||
1. **`gbrain extract` no longer hangs on large brains.** The N+1 dedup pre-load
|
||||
that ran 47K serial `getLinks()` calls before any work started is gone. Both
|
||||
engines already enforced uniqueness at the SQL layer; the in-memory dedup was
|
||||
redundant. Combined with new batched 100-row INSERTs, a full re-extract on a
|
||||
47K-page brain drops from "10+ min hang then ~minutes more" to "immediate
|
||||
work, ~30-60s total."
|
||||
|
||||
2. **v0.12.0 schema migration no longer times out on duplicate-heavy brains.**
|
||||
Migration v9 (timeline_dedup_index) and v8 (links uniqueness) now pre-create
|
||||
a btree helper index before the `DELETE ... USING` self-join, then drop it
|
||||
after dedup. Turns O(n²) dedup into O(n log n). On 80K+ duplicate rows the
|
||||
migration completes in under a second instead of timing out at 60 seconds.
|
||||
|
||||
## What you need to do
|
||||
|
||||
### If your v0.12.0 upgrade succeeded — you're already done
|
||||
|
||||
The migration is idempotent. The fix only matters if your migration FAILED on
|
||||
the v0.12.0 upgrade attempt. Run `gbrain init --migrate-only` once to confirm
|
||||
your schema is at version 10, then run `gbrain extract all --dir <brain>` if
|
||||
you want to reuse it now that it's fast.
|
||||
|
||||
### If your v0.12.0 upgrade FAILED on `idx_timeline_dedup` creation
|
||||
|
||||
You may have the brain in a partial-migration state with duplicate rows in
|
||||
`timeline_entries` (or `links`). Run:
|
||||
|
||||
```bash
|
||||
gbrain init --migrate-only
|
||||
```
|
||||
|
||||
Migration v9 will pre-create the helper index, dedup any existing duplicates
|
||||
(now sub-second instead of timing out), drop the helper, and create the unique
|
||||
index. Idempotent — safe to re-run.
|
||||
|
||||
If you previously ran the manual `CREATE TABLE _clean AS SELECT DISTINCT ON
|
||||
... + table swap` workaround Garry posted, your schema should already be at
|
||||
version 10. Confirm with:
|
||||
|
||||
```bash
|
||||
gbrain config get version
|
||||
```
|
||||
|
||||
If it shows `10`, you're done.
|
||||
|
||||
### If you previously did manual SQL surgery on duplicate timeline rows
|
||||
|
||||
The unique index `idx_timeline_dedup` should now be present after the workaround.
|
||||
Re-running `gbrain init --migrate-only` is a no-op for v9 (uses
|
||||
`CREATE UNIQUE INDEX IF NOT EXISTS`). The new migration code adds the helper
|
||||
btree on the dedup columns first — but the helper is dropped at the end of the
|
||||
migration, so even if v9 re-ran (it won't, version is already at 10), it would
|
||||
leave your schema unchanged.
|
||||
|
||||
### Re-running `gbrain extract` on a previously-stuck brain
|
||||
|
||||
This is the most common case. Run:
|
||||
|
||||
```bash
|
||||
gbrain extract all --dir <brain-dir>
|
||||
# or for live brains with no local checkout:
|
||||
gbrain extract all --source db
|
||||
```
|
||||
|
||||
Expect immediate output (`Links: created N from M pages` lines streaming as files
|
||||
process), not a 10-minute hang. On a re-run of a fully-extracted brain you should
|
||||
see `Done: 0 links, 0 timeline entries from N pages` — that's the truthful counter
|
||||
at work, confirming nothing changed.
|
||||
|
||||
## New engine API (informational, optional)
|
||||
|
||||
For plugin authors building integrations on the `BrainEngine` interface, two
|
||||
new methods are available:
|
||||
|
||||
- `addLinksBatch(LinkBatchInput[]) → Promise<number>`
|
||||
- `addTimelineEntriesBatch(TimelineBatchInput[]) → Promise<number>`
|
||||
|
||||
Both return the count of rows actually inserted (excluding ON CONFLICT no-ops
|
||||
and JOIN-dropped rows whose slugs don't exist). Existing per-row `addLink` /
|
||||
`addTimelineEntry` are unchanged — no migration required for plugin code.
|
||||
|
||||
## Verification
|
||||
|
||||
After the migration completes:
|
||||
|
||||
```bash
|
||||
# Confirm schema version
|
||||
gbrain config get version
|
||||
# Expect: 10
|
||||
|
||||
# Confirm the unique indexes exist (Postgres / Supabase only)
|
||||
gbrain stats
|
||||
# Expect: link_count and timeline_entry_count both populated, no duplicates
|
||||
```
|
||||
|
||||
If you hit any issue, file at https://github.com/garrytan/gbrain/issues with the
|
||||
output of `gbrain init --migrate-only` and `gbrain config get version`.
|
||||
+6
-1
@@ -18,7 +18,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'apply-migrations', 'skillpack-check']);
|
||||
const CLI_ONLY = new Set(['init', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'apply-migrations', 'skillpack-check', 'repair-jsonb']);
|
||||
|
||||
async function main() {
|
||||
const args = process.argv.slice(2);
|
||||
@@ -306,6 +306,11 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runApplyMigrations(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'repair-jsonb') {
|
||||
const { runRepairJsonbCli } = await import('./commands/repair-jsonb.ts');
|
||||
await runRepairJsonbCli(args);
|
||||
return;
|
||||
}
|
||||
if (command === 'skillpack-check') {
|
||||
// Agent-readable health report. Shells out to doctor + apply-migrations
|
||||
// internally; does not need its own DB connection.
|
||||
|
||||
+174
-57
@@ -18,11 +18,18 @@
|
||||
|
||||
import { readFileSync, readdirSync, lstatSync, existsSync } from 'fs';
|
||||
import { join, relative, dirname } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/engine.ts';
|
||||
import type { PageType } from '../core/types.ts';
|
||||
import { parseMarkdown } from '../core/markdown.ts';
|
||||
import { extractPageLinks, parseTimelineEntries, inferLinkType } from '../core/link-extraction.ts';
|
||||
|
||||
// Batch size for addLinksBatch / addTimelineEntriesBatch.
|
||||
// Postgres bind-parameter limit is 65535. Links use 4 cols/row → 16K hard ceiling;
|
||||
// timeline uses 5 cols/row → 13K hard ceiling. 100 is conservative on round-trip
|
||||
// count but safe at any future schema width and keeps per-batch error blast radius
|
||||
// small (a malformed row aborts at most 100, not thousands).
|
||||
const BATCH_SIZE = 100;
|
||||
|
||||
// --- Types ---
|
||||
|
||||
export interface ExtractedLink {
|
||||
@@ -69,19 +76,72 @@ export function walkMarkdownFiles(dir: string): { path: string; relPath: string
|
||||
|
||||
// --- Link extraction ---
|
||||
|
||||
/** Extract markdown links to .md files (relative paths only) */
|
||||
/**
|
||||
* Extract markdown links to .md files (relative paths only).
|
||||
*
|
||||
* Handles two syntaxes:
|
||||
* 1. Standard markdown: [text](relative/path.md)
|
||||
* 2. Wikilinks: [[relative/path]] or [[relative/path|Display Text]]
|
||||
*
|
||||
* Both are resolved relative to the file that contains them. External URLs
|
||||
* (containing ://) are always skipped. For wikilinks, the .md suffix is added
|
||||
* if absent and section anchors (#heading) are stripped.
|
||||
*/
|
||||
export function extractMarkdownLinks(content: string): { name: string; relTarget: string }[] {
|
||||
const results: { name: string; relTarget: string }[] = [];
|
||||
const pattern = /\[([^\]]+)\]\(([^)]+\.md)\)/g;
|
||||
|
||||
const mdPattern = /\[([^\]]+)\]\(([^)]+\.md)\)/g;
|
||||
let match;
|
||||
while ((match = pattern.exec(content)) !== null) {
|
||||
while ((match = mdPattern.exec(content)) !== null) {
|
||||
const target = match[2];
|
||||
if (target.includes('://')) continue; // skip external URLs
|
||||
if (target.includes('://')) continue;
|
||||
results.push({ name: match[1], relTarget: target });
|
||||
}
|
||||
|
||||
const wikiPattern = /\[\[([^|\]]+?)(?:\|[^\]]*?)?\]\]/g;
|
||||
while ((match = wikiPattern.exec(content)) !== null) {
|
||||
const rawPath = match[1].trim();
|
||||
if (rawPath.includes('://')) continue;
|
||||
const hashIdx = rawPath.indexOf('#');
|
||||
const pagePath = hashIdx >= 0 ? rawPath.slice(0, hashIdx) : rawPath;
|
||||
if (!pagePath) continue;
|
||||
const relTarget = pagePath.endsWith('.md') ? pagePath : pagePath + '.md';
|
||||
const pipeIdx = match[0].indexOf('|');
|
||||
const displayName = pipeIdx >= 0 ? match[0].slice(pipeIdx + 1, -2).trim() : rawPath;
|
||||
results.push({ name: displayName, relTarget });
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a wikilink target to a canonical slug, given the directory of the
|
||||
* containing page and the set of all known slugs in the brain.
|
||||
*
|
||||
* Wiki KBs often use inconsistent relative depths. Authors omit one or more
|
||||
* leading `../` because they think in "wiki-root-relative" terms. Resolution
|
||||
* order (first match wins):
|
||||
* 1. Standard `join(fileDir, relTarget)` — exact relative path as written
|
||||
* 2. Ancestor search — strip leading path components from fileDir, retry
|
||||
*
|
||||
* Returns null when no matching slug is found (dangling link).
|
||||
*/
|
||||
export function resolveSlug(fileDir: string, relTarget: string, allSlugs: Set<string>): string | null {
|
||||
const targetNoExt = relTarget.endsWith('.md') ? relTarget.slice(0, -3) : relTarget;
|
||||
|
||||
const s1 = join(fileDir, targetNoExt);
|
||||
if (allSlugs.has(s1)) return s1;
|
||||
|
||||
const parts = fileDir.split('/').filter(Boolean);
|
||||
for (let strip = 1; strip <= parts.length; strip++) {
|
||||
const ancestor = parts.slice(0, parts.length - strip).join('/');
|
||||
const candidate = ancestor ? join(ancestor, targetNoExt) : targetNoExt;
|
||||
if (allSlugs.has(candidate)) return candidate;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Infer link type from directory structure */
|
||||
function inferLinkType(fromDir: string, toDir: string, frontmatter?: Record<string, unknown>): string {
|
||||
const from = fromDir.split('/')[0];
|
||||
@@ -139,8 +199,8 @@ export function extractLinksFromFile(
|
||||
const fm = parseFrontmatterFromContent(content, relPath);
|
||||
|
||||
for (const { name, relTarget } of extractMarkdownLinks(content)) {
|
||||
const resolved = join(fileDir, relTarget).replace('.md', '');
|
||||
if (allSlugs.has(resolved)) {
|
||||
const resolved = resolveSlug(fileDir, relTarget, allSlugs);
|
||||
if (resolved !== null) {
|
||||
links.push({
|
||||
from_slug: slug, to_slug: resolved,
|
||||
link_type: inferLinkType(fileDir, dirname(resolved), fm),
|
||||
@@ -312,34 +372,42 @@ async function extractLinksFromDir(
|
||||
const files = walkMarkdownFiles(brainDir);
|
||||
const allSlugs = new Set(files.map(f => f.relPath.replace('.md', '')));
|
||||
|
||||
// Load existing links for O(1) dedup
|
||||
const existing = new Set<string>();
|
||||
try {
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
for (const page of pages) {
|
||||
for (const link of await engine.getLinks(page.slug)) {
|
||||
existing.add(`${link.from_slug}::${link.to_slug}`);
|
||||
}
|
||||
}
|
||||
} catch { /* fresh brain */ }
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
// Without this, the same link extracted from N files would print N times in --dry-run.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
let created = 0;
|
||||
const batch: LinkBatchInput[] = [];
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
try {
|
||||
created += await engine.addLinksBatch(batch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n');
|
||||
} else {
|
||||
console.error(` batch error (${batch.length} link rows lost): ${msg}`);
|
||||
}
|
||||
} finally {
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
try {
|
||||
const content = readFileSync(files[i].path, 'utf-8');
|
||||
const links = extractLinksFromFile(content, files[i].relPath, allSlugs);
|
||||
for (const link of links) {
|
||||
const key = `${link.from_slug}::${link.to_slug}`;
|
||||
if (existing.has(key)) continue;
|
||||
existing.add(key);
|
||||
if (dryRun) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${link.from_slug}::${link.to_slug}::${link.link_type}`;
|
||||
if (dryRunSeen.has(key)) continue;
|
||||
dryRunSeen.add(key);
|
||||
if (!jsonMode) console.log(` ${link.from_slug} → ${link.to_slug} (${link.link_type})`);
|
||||
created++;
|
||||
} else {
|
||||
try {
|
||||
await engine.addLink(link.from_slug, link.to_slug, link.context, link.link_type);
|
||||
created++;
|
||||
} catch { /* UNIQUE or page not found */ }
|
||||
batch.push(link);
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
} catch { /* skip unreadable */ }
|
||||
@@ -347,6 +415,7 @@ async function extractLinksFromDir(
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_links', done: i + 1, total: files.length }) + '\n');
|
||||
}
|
||||
}
|
||||
await flush();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
@@ -360,34 +429,41 @@ async function extractTimelineFromDir(
|
||||
): Promise<{ created: number; pages: number }> {
|
||||
const files = walkMarkdownFiles(brainDir);
|
||||
|
||||
// Load existing timeline entries for O(1) dedup
|
||||
const existing = new Set<string>();
|
||||
try {
|
||||
const pages = await engine.listPages({ limit: 100000 });
|
||||
for (const page of pages) {
|
||||
for (const entry of await engine.getTimeline(page.slug)) {
|
||||
existing.add(`${page.slug}::${entry.date}::${entry.summary}`);
|
||||
}
|
||||
}
|
||||
} catch { /* fresh brain */ }
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
let created = 0;
|
||||
const batch: TimelineBatchInput[] = [];
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
try {
|
||||
created += await engine.addTimelineEntriesBatch(batch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n');
|
||||
} else {
|
||||
console.error(` batch error (${batch.length} timeline rows lost): ${msg}`);
|
||||
}
|
||||
} finally {
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < files.length; i++) {
|
||||
try {
|
||||
const content = readFileSync(files[i].path, 'utf-8');
|
||||
const slug = files[i].relPath.replace('.md', '');
|
||||
for (const entry of extractTimelineFromContent(content, slug)) {
|
||||
const key = `${entry.slug}::${entry.date}::${entry.summary}`;
|
||||
if (existing.has(key)) continue;
|
||||
existing.add(key);
|
||||
if (dryRun) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${entry.slug}::${entry.date}::${entry.summary}`;
|
||||
if (dryRunSeen.has(key)) continue;
|
||||
dryRunSeen.add(key);
|
||||
if (!jsonMode) console.log(` ${entry.slug}: ${entry.date} — ${entry.summary}`);
|
||||
created++;
|
||||
} else {
|
||||
try {
|
||||
await engine.addTimelineEntry(entry.slug, { date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
|
||||
created++;
|
||||
} catch { /* page not in DB or constraint */ }
|
||||
batch.push({ slug: entry.slug, date: entry.date, source: entry.source, summary: entry.summary, detail: entry.detail });
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
} catch { /* skip unreadable */ }
|
||||
@@ -395,6 +471,7 @@ async function extractTimelineFromDir(
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_timeline', done: i + 1, total: files.length }) + '\n');
|
||||
}
|
||||
}
|
||||
await flush();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
@@ -455,6 +532,26 @@ async function extractLinksFromDB(
|
||||
const slugList = Array.from(allSlugs);
|
||||
let processed = 0, created = 0;
|
||||
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
const batch: LinkBatchInput[] = [];
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
try {
|
||||
created += await engine.addLinksBatch(batch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n');
|
||||
} else {
|
||||
console.error(` batch error (${batch.length} link rows lost): ${msg}`);
|
||||
}
|
||||
} finally {
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < slugList.length; i++) {
|
||||
const slug = slugList[i];
|
||||
const page = await engine.getPage(slug);
|
||||
@@ -471,7 +568,10 @@ async function extractLinksFromDB(
|
||||
|
||||
for (const c of candidates) {
|
||||
if (!allSlugs.has(c.targetSlug)) continue;
|
||||
if (dryRun) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${slug}::${c.targetSlug}::${c.linkType}`;
|
||||
if (dryRunSeen.has(key)) continue;
|
||||
dryRunSeen.add(key);
|
||||
if (jsonMode) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
action: 'add_link', from: slug, to: c.targetSlug,
|
||||
@@ -482,10 +582,8 @@ async function extractLinksFromDB(
|
||||
}
|
||||
created++;
|
||||
} else {
|
||||
try {
|
||||
await engine.addLink(slug, c.targetSlug, c.context, c.linkType);
|
||||
created++;
|
||||
} catch { /* FK violation or other */ }
|
||||
batch.push({ from_slug: slug, to_slug: c.targetSlug, link_type: c.linkType, context: c.context });
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
processed++;
|
||||
@@ -493,6 +591,7 @@ async function extractLinksFromDB(
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_links_db', done: processed, total: slugList.length }) + '\n');
|
||||
}
|
||||
}
|
||||
await flush();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
@@ -512,6 +611,26 @@ async function extractTimelineFromDB(
|
||||
const slugList = Array.from(allSlugs);
|
||||
let processed = 0, created = 0;
|
||||
|
||||
// Dedup in dry-run only — DB enforces uniqueness via ON CONFLICT in batch writes.
|
||||
const dryRunSeen = dryRun ? new Set<string>() : null;
|
||||
|
||||
const batch: TimelineBatchInput[] = [];
|
||||
async function flush() {
|
||||
if (batch.length === 0) return;
|
||||
try {
|
||||
created += await engine.addTimelineEntriesBatch(batch);
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
if (jsonMode) {
|
||||
process.stderr.write(JSON.stringify({ event: 'batch_error', size: batch.length, error: msg }) + '\n');
|
||||
} else {
|
||||
console.error(` batch error (${batch.length} timeline rows lost): ${msg}`);
|
||||
}
|
||||
} finally {
|
||||
batch.length = 0;
|
||||
}
|
||||
}
|
||||
|
||||
for (let i = 0; i < slugList.length; i++) {
|
||||
const slug = slugList[i];
|
||||
const page = await engine.getPage(slug);
|
||||
@@ -527,7 +646,10 @@ async function extractTimelineFromDB(
|
||||
const entries = parseTimelineEntries(fullContent);
|
||||
|
||||
for (const entry of entries) {
|
||||
if (dryRun) {
|
||||
if (dryRunSeen) {
|
||||
const key = `${slug}::${entry.date}::${entry.summary}`;
|
||||
if (dryRunSeen.has(key)) continue;
|
||||
dryRunSeen.add(key);
|
||||
if (jsonMode) {
|
||||
process.stdout.write(JSON.stringify({
|
||||
action: 'add_timeline', slug, date: entry.date,
|
||||
@@ -538,14 +660,8 @@ async function extractTimelineFromDB(
|
||||
}
|
||||
created++;
|
||||
} else {
|
||||
try {
|
||||
await engine.addTimelineEntry(
|
||||
slug,
|
||||
{ date: entry.date, summary: entry.summary, detail: entry.detail || '' },
|
||||
{ skipExistenceCheck: true },
|
||||
);
|
||||
created++;
|
||||
} catch { /* dedup constraint or other */ }
|
||||
batch.push({ slug, date: entry.date, summary: entry.summary, detail: entry.detail || '' });
|
||||
if (batch.length >= BATCH_SIZE) await flush();
|
||||
}
|
||||
}
|
||||
processed++;
|
||||
@@ -553,6 +669,7 @@ async function extractTimelineFromDB(
|
||||
process.stderr.write(JSON.stringify({ event: 'progress', phase: 'extracting_timeline_db', done: processed, total: slugList.length }) + '\n');
|
||||
}
|
||||
}
|
||||
await flush();
|
||||
|
||||
if (!jsonMode) {
|
||||
const label = dryRun ? '(dry run) would create' : 'created';
|
||||
|
||||
@@ -251,7 +251,7 @@ async function uploadRaw(args: string[]) {
|
||||
await sql`
|
||||
INSERT INTO files (page_slug, filename, storage_path, mime_type, size_bytes, content_hash, metadata)
|
||||
VALUES (${pageSlug}, ${filename}, ${storagePath}, ${mimeType}, ${stat.size}, ${'sha256:' + hash},
|
||||
${JSON.stringify({ type: fileType, upload_method: method })}::jsonb)
|
||||
${sql.json({ type: fileType, upload_method: method })})
|
||||
ON CONFLICT (storage_path) DO UPDATE SET
|
||||
content_hash = EXCLUDED.content_hash,
|
||||
size_bytes = EXCLUDED.size_bytes,
|
||||
|
||||
@@ -13,10 +13,12 @@
|
||||
import type { Migration } from './types.ts';
|
||||
import { v0_11_0 } from './v0_11_0.ts';
|
||||
import { v0_12_0 } from './v0_12_0.ts';
|
||||
import { v0_12_2 } from './v0_12_2.ts';
|
||||
|
||||
export const migrations: Migration[] = [
|
||||
v0_11_0,
|
||||
v0_12_0,
|
||||
v0_12_2,
|
||||
];
|
||||
|
||||
/** Look up a migration by exact version string. */
|
||||
|
||||
@@ -39,7 +39,10 @@ import { appendCompletedMigration } from '../../core/preferences.ts';
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
// 10-minute budget. Migrations v8/v9 dedup with helper-index should be sub-second
|
||||
// even on 80K-duplicate brains, but the outer wall-clock cap shouldn't be the
|
||||
// failure mode (the prior 60s ceiling tripped Garry's production upgrade).
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
/**
|
||||
* v0.12.2 migration orchestrator — JSONB double-encode repair.
|
||||
*
|
||||
* v0.12.0-and-earlier wrote JSONB columns via the buggy
|
||||
* `JSON.stringify(value)`-then-cast-to-jsonb interpolation pattern, which
|
||||
* postgres.js v3 stringified again on the wire. Result: every
|
||||
* `frontmatter->>'key'` query returned NULL on Postgres-backed brains and
|
||||
* GIN indexes on JSONB columns were inert. PGLite was unaffected (its
|
||||
* driver path uses parameterized binding, never interpolation).
|
||||
*
|
||||
* v0.12.2 fixes the writes (sql.json) AND repairs existing rows in place.
|
||||
* This is the migration. It's idempotent (only touches `jsonb_typeof = 'string'`
|
||||
* rows) and safe to re-run. PGLite engines no-op cleanly.
|
||||
*
|
||||
* Phases (all idempotent):
|
||||
* A. Schema — gbrain init --migrate-only (no schema changes in v0.12.2
|
||||
* but we still apply for consistency with v0.12.0).
|
||||
* B. Repair — gbrain repair-jsonb (the actual JSONB fix).
|
||||
* C. Verify — gbrain repair-jsonb --dry-run --json; assert 0 remaining.
|
||||
* D. Record — append completed.jsonl.
|
||||
*/
|
||||
|
||||
import { execSync } from 'child_process';
|
||||
import type { Migration, OrchestratorOpts, OrchestratorResult, OrchestratorPhaseResult } from './types.ts';
|
||||
import { appendCompletedMigration } from '../../core/preferences.ts';
|
||||
|
||||
// ── Phase A — Schema ────────────────────────────────────────
|
||||
|
||||
function phaseASchema(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'schema', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain init --migrate-only', { stdio: 'inherit', timeout: 60_000, env: process.env });
|
||||
return { name: 'schema', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'schema', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase B — JSONB repair ──────────────────────────────────
|
||||
|
||||
function phaseBRepair(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'jsonb_repair', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
execSync('gbrain repair-jsonb', { stdio: 'inherit', timeout: 600_000, env: process.env });
|
||||
return { name: 'jsonb_repair', status: 'complete' };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'jsonb_repair', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase C — Verify ────────────────────────────────────────
|
||||
|
||||
function phaseCVerify(opts: OrchestratorOpts): OrchestratorPhaseResult {
|
||||
if (opts.dryRun) return { name: 'verify', status: 'skipped', detail: 'dry-run' };
|
||||
try {
|
||||
const out = execSync('gbrain repair-jsonb --dry-run --json', {
|
||||
encoding: 'utf-8', timeout: 60_000, env: process.env,
|
||||
});
|
||||
const parsed = JSON.parse(out) as { total_repaired?: number; engine?: string };
|
||||
const remaining = parsed.total_repaired ?? 0;
|
||||
if (remaining > 0) {
|
||||
return {
|
||||
name: 'verify',
|
||||
status: 'failed',
|
||||
detail: `${remaining} string-typed JSONB rows remain after repair`,
|
||||
};
|
||||
}
|
||||
return { name: 'verify', status: 'complete', detail: parsed.engine ? `engine=${parsed.engine}` : undefined };
|
||||
} catch (e) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
return { name: 'verify', status: 'failed', detail: msg };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Orchestrator ────────────────────────────────────────────
|
||||
|
||||
async function orchestrator(opts: OrchestratorOpts): Promise<OrchestratorResult> {
|
||||
console.log('');
|
||||
console.log('=== v0.12.2 — JSONB double-encode repair ===');
|
||||
if (opts.dryRun) console.log(' (dry-run; no side effects)');
|
||||
console.log('');
|
||||
|
||||
const phases: OrchestratorPhaseResult[] = [];
|
||||
|
||||
const a = phaseASchema(opts);
|
||||
phases.push(a);
|
||||
if (a.status === 'failed') return finalizeResult(phases, 'failed');
|
||||
|
||||
const b = phaseBRepair(opts);
|
||||
phases.push(b);
|
||||
if (b.status === 'failed') return finalizeResult(phases, 'failed');
|
||||
|
||||
const c = phaseCVerify(opts);
|
||||
phases.push(c);
|
||||
|
||||
const overallStatus: 'complete' | 'partial' | 'failed' =
|
||||
a.status === 'failed' || b.status === 'failed' ? 'failed' :
|
||||
c.status === 'failed' ? 'partial' :
|
||||
'complete';
|
||||
|
||||
return finalizeResult(phases, overallStatus);
|
||||
}
|
||||
|
||||
function finalizeResult(phases: OrchestratorPhaseResult[], status: 'complete' | 'partial' | 'failed'): OrchestratorResult {
|
||||
if (status !== 'failed') {
|
||||
try {
|
||||
appendCompletedMigration({ version: '0.12.2', status: status as 'complete' | 'partial' });
|
||||
} catch {
|
||||
// Recording is best-effort.
|
||||
}
|
||||
}
|
||||
return {
|
||||
version: '0.12.2',
|
||||
status,
|
||||
phases,
|
||||
};
|
||||
}
|
||||
|
||||
export const v0_12_2: Migration = {
|
||||
version: '0.12.2',
|
||||
featurePitch: {
|
||||
headline: 'Postgres frontmatter queries now work — JSONB double-encode bug fixed and existing rows auto-repaired',
|
||||
description:
|
||||
'gbrain v0.12.0-and-earlier silently stored JSONB columns as quoted string literals on ' +
|
||||
'Postgres/Supabase (PGLite was unaffected). Every `frontmatter->>\'key\'` returned NULL ' +
|
||||
'and GIN indexes were inert. v0.12.2 fixes the writes AND auto-repairs every existing ' +
|
||||
'string-typed row in pages.frontmatter, raw_data.data, ingest_log.pages_updated, ' +
|
||||
'files.metadata, and page_versions.frontmatter. The migration is idempotent. Pages ' +
|
||||
'truncated by the splitBody horizontal-rule bug can be recovered with `gbrain sync --full`.',
|
||||
},
|
||||
orchestrator,
|
||||
};
|
||||
|
||||
/** Exported for unit tests. */
|
||||
export const __testing = {
|
||||
phaseASchema,
|
||||
phaseBRepair,
|
||||
phaseCVerify,
|
||||
};
|
||||
@@ -0,0 +1,151 @@
|
||||
/**
|
||||
* `gbrain repair-jsonb` — repair JSONB columns that were stored as string
|
||||
* literals due to the v0.12.0-and-earlier double-encode bug.
|
||||
*
|
||||
* Background: postgres-engine.ts wrote frontmatter and other JSONB columns
|
||||
* via the buggy `JSON.stringify(value)`-then-cast-to-jsonb interpolation
|
||||
* pattern, which postgres.js v3 stringified AGAIN on the wire. Result: every `frontmatter->>'key'` query returned NULL
|
||||
* on Postgres-backed brains; GIN indexes were inert. PGLite was unaffected
|
||||
* (different driver path). v0.12.1 fixes the writes (sql.json) but existing
|
||||
* rows stay broken until they're rewritten — that's what this command does.
|
||||
*
|
||||
* Strategy: for each affected JSONB column, detect rows where
|
||||
* `jsonb_typeof(col) = 'string'` and rewrite them via `(col #>> '{}')::jsonb`,
|
||||
* which extracts the string payload and re-parses it as JSONB. Idempotent:
|
||||
* re-running is a no-op (no rows match the guard). PGLite is a no-op too
|
||||
* (it never wrote string-typed JSONB).
|
||||
*
|
||||
* Affected columns (audit of src/schema.sql):
|
||||
* - pages.frontmatter (postgres-engine.ts:107 putPage)
|
||||
* - raw_data.data (postgres-engine.ts:668 putRawData)
|
||||
* - ingest_log.pages_updated (postgres-engine.ts:846 logIngest)
|
||||
* - files.metadata (commands/files.ts:254 file upload)
|
||||
* - page_versions.frontmatter (downstream of pages.frontmatter via
|
||||
* INSERT...SELECT FROM pages)
|
||||
*
|
||||
* Other JSONB columns (minion_jobs.{data,result,progress,stacktrace},
|
||||
* minion_inbox.payload) were always written via parameterized form ($N::jsonb
|
||||
* with a string parameter, not interpolation) so they were never affected.
|
||||
*/
|
||||
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
import type { EngineConfig } from '../core/types.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
|
||||
interface RepairTarget {
|
||||
table: string;
|
||||
column: string;
|
||||
/** Optional secondary key column for logging. */
|
||||
keyCol?: string;
|
||||
}
|
||||
|
||||
const TARGETS: RepairTarget[] = [
|
||||
{ table: 'pages', column: 'frontmatter', keyCol: 'slug' },
|
||||
{ table: 'raw_data', column: 'data', keyCol: 'source' },
|
||||
{ table: 'ingest_log', column: 'pages_updated', keyCol: 'source_ref' },
|
||||
{ table: 'files', column: 'metadata', keyCol: 'storage_path' },
|
||||
{ table: 'page_versions', column: 'frontmatter', keyCol: 'snapshot_at' },
|
||||
];
|
||||
|
||||
export interface RepairResult {
|
||||
engine: string;
|
||||
per_target: Array<{
|
||||
table: string;
|
||||
column: string;
|
||||
rows_repaired: number;
|
||||
}>;
|
||||
total_repaired: number;
|
||||
}
|
||||
|
||||
export interface RepairOpts {
|
||||
dryRun: boolean;
|
||||
/** Engine config override (for tests). Defaults to loadConfig() result. */
|
||||
engineConfig?: EngineConfig;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the repair against the currently-configured engine.
|
||||
*
|
||||
* On PGLite this finds 0 rows (the bug never affected the parameterized
|
||||
* encode path PGLite uses) and exits cleanly. On Postgres it issues one
|
||||
* idempotent UPDATE per target column.
|
||||
*/
|
||||
export async function repairJsonb(opts: RepairOpts = { dryRun: false }): Promise<RepairResult> {
|
||||
let engineCfg = opts.engineConfig;
|
||||
if (!engineCfg) {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
throw new Error('No brain configured. Run: gbrain init');
|
||||
}
|
||||
engineCfg = toEngineConfig(config);
|
||||
}
|
||||
const engineKind = engineCfg.engine || 'postgres';
|
||||
|
||||
const result: RepairResult = {
|
||||
engine: engineKind,
|
||||
per_target: [],
|
||||
total_repaired: 0,
|
||||
};
|
||||
|
||||
if (engineKind === 'pglite') {
|
||||
for (const t of TARGETS) {
|
||||
result.per_target.push({ table: t.table, column: t.column, rows_repaired: 0 });
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
await db.connect(engineCfg);
|
||||
const sql = db.getConnection();
|
||||
|
||||
for (const t of TARGETS) {
|
||||
let repaired = 0;
|
||||
|
||||
if (opts.dryRun) {
|
||||
const rows = await sql.unsafe(
|
||||
`SELECT count(*)::int AS n FROM ${t.table} WHERE jsonb_typeof(${t.column}) = 'string'`,
|
||||
);
|
||||
repaired = (rows[0] as { n: number }).n;
|
||||
} else {
|
||||
const rows = await sql.unsafe(
|
||||
`UPDATE ${t.table}
|
||||
SET ${t.column} = (${t.column} #>> '{}')::jsonb
|
||||
WHERE jsonb_typeof(${t.column}) = 'string'
|
||||
RETURNING 1`,
|
||||
);
|
||||
repaired = rows.length;
|
||||
}
|
||||
|
||||
result.per_target.push({ table: t.table, column: t.column, rows_repaired: repaired });
|
||||
result.total_repaired += repaired;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function runRepairJsonbCli(args: string[]): Promise<void> {
|
||||
const dryRun = args.includes('--dry-run');
|
||||
const jsonMode = args.includes('--json');
|
||||
|
||||
const result = await repairJsonb({ dryRun });
|
||||
|
||||
if (jsonMode) {
|
||||
console.log(JSON.stringify({ status: 'ok', dry_run: dryRun, ...result }));
|
||||
return;
|
||||
}
|
||||
|
||||
if (result.engine === 'pglite') {
|
||||
console.log('Engine: pglite — JSONB double-encode bug never affected this path. No-op.');
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(`${dryRun ? '[dry-run] ' : ''}Engine: postgres`);
|
||||
console.log(`${dryRun ? '[dry-run] ' : ''}JSONB repair across ${TARGETS.length} columns:`);
|
||||
for (const t of result.per_target) {
|
||||
const verb = dryRun ? 'would repair' : 'repaired';
|
||||
console.log(` ${t.table}.${t.column}: ${verb} ${t.rows_repaired} rows`);
|
||||
}
|
||||
console.log(`${dryRun ? '[dry-run] ' : ''}Total ${dryRun ? 'to repair' : 'repaired'}: ${result.total_repaired} rows`);
|
||||
if (!dryRun && result.total_repaired === 0) {
|
||||
console.log('Nothing to repair (already-valid JSONB or fresh install).');
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,23 @@ import type {
|
||||
EngineConfig,
|
||||
} from './types.ts';
|
||||
|
||||
/** Input row for addLinksBatch. Optional fields default to '' (matches NOT NULL DDL). */
|
||||
export interface LinkBatchInput {
|
||||
from_slug: string;
|
||||
to_slug: string;
|
||||
link_type?: string;
|
||||
context?: string;
|
||||
}
|
||||
|
||||
/** Input row for addTimelineEntriesBatch. Optional fields default to '' (matches NOT NULL DDL). */
|
||||
export interface TimelineBatchInput {
|
||||
slug: string;
|
||||
date: string;
|
||||
source?: string;
|
||||
summary: string;
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
/** Maximum results returned by search operations. Internal bulk operations (listPages) are not clamped. */
|
||||
export const MAX_SEARCH_LIMIT = 100;
|
||||
|
||||
@@ -53,6 +70,13 @@ export interface BrainEngine {
|
||||
|
||||
// Links
|
||||
addLink(from: string, to: string, context?: string, linkType?: string): Promise<void>;
|
||||
/**
|
||||
* Bulk insert links via a single multi-row INSERT...SELECT FROM (VALUES) JOIN pages
|
||||
* statement with ON CONFLICT DO NOTHING. Returns the count of rows actually inserted
|
||||
* (RETURNING clause excludes conflicts and JOIN-dropped rows whose slugs don't exist).
|
||||
* Used by extract.ts to avoid 47K sequential round-trips on large brains.
|
||||
*/
|
||||
addLinksBatch(links: LinkBatchInput[]): Promise<number>;
|
||||
/**
|
||||
* Remove links from `from` to `to`. If linkType is provided, only that specific
|
||||
* (from, to, type) row is removed. If omitted, ALL link types between the pair
|
||||
@@ -98,6 +122,13 @@ export interface BrainEngine {
|
||||
entry: TimelineInput,
|
||||
opts?: { skipExistenceCheck?: boolean },
|
||||
): Promise<void>;
|
||||
/**
|
||||
* Bulk insert timeline entries via a single multi-row INSERT...SELECT FROM (VALUES)
|
||||
* JOIN pages statement with ON CONFLICT DO NOTHING. Returns the count of rows
|
||||
* actually inserted (RETURNING excludes conflicts and JOIN-dropped rows whose
|
||||
* slugs don't exist). Used by extract.ts to avoid sequential round-trips.
|
||||
*/
|
||||
addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise<number>;
|
||||
getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]>;
|
||||
|
||||
// Raw data
|
||||
|
||||
+57
-23
@@ -22,14 +22,16 @@ export interface ParsedMarkdown {
|
||||
* tags: [startups, growth]
|
||||
* ---
|
||||
* Compiled truth content here...
|
||||
* ---
|
||||
*
|
||||
* <!-- timeline -->
|
||||
* Timeline content here...
|
||||
*
|
||||
* The first --- pair is YAML frontmatter (handled by gray-matter).
|
||||
* After frontmatter, the body is split at the first standalone ---
|
||||
* (a line containing only --- with optional whitespace).
|
||||
* Everything before is compiled_truth, everything after is timeline.
|
||||
* If no body --- exists, all content is compiled_truth.
|
||||
* After frontmatter, the body is split at the first recognized timeline
|
||||
* sentinel: `<!-- timeline -->` (preferred), `--- timeline ---` (decorated),
|
||||
* or a plain `---` immediately preceding a `## Timeline` / `## History`
|
||||
* heading (backward-compat for existing files). A bare `---` in body text
|
||||
* is treated as a markdown horizontal rule, not a timeline separator.
|
||||
*/
|
||||
export function parseMarkdown(content: string, filePath?: string): ParsedMarkdown {
|
||||
const { data: frontmatter, content: body } = matter(content);
|
||||
@@ -62,26 +64,21 @@ export function parseMarkdown(content: string, filePath?: string): ParsedMarkdow
|
||||
}
|
||||
|
||||
/**
|
||||
* Split body content at first standalone --- separator.
|
||||
* Split body content at the first recognized timeline sentinel.
|
||||
* Returns compiled_truth (before) and timeline (after).
|
||||
*
|
||||
* Recognized sentinels (in order of precedence):
|
||||
* 1. `<!-- timeline -->` — preferred, unambiguous, what serializeMarkdown emits
|
||||
* 2. `--- timeline ---` — decorated separator
|
||||
* 3. `---` ONLY when the next non-empty line is `## Timeline` or `## History`
|
||||
* (backward-compat fallback for older gbrain-written files)
|
||||
*
|
||||
* A plain `---` line is a markdown horizontal rule, NOT a timeline separator.
|
||||
* Treating bare `---` as a separator caused 83% content truncation on wiki corpora.
|
||||
*/
|
||||
export function splitBody(body: string): { compiled_truth: string; timeline: string } {
|
||||
// Match a line that is only --- (with optional whitespace)
|
||||
// Must not be at the very start (that would be frontmatter)
|
||||
const lines = body.split('\n');
|
||||
let splitIndex = -1;
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const trimmed = lines[i].trim();
|
||||
if (trimmed === '---') {
|
||||
// Skip if this is the very first non-empty line (leftover from frontmatter parsing)
|
||||
const beforeContent = lines.slice(0, i).join('\n').trim();
|
||||
if (beforeContent.length > 0) {
|
||||
splitIndex = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
const splitIndex = findTimelineSplitIndex(lines);
|
||||
|
||||
if (splitIndex === -1) {
|
||||
return { compiled_truth: body, timeline: '' };
|
||||
@@ -92,6 +89,33 @@ export function splitBody(body: string): { compiled_truth: string; timeline: str
|
||||
return { compiled_truth, timeline };
|
||||
}
|
||||
|
||||
function findTimelineSplitIndex(lines: string[]): number {
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const trimmed = lines[i].trim();
|
||||
|
||||
if (trimmed === '<!-- timeline -->' || trimmed === '<!--timeline-->') {
|
||||
return i;
|
||||
}
|
||||
|
||||
if (trimmed === '--- timeline ---' || /^---\s+timeline\s+---$/i.test(trimmed)) {
|
||||
return i;
|
||||
}
|
||||
|
||||
if (trimmed === '---') {
|
||||
const beforeContent = lines.slice(0, i).join('\n').trim();
|
||||
if (beforeContent.length === 0) continue;
|
||||
|
||||
for (let j = i + 1; j < lines.length; j++) {
|
||||
const next = lines[j].trim();
|
||||
if (next.length === 0) continue;
|
||||
if (/^##\s+(timeline|history)\b/i.test(next)) return i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize a page back to markdown format.
|
||||
* Produces: frontmatter + compiled_truth + --- + timeline
|
||||
@@ -116,7 +140,7 @@ export function serializeMarkdown(
|
||||
|
||||
let body = compiled_truth;
|
||||
if (timeline) {
|
||||
body += '\n\n---\n\n' + timeline;
|
||||
body += '\n\n<!-- timeline -->\n\n' + timeline;
|
||||
}
|
||||
|
||||
return yamlContent + '\n\n' + body + '\n';
|
||||
@@ -125,8 +149,18 @@ export function serializeMarkdown(
|
||||
function inferType(filePath?: string): PageType {
|
||||
if (!filePath) return 'concept';
|
||||
|
||||
// Normalize: add leading / for consistent matching
|
||||
// Normalize: add leading / for consistent matching.
|
||||
// Wiki subtypes and /writing/ check FIRST — they're stronger signals than
|
||||
// ancestor directories. e.g. `projects/blog/writing/essay.md` is a piece of
|
||||
// writing, not a project page; `tech/wiki/analysis/foo.md` is analysis,
|
||||
// not a hit on the broader `tech/` ancestor.
|
||||
const lower = ('/' + filePath).toLowerCase();
|
||||
if (lower.includes('/writing/')) return 'writing';
|
||||
if (lower.includes('/wiki/analysis/')) return 'analysis';
|
||||
if (lower.includes('/wiki/guides/') || lower.includes('/wiki/guide/')) return 'guide';
|
||||
if (lower.includes('/wiki/hardware/')) return 'hardware';
|
||||
if (lower.includes('/wiki/architecture/')) return 'architecture';
|
||||
if (lower.includes('/wiki/concepts/') || lower.includes('/wiki/concept/')) return 'concept';
|
||||
if (lower.includes('/people/') || lower.includes('/person/')) return 'person';
|
||||
if (lower.includes('/companies/') || lower.includes('/company/')) return 'company';
|
||||
if (lower.includes('/deals/') || lower.includes('/deal/')) return 'deal';
|
||||
|
||||
+15
-1
@@ -23,7 +23,9 @@ interface Migration {
|
||||
|
||||
// Migrations are embedded here, not loaded from files.
|
||||
// Add new migrations at the end. Never modify existing ones.
|
||||
const MIGRATIONS: Migration[] = [
|
||||
// Exported for tests that structurally assert migration contents (e.g., "v9 must
|
||||
// pre-create idx_timeline_dedup_helper before the DELETE..."). Read-only contract.
|
||||
export const MIGRATIONS: Migration[] = [
|
||||
// Version 1 is the baseline (schema.sql creates everything with IF NOT EXISTS).
|
||||
{
|
||||
version: 2,
|
||||
@@ -230,14 +232,20 @@ const MIGRATIONS: Migration[] = [
|
||||
// Idempotent for both upgrade and fresh-install paths.
|
||||
// Fresh installs already have links_from_to_type_unique from schema.sql; we drop it
|
||||
// (along with the legacy from-to-only constraint) before re-adding it cleanly.
|
||||
// Helper btree on the dedup columns turns the DELETE...USING self-join from O(n²)
|
||||
// into O(n log n). Without it, a brain with 80K+ duplicate link rows hits
|
||||
// Supabase Management API's 60s ceiling during upgrade.
|
||||
sql: `
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_page_id_to_page_id_key;
|
||||
ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_unique;
|
||||
CREATE INDEX IF NOT EXISTS idx_links_dedup_helper
|
||||
ON links(from_page_id, to_page_id, link_type);
|
||||
DELETE FROM links a USING links b
|
||||
WHERE a.from_page_id = b.from_page_id
|
||||
AND a.to_page_id = b.to_page_id
|
||||
AND a.link_type = b.link_type
|
||||
AND a.id > b.id;
|
||||
DROP INDEX IF EXISTS idx_links_dedup_helper;
|
||||
ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique
|
||||
UNIQUE(from_page_id, to_page_id, link_type);
|
||||
`,
|
||||
@@ -247,12 +255,18 @@ const MIGRATIONS: Migration[] = [
|
||||
name: 'timeline_dedup_index',
|
||||
// Idempotent: CREATE UNIQUE INDEX IF NOT EXISTS handles fresh + upgrade.
|
||||
// Dedup any existing duplicates first so the index can be created.
|
||||
// Helper btree turns the DELETE...USING self-join from O(n²) into O(n log n).
|
||||
// Without it, a brain with 80K+ duplicate timeline rows hits Supabase
|
||||
// Management API's 60s ceiling. See migration v8 for the same pattern.
|
||||
sql: `
|
||||
CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper
|
||||
ON timeline_entries(page_id, date, summary);
|
||||
DELETE FROM timeline_entries a USING timeline_entries b
|
||||
WHERE a.page_id = b.page_id
|
||||
AND a.date = b.date
|
||||
AND a.summary = b.summary
|
||||
AND a.id > b.id;
|
||||
DROP INDEX IF EXISTS idx_timeline_dedup_helper;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup
|
||||
ON timeline_entries(page_id, date, summary);
|
||||
`,
|
||||
|
||||
@@ -2,7 +2,7 @@ import { PGlite } from '@electric-sql/pglite';
|
||||
import { vector } from '@electric-sql/pglite/vector';
|
||||
import { pg_trgm } from '@electric-sql/pglite/contrib/pg_trgm';
|
||||
import type { Transaction } from '@electric-sql/pglite';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from './engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
||||
import { runMigrations } from './migrate.ts';
|
||||
import { PGLITE_SCHEMA_SQL } from './pglite-schema.ts';
|
||||
@@ -333,6 +333,29 @@ export class PGLiteEngine implements BrainEngine {
|
||||
);
|
||||
}
|
||||
|
||||
async addLinksBatch(links: LinkBatchInput[]): Promise<number> {
|
||||
if (links.length === 0) return 0;
|
||||
// unnest() pattern: 4 array-typed bound parameters regardless of batch size.
|
||||
// Same shape as PostgresEngine. Avoids the 65535-parameter cap entirely.
|
||||
const fromSlugs = links.map(l => l.from_slug);
|
||||
const toSlugs = links.map(l => l.to_slug);
|
||||
// Normalize optional fields to '' to match per-row addLink + NOT NULL DDL.
|
||||
const linkTypes = links.map(l => l.link_type || '');
|
||||
const contexts = links.map(l => l.context || '');
|
||||
const result = await this.db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context)
|
||||
SELECT f.id, t.id, v.link_type, v.context
|
||||
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[])
|
||||
AS v(from_slug, to_slug, link_type, context)
|
||||
JOIN pages f ON f.slug = v.from_slug
|
||||
JOIN pages t ON t.slug = v.to_slug
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type) DO NOTHING
|
||||
RETURNING 1`,
|
||||
[fromSlugs, toSlugs, linkTypes, contexts]
|
||||
);
|
||||
return result.rows.length;
|
||||
}
|
||||
|
||||
async removeLink(from: string, to: string, linkType?: string): Promise<void> {
|
||||
if (linkType !== undefined) {
|
||||
await this.db.query(
|
||||
@@ -593,6 +616,28 @@ export class PGLiteEngine implements BrainEngine {
|
||||
);
|
||||
}
|
||||
|
||||
async addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise<number> {
|
||||
if (entries.length === 0) return 0;
|
||||
// unnest() pattern: 5 array-typed bound parameters regardless of batch size.
|
||||
const slugs = entries.map(e => e.slug);
|
||||
const dates = entries.map(e => e.date);
|
||||
// Normalize optional fields to '' to match per-row addTimelineEntry + NOT NULL DDL.
|
||||
const sources = entries.map(e => e.source || '');
|
||||
const summaries = entries.map(e => e.summary);
|
||||
const details = entries.map(e => e.detail || '');
|
||||
const result = await this.db.query(
|
||||
`INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
||||
SELECT p.id, v.date::date, v.source, v.summary, v.detail
|
||||
FROM unnest($1::text[], $2::text[], $3::text[], $4::text[], $5::text[])
|
||||
AS v(slug, date, source, summary, detail)
|
||||
JOIN pages p ON p.slug = v.slug
|
||||
ON CONFLICT (page_id, date, summary) DO NOTHING
|
||||
RETURNING 1`,
|
||||
[slugs, dates, sources, summaries, details]
|
||||
);
|
||||
return result.rows.length;
|
||||
}
|
||||
|
||||
async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
|
||||
const limit = opts?.limit || 100;
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import postgres from 'postgres';
|
||||
import type { BrainEngine } from './engine.ts';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from './engine.ts';
|
||||
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
||||
import { runMigrations } from './migrate.ts';
|
||||
import { SCHEMA_SQL } from './schema-embedded.ts';
|
||||
@@ -17,7 +17,7 @@ import type {
|
||||
} from './types.ts';
|
||||
import { GBrainError } from './types.ts';
|
||||
import * as db from './db.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding } from './utils.ts';
|
||||
|
||||
export class PostgresEngine implements BrainEngine {
|
||||
private _sql: ReturnType<typeof postgres> | null = null;
|
||||
@@ -104,7 +104,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
const rows = await sql`
|
||||
INSERT INTO pages (slug, type, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
|
||||
VALUES (${slug}, ${page.type}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${JSON.stringify(frontmatter)}::jsonb, ${hash}, now())
|
||||
VALUES (${slug}, ${page.type}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter)}, ${hash}, now())
|
||||
ON CONFLICT (slug) DO UPDATE SET
|
||||
type = EXCLUDED.type,
|
||||
title = EXCLUDED.title,
|
||||
@@ -272,7 +272,8 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
const result = new Map<number, Float32Array>();
|
||||
for (const row of rows) {
|
||||
if (row.embedding) result.set(row.id as number, row.embedding as Float32Array);
|
||||
const parsed = parseEmbedding(row.embedding);
|
||||
if (parsed) result.set(row.id as number, parsed);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
@@ -372,6 +373,30 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
}
|
||||
|
||||
async addLinksBatch(links: LinkBatchInput[]): Promise<number> {
|
||||
if (links.length === 0) return 0;
|
||||
const sql = this.sql;
|
||||
// unnest() pattern: 4 array-typed bound parameters regardless of batch size.
|
||||
// Avoids the 65535-parameter cap and the postgres-js sql(rows, ...) helper's
|
||||
// identifier-escape gotcha when used inside a (VALUES) subquery.
|
||||
const fromSlugs = links.map(l => l.from_slug);
|
||||
const toSlugs = links.map(l => l.to_slug);
|
||||
// Normalize optional fields to '' to match per-row addLink + NOT NULL DDL.
|
||||
const linkTypes = links.map(l => l.link_type || '');
|
||||
const contexts = links.map(l => l.context || '');
|
||||
const result = await sql`
|
||||
INSERT INTO links (from_page_id, to_page_id, link_type, context)
|
||||
SELECT f.id, t.id, v.link_type, v.context
|
||||
FROM unnest(${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[], ${contexts}::text[])
|
||||
AS v(from_slug, to_slug, link_type, context)
|
||||
JOIN pages f ON f.slug = v.from_slug
|
||||
JOIN pages t ON t.slug = v.to_slug
|
||||
ON CONFLICT (from_page_id, to_page_id, link_type) DO NOTHING
|
||||
RETURNING 1
|
||||
`;
|
||||
return result.length;
|
||||
}
|
||||
|
||||
async removeLink(from: string, to: string, linkType?: string): Promise<void> {
|
||||
const sql = this.sql;
|
||||
if (linkType !== undefined) {
|
||||
@@ -629,6 +654,28 @@ export class PostgresEngine implements BrainEngine {
|
||||
`;
|
||||
}
|
||||
|
||||
async addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise<number> {
|
||||
if (entries.length === 0) return 0;
|
||||
const sql = this.sql;
|
||||
// unnest() pattern: 5 array-typed bound parameters regardless of batch size.
|
||||
const slugs = entries.map(e => e.slug);
|
||||
const dates = entries.map(e => e.date);
|
||||
// Normalize optional fields to '' to match per-row addTimelineEntry + NOT NULL DDL.
|
||||
const sources = entries.map(e => e.source || '');
|
||||
const summaries = entries.map(e => e.summary);
|
||||
const details = entries.map(e => e.detail || '');
|
||||
const result = await sql`
|
||||
INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
||||
SELECT p.id, v.date::date, v.source, v.summary, v.detail
|
||||
FROM unnest(${slugs}::text[], ${dates}::text[], ${sources}::text[], ${summaries}::text[], ${details}::text[])
|
||||
AS v(slug, date, source, summary, detail)
|
||||
JOIN pages p ON p.slug = v.slug
|
||||
ON CONFLICT (page_id, date, summary) DO NOTHING
|
||||
RETURNING 1
|
||||
`;
|
||||
return result.length;
|
||||
}
|
||||
|
||||
async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
|
||||
const sql = this.sql;
|
||||
const limit = opts?.limit || 100;
|
||||
@@ -665,7 +712,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
const sql = this.sql;
|
||||
const result = await sql`
|
||||
INSERT INTO raw_data (page_id, source, data)
|
||||
SELECT id, ${source}, ${JSON.stringify(data)}::jsonb
|
||||
SELECT id, ${source}, ${sql.json(data as Record<string, unknown>)}
|
||||
FROM pages WHERE slug = ${slug}
|
||||
ON CONFLICT (page_id, source) DO UPDATE SET
|
||||
data = EXCLUDED.data,
|
||||
@@ -843,7 +890,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
const sql = this.sql;
|
||||
await sql`
|
||||
INSERT INTO ingest_log (source_type, source_ref, pages_updated, summary)
|
||||
VALUES (${entry.source_type}, ${entry.source_ref}, ${JSON.stringify(entry.pages_updated)}::jsonb, ${entry.summary})
|
||||
VALUES (${entry.source_type}, ${entry.source_ref}, ${sql.json(entry.pages_updated)}, ${entry.summary})
|
||||
`;
|
||||
}
|
||||
|
||||
|
||||
+1
-1
@@ -1,5 +1,5 @@
|
||||
// Page types
|
||||
export type PageType = 'person' | 'company' | 'deal' | 'yc' | 'civic' | 'project' | 'concept' | 'source' | 'media';
|
||||
export type PageType = 'person' | 'company' | 'deal' | 'yc' | 'civic' | 'project' | 'concept' | 'source' | 'media' | 'writing' | 'analysis' | 'guide' | 'hardware' | 'architecture';
|
||||
|
||||
export interface Page {
|
||||
id: number;
|
||||
|
||||
+46
-1
@@ -43,6 +43,51 @@ export function rowToPage(row: Record<string, unknown>): Page {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize an embedding value into a Float32Array.
|
||||
*
|
||||
* pgvector returns embeddings in different shapes depending on driver/path:
|
||||
* - postgres.js (Postgres): often a string like `"[0.1,0.2,...]"`
|
||||
* - pglite: typically a numeric array or Float32Array
|
||||
* - pgvector node binding: numeric array
|
||||
* - Some queries that JSON-aggregate embeddings: JSON-string array
|
||||
*
|
||||
* Without normalization, downstream cosine math sees a string and produces
|
||||
* NaN scores silently. This helper guarantees a Float32Array or throws
|
||||
* loudly on malformed input — never returns NaN.
|
||||
*/
|
||||
export function parseEmbedding(value: unknown): Float32Array | null {
|
||||
if (value === null || value === undefined) return null;
|
||||
if (value instanceof Float32Array) return value;
|
||||
if (Array.isArray(value)) {
|
||||
if (value.length === 0) return new Float32Array(0);
|
||||
if (typeof value[0] !== 'number') {
|
||||
throw new Error(`parseEmbedding: array contains non-numeric element (${typeof value[0]})`);
|
||||
}
|
||||
return Float32Array.from(value as number[]);
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const trimmed = value.trim();
|
||||
// Plain non-vector strings: treat as "no embedding here", return null.
|
||||
// Strings that LOOK like vector literals but contain garbage: throw,
|
||||
// because that's a real corruption signal worth surfacing loudly.
|
||||
if (!trimmed.startsWith('[') || !trimmed.endsWith(']')) return null;
|
||||
const inner = trimmed.slice(1, -1).trim();
|
||||
if (inner.length === 0) return new Float32Array(0);
|
||||
const parts = inner.split(',');
|
||||
const out = new Float32Array(parts.length);
|
||||
for (let i = 0; i < parts.length; i++) {
|
||||
const n = Number(parts[i].trim());
|
||||
if (!Number.isFinite(n)) {
|
||||
throw new Error(`parseEmbedding: non-finite value at index ${i}: ${parts[i]}`);
|
||||
}
|
||||
out[i] = n;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export function rowToChunk(row: Record<string, unknown>, includeEmbedding = false): Chunk {
|
||||
return {
|
||||
id: row.id as number,
|
||||
@@ -50,7 +95,7 @@ export function rowToChunk(row: Record<string, unknown>, includeEmbedding = fals
|
||||
chunk_index: row.chunk_index as number,
|
||||
chunk_text: row.chunk_text as string,
|
||||
chunk_source: row.chunk_source as 'compiled_truth' | 'timeline',
|
||||
embedding: includeEmbedding && row.embedding ? row.embedding as Float32Array : null,
|
||||
embedding: includeEmbedding ? parseEmbedding(row.embedding) : null,
|
||||
model: row.model as string,
|
||||
token_count: row.token_count as number | null,
|
||||
embedded_at: row.embedded_at ? new Date(row.embedded_at as string) : null,
|
||||
|
||||
@@ -102,9 +102,10 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
|
||||
expect(plan.applied).toEqual([]);
|
||||
expect(plan.partial).toEqual([]);
|
||||
expect(plan.pending.map(m => m.version)).toContain('0.11.0');
|
||||
// v0.12.0 (Knowledge Graph auto-wire) is registered but installed VERSION
|
||||
// is 0.11.1, so it lands in skippedFuture until the binary catches up.
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0']);
|
||||
// v0.12.0 (Knowledge Graph) and v0.12.2 (JSONB repair) are registered but
|
||||
// installed VERSION is 0.11.1, so they land in skippedFuture until the
|
||||
// binary catches up.
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.0', '0.12.2']);
|
||||
});
|
||||
|
||||
test('already applied → v0.11.0 lands in `applied` bucket, not pending', () => {
|
||||
@@ -140,7 +141,10 @@ describe('buildPlan — diff against completed + installed VERSION', () => {
|
||||
const idx = indexCompleted([]);
|
||||
const plan = buildPlan(idx, '0.12.0');
|
||||
expect(plan.pending.map(m => m.version)).toContain('0.11.0');
|
||||
expect(plan.skippedFuture).toEqual([]);
|
||||
// v0.12.2 was added later (JSONB repair); installed=0.12.0 means it
|
||||
// belongs in skippedFuture, not pending. v0.11.0 and v0.12.0 stay
|
||||
// pending despite being ≤ installed — that is the H9 invariant.
|
||||
expect(plan.skippedFuture.map(m => m.version)).toEqual(['0.12.2']);
|
||||
});
|
||||
|
||||
test('--migration filter narrows to one version', () => {
|
||||
|
||||
@@ -285,6 +285,136 @@ describeE2E('E2E: Timeline', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Batch methods (addLinksBatch / addTimelineEntriesBatch)
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Postgres-engine batch methods use postgres-js's sql(rows, 'col1', ...) helper,
|
||||
// which is structurally different from PGLite's manual $N placeholder construction
|
||||
// (covered in test/pglite-engine.test.ts). These tests verify the postgres-js code
|
||||
// path against a real Postgres against the same invariants.
|
||||
|
||||
describeE2E('E2E: addLinksBatch (postgres-engine)', () => {
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
await importFixtures();
|
||||
});
|
||||
afterAll(teardownDB);
|
||||
|
||||
test('empty batch returns 0 with no DB call', async () => {
|
||||
const engine = getEngine();
|
||||
expect(await engine.addLinksBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('within-batch duplicates dedup via ON CONFLICT (no 21000 cardinality error)', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
// Deterministic cleanup so re-runs aren't perturbed by prior fixture state.
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-dup'`;
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-dup' },
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-dup' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-dup'`;
|
||||
});
|
||||
|
||||
test('rows with missing slug silently dropped by JOIN', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-missing'`;
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/does-not-exist', to_slug: 'companies/novamind', link_type: 'e2e-batch-missing' },
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-missing' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-missing'`;
|
||||
});
|
||||
|
||||
test('half-existing batch returns count of new only', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-half'`;
|
||||
await engine.addLink('people/sarah-chen', 'companies/novamind', 'pre-existing', 'e2e-batch-half');
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-half' },
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'people/marcus-reid', link_type: 'e2e-batch-half' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-half'`;
|
||||
});
|
||||
|
||||
test('missing optional fields normalize to empty strings (NOT NULL safety)', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM links WHERE link_type = ''`;
|
||||
// No link_type, no context — must default to '' to satisfy NOT NULL.
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
const rows = await conn`
|
||||
SELECT link_type, context FROM links
|
||||
WHERE from_page_id = (SELECT id FROM pages WHERE slug = 'people/sarah-chen')
|
||||
AND to_page_id = (SELECT id FROM pages WHERE slug = 'companies/novamind')
|
||||
AND link_type = ''
|
||||
`;
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].context).toBe('');
|
||||
await conn`DELETE FROM links WHERE link_type = ''`;
|
||||
});
|
||||
});
|
||||
|
||||
describeE2E('E2E: addTimelineEntriesBatch (postgres-engine)', () => {
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
await importFixtures();
|
||||
});
|
||||
afterAll(teardownDB);
|
||||
|
||||
test('empty batch returns 0', async () => {
|
||||
const engine = getEngine();
|
||||
expect(await engine.addTimelineEntriesBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('within-batch duplicates dedup via ON CONFLICT', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-dup'`;
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-01', summary: 'e2e-batch-tl-dup' },
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-01', summary: 'e2e-batch-tl-dup' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-dup'`;
|
||||
});
|
||||
|
||||
test('rows with missing slug silently dropped by JOIN', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-missing'`;
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'people/no-such-page', date: '2025-05-02', summary: 'e2e-batch-tl-missing' },
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-02', summary: 'e2e-batch-tl-missing' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-missing'`;
|
||||
});
|
||||
|
||||
test('mix of new + existing returns count of new only', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM timeline_entries WHERE summary IN ('e2e-batch-tl-half-1', 'e2e-batch-tl-half-2')`;
|
||||
await engine.addTimelineEntry('people/sarah-chen', { date: '2025-05-03', summary: 'e2e-batch-tl-half-1' });
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-03', summary: 'e2e-batch-tl-half-1' },
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-04', summary: 'e2e-batch-tl-half-2' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM timeline_entries WHERE summary IN ('e2e-batch-tl-half-1', 'e2e-batch-tl-half-2')`;
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Versions
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
/**
|
||||
* E2E JSONB round-trip tests — the test that should have caught the v0.12.0
|
||||
* silent-data-loss bug originally.
|
||||
*
|
||||
* v0.12.0-and-earlier wrote JSONB columns via `${JSON.stringify(value)}::jsonb`
|
||||
* which postgres.js v3 stringified again on the wire. Result: every JSONB
|
||||
* column stored a quoted-string literal instead of an object. Every
|
||||
* `frontmatter->>'key'` query returned NULL. PGLite was unaffected (different
|
||||
* driver path), which is why every previous unit test passed while real
|
||||
* Postgres-backed brains silently lost data.
|
||||
*
|
||||
* These tests exercise each of the four JSONB write sites and assert that:
|
||||
* 1. `jsonb_typeof(col) = 'object'` (or 'array' for array-shaped values)
|
||||
* — proves the column is a real JSONB structure, not a string literal.
|
||||
* 2. `col->>'key'` returns the expected scalar — proves downstream queries
|
||||
* and GIN indexes will work as intended.
|
||||
*
|
||||
* Without these E2E assertions, the CI grep guard in scripts/check-jsonb-pattern.sh
|
||||
* is the only protection — and it doesn't catch helper-wrapped or multi-line
|
||||
* variants of the buggy pattern.
|
||||
*
|
||||
* Run: DATABASE_URL=... bun test test/e2e/postgres-jsonb.test.ts
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import {
|
||||
hasDatabase, setupDB, teardownDB, getEngine, getConn,
|
||||
} from './helpers.ts';
|
||||
|
||||
const skip = !hasDatabase();
|
||||
const describeE2E = skip ? describe.skip : describe;
|
||||
|
||||
if (skip) {
|
||||
console.log('Skipping E2E JSONB round-trip tests (DATABASE_URL not set)');
|
||||
}
|
||||
|
||||
describeE2E('Postgres JSONB round-trip — frontmatter / data / pages_updated / metadata', () => {
|
||||
beforeAll(async () => { await setupDB(); });
|
||||
afterAll(async () => { await teardownDB(); });
|
||||
|
||||
test('pages.frontmatter — putPage stores object, not string literal', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
|
||||
await engine.putPage('jsonb-test/frontmatter', {
|
||||
type: 'concept',
|
||||
title: 'JSONB roundtrip',
|
||||
compiled_truth: 'body',
|
||||
frontmatter: { author: 'garry', score: 7, tags: ['x', 'y'] },
|
||||
});
|
||||
|
||||
const rows = await conn.unsafe(`
|
||||
SELECT
|
||||
jsonb_typeof(frontmatter) AS jt,
|
||||
frontmatter->>'author' AS author,
|
||||
frontmatter->>'score' AS score,
|
||||
frontmatter->'tags' AS tags
|
||||
FROM pages
|
||||
WHERE slug = 'jsonb-test/frontmatter'
|
||||
`);
|
||||
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0].jt).toBe('object');
|
||||
expect(rows[0].author).toBe('garry');
|
||||
expect(rows[0].score).toBe('7');
|
||||
expect(rows[0].tags).toEqual(['x', 'y']);
|
||||
});
|
||||
|
||||
test('raw_data.data — putRawData stores object, not string literal', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
|
||||
await engine.putPage('jsonb-test/raw', { type: 'concept', title: 't', compiled_truth: '' });
|
||||
await engine.putRawData('jsonb-test/raw', 'unit-test', { kind: 'fixture', count: 42 });
|
||||
|
||||
const rows = await conn.unsafe(`
|
||||
SELECT
|
||||
jsonb_typeof(rd.data) AS jt,
|
||||
rd.data->>'kind' AS kind,
|
||||
rd.data->>'count' AS count
|
||||
FROM raw_data rd
|
||||
JOIN pages p ON p.id = rd.page_id
|
||||
WHERE p.slug = 'jsonb-test/raw' AND rd.source = 'unit-test'
|
||||
`);
|
||||
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0].jt).toBe('object');
|
||||
expect(rows[0].kind).toBe('fixture');
|
||||
expect(rows[0].count).toBe('42');
|
||||
});
|
||||
|
||||
test('ingest_log.pages_updated — logIngest stores array, not string literal', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
|
||||
await engine.logIngest({
|
||||
source_type: 'unit-test',
|
||||
source_ref: 'jsonb-roundtrip',
|
||||
pages_updated: ['a/b', 'c/d', 'e/f'],
|
||||
summary: 'roundtrip-check',
|
||||
});
|
||||
|
||||
const rows = await conn.unsafe(`
|
||||
SELECT
|
||||
jsonb_typeof(pages_updated) AS jt,
|
||||
pages_updated->>0 AS first,
|
||||
jsonb_array_length(pages_updated) AS len
|
||||
FROM ingest_log
|
||||
WHERE source_ref = 'jsonb-roundtrip'
|
||||
`);
|
||||
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0].jt).toBe('array');
|
||||
expect(rows[0].first).toBe('a/b');
|
||||
expect(rows[0].len).toBe(3);
|
||||
});
|
||||
|
||||
test('files.metadata — write site uses sql.json, not string interpolation', async () => {
|
||||
const conn = getConn();
|
||||
|
||||
// Mimic the write at src/commands/files.ts:254 (the bonus fix).
|
||||
await conn`
|
||||
INSERT INTO files (filename, storage_path, mime_type, size_bytes, content_hash, metadata)
|
||||
VALUES (
|
||||
'roundtrip.bin',
|
||||
'unit-test/roundtrip.bin',
|
||||
'application/octet-stream',
|
||||
${0},
|
||||
'sha256:test',
|
||||
${conn.json({ type: 'archive', upload_method: 'unit-test' })}
|
||||
)
|
||||
`;
|
||||
|
||||
const rows = await conn.unsafe(`
|
||||
SELECT
|
||||
jsonb_typeof(metadata) AS jt,
|
||||
metadata->>'type' AS type,
|
||||
metadata->>'upload_method' AS method
|
||||
FROM files
|
||||
WHERE storage_path = 'unit-test/roundtrip.bin'
|
||||
`);
|
||||
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0].jt).toBe('object');
|
||||
expect(rows[0].type).toBe('archive');
|
||||
expect(rows[0].method).toBe('unit-test');
|
||||
});
|
||||
|
||||
test('page_versions.frontmatter — INSERT...SELECT propagates object shape', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
|
||||
await engine.putPage('jsonb-test/versioned', {
|
||||
type: 'concept',
|
||||
title: 'versioned',
|
||||
compiled_truth: 'v1',
|
||||
frontmatter: { mood: 'happy' },
|
||||
});
|
||||
await engine.createVersion('jsonb-test/versioned');
|
||||
|
||||
const rows = await conn.unsafe(`
|
||||
SELECT
|
||||
jsonb_typeof(pv.frontmatter) AS jt,
|
||||
pv.frontmatter->>'mood' AS mood
|
||||
FROM page_versions pv
|
||||
JOIN pages p ON p.id = pv.page_id
|
||||
WHERE p.slug = 'jsonb-test/versioned'
|
||||
`);
|
||||
|
||||
expect(rows.length).toBeGreaterThan(0);
|
||||
expect(rows[0].jt).toBe('object');
|
||||
expect(rows[0].mood).toBe('happy');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,161 @@
|
||||
/**
|
||||
* Tests for `gbrain extract --source fs` (the default, FS-walking path).
|
||||
*
|
||||
* Companion to test/extract-db.test.ts. Specifically guards against the
|
||||
* v0.12.0 N+1 hang: extractLinksFromDir / extractTimelineFromDir used to
|
||||
* pre-load the entire dedup set with one engine.getLinks() per page across
|
||||
* engine.listPages(), which on a 47K-page brain meant 47K sequential
|
||||
* round-trips before any work happened.
|
||||
*
|
||||
* Verifies:
|
||||
* 1. Single run extracts the expected links + timeline entries.
|
||||
* 2. Second run reports `created: 0` (proves DO NOTHING in batch + accurate
|
||||
* counter via RETURNING).
|
||||
* 3. --dry-run prints the same link found across multiple files exactly
|
||||
* once (proves the dry-run-only dedup Set works).
|
||||
* 4. Second run wall-clock < 2s (regression guard against any future change
|
||||
* that re-introduces the N+1 read pre-load).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runExtract } from '../src/commands/extract.ts';
|
||||
import type { PageInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let brainDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function truncateAll() {
|
||||
for (const t of ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages']) {
|
||||
await (engine as any).db.exec(`DELETE FROM ${t}`);
|
||||
}
|
||||
}
|
||||
|
||||
const personPage = (title: string, body = ''): PageInput => ({
|
||||
type: 'person', title, compiled_truth: body, timeline: '',
|
||||
});
|
||||
|
||||
const companyPage = (title: string, body = ''): PageInput => ({
|
||||
type: 'company', title, compiled_truth: body, timeline: '',
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-extract-fs-'));
|
||||
});
|
||||
|
||||
function writeFile(rel: string, content: string) {
|
||||
const full = join(brainDir, rel);
|
||||
mkdirSync(join(full, '..'), { recursive: true });
|
||||
writeFileSync(full, content);
|
||||
}
|
||||
|
||||
describe('gbrain extract links --source fs', () => {
|
||||
test('first run inserts links, second run reports 0 (idempotent + truthful counter)', async () => {
|
||||
// Set up brain in DB matching the file structure
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
await engine.putPage('people/bob', personPage('Bob'));
|
||||
await engine.putPage('companies/acme', companyPage('Acme'));
|
||||
|
||||
// Set up matching markdown files on disk
|
||||
writeFile('people/alice.md', '---\ntitle: Alice\n---\n\n[Bob](../people/bob.md) is a friend.\n');
|
||||
writeFile('people/bob.md', '---\ntitle: Bob\n---\n\nWorks at [Acme](../companies/acme.md).\n');
|
||||
writeFile('companies/acme.md', '---\ntitle: Acme\n---\n\nFounded by [Alice](../people/alice.md).\n');
|
||||
|
||||
// First run — write batch path
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
const linksAfter1 = (await engine.getLinks('people/alice'))
|
||||
.concat(await engine.getLinks('people/bob'))
|
||||
.concat(await engine.getLinks('companies/acme'));
|
||||
expect(linksAfter1.length).toBeGreaterThanOrEqual(3);
|
||||
|
||||
// Second run — must dedup via ON CONFLICT and report 0 new (truthful counter)
|
||||
const start = Date.now();
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
const linksAfter2 = (await engine.getLinks('people/alice'))
|
||||
.concat(await engine.getLinks('people/bob'))
|
||||
.concat(await engine.getLinks('companies/acme'));
|
||||
expect(linksAfter2.length).toBe(linksAfter1.length);
|
||||
|
||||
// Perf regression guard: re-run on tiny fixture must not loop through
|
||||
// listPages + per-page getLinks. ~10 files should complete in well under
|
||||
// 2s even on a slow CI box.
|
||||
expect(elapsedMs).toBeLessThan(2000);
|
||||
});
|
||||
|
||||
test('--dry-run dedups duplicate candidates across files (printed once, not N times)', async () => {
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
await engine.putPage('companies/acme', companyPage('Acme'));
|
||||
|
||||
// Same link target appears in 3 different files. The target file must
|
||||
// exist on disk so the FS extractor's allSlugs Set includes it.
|
||||
writeFile('companies/acme.md', '---\ntitle: Acme\n---\n');
|
||||
writeFile('a.md', '[Acme](companies/acme.md)\n');
|
||||
writeFile('b.md', '[Acme](companies/acme.md)\n');
|
||||
writeFile('c.md', '[Acme](companies/acme.md)\n');
|
||||
|
||||
// Capture stdout to check print frequency
|
||||
const lines: string[] = [];
|
||||
const origLog = console.log;
|
||||
console.log = (...args: unknown[]) => { lines.push(args.join(' ')); };
|
||||
try {
|
||||
await runExtract(engine, ['links', '--dry-run', '--dir', brainDir]);
|
||||
} finally {
|
||||
console.log = origLog;
|
||||
}
|
||||
|
||||
// Each (from, to, link_type) tuple should print at most once.
|
||||
// Three distinct from_slugs (a, b, c) all link to companies/acme, so
|
||||
// we expect 3 link lines (one per source file), not 9.
|
||||
const linkLines = lines.filter(l => l.includes('→') && l.includes('companies/acme'));
|
||||
expect(linkLines.length).toBe(3);
|
||||
|
||||
// No actual writes happened
|
||||
const links = await engine.getLinks('companies/acme');
|
||||
expect(links.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain extract timeline --source fs', () => {
|
||||
test('first run inserts entries, second run reports 0 (idempotent + truthful counter)', async () => {
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
|
||||
writeFile('people/alice.md', `---
|
||||
title: Alice
|
||||
---
|
||||
|
||||
## Timeline
|
||||
|
||||
- **2024-01-15** | source — Founded NovaMind
|
||||
- **2024-06-01** | source — Raised seed round
|
||||
`);
|
||||
|
||||
await runExtract(engine, ['timeline', '--dir', brainDir]);
|
||||
const after1 = await engine.getTimeline('people/alice');
|
||||
expect(after1.length).toBe(2);
|
||||
|
||||
const start = Date.now();
|
||||
await runExtract(engine, ['timeline', '--dir', brainDir]);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
const after2 = await engine.getTimeline('people/alice');
|
||||
expect(after2.length).toBe(2);
|
||||
|
||||
expect(elapsedMs).toBeLessThan(2000);
|
||||
});
|
||||
});
|
||||
@@ -252,7 +252,7 @@ title: Chunked
|
||||
|
||||
This is compiled truth content that should be chunked as compiled_truth source.
|
||||
|
||||
---
|
||||
<!-- timeline -->
|
||||
|
||||
- 2024-01-01: This is timeline content that should be chunked as timeline source.
|
||||
`);
|
||||
|
||||
+100
-18
@@ -2,7 +2,7 @@ import { describe, test, expect } from 'bun:test';
|
||||
import { parseMarkdown, serializeMarkdown, splitBody } from '../src/core/markdown.ts';
|
||||
|
||||
describe('Markdown Parser', () => {
|
||||
test('parses frontmatter + compiled_truth + timeline', () => {
|
||||
test('parses frontmatter + compiled_truth + timeline (explicit sentinel)', () => {
|
||||
const md = `---
|
||||
type: concept
|
||||
title: Do Things That Don't Scale
|
||||
@@ -11,7 +11,7 @@ tags: [startups, growth]
|
||||
|
||||
Paul Graham argues that startups should do unscalable things early on.
|
||||
|
||||
---
|
||||
<!-- timeline -->
|
||||
|
||||
- 2013-07-01: Published on paulgraham.com
|
||||
- 2024-11-15: Referenced in batch kickoff talk
|
||||
@@ -90,30 +90,75 @@ Content
|
||||
});
|
||||
|
||||
describe('splitBody', () => {
|
||||
test('splits at first standalone ---', () => {
|
||||
const body = 'Above the line\n\n---\n\nBelow the line';
|
||||
test('splits at <!-- timeline --> sentinel', () => {
|
||||
const body = 'Above the line\n\n<!-- timeline -->\n\nBelow the line';
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
expect(compiled_truth).toContain('Above the line');
|
||||
expect(timeline).toContain('Below the line');
|
||||
});
|
||||
|
||||
test('returns all as compiled_truth if no separator', () => {
|
||||
test('splits at --- timeline --- sentinel', () => {
|
||||
const body = 'Above the line\n\n--- timeline ---\n\nBelow the line';
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
expect(compiled_truth).toContain('Above the line');
|
||||
expect(timeline).toContain('Below the line');
|
||||
});
|
||||
|
||||
test('splits at --- when followed by ## Timeline heading', () => {
|
||||
const body = 'Article content\n\n---\n\n## Timeline\n\n- 2024: Event happened';
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
expect(compiled_truth).toContain('Article content');
|
||||
expect(timeline).toContain('## Timeline');
|
||||
expect(timeline).toContain('Event happened');
|
||||
});
|
||||
|
||||
test('splits at --- when followed by ## History heading', () => {
|
||||
const body = 'Article content\n\n---\n\n## History\n\n- 2020: Founded';
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
expect(compiled_truth).toContain('Article content');
|
||||
expect(timeline).toContain('## History');
|
||||
});
|
||||
|
||||
test('does NOT split at plain --- (horizontal rule in article body)', () => {
|
||||
const body = 'Above the line\n\n---\n\nBelow the line';
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
expect(compiled_truth).toBe(body);
|
||||
expect(timeline).toBe('');
|
||||
});
|
||||
|
||||
test('does NOT split on multiple plain --- horizontal rules', () => {
|
||||
const body = 'Section 1\n\n---\n\nSection 2\n\n---\n\nSection 3';
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
expect(compiled_truth).toBe(body);
|
||||
expect(timeline).toBe('');
|
||||
});
|
||||
|
||||
test('returns all as compiled_truth if no sentinel', () => {
|
||||
const body = 'Just some content\nWith multiple lines';
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
expect(compiled_truth).toBe(body);
|
||||
expect(timeline).toBe('');
|
||||
});
|
||||
|
||||
test('handles --- at end of content', () => {
|
||||
test('plain --- at end of content stays in compiled_truth', () => {
|
||||
const body = 'Content here\n\n---\n';
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
expect(compiled_truth).toContain('Content here');
|
||||
expect(timeline.trim()).toBe('');
|
||||
expect(compiled_truth).toBe(body);
|
||||
expect(timeline).toBe('');
|
||||
});
|
||||
|
||||
test('<!-- timeline --> with content before and after', () => {
|
||||
const body = '## Summary\n\nArticle summary here.\n\n---\n\nMore body content.\n\n<!-- timeline -->\n\n- 2024: Timeline entry';
|
||||
const { compiled_truth, timeline } = splitBody(body);
|
||||
expect(compiled_truth).toContain('## Summary');
|
||||
expect(compiled_truth).toContain('More body content.');
|
||||
expect(compiled_truth).not.toContain('Timeline entry');
|
||||
expect(timeline).toContain('Timeline entry');
|
||||
});
|
||||
});
|
||||
|
||||
describe('serializeMarkdown', () => {
|
||||
test('round-trips through parse and serialize', () => {
|
||||
test('round-trips through parse and serialize (explicit sentinel)', () => {
|
||||
const original = `---
|
||||
type: concept
|
||||
title: Do Things That Don't Scale
|
||||
@@ -125,7 +170,7 @@ custom: value
|
||||
|
||||
Paul Graham argues that startups should do unscalable things early on.
|
||||
|
||||
---
|
||||
<!-- timeline -->
|
||||
|
||||
- 2013-07-01: Published on paulgraham.com
|
||||
`;
|
||||
@@ -148,7 +193,7 @@ Paul Graham argues that startups should do unscalable things early on.
|
||||
});
|
||||
|
||||
describe('parseMarkdown edge cases', () => {
|
||||
test('handles content with multiple --- separators', () => {
|
||||
test('does NOT split on plain --- separators (horizontal rules stay in compiled_truth)', () => {
|
||||
const md = `---
|
||||
type: concept
|
||||
title: Test
|
||||
@@ -158,16 +203,38 @@ First section.
|
||||
|
||||
---
|
||||
|
||||
Timeline part 1.
|
||||
Second section.
|
||||
|
||||
---
|
||||
|
||||
More timeline.`;
|
||||
Third section.`;
|
||||
const parsed = parseMarkdown(md);
|
||||
// Only splits at the FIRST standalone ---
|
||||
expect(parsed.compiled_truth.trim()).toBe('First section.');
|
||||
expect(parsed.timeline).toContain('Timeline part 1.');
|
||||
expect(parsed.timeline).toContain('More timeline.');
|
||||
expect(parsed.compiled_truth).toContain('First section.');
|
||||
expect(parsed.compiled_truth).toContain('Second section.');
|
||||
expect(parsed.compiled_truth).toContain('Third section.');
|
||||
expect(parsed.timeline).toBe('');
|
||||
});
|
||||
|
||||
test('splits on <!-- timeline --> sentinel with horizontal rules in body', () => {
|
||||
const md = `---
|
||||
type: concept
|
||||
title: Test
|
||||
---
|
||||
|
||||
First section.
|
||||
|
||||
---
|
||||
|
||||
Second section.
|
||||
|
||||
<!-- timeline -->
|
||||
|
||||
- 2024: Timeline entry`;
|
||||
const parsed = parseMarkdown(md);
|
||||
expect(parsed.compiled_truth).toContain('First section.');
|
||||
expect(parsed.compiled_truth).toContain('Second section.');
|
||||
expect(parsed.compiled_truth).not.toContain('Timeline entry');
|
||||
expect(parsed.timeline).toContain('Timeline entry');
|
||||
});
|
||||
|
||||
test('handles frontmatter without type or title', () => {
|
||||
@@ -177,7 +244,7 @@ custom_field: hello
|
||||
|
||||
Some content.`;
|
||||
const parsed = parseMarkdown(md);
|
||||
expect(parsed.type).toBeTruthy(); // should have a default
|
||||
expect(parsed.type).toBeTruthy();
|
||||
expect(parsed.compiled_truth.trim()).toBe('Some content.');
|
||||
expect(parsed.frontmatter.custom_field).toBe('hello');
|
||||
});
|
||||
@@ -199,4 +266,19 @@ Some content.`;
|
||||
expect(parseMarkdown('', 'concepts/thing.md').type).toBe('concept');
|
||||
expect(parseMarkdown('', 'companies/acme.md').type).toBe('company');
|
||||
});
|
||||
|
||||
test('infers type from wiki subdirectory paths', () => {
|
||||
expect(parseMarkdown('', 'tech/wiki/concepts/longevity-science.md').type).toBe('concept');
|
||||
expect(parseMarkdown('', 'tech/wiki/guides/team-os-claude-code.md').type).toBe('guide');
|
||||
expect(parseMarkdown('', 'tech/wiki/analysis/agi-timeline-debate.md').type).toBe('analysis');
|
||||
expect(parseMarkdown('', 'tech/wiki/hardware/h100-vs-gb200-training-benchmarks.md').type).toBe('hardware');
|
||||
expect(parseMarkdown('', 'tech/wiki/architecture/kb-infrastructure.md').type).toBe('architecture');
|
||||
expect(parseMarkdown('', 'finance/wiki/analysis/polymarket-bot-automation-thesis.md').type).toBe('analysis');
|
||||
expect(parseMarkdown('', 'personal/wiki/concepts/career-regrets-2026-framework.md').type).toBe('concept');
|
||||
});
|
||||
|
||||
test('infers writing type from /writing/ paths', () => {
|
||||
expect(parseMarkdown('', 'writing/post.md').type).toBe('writing');
|
||||
expect(parseMarkdown('', 'projects/blog/writing/essay.md').type).toBe('writing');
|
||||
});
|
||||
});
|
||||
|
||||
+186
-3
@@ -1,5 +1,6 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { LATEST_VERSION } from '../src/core/migrate.ts';
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { LATEST_VERSION, runMigrations, MIGRATIONS } from '../src/core/migrate.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
describe('migrate', () => {
|
||||
test('LATEST_VERSION is a number >= 1', () => {
|
||||
@@ -8,10 +9,192 @@ describe('migrate', () => {
|
||||
});
|
||||
|
||||
test('runMigrations is exported and callable', async () => {
|
||||
const { runMigrations } = await import('../src/core/migrate.ts');
|
||||
expect(typeof runMigrations).toBe('function');
|
||||
});
|
||||
|
||||
// Integration tests for actual migration execution require DATABASE_URL
|
||||
// and are covered in the E2E suite (test/e2e/mechanical.test.ts)
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// REGRESSION TESTS — migrations v8 + v9 perf on duplicate-heavy tables
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Garry's production brain hit Supabase Management API's 60s ceiling because
|
||||
// the DELETE...USING self-join in migrations v8 + v9 was O(n²) without an
|
||||
// index on the dedup columns. The fix pre-creates a btree helper index
|
||||
// before the DELETE, then drops it. These tests guard against any future
|
||||
// change that re-introduces the missing helper index.
|
||||
//
|
||||
// Two-layer guard:
|
||||
// 1. Structural — assert the migration SQL literally contains the helper
|
||||
// CREATE INDEX + DROP INDEX (deterministic, fast, catches the regression
|
||||
// even at 0-row scale where wall-clock can't distinguish O(n²) from O(1)).
|
||||
// 2. Behavioral — populate 1000 duplicates and assert the migration completes
|
||||
// under the wall-clock cap. Sanity check at small scale; the structural
|
||||
// assertion is the real guard.
|
||||
|
||||
describe('migrations v8 + v9 — structural guard for helper-index fix', () => {
|
||||
test('migration v8 SQL contains idx_links_dedup_helper CREATE+DROP around the DELETE', () => {
|
||||
const v8 = MIGRATIONS.find(m => m.version === 8);
|
||||
expect(v8).toBeDefined();
|
||||
const sql = v8!.sql;
|
||||
|
||||
// The fix must: (a) create the helper btree, (b) DELETE...USING, (c) drop the helper, (d) add the unique constraint.
|
||||
// If anyone reorders or removes the helper-index lines, this fails.
|
||||
expect(sql).toContain('CREATE INDEX IF NOT EXISTS idx_links_dedup_helper');
|
||||
expect(sql).toContain('ON links(from_page_id, to_page_id, link_type)');
|
||||
expect(sql).toContain('DROP INDEX IF EXISTS idx_links_dedup_helper');
|
||||
expect(sql).toContain('DELETE FROM links a USING links b');
|
||||
expect(sql).toContain('ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique');
|
||||
|
||||
// Order matters: CREATE INDEX before DELETE, DROP INDEX after DELETE, before ADD CONSTRAINT.
|
||||
const createIdx = sql.indexOf('CREATE INDEX IF NOT EXISTS idx_links_dedup_helper');
|
||||
const deleteUsing = sql.indexOf('DELETE FROM links a USING links b');
|
||||
const dropIdx = sql.indexOf('DROP INDEX IF EXISTS idx_links_dedup_helper');
|
||||
const addConstraint = sql.indexOf('ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique');
|
||||
expect(createIdx).toBeLessThan(deleteUsing);
|
||||
expect(deleteUsing).toBeLessThan(dropIdx);
|
||||
expect(dropIdx).toBeLessThan(addConstraint);
|
||||
});
|
||||
|
||||
test('migration v9 SQL contains idx_timeline_dedup_helper CREATE+DROP around the DELETE', () => {
|
||||
const v9 = MIGRATIONS.find(m => m.version === 9);
|
||||
expect(v9).toBeDefined();
|
||||
const sql = v9!.sql;
|
||||
|
||||
expect(sql).toContain('CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper');
|
||||
expect(sql).toContain('ON timeline_entries(page_id, date, summary)');
|
||||
expect(sql).toContain('DROP INDEX IF EXISTS idx_timeline_dedup_helper');
|
||||
expect(sql).toContain('DELETE FROM timeline_entries a USING timeline_entries b');
|
||||
expect(sql).toContain('CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup');
|
||||
|
||||
const createHelper = sql.indexOf('CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper');
|
||||
const deleteUsing = sql.indexOf('DELETE FROM timeline_entries a USING timeline_entries b');
|
||||
const dropHelper = sql.indexOf('DROP INDEX IF EXISTS idx_timeline_dedup_helper');
|
||||
const createUnique = sql.indexOf('CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup');
|
||||
expect(createHelper).toBeLessThan(deleteUsing);
|
||||
expect(deleteUsing).toBeLessThan(dropHelper);
|
||||
expect(dropHelper).toBeLessThan(createUnique);
|
||||
});
|
||||
});
|
||||
|
||||
describe('migrate: v8 (links_dedup) regression — must be fast on 1K duplicate rows', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('1000 duplicate links dedup completes in <5s and leaves table deduped', async () => {
|
||||
// Set up: drop the unique constraint so duplicates can be inserted, then reset
|
||||
// version so v8 re-runs. Schema-embedded.ts already has the constraint, so
|
||||
// initSchema() above set it up; explicit DROP makes the test premise valid.
|
||||
const db = (engine as any).db;
|
||||
await db.exec(`ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_unique`);
|
||||
|
||||
// Two pages so the FK is satisfied
|
||||
await engine.putPage('p/from', { type: 'concept', title: 'F', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('p/to', { type: 'concept', title: 'T', compiled_truth: '', timeline: '' });
|
||||
const fromId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/from'`)).rows[0].id;
|
||||
const toId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/to'`)).rows[0].id;
|
||||
|
||||
// Insert 1000 duplicates of the same (from, to, type) row
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
await db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context) VALUES ($1, $2, $3, $4)`,
|
||||
[fromId, toId, 'mention', `dup-${i}`]
|
||||
);
|
||||
}
|
||||
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c;
|
||||
expect(beforeCount).toBe(1000);
|
||||
|
||||
// Reset version to 7 so v8 + v9 + v10 re-run
|
||||
await engine.setConfig('version', '7');
|
||||
|
||||
// Run migrations and assert wall-clock + correctness
|
||||
const start = Date.now();
|
||||
await runMigrations(engine);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
expect(elapsedMs).toBeLessThan(5000);
|
||||
|
||||
const afterCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c;
|
||||
expect(afterCount).toBe(1); // deduped to one row
|
||||
|
||||
// Unique constraint reinstated
|
||||
const constraints = (await db.query(`
|
||||
SELECT conname FROM pg_constraint
|
||||
WHERE conrelid = 'links'::regclass AND contype = 'u'
|
||||
`)).rows;
|
||||
expect(constraints.some((c: { conname: string }) => c.conname === 'links_from_to_type_unique')).toBe(true);
|
||||
|
||||
// Helper index was dropped after dedup
|
||||
const helperIdx = (await db.query(`
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'links' AND indexname = 'idx_links_dedup_helper'
|
||||
`)).rows;
|
||||
expect(helperIdx.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 1K duplicate rows', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('1000 duplicate timeline entries dedup completes in <5s and leaves table deduped', async () => {
|
||||
const db = (engine as any).db;
|
||||
await db.exec(`DROP INDEX IF EXISTS idx_timeline_dedup`);
|
||||
|
||||
await engine.putPage('p/timeline', { type: 'concept', title: 'TL', compiled_truth: '', timeline: '' });
|
||||
const pageId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/timeline'`)).rows[0].id;
|
||||
|
||||
// Insert 1000 duplicates of the same (page_id, date, summary) row
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
await db.query(
|
||||
`INSERT INTO timeline_entries (page_id, date, source, summary, detail) VALUES ($1, $2::date, $3, $4, $5)`,
|
||||
[pageId, '2024-01-15', `src-${i}`, 'Founded NovaMind', `detail-${i}`]
|
||||
);
|
||||
}
|
||||
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c;
|
||||
expect(beforeCount).toBe(1000);
|
||||
|
||||
await engine.setConfig('version', '7');
|
||||
|
||||
const start = Date.now();
|
||||
await runMigrations(engine);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
expect(elapsedMs).toBeLessThan(5000);
|
||||
|
||||
const afterCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c;
|
||||
expect(afterCount).toBe(1);
|
||||
|
||||
const uniqueIdx = (await db.query(`
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'timeline_entries' AND indexname = 'idx_timeline_dedup'
|
||||
`)).rows;
|
||||
expect(uniqueIdx.length).toBe(1);
|
||||
|
||||
const helperIdx = (await db.query(`
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'timeline_entries' AND indexname = 'idx_timeline_dedup_helper'
|
||||
`)).rows;
|
||||
expect(helperIdx.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
/**
|
||||
* Tests for the v0.12.2 JSONB-double-encode-repair orchestrator.
|
||||
*
|
||||
* Covers the contract that makes this migration safe to ship:
|
||||
* - Registered in the TS registry (so apply-migrations sees it).
|
||||
* - Phase functions exported via __testing for unit-level coverage.
|
||||
* - Dry-run skips all side-effect phases.
|
||||
* - Feature pitch explains what the user can NOW do that they couldn't.
|
||||
*
|
||||
* Idempotency, repair correctness, and PGLite-no-op behavior are exercised
|
||||
* end-to-end against real Postgres in test/e2e/postgres-jsonb.test.ts.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
|
||||
describe('v0.12.2 — JSONB double-encode repair migration', () => {
|
||||
test('registered in the TS migration registry', async () => {
|
||||
const { migrations, getMigration } = await import('../src/commands/migrations/index.ts');
|
||||
const versions = migrations.map(m => m.version);
|
||||
expect(versions).toContain('0.12.2');
|
||||
const m = getMigration('0.12.2');
|
||||
expect(m).not.toBeNull();
|
||||
expect(m!.featurePitch.headline).toContain('JSONB');
|
||||
expect(typeof m!.orchestrator).toBe('function');
|
||||
});
|
||||
|
||||
test('feature pitch lists the affected columns and the recovery path', async () => {
|
||||
const { v0_12_2 } = await import('../src/commands/migrations/v0_12_2.ts');
|
||||
const desc = v0_12_2.featurePitch.description ?? '';
|
||||
expect(desc).toContain('pages.frontmatter');
|
||||
expect(desc).toContain('raw_data.data');
|
||||
expect(desc).toContain('ingest_log.pages_updated');
|
||||
expect(desc).toContain('files.metadata');
|
||||
expect(desc).toContain('page_versions.frontmatter');
|
||||
expect(desc).toContain('gbrain sync --full');
|
||||
});
|
||||
|
||||
test('phase functions exported for unit testing', async () => {
|
||||
const { __testing } = await import('../src/commands/migrations/v0_12_2.ts');
|
||||
expect(typeof __testing.phaseASchema).toBe('function');
|
||||
expect(typeof __testing.phaseBRepair).toBe('function');
|
||||
expect(typeof __testing.phaseCVerify).toBe('function');
|
||||
});
|
||||
|
||||
test('dry-run skips all side-effect phases', async () => {
|
||||
const { v0_12_2 } = await import('../src/commands/migrations/v0_12_2.ts');
|
||||
const result = await v0_12_2.orchestrator({
|
||||
yes: true,
|
||||
dryRun: true,
|
||||
noAutopilotInstall: true,
|
||||
});
|
||||
expect(result.version).toBe('0.12.2');
|
||||
expect(result.phases.length).toBeGreaterThanOrEqual(3);
|
||||
for (const p of result.phases) {
|
||||
expect(p.status).toBe('skipped');
|
||||
expect(p.detail).toContain('dry-run');
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -350,6 +350,118 @@ describe('PGLiteEngine: Timeline', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Batch methods (addLinksBatch / addTimelineEntriesBatch)
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
describe('PGLiteEngine: addLinksBatch', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
await engine.putPage('a', { type: 'concept', title: 'A', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('b', { type: 'concept', title: 'B', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('c', { type: 'concept', title: 'C', compiled_truth: '', timeline: '' });
|
||||
});
|
||||
|
||||
test('empty batch returns 0 with no DB call', async () => {
|
||||
expect(await engine.addLinksBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('batch of 1 with missing optional fields inserts row with empty defaults', async () => {
|
||||
const inserted = await engine.addLinksBatch([{ from_slug: 'a', to_slug: 'b' }]);
|
||||
expect(inserted).toBe(1);
|
||||
const links = await engine.getLinks('a');
|
||||
expect(links.length).toBe(1);
|
||||
expect(links[0].context).toBe('');
|
||||
expect(links[0].link_type).toBe('');
|
||||
});
|
||||
|
||||
test('within-batch duplicates are deduped via ON CONFLICT (no 21000 error)', async () => {
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'a', to_slug: 'b', link_type: 'mention' },
|
||||
{ from_slug: 'a', to_slug: 'b', link_type: 'mention' },
|
||||
{ from_slug: 'a', to_slug: 'c', link_type: 'mention' },
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
});
|
||||
|
||||
test('rows with missing slug are silently dropped by JOIN', async () => {
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'doesnt-exist', to_slug: 'b' },
|
||||
{ from_slug: 'a', to_slug: 'b' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
|
||||
test('half-existing batch returns count of new only', async () => {
|
||||
await engine.addLink('a', 'b', '', 'mention');
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'a', to_slug: 'b', link_type: 'mention' },
|
||||
{ from_slug: 'a', to_slug: 'c', link_type: 'mention' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
|
||||
test('batch of 100 fresh rows returns 100', async () => {
|
||||
// Create 100 target pages
|
||||
for (let i = 0; i < 100; i++) {
|
||||
await engine.putPage(`target/${i}`, { type: 'concept', title: `T${i}`, compiled_truth: '', timeline: '' });
|
||||
}
|
||||
const batch = Array.from({ length: 100 }, (_, i) => ({
|
||||
from_slug: 'a', to_slug: `target/${i}`, link_type: 'mention',
|
||||
}));
|
||||
expect(await engine.addLinksBatch(batch)).toBe(100);
|
||||
});
|
||||
});
|
||||
|
||||
describe('PGLiteEngine: addTimelineEntriesBatch', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
await engine.putPage('p1', { type: 'concept', title: 'P1', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('p2', { type: 'concept', title: 'P2', compiled_truth: '', timeline: '' });
|
||||
});
|
||||
|
||||
test('empty batch returns 0', async () => {
|
||||
expect(await engine.addTimelineEntriesBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('batch of 1 with missing optionals inserts with empty defaults', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
const entries = await engine.getTimeline('p1');
|
||||
expect(entries.length).toBe(1);
|
||||
expect(entries[0].source).toBe('');
|
||||
expect(entries[0].detail).toBe('');
|
||||
});
|
||||
|
||||
test('within-batch duplicates are deduped via ON CONFLICT', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
{ slug: 'p1', date: '2024-02-01', summary: 'Launched' },
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
});
|
||||
|
||||
test('rows with missing slug are silently dropped by JOIN', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'no-such-page', date: '2024-01-15', summary: 'Phantom' },
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Real' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
|
||||
test('mix of new + existing returns count of new only', async () => {
|
||||
await engine.addTimelineEntry('p1', { date: '2024-01-15', summary: 'Founded' });
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
{ slug: 'p1', date: '2024-02-01', summary: 'Launched' },
|
||||
{ slug: 'p2', date: '2024-03-01', summary: 'Spun off' },
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Raw Data, Versions, Config, IngestLog
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
/**
|
||||
* Unit tests for `gbrain repair-jsonb`.
|
||||
*
|
||||
* The actual repair logic runs against real Postgres in
|
||||
* test/e2e/postgres-jsonb.test.ts (covers the round-trip + the migration
|
||||
* orchestrator end to end). Here we cover only the engine-detection
|
||||
* short-circuit: PGLite was never affected by the JSONB double-encode bug,
|
||||
* so the command must report 0 repaired rows and never connect.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { repairJsonb } from '../src/commands/repair-jsonb.ts';
|
||||
|
||||
describe('repairJsonb — PGLite short-circuit', () => {
|
||||
test('PGLite engines short-circuit: no DB connection, all targets report 0 repaired', async () => {
|
||||
const result = await repairJsonb({
|
||||
dryRun: false,
|
||||
engineConfig: { engine: 'pglite' },
|
||||
});
|
||||
expect(result.engine).toBe('pglite');
|
||||
expect(result.total_repaired).toBe(0);
|
||||
// All 5 columns reported: pages.frontmatter, raw_data.data,
|
||||
// ingest_log.pages_updated, files.metadata, page_versions.frontmatter.
|
||||
expect(result.per_target.length).toBe(5);
|
||||
for (const t of result.per_target) {
|
||||
expect(t.rows_repaired).toBe(0);
|
||||
}
|
||||
const tables = result.per_target.map(t => `${t.table}.${t.column}`).sort();
|
||||
expect(tables).toEqual([
|
||||
'files.metadata',
|
||||
'ingest_log.pages_updated',
|
||||
'page_versions.frontmatter',
|
||||
'pages.frontmatter',
|
||||
'raw_data.data',
|
||||
]);
|
||||
});
|
||||
});
|
||||
+47
-1
@@ -1,5 +1,5 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from '../src/core/utils.ts';
|
||||
import { validateSlug, contentHash, parseEmbedding, rowToPage, rowToChunk, rowToSearchResult } from '../src/core/utils.ts';
|
||||
|
||||
describe('validateSlug', () => {
|
||||
test('accepts valid slugs', () => {
|
||||
@@ -98,6 +98,52 @@ describe('rowToChunk', () => {
|
||||
}, true);
|
||||
expect(chunk.embedding).not.toBeNull();
|
||||
});
|
||||
|
||||
test('parses pgvector string embeddings when requested', () => {
|
||||
const chunk = rowToChunk({
|
||||
id: 1, page_id: 1, chunk_index: 0, chunk_text: 'text',
|
||||
chunk_source: 'compiled_truth', embedding: '[0.1, 0.2, 0.3]',
|
||||
model: 'test', token_count: 5, embedded_at: '2024-01-01',
|
||||
}, true);
|
||||
expect(chunk.embedding).toBeInstanceOf(Float32Array);
|
||||
expect(Array.from(chunk.embedding || [])).toHaveLength(3);
|
||||
expect(chunk.embedding?.[0]).toBeCloseTo(0.1, 6);
|
||||
expect(chunk.embedding?.[1]).toBeCloseTo(0.2, 6);
|
||||
expect(chunk.embedding?.[2]).toBeCloseTo(0.3, 6);
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseEmbedding', () => {
|
||||
test('returns Float32Array unchanged', () => {
|
||||
const emb = new Float32Array([0.1, 0.2]);
|
||||
expect(parseEmbedding(emb)).toBe(emb);
|
||||
});
|
||||
|
||||
test('parses pgvector text into Float32Array', () => {
|
||||
const parsed = parseEmbedding('[0.1, 0.2, 0.3]');
|
||||
expect(parsed).toBeInstanceOf(Float32Array);
|
||||
expect(Array.from(parsed || [])).toHaveLength(3);
|
||||
expect(parsed?.[0]).toBeCloseTo(0.1, 6);
|
||||
expect(parsed?.[1]).toBeCloseTo(0.2, 6);
|
||||
expect(parsed?.[2]).toBeCloseTo(0.3, 6);
|
||||
});
|
||||
|
||||
test('returns null for unsupported embedding values', () => {
|
||||
expect(parseEmbedding(null)).toBeNull();
|
||||
expect(parseEmbedding(undefined)).toBeNull();
|
||||
expect(parseEmbedding('not-a-vector')).toBeNull();
|
||||
});
|
||||
|
||||
test('parses numeric array into Float32Array', () => {
|
||||
const parsed = parseEmbedding([0.5, 0.25, 0.125]);
|
||||
expect(parsed).toBeInstanceOf(Float32Array);
|
||||
expect(parsed?.[0]).toBeCloseTo(0.5, 6);
|
||||
});
|
||||
|
||||
test('throws on vector-like string with non-numeric content (no silent NaN)', () => {
|
||||
expect(() => parseEmbedding('[abc, def]')).toThrow();
|
||||
expect(() => parseEmbedding('[1, NaN, 3]')).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('rowToSearchResult', () => {
|
||||
|
||||
Reference in New Issue
Block a user