Files
gbrain/test/e2e/multi-source.test.ts
T
89ae720959 v0.31.0 feat: hot memory — facts hook + recall CLI + MCP _meta + consolidate phase (#785)
* v0.31 feat(migrate): facts hot memory schema (migration v40)

Phase 1 of v0.31 hot-memory.

- New facts table with source_id (TEXT FK to sources, per-source isolation),
  kind CHECK (event/preference/commitment/belief/fact), visibility CHECK
  (private/world for takes-style ACL parity), valid_from/valid_until/
  expired_at/superseded_by for temporal + supersession audit, and
  consolidated_at/consolidated_into pointing at takes(id) for the dream-
  cycle hot→cold bridge.
- Embedding column dim resolved at migration time from
  config.embedding_dimensions so non-OpenAI brains (Voyage etc) work
  out-of-the-box. HALFVEC where pgvector >= 0.7; falls back to VECTOR
  with stderr warn on older versions. Matching opclass per column type
  (halfvec_cosine_ops vs vector_cosine_ops).
- 5 partial indexes leading on source_id so every read uses the trust
  boundary as part of the index, not a callback. HNSW partial index
  excludes expired/null rows so footprint stays proportional to active
  fact count.
- RLS DO-block matches takes pattern (Postgres BYPASSRLS gate; PGLite
  no-op).
- v0_31_0.ts orchestrator follows v0_28_0.ts pattern — phase A asserts
  schema version >= 40 + facts table presence; runner owns ledger.

All 87 existing migrate.test.ts cases pass. PGLite smoke test confirms
table + indexes + CHECK constraints + ON DELETE CASCADE all behave.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 chore(version): bump VERSION + package.json to 0.31.0

Phase 1 closer. CHANGELOG entry written when Phase 7 lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(engine): facts hot memory engine API (Phase 2)

Phase 2 of v0.31 hot-memory.

Adds 8 facts methods to BrainEngine implemented on both PGLite and
Postgres engines:

- insertFact(input, ctx) — INSERT with optional supersedeId; expires the
  named row in the same transaction. Per-entity advisory lock on Postgres
  (`pg_advisory_xact_lock(hashtextextended(source_id::text || ':' ||
  entity_slug, 0))`) for the dedup window. PGLite is single-process so
  the lock is a no-op.
- expireFact(id, opts) — sets expired_at + optional superseded_by.
  Idempotent-as-false (already-expired returns false).
- listFactsByEntity / listFactsSince / listFactsBySession — list surfaces
  with FactListOpts filters (activeOnly, kinds, visibility, limit/offset).
  Every query starts WHERE source_id = $X so the trust boundary is part
  of the index path.
- listSupersessions — audit log; activeOnly:false + expired_at IS NOT NULL
  + superseded_by IS NOT NULL.
- findCandidateDuplicates(source_id, entity_slug, factText, k) —
  entity-prefiltered (mandatory), k=5 default, hard cap 20. Embedding-
  cosine ordering when caller supplies an embedding, recency fallback
  otherwise. Bounds the contradiction-classifier blast radius.
- consolidateFact(id, takeId) — sets consolidated_at + consolidated_into.
  Never DELETE; facts stay as audit trail for the resulting take.
- getFactsHealth(source_id) — per-source counters consumed by `gbrain
  doctor` facts_health check.

Public types in engine.ts: FactKind (5-value union), FactVisibility,
FactInsertStatus, FactRow, NewFact, FactListOpts, FactsHealth.

PGLite + Postgres helpers: rowToFact / rowToFactPg parse the
text-format pgvector embedding back into Float32Array; toPgVectorLiteral
encodes for the supersede-path INSERT (postgres-js can't bind Float32Array
directly to a vector column without an explicit literal cast).

Smoke test confirms every method end-to-end on PGLite. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(facts): extraction code path (Phase 3)

Phase 3 of v0.31 hot-memory.

Five new modules under src/core/facts/ + src/core/entities/:

- src/core/facts/decay.ts — pure helper. effectiveConfidence(fact, now)
  applies confidence × exp(-age/halflife) with per-kind halflife table
  (event 7d, commitment 90d, preference 90d, belief 365d, fact 365d).
  Returns 0 for expired or past-valid_until rows. Single source of truth
  consumed by recall, supersession audit, facts_health, and the MCP _meta
  injector (eD8 DRY).

- src/core/facts/queue.ts — bounded in-memory queue. Cap 100 default,
  drop-oldest on overflow with counter. Per-session in-flight=1 serializes
  burst chat. AbortSignal threading from server SIGTERM (mirrors minion
  worker pattern per eD7): 5s grace for in-flight, then drop pending with
  counter. getFactsQueue() process-singleton; __resetFactsQueueForTests
  for hermetic tests.

- src/core/facts/classify.ts — contradiction classifier with cosine
  fast-path (D13: ≥0.95 → duplicate, skip LLM) and classifier-failure
  fallback (D12: cosine ≥0.92 → duplicate, else INSERT). Pure cosine
  helper exported. JSON-strict output with 4-strategy parse fallback;
  refusal stop-reason maps to fallback path. Caller-provided abort
  signal propagated to the gateway chat call.

- src/core/facts/extract.ts — Haiku turn-extractor. Reuses
  INJECTION_PATTERNS from src/core/think/sanitize.ts on the way IN
  (turn_text) AND on the way OUT (each fact). Tight system prompt with
  5-kind taxonomy, 0..1 confidence scoring, entity slug or display name.
  Anti-loop check on isDreamGenerated (reuses v0.23.2 marker semantics).
  Synchronous embedOne() per fact via the gateway so classifier paths
  have embeddings available; AbortError re-thrown explicitly so SIGTERM
  during embed never writes a NULL-embedding row meant to be cancelled
  (eE8 distinction).

- src/core/entities/resolve.ts — slug canonicalization shared by
  signal-detector AND facts. Resolution order: exact slug match →
  pg_trgm fuzzy match (similarity ≥0.4) → deterministic slugify
  fallback. slugify exported standalone for tests + callers that want
  the floor.

Smoke tests confirm decay table, cosine math, slugify rules, queue
drop-oldest under overflow, and shutdown grace + drop-pending semantics.
Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(mcp+cli): MCP ops + recall CLI + _meta + transport refactor (Phase 4)

Phase 4 of v0.31 hot-memory.

Three new MCP ops on the contract-first surface:

- `extract_facts` (write scope, localOnly:false): extracts facts from a
  conversation turn via the Haiku extractor, runs the cosine fast-path
  dedup, INSERTs into per-source hot memory. Returns counts +
  fact_ids[]. Skips on is_dream_generated:true (anti-loop).
- `recall` (read scope): query the per-source hot memory by
  entity / since / session / supersessions / grep filter. Visibility-
  aware: remote callers see visibility='world' rows only (takes-style
  ACL parity, eD21). Returns most-recent first; pagination via limit.
- `forget_fact` (write scope): expireFact wrapper. Idempotent-as-error
  on unknown id; uses the new 'fact_not_found' ErrorCode.

ErrorCode union opened (eD6 / eE7): TS forward-compat via the
`(string & {})` autocomplete-friendly hack so downstream consumers
(gbrain-evals etc) don't break their typecheck on every new code.
Three new codes: 'rate_limited', 'extraction_failed', 'fact_not_found'.

OperationContext gains source_id?:string (eD4 / eE2 — TEXT not INTEGER
per schema reality). Resolved once in buildOperationContext from
DispatchOpts.sourceId. Stdio MCP defaults to GBRAIN_SOURCE env or
'default'; HTTP MCP reads it from the per-token sources scope (eE3).

ToolResult gains _meta?: Record<string, unknown> (eD3). Dispatcher
calls a configurable metaHook AFTER op.handler succeeds, wrapped in
its own try/catch so a DB blip degrades to no-_meta rather than
flipping the whole tool call to error (eE4).

New module src/core/facts/meta-hook.ts:
- getBrainHotMemoryMeta(name, ctx) builds the _meta.brain_hot_memory
  payload. Cache key (source_id, session_id, hash(takesHoldersAllowList
  sorted)) (eD10 / eE5). 30s TTL per session. Visibility filter applies:
  remote → world only; local → all. Top-K=10 ranked by effective
  confidence (decay). Skips injection on recall/extract_facts/forget_fact
  themselves. bumpHotMemoryCache() invalidates per (source_id,
  session_id) on extraction event.

D12 (eE1) accepted: serve-http.ts:801 inlined dispatch path REFACTORED
to call dispatchToolCall. HTTP MCP now inherits source_id, _meta
injection, error envelope unification, and OperationContext shape from
the same code path stdio uses. Scope check + mcp_request_log + SSE
broadcast stay in serve-http.ts (HTTP-specific concerns); the dispatcher
returns ToolResult and the HTTP handler reads isError + content + _meta
to fan into the audit + broadcast paths.

put_page compliance backstop (D23): when a conversation-shape page is
written (note/meeting/slack/email/calendar-event/source/writing) with
a substantive body (>=80 chars) on a non-subagent slug AND no
dream_generated:true marker, fire-and-forget enqueue an extraction job
into the bounded queue. Never blocks the put_page response. Skipped
reasons (no_parsed_page / subagent_namespace / dream_generated /
kind:* / too_short / queue_shutdown / backstop_error) are stable
strings consumed by tests.

`gbrain recall` + `gbrain forget` CLI commands (src/commands/recall.ts):
- recall <entity> | --since DUR | --session ID | --today (markdown
  with kind icons 📅🎯🤝💭📌) | --grep TEXT | --supersessions |
  --include-expired | --as-context (prompt-injection-ready) | --json
- forget <fact-id> shorthand for expireFact

Wired into src/cli.ts dispatch table next to takes / think.

Smoke tests confirm: dispatch surfaces (extract_facts → ops →
listFactsByEntity), forget_fact + idempotent re-call, _meta visibility
filter (remote sees world only, local sees all), CLI markdown render
with kind icons + age strings + decayed confidence.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(cycle): consolidate phase — facts → takes promotion (Phase 5)

Phase 5 of v0.31 hot-memory.

New 10th cycle phase `consolidate` between `patterns` and `embed`:

- src/core/cycle.ts:
  * CyclePhase union extended with 'consolidate'
  * ALL_PHASES gets 'consolidate' between patterns and embed (graph-fresh
    after patterns; embed runs after so the new takes get embedded
    same-cycle)
  * NEEDS_LOCK_PHASES gets 'consolidate' (writes takes + UPDATEs facts)
  * CycleReport.totals gains facts_consolidated + consolidate_takes_written
  * runCycle dispatches the new phase via dynamic import

- src/core/cycle/phases/consolidate.ts (new):
  * Scans (source_id, entity_slug) buckets where COUNT(unconsolidated
    facts) >= 3 (uses idx_facts_unconsolidated partial index)
  * Skips buckets where the OLDEST fact is < 24h old (gives signal time
    to settle before locking it into cold memory)
  * Greedy cosine clustering at threshold 0.85; head-element centroid
    keeps it deterministic + cheap. Singletons (no embedding) stay
    unconsolidated this cycle.
  * For each cluster size >= 2: picks the highest-confidence fact's text
    as the take claim (v0.31 deterministic; v0.32 swaps to Sonnet
    synthesis pass). avg confidence → take weight, earliest valid_from →
    take since_date, concatenated source_sessions → take.source.
  * Resolves entity_slug → page_id via pages.slug (per source). Skips
    cluster if page is missing in this source — no auto-page-creation
    in v0.31.
  * INSERT into takes(kind='fact', holder='self') with row_num =
    MAX(existing) + 1.
  * UPDATE contributing facts: consolidated_at = now() +
    consolidated_into = takes.id. NEVER DELETE — facts are the audit
    trail for the resulting take.
  * dryRun honored: pretends the writes happened; counters still tick
    so operators can preview load before the first real run.
  * yieldDuringPhase keepalive between buckets so the Minions worker
    job lock + cycle-lock TTL don't drift on long runs.

Smoke test on PGLite confirms: 4 unconsolidated facts → clustered
(cosine 1.0 since same vector) → 1 take row created → all 4 facts
marked consolidated_into. runCycle({phases:['consolidate']}) wires
through to the report totals. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 test: 18 facts test files (Phase 6)

Phase 6 of v0.31 hot-memory: comprehensive coverage across the new
substrate. 110 unit tests pass; 5 E2E test files added (skip gracefully
without DATABASE_URL).

Unit tests (PGLite in-memory, no DATABASE_URL):
- test/facts-decay.test.ts (12 cases) — HALFLIFE_DAYS pinned per kind,
  effectiveConfidence math: age=0 / age=halflife (~1/e) / age=2×halflife
  (~1/e²) / expired returns 0 / valid_until past returns 0 /
  preference-vs-event slower decay / belief-vs-commitment crossover.
- test/facts-queue.test.ts (10 cases) — FIFO within session, drop-oldest
  on overflow, per-session in-flight=1 serializes, different sessions
  parallelize, failed jobs counter, shutdown grace + drop_pending +
  external AbortController triggers shutdown.
- test/facts-classify.test.ts (8 cases) — cosineSimilarity edge cases,
  empty candidates → independent, cheap fast-path ≥0.95 → duplicate
  no LLM, threshold-configurable cosine_fallback path.
- test/facts-engine.test.ts (13 cases) — every BrainEngine fact method
  end-to-end: insertFact (insert/supersede), expireFact idempotency,
  list*, findCandidateDuplicates entity-prefiltered + k cap + cosine
  ordering, consolidateFact never DELETE, getFactsHealth shape +
  total_today ⊆ total_week.
- test/facts-multi-tenant.test.ts (6 cases) — cross-source isolation
  on every list method + CASCADE delete on sources.
- test/facts-visibility.test.ts (6 cases) — visibility column private/
  world; remote=true filters to world-only via dispatchToolCall;
  remote=false sees all.
- test/facts-canonicality.test.ts (10 cases) — slugify rules including
  NFKD diacritic strip ("Crème Brûlée" → "creme-brulee"), exact slug
  match, fallback to slugify when no fuzzy match.
- test/facts-extract.test.ts (4 cases) — empty turn returns [], dream-
  generated short-circuit, graceful no-API-key return.
- test/facts-backstop-gating.test.ts (5 cases) — put_page backstop:
  too_short, subagent_namespace, dream_generated, eligible note path,
  non-eligible kind:guide.
- test/facts-anti-loop.test.ts (4 cases) — extractor + put_page both
  respect dream_generated:true marker.
- test/facts-doctor-shape.test.ts (4 cases) — facts_health JSON shape
  pinned for downstream consumers.
- test/facts-mcp-allowlist.serial.test.ts (5 cases) — extract_facts
  write-scope, recall read-scope, forget_fact write-scope, forget_fact
  fact_not_found error code, extract_facts no-API-key zero counts.
- test/facts-context-injection.serial.test.ts (6 cases) — _meta
  injection on success, world-only filter under remote=true, anti-loop
  on facts ops themselves, best-effort degrade on hook error,
  cache-key includes allow-list hash.
- test/facts-separation-pglite.test.ts (2 cases) — Garry's Separation
  Test as primary ship gate, plus expired hidden-by-default contract.
- test/facts-recall-render.test.ts (3 cases) — --today markdown render
  with all 5 kind icons, --json shape with effective_confidence,
  --as-context emits comment-wrapped block.
- test/facts-migration-dim.test.ts (4 cases) — embedding column type
  is HALFVEC/VECTOR (not arbitrary), dim matches gateway-configured
  embedding_dimensions, HNSW opclass agrees with column type, idempotent
  re-init.
- test/cycle-consolidate.test.ts (5 cases) — below-count + below-age
  thresholds skip, happy path 4 facts → 1 take + all consolidated never
  DELETE, dryRun honored, missing page → bucket skipped.

E2E tests (skip gracefully on DATABASE_URL unset; required gates by
CLAUDE.md test policy):
- test/e2e/facts-separation-postgres.test.ts — Postgres parity for the
  ship gate.
- test/e2e/facts-cross-source-isolation.test.ts — cross-source ACL on PG
  + CASCADE delete.
- test/e2e/facts-forget.test.ts — full forget_fact MCP roundtrip.
- test/e2e/facts-context-injection-postgres.test.ts — _meta injection
  end-to-end on PG.
- test/e2e/facts-recall-render.test.ts — recall --today markdown on PG.
- test/e2e/serve-http-meta.test.ts — eE1 regression: HTTP MCP transport
  inherits _meta + sourceId + scope correctness via dispatchToolCall.

Side-effect: src/core/entities/resolve.ts NFKD post-decompose strips
combining marks (U+0300..U+036F) before hyphenating non-alphanumerics,
so "Crème" → "creme", not "cre-me-".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(operational): kill switch + doctor check + CHANGELOG + README (Phase 7)

Phase 7 of v0.31 hot-memory.

- src/core/facts/extract.ts: new isFactsExtractionEnabled(engine) helper
  reads `facts.extraction_enabled` config row. Defaults to TRUE; flip to
  'false'/'0'/'no'/'off' (case-insensitive) via `gbrain config set
  facts.extraction_enabled false` to kill extraction across the brain
  without binary downgrade.
- extract_facts MCP op short-circuits with zero-counts envelope + a
  'skipped: extraction_disabled' field when the flag is off (clean
  success, not permission_denied).
- put_page facts backstop respects the same flag — eligibility check now
  returns 'extraction_disabled' as the skipped reason.
- src/commands/doctor.ts: new facts_health check (runs after queue_health,
  before index_audit). Probes for the facts table existence (post-v40
  guard), then surfaces total_active / total_today / total_week /
  total_consolidated + top-3 entities for the default source. Pre-v0.31
  brains report "facts table not present (pre-v0.31 brain or migration
  pending)".
- CHANGELOG.md: full v0.31.0 entry in the GStack release-summary voice.
  Headline + numbers-table + what-it-ships + itemized changes + "To take
  advantage of v0.31" upgrade block + out-of-scope. Honest about the
  HALFVEC + serve-http refactor + ErrorCode-open-union complications.
- README.md: cycle phase list updated 8 → 10 (consolidate + purge). New
  "v0.31 Hot Memory" command block under Commands with recall + forget
  variants, kind icons, --as-context surface for headless agents.

Test gates: 28 facts unit tests pass after the kill-switch wiring + doctor
check ride-along. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(migrate): add facts→sources FK explicitly via ALTER TABLE

The inline column-level FK declaration on facts.source_id worked on
PGLite but silently got dropped on Postgres in the v0.31 e2e run —
the migration handler ran via postgres-js's `unsafe()` multi-statement
path and the resulting facts table came back without the
`facts_source_id_fkey` constraint. Same psql input run directly
against the same database produced the FK; the difference was the
unsafe() pipeline, not the SQL itself.

Splitting the FK into a separate ALTER TABLE inside a DO block makes
the constraint declaration explicit and idempotent: the named
constraint either exists or it doesn't, the ALTER is a no-op on
re-runs, and the failure mode is loud rather than silently leaving
a CASCADE-less foreign key behind.

Without this fix, deleting a source row leaves orphaned facts rows
(test/e2e/facts-cross-source-isolation.test.ts CASCADE-on-sources-
delete case caught it). With this fix the constraint is in place,
the cascade fires, and both PG + PGLite e2e suites stay green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 test: update phase-count assertions for the new consolidate phase

Three e2e/unit tests pinned the cycle phase count or order, all now
updated to reflect v0.31's 10-phase cycle:

- test/e2e/dream-cycle-eight-phase-pglite.test.ts:
  describe rename "8-phase cycle" → "10-phase cycle"; ALL_PHASES
  expectation extended to include 'consolidate' (between patterns +
  embed) and 'purge' (the v0.26.5 addition that was already in
  ALL_PHASES but missing from the test's assertion list). totals
  match adds the new facts_consolidated + consolidate_takes_written
  fields plus the pre-existing purged_sources_count + purged_pages_count
  that should have been added when v0.26.5 landed.

- test/e2e/cycle.test.ts: dry-run full cycle now expects
  report.phases.length === 10 (was 9).

- test/core/cycle.serial.test.ts: yieldBetweenPhases hook count + full
  cycle phases.length both updated 9 → 10. Comments call out the
  v0.31 addition lineage so the next person to add a phase sees the
  precedent.

These are mechanical assertion bumps. The tests pass against the
updated assertions on PGLite and Postgres.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(test): truncate facts table between e2e describe blocks

setupDB() truncates ALL_TABLES between every describe block's
beforeAll() hook. The list missed the new v0.31 facts table, so
facts seeded by an earlier describe block leaked into Garry's
Separation Test on Postgres — listFactsByEntity('travel') returned
2 rows instead of 1 because a prior facts-context-injection test had
also seeded a 'travel' fact.

Adding 'facts' to the truncate list (before 'pages' to respect FK
ordering) makes every describe-block start from an empty facts table.

Pinned by re-running the e2e file ordering that originally caught it
(facts-recall-render → cross-source-isolation → serve-http-meta →
context-injection → separation-postgres → facts-forget) — 13 pass /
0 fail after the fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 test: meta-hook cache + Postgres consolidate phase coverage

Two net-new test files filling real coverage gaps the earlier sweep missed:

- test/facts-meta-cache.test.ts (5 cases) — pins the eD3/eD10 cache
  contract that the dispatcher relies on. 30s TTL hit path, post-bump
  fresh-query, scoped invalidation (bump for sess-A leaves sess-B cache
  warm — closes the cross-source leak risk codex F5 originally surfaced
  on the recall payload), facts-self ops skip injection (anti-loop on
  recall / extract_facts / forget_fact), distinct allow-lists produce
  distinct cache entries.

- test/e2e/cycle-consolidate-postgres.test.ts (3 cases) — Postgres
  parity for the dream-cycle consolidate phase. Mirrors the PGLite
  unit test but exercises the real postgres-engine codepaths: sql.begin
  transactions, advisory locks on insertFact's entity-slug dedup window,
  unsafe('::vector') casts on findCandidateDuplicates ordering,
  addTakesBatch postgres-js unnest path. Happy path (4 facts → 1 take +
  all consolidated_into set), age-threshold skip, dry-run no-write.

All 5 unit + 3 e2e tests pass. Closes the unit-only gap on the
consolidate phase (was only PGLite-tested) and pins meta-cache
invariants the dispatcher depends on.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix: thread auth + sourceId, JSON-shape every error envelope

Three bugs surfaced during the full e2e sweep that all trace back to my
v0.31 dispatch refactor (D12/eE1) silently dropping auth threading +
non-OperationError exceptions emitting plain strings:

1. **HTTP MCP transport lost ctx.auth.** Refactoring serve-http.ts to call
   dispatchToolCall meant auth had to come through DispatchOpts, but the
   field didn't exist yet. Every HTTP whoami call returned
   `unknown_transport` because ctx.auth was undefined. Added `auth?:
   AuthInfo` to DispatchOpts, plumbed it through buildOperationContext,
   and updated serve-http.ts:816 to pass `auth: authInfo` alongside
   sourceId/takesHoldersAllowList. Pinned by sources-remote-mcp e2e
   `whoami reports oauth transport + sources_admin scope`.

2. **Non-OperationError exceptions emitted plain strings, not JSON.**
   The pre-v0.31 serve-http.ts always wrapped errors in JSON envelope
   `{error, message}`; my dispatch refactor missed the unknown-tool +
   uncaught-throw paths and emitted `Error: ${msg}` text content. Every
   caller that did `JSON.parse(content)` (sources-remote-mcp callMcp
   helper at line 104) crashed with `Unexpected identifier "Error"`.
   Both error paths in dispatchToolCall now return JSON-shaped content
   matching the OperationError pattern.

3. **Files→sources FK silently lost on rewound bootstrap path.**
   test/e2e/postgres-bootstrap.test.ts simulates a pre-v0.21 brain by
   `DROP TABLE IF EXISTS sources CASCADE` which removes
   files_source_id_fkey while leaving files.source_id intact. The v23
   migration's `ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id ...
   REFERENCES sources(id) ON DELETE CASCADE` is a no-op when the column
   exists, so the FK never came back on upgrade — and any sources-remove
   afterward stopped cascading to files. Added a defensive
   `IF NOT EXISTS files_source_id_fkey ... ALTER TABLE ADD CONSTRAINT`
   block inside v23's handler. Pinned by `multi-source — cascade delete
   covers every dependent row` after running postgres-bootstrap.

Plus: src/core/preferences.ts now honors GBRAIN_HOME for
`~/.gbrain/migrations/completed.jsonl`. Without this, the doctor
exits-0 mechanical test inherits the developer machine's stale
partial-migration ledger entries (0.21.0, 0.22.4, 0.28.0, 0.29.1
prior dev work) and surfaces them as the [FAIL] minions_migration check.
GBRAIN_HOME-scoped tempdir per test now isolates this state cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 chore: scrub personal references from public artifacts

Per the CLAUDE.md privacy rule on `Garry's Separation Test`, replace
personally-coded references in v0.31 artifacts with neutral examples:

- CHANGELOG.md v0.31 entry: rename "Garry's Separation Test" header to
  "The cross-session test" + drop the "topic-2659/topic-1941, 7 AM/2 PM,
  flying to Tokyo" narrative.
- src/commands/migrations/v0_31_0.ts feature pitch: same scrub.
- test/facts-separation-pglite.test.ts + test/e2e/facts-separation-postgres.test.ts:
  rename describe blocks; replace specific topic-NNNN session ids with
  session-A / session-B; replace personal sample fact with
  "sample event Tuesday".
- src/core/facts/extract.ts extractor system prompt example slugs:
  people/sam-altman → people/alice-example; companies/anthropic → companies/acme.
- src/core/entities/resolve.ts comment: Sam Altman → Alice Example.
- All v0.31 test fixtures: people/sam → people/alice-example,
  Sam Altman → Alice Example, sam-the-cofounder → alice-the-cofounder.
  Test names referencing real-world entities replaced with neutral slugs.

Pre-existing references to "Garry" elsewhere in CHANGELOG (v0.17, v0.19,
v0.21+ entries) are untouched — that's a separate scope from this v0.31
ship.

Plus: the truncate fix for the Bun-script-induced syntax error in
test/e2e/mechanical.test.ts (cliEnv arrow function had ", 30_000)" tacked
onto its closing brace by the bulk-add-timeouts script — repaired to a
clean function definition).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(test): bump E2E phase-count assertions for 11-phase cycle

Two E2E tests still asserted the v0.31 pre-merge 10-phase shape
(consolidate inserted, but recompute_emotional_weight from v0.29 not yet
absorbed). With master's v0.29 work merged in, the cycle is now 11 phases:
lint → backlinks → sync → synthesize → extract → patterns →
recompute_emotional_weight → consolidate → embed → orphans → purge.

- test/e2e/cycle.test.ts: 10 → 11
- test/e2e/dream-cycle-eight-phase-pglite.test.ts: ALL_PHASES + dry-run order

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(merge): close brace between v44 and v45 migration objects

The v0.30.2 merge resolution stitched master's v40-v44 migrations onto
HEAD's v45 (facts hot memory) migration but lost the closing `},` between
v44 and v45. tsc caught it as TS1136 Property assignment expected at
migrate.ts:2188.

This is a one-line bracket fix; the rest of the merge resolution is
correct and tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix: put_page cliHints + buildPlan v0.31.0 in skippedFuture

Two unit-test failures surfaced after the v0.30.2 merge:

1. operations.ts: put_page had `cliHints: { name: 'put', positional: ['stdin'] }`
   from earlier v0.31 development. The parity test enforces that every name
   in `positional` is a real param. Restored master's correct shape:
   `{ name: 'put', positional: ['slug'], stdin: 'content' }`.

2. test/apply-migrations.test.ts: the H9 regression tests pin the exact
   skippedFuture list. Adding v0.31.0 to the registry meant the list grew
   by one. Updated both `expect(...).toEqual([...])` assertions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 docs: clarify consolidate is 11th phase + regen llms-full.txt

CHANGELOG.md narrative said "new 10th phase consolidate"; with v0.29's
recompute_emotional_weight already on master, consolidate is the 11th phase
(between recompute and embed). Schema migration is v45, not v40, after the
merge resolution renumbered it to clear master's v40-v44.

llms-full.txt regenerated to reflect the README's 11-phase dream-cycle
phrasing (the build-llms test enforces commit-time parity).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 16:57:47 -07:00

610 lines
26 KiB
TypeScript

/**
* E2E: v0.18.0 multi-source migrations against REAL Postgres.
*
* PGLite doesn't have a files table (see pglite-schema.ts header), so the
* v23 migration's files.source_id + files.page_id rewrite + ledger seed
* is NEVER executed by the PGLite integration test. This file closes
* that gap by exercising the full v20-v23 chain against a real Postgres
* DB with pre-existing data.
*
* Also covers the gaps in the PR's pre-shipping test matrix that the
* author self-audited:
* - files.page_slug → page_id backfill against real rows
* - file_migration_ledger seeding
* - cascade delete via sources.remove (pages + chunks + timeline +
* files + links all gone)
* - sync --source <id> routing reads + writes per-source sync anchors
* instead of the global config keys
*
* Gated by DATABASE_URL — skips gracefully when unset, per the CLAUDE.md
* E2E lifecycle pattern.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
import { runSources } from '../../src/commands/sources.ts';
import { performSync } from '../../src/commands/sync.ts';
import { runStorageBackfill } from '../../src/commands/migrations/v0_18_0-storage-backfill.ts';
import type { StorageBackend } from '../../src/core/storage.ts';
import { hasDatabase, setupDB, teardownDB, getConn, getEngine } from './helpers.ts';
const SKIP = !hasDatabase();
const describeE2E = SKIP ? describe.skip : describe;
describeE2E('v0.18.0 multi-source — Postgres schema shape (fresh install)', () => {
beforeAll(async () => {
await setupDB();
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
// residual rows from prior test runs can shadow new INSERTs. Wipe
// non-default sources at the top of every describe to keep each
// block hermetic. file_migration_ledger cascades from files which
// setupDB already truncates, but wipe explicitly in case files did
// not cascade it.
const conn = getConn();
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
await conn.unsafe(`DELETE FROM file_migration_ledger`);
}, 30_000);
afterAll(async () => {
await teardownDB();
});
test("sources('default') exists after initSchema + migration chain", async () => {
const conn = getConn();
const rows = await conn.unsafe(
`SELECT id, name, config FROM sources WHERE id = 'default'`,
);
expect(rows.length).toBe(1);
expect(rows[0].name).toBe('default');
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
expect(config.federated).toBe(true);
});
test('pages.source_id NOT NULL with DEFAULT default (v21)', async () => {
const conn = getConn();
const rows = await conn.unsafe(
`SELECT column_name, column_default, is_nullable
FROM information_schema.columns
WHERE table_name = 'pages' AND column_name = 'source_id'`,
);
expect(rows.length).toBe(1);
expect(rows[0].is_nullable).toBe('NO');
expect(String(rows[0].column_default)).toContain('default');
});
test('composite UNIQUE pages(source_id, slug) replaces global UNIQUE(slug)', async () => {
const conn = getConn();
const composite = await conn.unsafe(
`SELECT conname FROM pg_constraint WHERE conname = 'pages_source_slug_key'`,
);
expect(composite.length).toBe(1);
const oldGlobal = await conn.unsafe(
`SELECT conname FROM pg_constraint WHERE conname = 'pages_slug_key'`,
);
expect(oldGlobal.length).toBe(0);
});
test('links.resolution_type column exists with CHECK (v22)', async () => {
const conn = getConn();
const rows = await conn.unsafe(
`SELECT column_name FROM information_schema.columns
WHERE table_name = 'links' AND column_name = 'resolution_type'`,
);
expect(rows.length).toBe(1);
const check = await conn.unsafe(
`SELECT conname FROM pg_constraint WHERE conname = 'links_resolution_type_check'`,
);
expect(check.length).toBe(1);
});
test('files.source_id + files.page_id columns exist (v23, Postgres-only)', async () => {
const conn = getConn();
const cols = await conn.unsafe(
`SELECT column_name FROM information_schema.columns
WHERE table_name = 'files' AND column_name IN ('source_id', 'page_id')`,
);
// postgres.js returns RowList with an iterable-row shape; cast via
// unknown before narrowing to plain objects (TS2352 otherwise).
const names = new Set(
(cols as unknown as Array<{ column_name: string }>).map(r => r.column_name),
);
expect(names.has('source_id')).toBe(true);
expect(names.has('page_id')).toBe(true);
});
test('file_migration_ledger table exists with status CHECK (v23)', async () => {
const conn = getConn();
const tables = await conn.unsafe(
`SELECT table_name FROM information_schema.tables
WHERE table_name = 'file_migration_ledger'`,
);
expect(tables.length).toBe(1);
const check = await conn.unsafe(
`SELECT conname FROM pg_constraint WHERE conname = 'chk_ledger_status'`,
);
expect(check.length).toBe(1);
});
});
describeE2E('v0.18.0 multi-source — composite UNIQUE semantics on real Postgres', () => {
beforeAll(async () => {
await setupDB();
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
// residual rows from prior test runs can shadow new INSERTs. Wipe
// non-default sources at the top of every describe to keep each
// block hermetic. file_migration_ledger cascades from files which
// setupDB already truncates, but wipe explicitly in case files did
// not cascade it.
const conn = getConn();
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
await conn.unsafe(`DELETE FROM file_migration_ledger`);
}, 30_000);
afterAll(async () => {
await teardownDB();
});
test('same slug in two sources coexists (REGRESSION GUARD — Codex critical)', async () => {
const conn = getConn();
// Create a second source.
const engine = getEngine();
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['add', 'wiki', '--federated']);
// Insert the same slug under 'default' (via putPage) and 'wiki' (raw INSERT).
await engine.putPage('topics/ai', {
type: 'concept', title: 'AI from default', compiled_truth: 'default source take',
});
await conn.unsafe(
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
VALUES ('wiki', 'topics/ai', 'concept', 'AI from wiki', 'wiki source take', '', '{}'::jsonb, 'wikihash')`,
);
const rows = await conn.unsafe(
`SELECT source_id, slug, title FROM pages WHERE slug = 'topics/ai' ORDER BY source_id`,
);
expect(rows.length).toBe(2);
expect(rows.map((r: any) => r.source_id).sort()).toEqual(['default', 'wiki']);
});
test('duplicate (source_id, slug) hits composite UNIQUE', async () => {
const conn = getConn();
let err: Error | null = null;
try {
await conn.unsafe(
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
VALUES ('wiki', 'topics/ai', 'concept', 'dup', '', '', '{}'::jsonb, 'dup')`,
);
} catch (e) {
err = e as Error;
}
expect(err).not.toBeNull();
expect(err!.message.toLowerCase()).toMatch(/unique|duplicate/);
});
test('putPage (engine API) targets default source by schema DEFAULT', async () => {
const engine = getEngine();
await engine.putPage('topics/from-putpage', {
type: 'note', title: 'Via putPage', compiled_truth: 'body',
});
const conn = getConn();
const rows = await conn.unsafe(
`SELECT source_id FROM pages WHERE slug = 'topics/from-putpage'`,
);
expect(rows.length).toBe(1);
expect(rows[0].source_id).toBe('default');
});
});
describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row', () => {
beforeAll(async () => {
await setupDB();
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
// residual rows from prior test runs can shadow new INSERTs. Wipe
// non-default sources at the top of every describe to keep each
// block hermetic. file_migration_ledger cascades from files which
// setupDB already truncates, but wipe explicitly in case files did
// not cascade it.
const conn = getConn();
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
await conn.unsafe(`DELETE FROM file_migration_ledger`);
}, 30_000);
afterAll(async () => {
await teardownDB();
});
test('sources remove cascades to pages + chunks + timeline + links + files', async () => {
const conn = getConn();
const engine = getEngine();
// Build a fully populated source: page, chunks, timeline entries,
// links, a file row. Then remove the source and verify nothing
// for that source survives.
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['add', 'cascadetest', '--federated']);
// Page under cascadetest
await conn.unsafe(
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
VALUES ('cascadetest', 'people/alice', 'person', 'Alice', 'Alice body', '', '{}'::jsonb, 'h1')`,
);
const alicePage = await conn.unsafe(
`SELECT id FROM pages WHERE source_id = 'cascadetest' AND slug = 'people/alice'`,
);
const aliceId = alicePage[0].id as number;
// A second page for link target
await conn.unsafe(
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
VALUES ('cascadetest', 'companies/acme', 'company', 'Acme', 'Acme body', '', '{}'::jsonb, 'h2')`,
);
const acmePage = await conn.unsafe(
`SELECT id FROM pages WHERE source_id = 'cascadetest' AND slug = 'companies/acme'`,
);
const acmeId = acmePage[0].id as number;
// Chunk
await conn.unsafe(
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source)
VALUES (${aliceId}, 0, 'Alice body chunk', 'compiled_truth')`,
);
// Timeline
await conn.unsafe(
`INSERT INTO timeline_entries (page_id, date, source, summary, detail)
VALUES (${aliceId}, '2026-01-15', 'test', 'Joined Acme', 'detail')`,
);
// Link Alice → Acme
await conn.unsafe(
`INSERT INTO links (from_page_id, to_page_id, link_type, link_source)
VALUES (${aliceId}, ${acmeId}, 'works_at', 'markdown')`,
);
// File row pointing at Alice
await conn.unsafe(
`INSERT INTO files (source_id, page_id, filename, storage_path, content_hash)
VALUES ('cascadetest', ${aliceId}, 'alice.pdf', 'cascadetest/people/alice/alice.pdf', 'fh1')`,
);
// Sanity: everything exists
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'cascadetest'`))[0].n).toBe(2);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM content_chunks WHERE page_id = ${aliceId}`))[0].n).toBe(1);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM timeline_entries WHERE page_id = ${aliceId}`))[0].n).toBe(1);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM links WHERE from_page_id = ${aliceId}`))[0].n).toBe(1);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(1);
// Remove the source.
// v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected.
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['remove', 'cascadetest', '--confirm-destructive']);
// Everything for that source is gone.
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'cascadetest'`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM content_chunks WHERE page_id = ${aliceId}`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM timeline_entries WHERE page_id = ${aliceId}`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM links WHERE from_page_id = ${aliceId}`))[0].n).toBe(0);
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(0);
// The sources row itself is gone.
const src = await conn.unsafe(`SELECT id FROM sources WHERE id = 'cascadetest'`);
expect(src.length).toBe(0);
});
});
describeE2E('v0.18.0 multi-source — sync --source routes through sources table', () => {
beforeAll(async () => {
await setupDB();
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
// residual rows from prior test runs can shadow new INSERTs. Wipe
// non-default sources at the top of every describe to keep each
// block hermetic. file_migration_ledger cascades from files which
// setupDB already truncates, but wipe explicitly in case files did
// not cascade it.
const conn = getConn();
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
await conn.unsafe(`DELETE FROM file_migration_ledger`);
}, 30_000);
afterAll(async () => {
await teardownDB();
});
test('performSync with sourceId reads local_path from sources row', async () => {
const engine = getEngine();
const conn = getConn();
// Register a source with a bogus path (we're not actually walking a
// repo — this test asserts that performSync correctly RESOLVES the
// source row vs hitting the global config).
await runSources(engine as unknown as Parameters<typeof runSources>[0], [
'add', 'syncsrc', '--path', '/nonexistent/syncsrc/path', '--no-federated',
]);
// Also set a DIFFERENT path in the global config so we can verify
// sourceId actually disambiguates.
await engine.setConfig('sync.repo_path', '/some/other/default/path');
// performSync({sourceId: 'syncsrc'}) should attempt to use
// /nonexistent/syncsrc/path, NOT /some/other/default/path.
let err: Error | null = null;
try {
await performSync(engine, { sourceId: 'syncsrc' });
} catch (e) {
err = e as Error;
}
expect(err).not.toBeNull();
// The error message references the source-scoped path, not the
// global config path. (Could be "Not a git repository"
// or "No commits in repo" — either way the path it cites should
// be the source's.)
expect(err!.message).toContain('/nonexistent/syncsrc/path');
expect(err!.message).not.toContain('/some/other/default/path');
});
test('performSync with no sourceId falls back to global sync.repo_path', async () => {
const engine = getEngine();
// Global config is still '/some/other/default/path' from the
// previous test. Without --source, performSync uses it.
let err: Error | null = null;
try {
await performSync(engine, {});
} catch (e) {
err = e as Error;
}
expect(err).not.toBeNull();
expect(err!.message).toContain('/some/other/default/path');
});
});
describeE2E('v0.18.0 multi-source — sources table surface', () => {
beforeAll(async () => {
await setupDB();
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
// residual rows from prior test runs can shadow new INSERTs. Wipe
// non-default sources at the top of every describe to keep each
// block hermetic. file_migration_ledger cascades from files which
// setupDB already truncates, but wipe explicitly in case files did
// not cascade it.
const conn = getConn();
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
await conn.unsafe(`DELETE FROM file_migration_ledger`);
}, 30_000);
afterAll(async () => {
await teardownDB();
});
test('default source is seeded federated=true; new sources default to isolated', async () => {
const conn = getConn();
const engine = getEngine();
const def = await conn.unsafe(`SELECT config FROM sources WHERE id = 'default'`);
const defConfig = typeof def[0].config === 'string' ? JSON.parse(def[0].config) : def[0].config;
expect(defConfig.federated).toBe(true);
// Defensive cleanup: sources isn't in helpers.ALL_TABLES, so residual
// rows from prior test runs can shadow this INSERT via ON CONFLICT
// DO NOTHING. Delete first, then create.
await conn.unsafe(`DELETE FROM sources WHERE id = 'isolatedsrc'`);
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['add', 'isolatedsrc']);
const iso = await conn.unsafe(`SELECT config FROM sources WHERE id = 'isolatedsrc'`);
const isoConfig = typeof iso[0].config === 'string' ? JSON.parse(iso[0].config) : iso[0].config;
expect(isoConfig.federated).toBeUndefined(); // omitted → isolated-by-default
});
test('federate / unfederate flips config.federated on real DB', async () => {
const conn = getConn();
const engine = getEngine();
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['federate', 'isolatedsrc']);
let row = await conn.unsafe(`SELECT config FROM sources WHERE id = 'isolatedsrc'`);
let config = typeof row[0].config === 'string' ? JSON.parse(row[0].config) : row[0].config;
expect(config.federated).toBe(true);
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['unfederate', 'isolatedsrc']);
row = await conn.unsafe(`SELECT config FROM sources WHERE id = 'isolatedsrc'`);
config = typeof row[0].config === 'string' ? JSON.parse(row[0].config) : row[0].config;
expect(config.federated).toBe(false);
});
test('rename changes name, id stays stable', async () => {
const conn = getConn();
const engine = getEngine();
await runSources(engine as unknown as Parameters<typeof runSources>[0], [
'rename', 'isolatedsrc', 'My Isolated Source',
]);
const row = await conn.unsafe(`SELECT id, name FROM sources WHERE id = 'isolatedsrc'`);
expect(row[0].id).toBe('isolatedsrc');
expect(row[0].name).toBe('My Isolated Source');
});
});
describeE2E('v0.18.0 multi-source — storage backfill against file_migration_ledger', () => {
beforeAll(async () => {
await setupDB();
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
// residual rows from prior test runs can shadow new INSERTs. Wipe
// non-default sources at the top of every describe to keep each
// block hermetic. file_migration_ledger cascades from files which
// setupDB already truncates, but wipe explicitly in case files did
// not cascade it.
const conn = getConn();
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
await conn.unsafe(`DELETE FROM file_migration_ledger`);
}, 30_000);
afterAll(async () => {
await teardownDB();
});
test('seeded ledger + stub storage: pending → complete end-to-end', async () => {
const conn = getConn();
const engine = getEngine();
// Seed a page + file (via raw INSERT so the test doesn't depend on
// sync running).
await conn.unsafe(
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
VALUES ('default', 'topics/storage', 'note', 'Storage test', 'body', '', '{}'::jsonb, 'sh1')`,
);
const pageRow = await conn.unsafe(
`SELECT id FROM pages WHERE source_id = 'default' AND slug = 'topics/storage'`,
);
const pageId = pageRow[0].id as number;
await conn.unsafe(
`INSERT INTO files (source_id, page_id, filename, storage_path, content_hash)
VALUES ('default', ${pageId}, 'doc.pdf', 'topics/storage/doc.pdf', 'fh1')`,
);
const fileRow = await conn.unsafe(
`SELECT id FROM files WHERE storage_path = 'topics/storage/doc.pdf'`,
);
const fileId = fileRow[0].id as number;
// Seed the ledger manually so we don't depend on the v23 seed SQL
// (the TRUNCATE CASCADE in setupDB wipes ledger rows).
await conn.unsafe(
`INSERT INTO file_migration_ledger (file_id, storage_path_old, storage_path_new, status)
VALUES (${fileId}, 'topics/storage/doc.pdf', 'default/topics/storage/doc.pdf', 'pending')
ON CONFLICT (file_id) DO NOTHING`,
);
// Stub storage: downloads return bytes, uploads track what was written.
const uploaded = new Set<string>();
const stub: StorageBackend = {
upload: async (p: string) => { uploaded.add(p); },
download: async (p: string) => Buffer.from('bytes-for:' + p),
delete: async (p: string) => { uploaded.delete(p); },
exists: async (p: string) => uploaded.has(p),
list: async () => [],
getUrl: async (p) => `https://stub/${p}`,
};
const report = await runStorageBackfill(engine, stub);
expect(report.total).toBe(1);
expect(report.nowComplete).toBe(1);
expect(report.failed).toBe(0);
// Ledger row transitioned to complete.
const ledger = await conn.unsafe(
`SELECT status FROM file_migration_ledger WHERE file_id = ${fileId}`,
);
expect(ledger[0].status).toBe('complete');
// Files row now points at the new path.
const filesAfter = await conn.unsafe(
`SELECT storage_path FROM files WHERE id = ${fileId}`,
);
expect(filesAfter[0].storage_path).toBe('default/topics/storage/doc.pdf');
// Stub storage saw the upload happen at the new path.
expect(uploaded.has('default/topics/storage/doc.pdf')).toBe(true);
});
});
// v0.18.0: real-Postgres regression guard for the addLinksBatch /
// addTimelineEntriesBatch JOIN fan-out bug. Before the fix, the JOIN was
// `pages.slug = v.from_slug` unqualified — so two pages sharing the same
// slug across sources would silently duplicate edges and timeline rows.
// postgres-js binds arrays through `unnest()` rather than inline VALUES,
// so the query shape is structurally different from PGLite's and gets its
// own coverage.
describeE2E('v0.18.0 multi-source — addLinksBatch / addTimelineEntriesBatch source-awareness', () => {
beforeAll(async () => {
await setupDB();
const conn = getConn();
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
await conn.unsafe(`DELETE FROM file_migration_ledger`);
}, 30_000);
afterAll(async () => { await teardownDB(); });
async function seedSameSlugTwoSources() {
const conn = getConn();
const engine = getEngine() as PostgresEngine;
// Second source alongside 'default'.
await conn.unsafe(
`INSERT INTO sources (id, name) VALUES ('alt', 'alt') ON CONFLICT (id) DO NOTHING`
);
// Create same-slug pages in both sources. putPage defaults to 'default'.
await engine.putPage('topics/ai', { type: 'concept', title: 'AI (default)', compiled_truth: '', timeline: '' });
await engine.putPage('topics/ml', { type: 'concept', title: 'ML (default)', compiled_truth: '', timeline: '' });
await conn.unsafe(
`INSERT INTO pages (slug, type, title, compiled_truth, timeline, frontmatter, content_hash, source_id, updated_at)
VALUES ('topics/ai', 'concept', 'AI (alt)', '', '', '{}'::jsonb, 'alt-ai-hash', 'alt', now()),
('topics/ml', 'concept', 'ML (alt)', '', '', '{}'::jsonb, 'alt-ml-hash', 'alt', now())`
);
}
test('addLinksBatch without explicit source_id does NOT fan out across sources', async () => {
await seedSameSlugTwoSources();
const conn = getConn();
const engine = getEngine() as PostgresEngine;
// Reset links from any prior describe block.
await conn.unsafe(`DELETE FROM links`);
const inserted = await engine.addLinksBatch([
{ from_slug: 'topics/ai', to_slug: 'topics/ml', link_type: 'mention' },
]);
// Exactly one edge (default → default). Before the fix this was 2.
expect(inserted).toBe(1);
const rows = await conn.unsafe(
`SELECT f.source_id AS from_src, t.source_id AS to_src
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id`
);
expect(rows.length).toBe(1);
expect(rows[0].from_src).toBe('default');
expect(rows[0].to_src).toBe('default');
});
test('addLinksBatch supports cross-source edges when explicit source_ids differ', async () => {
const conn = getConn();
const engine = getEngine() as PostgresEngine;
await conn.unsafe(`DELETE FROM links`);
const inserted = await engine.addLinksBatch([
{
from_slug: 'topics/ai', to_slug: 'topics/ml', link_type: 'mention',
from_source_id: 'default', to_source_id: 'alt',
},
]);
expect(inserted).toBe(1);
const rows = await conn.unsafe(
`SELECT f.source_id AS from_src, t.source_id AS to_src
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id`
);
expect(rows.length).toBe(1);
expect(rows[0].from_src).toBe('default');
expect(rows[0].to_src).toBe('alt');
});
test('addTimelineEntriesBatch without explicit source_id does NOT fan out across sources', async () => {
const conn = getConn();
const engine = getEngine() as PostgresEngine;
await conn.unsafe(`DELETE FROM timeline_entries`);
const inserted = await engine.addTimelineEntriesBatch([
{ slug: 'topics/ai', date: '2024-01-15', summary: 'Founded' },
]);
expect(inserted).toBe(1);
const rows = await conn.unsafe(
`SELECT p.source_id
FROM timeline_entries te
JOIN pages p ON p.id = te.page_id`
);
expect(rows.length).toBe(1);
expect(rows[0].source_id).toBe('default');
});
test('addTimelineEntriesBatch with explicit alt source_id lands only in alt', async () => {
const conn = getConn();
const engine = getEngine() as PostgresEngine;
await conn.unsafe(`DELETE FROM timeline_entries`);
const inserted = await engine.addTimelineEntriesBatch([
{ slug: 'topics/ai', date: '2024-02-01', summary: 'Alt-only event', source_id: 'alt' },
]);
expect(inserted).toBe(1);
const rows = await conn.unsafe(
`SELECT p.source_id
FROM timeline_entries te
JOIN pages p ON p.id = te.page_id`
);
expect(rows.length).toBe(1);
expect(rows[0].source_id).toBe('alt');
});
});