mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.42.21.0 fix(postgres): module-singleton ownership — canonical landing for the dream-cycle "connect() has not been called" class (#1404/#1471/#1619) (#1805)
* fix(postgres): module-singleton ownership — borrower disconnect no longer nulls the cycle's connection (#1404/#1471/#1619) gbrain dream on Postgres failed every DB phase with "No database connection: connect() has not been called": a short-lived borrower probe engine (lint/doctor config-lift, no poolSize) called db.disconnect() in its own disconnect(), nulling the shared module singleton the long-lived cycle owner was still using. The module `sql` is only ever nulled by db.disconnect() (postgres.js auto-reconnects its own pool), so the failure was always a borrower-disconnect, never an idle-pooler drop. Fix: db.connect() returns whether THIS call created the singleton (atomic — no await between the null-check and the sql=postgres() assignment), PostgresEngine stores it as _ownsModuleSingleton, and disconnect() only calls db.disconnect() when it owns the connection. Borrowers no-op. Hardening: db.disconnect() snapshots +nulls sql before awaiting end(); reconnect() shares one in-flight _reconnectPromise. Tests: new postgres-engine-singleton-ownership.test.ts; expanded DB-gated e2e matrix (owner/borrower, creation-not-role, symmetric CLI-exit, owner-reconnect- with-live-borrower); module-style getter asymmetry; #1570 shared-recovery regression updated to assert the fixed contract. Co-Authored-By: nullhex-io <noreply@github.com> Co-Authored-By: joelwp <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.42.15.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: nullhex-io <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
nullhex-io
Claude Opus 4.8
nullhex-io
Claude Opus 4.8
parent
ec5fed2921
commit
f3ade6c0c3
@@ -2,6 +2,53 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.21.0] - 2026-06-02
|
||||
|
||||
**Your nightly `gbrain dream` stops silently losing every database phase.** If you run gbrain on Postgres (local or Supabase), the dream cycle has been quietly failing: the `lint` and `backlinks` phases work, then `sync`, `synthesize`, `embed`, and the rest all blow up with `No database connection: connect() has not been called`. The extract phase reports "created 0 links" while actually dropping every row. Run the same phases one at a time in separate commands and they all work — only the full cycle breaks. The result: your brain quietly stops staying up to date, and the cycle leaves a stuck lock behind.
|
||||
|
||||
The cause turned out to be a single sentence of logic. The dream cycle opens one long-lived database connection and reuses it for the whole run. But along the way, small helper steps (the lint config probe, doctor checks) briefly open their own connection handle. When those short-lived helpers finished and closed *their* handle, they were actually closing the *shared* connection the rest of the cycle still needed. Every later phase then found the connection gone.
|
||||
|
||||
This release teaches gbrain who actually owns the shared connection. Only the engine that opened it is allowed to close it; the short-lived helpers leave it alone. The fix lands automatically:
|
||||
|
||||
```
|
||||
gbrain upgrade
|
||||
gbrain dream --dir <your-brain> # every DB phase now reports ✓, lock releases cleanly
|
||||
```
|
||||
|
||||
Nothing to configure. If your dream cycle was the broken kind, it just starts working.
|
||||
|
||||
### Under the hood
|
||||
|
||||
The shared connection is the module-level `sql` singleton in `src/core/db.ts`. It was only ever nulled by `db.disconnect()` — postgres.js auto-reconnects its own internal pool and never touches our reference — so the singleton going null mid-cycle was always a "borrower" engine's disconnect cascading into `db.disconnect()`, never an idle-pooler drop. `PostgresEngine.disconnect()` called `db.disconnect()` for any `_connectionStyle === 'module'` engine with no check for whether that engine actually created the singleton.
|
||||
|
||||
The fix adds an ownership token. `db.connect()` now returns whether THIS call created the singleton — decided atomically inside `connect()`, since there is no `await` between its `if (sql)` null-check and the synchronous `sql = postgres(...)` assignment, so two connects can't both claim creation. `PostgresEngine` stores that as `_ownsModuleSingleton` and only calls `db.disconnect()` when it owns the connection; borrowers clear their own marker and leave the shared pool alone. Two adjacent hardenings landed in the same pass: `db.disconnect()` now snapshots and nulls `sql` *before* awaiting `end()` (so a concurrent connect can't join a pool that's already closing), and `reconnect()` uses a shared in-flight promise so concurrent callers await the same reconnect instead of racing a half-rebuilt pool.
|
||||
|
||||
Earlier partial fixes addressed symptoms: v0.41.27.0's retry-with-reconnect rescued the batch-write phases (extract) but not `sync`/`synthesize`, which don't go through the retry path; v0.42.5.0 stopped the lint phase from being a borrower but left the structural hole open for every other helper. This closes the hole at the source.
|
||||
|
||||
### To take advantage of v0.42.21.0
|
||||
|
||||
`gbrain upgrade` applies this automatically — there are no migrations or config to set. To confirm it took:
|
||||
|
||||
1. Run a cycle and watch the phases:
|
||||
```bash
|
||||
gbrain dream --dir <your-brain> --dry-run
|
||||
```
|
||||
Every DB phase should report `✓`, not `✗ ... connect() has not been called`.
|
||||
2. Confirm no stuck lock:
|
||||
```bash
|
||||
psql <your-db> -c "SELECT count(*) FROM gbrain_cycle_locks;" # expect 0
|
||||
```
|
||||
3. If a DB phase still fails, please file an issue at https://github.com/garrytan/gbrain/issues with the output of `gbrain doctor` and the cycle's stderr.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- **`gbrain dream` on Postgres completes every phase.** The module-singleton ownership fix (`_ownsModuleSingleton` in `src/core/postgres-engine.ts`, `db.connect()` returning the creator token in `src/core/db.ts`) means a short-lived probe engine's `disconnect()` can no longer null the shared connection the cycle is using. Closes #1404, #1471, #1619 (and the symptom tracked in #1570/#1535).
|
||||
- **Connection teardown is concurrency-safe.** `db.disconnect()` snapshots + nulls the singleton before awaiting `end()`; `PostgresEngine.reconnect()` shares one in-flight `_reconnectPromise` so concurrent callers await it instead of racing.
|
||||
- **Tests:** new `test/postgres-engine-singleton-ownership.test.ts` (source-level guardrails for the ownership contract), an expanded DB-gated behavioral matrix in `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (owner/borrower, creation-not-role, symmetric CLI-exit, owner-reconnect-with-live-borrower), a module-style asymmetry case in `test/postgres-engine-getter-selfheal.test.ts`, and the #1570 shared-recovery regression updated to assert the fixed contract.
|
||||
|
||||
### For contributors
|
||||
|
||||
This is the canonical landing for a long-standing class with ~15 competing community PRs and zero merged. Thanks to **@nullhex-io** (#1651) and **@joelwp** (#1667) — the ownership-flag approach that shipped, and to **@BrendanGahan**, **@xaviroblessarries**, and the other #1471 reporters for the precise root-cause walkthroughs. Three follow-ups are filed in `TODOS.md` (connection-lifecycle hardening under concurrent module connects, facts-queue drain on the dream/fall-through paths, stale ConnectionManager refresh after reconnect) — all pre-existing and not reachable in current gbrain.
|
||||
## [0.42.20.0] - 2026-06-03
|
||||
|
||||
**Three ways GBrain could freeze or go silent are fixed.** This release closes a
|
||||
@@ -194,6 +241,7 @@ Things to know about: the sync bookmark (`last_commit`) still only advances when
|
||||
- **Vanished-on-disk added files are skipped, not failed.** A file added in the diff but deleted from disk by a commit after the pin (normal forward delete) is skipped and checkpointed instead of blocking the run.
|
||||
- **Cleaner single-flight backpressure.** `performSync` throws a typed `SyncLockBusyError`; the Minion `sync` handler catches it and marks the job *skipped* (not failed), so a cron/autopilot tick that hits a held lock defers to the holder without polluting the failed-job/crash metrics.
|
||||
- **Tests.** `test/sync-resumable-import.serial.test.ts` (13 cases): convergence regression, resume-skips-checkpointed, pinned-target/forward-drift, history-rewrite re-pin, `last_sync_at` not bumped on a blocked run + good-file banking, vanished-file skip, dry-run/empty-diff, plus pure-helper coverage for the fingerprint, clamp, and pool-budget math.
|
||||
|
||||
## [0.42.16.0] - 2026-06-02
|
||||
|
||||
**`gbrain doctor` now tells you the brain is OOM-looping in one line, ranks every
|
||||
|
||||
@@ -1,5 +1,65 @@
|
||||
# TODOS
|
||||
|
||||
## v0.42.21.0 module-singleton ownership follow-ups (v0.42+)
|
||||
|
||||
Filed from the v0.42.21.0 wave (#1404/#1471/#1619 — the dream-cycle
|
||||
"connect() has not been called" class, fixed via `_ownsModuleSingleton`).
|
||||
Surfaced by the Codex outside-voice review (finding #4) and deliberately scoped
|
||||
OUT — pre-existing, and the ownership fix *reduces* its window. See plan +
|
||||
GSTACK REVIEW REPORT at
|
||||
`~/.claude/plans/system-instruction-you-are-working-lazy-allen.md`.
|
||||
|
||||
- [ ] **P3 — Stale `ConnectionManager` read-pool after an owner `reconnect()`.**
|
||||
A module-style borrower engine caches the singleton at connect time via
|
||||
`connectionManager.setReadPool(db.getConnection())` (`postgres-engine.ts:~208`).
|
||||
When the OWNER engine calls `reconnect()` (the batchRetry path), it tears down
|
||||
the old module singleton and builds a fresh one — but the borrower's
|
||||
`connectionManager` still holds the OLD (ended) pool. The borrower's normal
|
||||
query path is fine (`this.sql` → `db.getConnection()` resolves the NEW
|
||||
singleton), so this is invisible on read/write. The edge is
|
||||
`initSchema()`, which routes DDL through `connectionManager.ddl()`
|
||||
(`postgres-engine.ts:~253`) — a borrower running initSchema after an owner
|
||||
reconnect would hit the dead pool. Pre-existing (not introduced by #1471), and
|
||||
the ownership fix makes owner reconnects *rarer* (the singleton no longer gets
|
||||
nulled by borrowers, so reconnect only fires on genuine transient drops), which
|
||||
shrinks the window. Real fix: refresh a borrower's `connectionManager` read
|
||||
pool lazily from `db.getConnection()` on use, or have `db.connect()`/reconnect
|
||||
publish a generation counter the manager checks. Defer until a borrower is
|
||||
observed running `initSchema()` mid-process (no current caller does).
|
||||
|
||||
- [ ] **P2 — Ownership state can desync from the shared singleton under
|
||||
CONCURRENT module connect/reconnect.** Both adversarial reviewers (Codex +
|
||||
Claude) independently flagged this. `_ownsModuleSingleton` is per-engine state
|
||||
about a shared (module-level) resource, so it can migrate: if a borrower calls
|
||||
`connect()`/`reconnect()` during the window when an owner's `reconnect()` has
|
||||
nulled `sql` (`db.ts` snapshot-early-null) but not yet rebuilt it, the borrower
|
||||
creates the new singleton and becomes owner; the owner re-connects as a
|
||||
borrower; the short-lived borrower's later `disconnect()` then closes the live
|
||||
pool the demoted owner still uses — the original bug, in reverse. ALSO: the
|
||||
audit-import + `connectionManager.disconnect()` awaits in `PostgresEngine.disconnect()`
|
||||
and the publish-before-`SELECT 1` window in `db.connect()` let a concurrent
|
||||
connect join a dying/unverified pool. NOT REACHABLE in current gbrain — cycle
|
||||
phases are sequential on one awaited engine, borrowers are nested within a
|
||||
phase, the parallel-sync worker pool uses INSTANCE engines (not the singleton),
|
||||
and facts/last-retrieved background writes reuse the owner engine (no second
|
||||
module engine). The ownership fix is correct for every reachable path and is
|
||||
fully tested. The structural fix (which removes the unenforced "no concurrent
|
||||
module connect" invariant) is the refcount/lease-in-db.ts approach Codex argued
|
||||
in the plan review: keep the lifecycle state WITH the shared resource so it
|
||||
can't desync per-engine, bounded against CLI-hang by a top-level forced
|
||||
cleanup. Do this BEFORE introducing any concurrent module-engine connect path.
|
||||
|
||||
- [ ] **P3 — `dream` + CLI_ONLY fall-through paths don't drain the facts /
|
||||
last-retrieved queues before the owner disconnect.** The op-dispatch path
|
||||
(`cli.ts:~282-314`) drains `getFactsQueue().drainPending()` +
|
||||
`awaitPendingLastRetrievedWrites()` before `engine.disconnect()`; the `dream`
|
||||
owner-disconnect (`cli.ts:~1164`) and the fall-through owner-disconnect
|
||||
(`cli.ts:~1785`) do not. If the dream cycle ever enqueues a facts:absorb /
|
||||
last-retrieved write that's still in flight at disconnect, the owner nulls the
|
||||
singleton and the write throws "No database connection". Pre-existing (not
|
||||
introduced by the #1471 ownership fix), surfaced by the Claude adversarial
|
||||
review (F5). Fix: hoist the same drain-before-disconnect block the op-dispatch
|
||||
path uses into a shared helper and call it on all three owner-disconnect sites.
|
||||
## v0.42.x AI SDK v6 tool-schema fix follow-ups (#1782/#1764)
|
||||
|
||||
Surfaced by the codex outside-voice pass during `/plan-eng-review` and
|
||||
@@ -135,6 +195,7 @@ complete and tested; these are hardening/cleanup.
|
||||
run unbounded LLM rollouts with no deadline check. BudgetTracker still caps spend; the
|
||||
runtime guarantee is best-effort. Thread the deadline + abortSignal into those phases, or
|
||||
document runtime as best-effort.
|
||||
|
||||
## v0.42.7.0 extract-in-default-loop follow-ups (v0.42+)
|
||||
|
||||
Filed from the v0.42.2.0 wave (#1696 link/timeline extraction freshness
|
||||
|
||||
@@ -22,7 +22,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/commands/doctor.ts` extension — `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded. Pinned by cases in `test/doctor.test.ts`.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`).
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch`/`addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error (macOS 26.3 WASM bug #223, points at `gbrain doctor`); the lock is released on failure so the next process can retry cleanly. `searchKeyword`/`searchKeywordChunks` multiply `ts_rank` by the source-factor CASE at chunk grain; `searchVector` is a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`, inner LIMIT scales with offset to preserve pagination. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for forward-referenced state the embedded blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target, plus `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`) and adds only what's missing; threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope; no-op on fresh installs and modern brains (closes the upgrade-wedge bug class #239/#243/#266/#357/#366/#374/#375/#378/#395/#396/#1018/#974/#820). `getBrainScore` returns 100/100 with full breakdown (35/25/15/15/10) when `pageCount === 0` (vacuous truth — empty brain has no coverage problem); Pinned by `test/brain-score-breakdown.test.ts` empty-brain assertion + `test/doctor-report-remote.serial.test.ts`. `disconnect()` uses snapshot+early-null (snapshot `_db`/`_lock`, null instance fields BEFORE any `await` so a concurrent `connect()` can't see a partial mid-close state) wrapped in try/finally guaranteeing lock-release even if `db.close()` throws; KEEPS close-then-release order (release-then-close was rejected: it would widen the window where a sibling process connects to a still-closing brain); Pinned by `test/pglite-engine-disconnect.serial.test.ts`. Exports `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` routing the catch-block hint by failure shape (`bunfs` matches literal `$$bunfs` OR `ENOENT[\s\S]*pglite\.data` co-occurrence, surfaces a paste-ready `bun upgrade` + Node fallback; `macos-26-3` keeps the #223 link; `unknown` falls through); Pinned by `test/pglite-init-classifier.test.ts`. Implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding, caller-chunking primitive throwing when input exceeds `DELETE_BATCH_SIZE`, `deletePages` returns `RETURNING slug` rows so callers filter `pagesAffected` to confirmed deletes. Implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, widened `countStaleChunks({sourceId?, signature?})` (the `signature` opt widens via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)`, NULL grandfathered); parity SQL with postgres-engine.ts. PGLite-specific DDL (pgvector, pg_trgm, triggers).
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch`/`addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. `searchKeyword`/`searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection. `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude from `src/core/search/sql-ranking.ts`; `searchVector` is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying `p.source_id` inner→outer. `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. `countStaleChunks()`+`listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale` (eliminates ~76 MB/call client-side pull); `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same probe set as PGLite (extended for column-only forward-reference cases: `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`/`archived_at`/`archive_expires_at`); the entire probe path runs on the DDL connection threaded from `initSchema` (closing a concurrent-bootstrap race for Supabase pooler users); closes #1018/#974/#820. `disconnect()` is idempotent — `_connectionStyle` tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts`. `getBrainScore` empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when `pageCount === 0` (both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic). Implements `deletePages(slugs, {sourceId}): Promise<string[]>` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single round-trip; caller chunks); `resolveSlugsByPaths` does `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2`; FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`, `files.page_id`+`links.origin_page_id` go SET NULL; throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (`sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, widened `countStaleChunks`, all accept optional `signature` extending "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN, NULL grandfathered; the `embedding IS NULL` server-side filter is preserved as the no-signature fast path); Pinned by `test/e2e/engine-parity.test.ts`.
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch`/`addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. `searchKeyword`/`searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection. `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude from `src/core/search/sql-ranking.ts`; `searchVector` is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying `p.source_id` inner→outer. `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures, and by `batchRetry` on a retryable connection error). Concurrent callers share one in-flight `_reconnectPromise` (they await the single reconnect rather than racing a half-rebuilt pool); ownership re-samples through the atomic `db.connect()` token on the connect leg. `reconnect(ctx?)` accepts the triggering error and records a pool-recovery audit event (`reap_detected`/`reconnect_other`/`reconnect_succeeded`/`reconnect_failed`) for the `pool_reap_health` doctor check. `executeRaw` is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. `countStaleChunks()`+`listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale` (eliminates ~76 MB/call client-side pull); `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same probe set as PGLite (extended for column-only forward-reference cases: `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`/`archived_at`/`archive_expires_at`); the entire probe path runs on the DDL connection threaded from `initSchema` (closing a concurrent-bootstrap race for Supabase pooler users); closes #1018/#974/#820. `disconnect()` is idempotent — `_connectionStyle` tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; and a module-style engine only calls `db.disconnect()` when it owns the singleton (`_ownsModuleSingleton`, set from the `db.connect()` creation token), so a borrower probe engine's teardown leaves the cycle owner's connection intact. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` + `test/postgres-engine-singleton-ownership.test.ts`. `getBrainScore` empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when `pageCount === 0` (both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic). Implements `deletePages(slugs, {sourceId}): Promise<string[]>` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single round-trip; caller chunks); `resolveSlugsByPaths` does `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2`; FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`, `files.page_id`+`links.origin_page_id` go SET NULL; throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (`sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, widened `countStaleChunks`, all accept optional `signature` extending "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN, NULL grandfathered; the `embedding IS NULL` server-side filter is preserved as the no-signature fast path); Pinned by `test/e2e/engine-parity.test.ts`.
|
||||
- `src/core/cjk.ts` — Single source of truth for CJK detection. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables). Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`.
|
||||
- `src/core/audit-slug-fallback.ts` — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()`. Separate surface from `sync-failures.jsonl` — that file carries bookmark-gating semantics that info events shouldn't trigger.
|
||||
- `src/core/embedding-pricing.ts` — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token. Unknown providers degrade to "estimate unavailable" instead of fabricating numbers.
|
||||
@@ -31,7 +31,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/commands/reindex-code.ts` — `gbrain reindex --code [--source ID] [--dry-run] [--yes] [--json] [--force] [--no-embed]`. Walks `pages WHERE type = 'code'` in 100-row batches, replays through `importCodeFile` for chunk + embed + content_hash folding. Idempotent unless `--force` bypasses the content_hash early-return. Cost-preview model field reads `getEmbeddingModelName()` from the gateway so preview reflects what the gateway will actually embed with. An informational stderr nudge inside `runReindexCode` (so dry-run + execute both surface it): when the configured embedding model isn't code-tuned (allowlist `{'voyage-code-3'}`, case-insensitive bare match), prints a recommendation to switch to `voyage:voyage-code-3`; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1`, `--no-embed`, or `--json`. Pure `shouldNudgeCodeModel(bareName)` returns a tagged `NudgeDecision` union (takes the bare model name, emits qualified `voyage:voyage-code-3` for the paste-ready `gbrain config set` line). When `--yes` is absent and the caller is non-TTY or passed `--json`, the cost gate refuses (exit 2, no spend) via the pure exported `buildCostRefusal({json, previewMsg, preview, costUsd, model}): {stdout?, stderr?}` — JSON envelope only when `--json` is explicit, otherwise a human refusal on stderr (the spend guardrail is independent of the output format). Pinned by `test/ai/voyage-code-3-recipe.test.ts`, `test/reindex-code-nudge.serial.test.ts`, `test/reindex-code-model-source.serial.test.ts` (IRON-RULE regression for the cost-preview fix), `test/reindex-cost-refusal.test.ts`.
|
||||
- `src/commands/sync.ts:resolveSlugByPathOrSourcePath` — Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path.
|
||||
- `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column).
|
||||
- `src/core/db.ts` — Connection management, schema initialization. `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT`/`GBRAIN_IDLE_TX_TIMEOUT`/`GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (`setSessionDefaults` kept as a back-compat no-op shim).
|
||||
- `src/core/db.ts` — Connection management, schema initialization. `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT`/`GBRAIN_IDLE_TX_TIMEOUT`/`GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (`setSessionDefaults` kept as a back-compat no-op shim). `connect()` returns `Promise<boolean>` — `true` iff THIS call created the module singleton, `false` if it joined an existing one; the decision is atomic (no `await` between the `if (sql)` null-check and the synchronous `sql = postgres(...)` assignment), so two concurrent module connects can't both claim creation. `PostgresEngine` stores the return as its `_ownsModuleSingleton` token and only the creating engine may `db.disconnect()` the singleton — a borrower probe engine (lint/doctor config-lift) no-ops its disconnect, so its teardown can't null the connection the long-lived cycle owner is still using (the dream-cycle "connect() has not been called" failure). The module `sql` is only ever nulled by `db.disconnect()` (postgres.js auto-reconnects its own internal pool and never touches our reference). `disconnect()` snapshots + nulls `sql` before awaiting `s.end()` so a concurrent connect can't join a pool that's already closing.
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`).
|
||||
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) so a model/dims swap is detectable as stale; `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`), mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle those). `importFromContent`'s tag reconciliation is ADD-ONLY: it only `addTag` (idempotent, ON CONFLICT DO NOTHING). The `tags` table has no provenance column and frontmatter tags are stripped from stored `pages.frontmatter` (markdown.ts:118), so a frontmatter-origin tag can't be distinguished from a DB-enrichment tag (auto-tag / dream synthesize / signal-detector) at re-import — deletion is unsafe (would wipe enrichment under `gbrain reindex --markdown`). Accepted trade-off: removing a tag from frontmatter no longer removes it from the DB on next sync (needs a `tag_source` provenance column). Pinned by `test/reindex-preserve-tags.test.ts` + `test/import-file.test.ts`.
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). Exported `pruneDir(name: string): boolean` is the single source of truth for descent-time directory exclusion across walkers — blocks `node_modules` (no leading dot, so naive walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars; `isSyncable` applies it per path segment, and `walkMarkdownFiles` in `src/commands/extract.ts` + `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing to save the IO of walking thousands of vendor files (closes #923 + #202). `manageGitignore` worktree discriminator matches the gitdir path segment (`/modules/<name>` = submodule, `/worktrees/<name>` = worktree, per Git's documented layout) so Conductor worktrees (first-class repos) get `.gitignore` management for storage-tiering (closes #889). `classifyErrorCode(errorMsg)` regex classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN`. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`, backfilled at ack time on older entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. `MISSING_OPEN`/`MISSING_CLOSE`/`EMPTY_FRONTMATTER` regexes match the actual `markdown.ts:159-244` validator strings; `FILE_TOO_LARGE` covers `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers `:347`.
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.20.0"
|
||||
"version": "0.42.21.0"
|
||||
}
|
||||
|
||||
@@ -1237,6 +1237,12 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
try {
|
||||
await runDream(eng, args);
|
||||
} finally {
|
||||
// #1471 invariant tripwire (the dream-cycle owner): `eng` created the
|
||||
// module singleton (first module connector) and is disconnected LAST,
|
||||
// here, after the whole cycle. The ownership fix relies on this owner's
|
||||
// lifetime strictly dominating every borrower (lint/doctor probe engines
|
||||
// created mid-cycle). Do NOT disconnect `eng` before runDream returns, or
|
||||
// a borrower could outlive the owner and lose the shared singleton.
|
||||
if (eng) await eng.disconnect();
|
||||
}
|
||||
return;
|
||||
@@ -1898,6 +1904,8 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
// hung Haiku), THEN disconnect. The drain-before-disconnect is the causal
|
||||
// fix; the force-exit defense below is secondary (it CANNOT preempt a WASM
|
||||
// busy-loop on a pinned JS thread — that's exactly why the drain matters).
|
||||
// #1471: this is also the fall-through OWNER-disconnect — the owner is torn
|
||||
// down LAST (after the drain), so module-singleton borrowers never outlive it.
|
||||
if (command !== 'serve') {
|
||||
const forceExit = shouldForceExitAfterMain();
|
||||
let hardExitTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
|
||||
+25
-7
@@ -159,13 +159,27 @@ export function getConnection(): ReturnType<typeof postgres> {
|
||||
return sql;
|
||||
}
|
||||
|
||||
export async function connect(config: EngineConfig): Promise<void> {
|
||||
/**
|
||||
* Connect the module-level singleton. Returns `true` iff THIS call created the
|
||||
* singleton, `false` if it joined an existing one.
|
||||
*
|
||||
* #1471 ownership: the create-vs-join decision is made HERE, atomically. There
|
||||
* is no `await` between the `if (sql)` null-check below and the synchronous
|
||||
* `sql = postgres(url, opts)` assignment, so two concurrent module connects
|
||||
* cannot both observe `sql === null` and both create. Callers store the return
|
||||
* as their ownership token (`PostgresEngine._ownsModuleSingleton`); only the
|
||||
* creator may later tear the singleton down. Borrowers (probe engines created
|
||||
* while the singleton already exists) get `false` and must NOT disconnect it.
|
||||
*
|
||||
* Back-compat: callers that ignore the return value are unaffected.
|
||||
*/
|
||||
export async function connect(config: EngineConfig): Promise<boolean> {
|
||||
if (sql) {
|
||||
// Warn if a different URL is passed — the old connection is still in use
|
||||
if (config.database_url && connectedUrl && config.database_url !== connectedUrl) {
|
||||
console.warn('[gbrain] connect() called with a different database_url but a connection already exists. Using existing connection.');
|
||||
}
|
||||
return;
|
||||
return false; // joined an existing singleton — caller is a borrower
|
||||
}
|
||||
|
||||
const url = config.database_url;
|
||||
@@ -212,6 +226,7 @@ export async function connect(config: EngineConfig): Promise<void> {
|
||||
connectedUrl = url;
|
||||
|
||||
await setSessionDefaults(sql);
|
||||
return true; // we created the singleton — caller is the owner
|
||||
} catch (e: unknown) {
|
||||
sql = null;
|
||||
connectedUrl = null;
|
||||
@@ -236,11 +251,14 @@ export async function disconnect(): Promise<void> {
|
||||
// instance-pool callers go through here.
|
||||
logDbDisconnect('postgres', 'module');
|
||||
} catch { /* best-effort; never block disconnect on audit failure */ }
|
||||
if (sql) {
|
||||
await sql.end();
|
||||
sql = null;
|
||||
connectedUrl = null;
|
||||
}
|
||||
// #1471 (codex #6): snapshot + null the singleton BEFORE awaiting end(), so a
|
||||
// concurrent module connect() can't observe a non-null `sql` mid-teardown and
|
||||
// join a pool that's already closing. Mirrors the v0.41.8.0 PGLite-disconnect
|
||||
// snapshot+early-null pattern.
|
||||
const s = sql;
|
||||
sql = null;
|
||||
connectedUrl = null;
|
||||
if (s) await s.end();
|
||||
}
|
||||
|
||||
export async function initSchema(): Promise<void> {
|
||||
|
||||
@@ -96,6 +96,19 @@ export class PostgresEngine implements BrainEngine {
|
||||
private _savedConfig: (EngineConfig & { poolSize?: number; parentConnectionManager?: ConnectionManager }) | null = null;
|
||||
/** Whether a reconnect is in progress (prevents concurrent reconnects). */
|
||||
private _reconnecting = false;
|
||||
/**
|
||||
* #1471: module-singleton OWNERSHIP token. `true` only for the engine whose
|
||||
* connect() actually created the shared db.ts `sql` singleton (returned
|
||||
* atomically by db.connect()). Borrowers — probe engines constructed while the
|
||||
* singleton already exists (resolveLintContentSanity config-lift, doctor,
|
||||
* integrity) — get `false` and must NOT db.disconnect() it, or they null the
|
||||
* `sql` the long-lived owner (the cycle engine) still uses and every later
|
||||
* phase throws "connect() has not been called". `_connectionStyle` alone can't
|
||||
* separate owner from borrower: both are 'module'. Correct because the
|
||||
* creator's lifetime dominates all borrowers — the CLI engine is created first
|
||||
* and disconnected last (cli.ts), and borrowers are strictly nested.
|
||||
*/
|
||||
private _ownsModuleSingleton = false;
|
||||
/**
|
||||
* Tracks which connection path this engine is using so disconnect() is
|
||||
* idempotent. 'instance' = own _sql pool (poolSize was set);
|
||||
@@ -194,8 +207,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
});
|
||||
this.connectionManager.setReadPool(this._sql);
|
||||
} else {
|
||||
// Module-level singleton (backward compat for CLI main engine)
|
||||
await db.connect(config);
|
||||
// Module-level singleton (backward compat for CLI main engine).
|
||||
// #1471: db.connect() returns whether THIS call created the singleton —
|
||||
// decided atomically inside connect() (no await between its null-check and
|
||||
// pool assignment), so two concurrent module connects can't both claim
|
||||
// ownership. Store the token; only the owner tears the singleton down.
|
||||
this._ownsModuleSingleton = await db.connect(config);
|
||||
this._connectionStyle = 'module';
|
||||
|
||||
// v0.30.1: connection-manager wraps the module singleton.
|
||||
@@ -237,7 +254,13 @@ export class PostgresEngine implements BrainEngine {
|
||||
return;
|
||||
}
|
||||
if (this._connectionStyle === 'module') {
|
||||
await db.disconnect();
|
||||
// #1471: only the engine that created the shared singleton may tear it
|
||||
// down. A borrower clears its own markers WITHOUT calling db.disconnect(),
|
||||
// so a probe engine's teardown can't clobber the owner's live connection.
|
||||
if (this._ownsModuleSingleton) {
|
||||
await db.disconnect();
|
||||
this._ownsModuleSingleton = false;
|
||||
}
|
||||
this._connectionStyle = null;
|
||||
}
|
||||
// else: nothing to disconnect (already done or never connected)
|
||||
@@ -1984,7 +2007,7 @@ export class PostgresEngine implements BrainEngine {
|
||||
},
|
||||
// v0.41.25.0 (#1570): on null-singleton retryable errors, rebuild
|
||||
// the connection BEFORE the inter-attempt sleep so the next attempt
|
||||
// sees a live pool. `this.reconnect()` is race-safe via
|
||||
// sees a live pool. `this.reconnect()` is race-safe via the
|
||||
// `_reconnecting` guard, handles both module and instance pools,
|
||||
// and is a fast no-op when the underlying client is still healthy
|
||||
// (postgres.js's own connection-replacement covers that case).
|
||||
|
||||
+1
-1
@@ -138,7 +138,7 @@ export interface WithRetryOpts {
|
||||
*
|
||||
* Engine-level callers (PostgresEngine.batchRetry) inject
|
||||
* `(ctx) => this.reconnect(ctx)` which already handles both module and
|
||||
* instance pools, race-safe via `_reconnecting` guard.
|
||||
* instance pools, race-safe via the `_reconnecting` guard.
|
||||
*
|
||||
* v0.42.x (#1685 CODEX #8): receives the triggering error so the engine can
|
||||
* classify it (pooler reap vs network/auth) for the pool-recovery audit. The
|
||||
|
||||
@@ -130,27 +130,6 @@ describe('classifyWorkerExit', () => {
|
||||
// --- Mock-based tests for reconnect logic ---
|
||||
|
||||
describe('PostgresEngine reconnect behavior', () => {
|
||||
it('reconnect flag prevents concurrent reconnections', async () => {
|
||||
// Simulate the _reconnecting guard
|
||||
let reconnecting = false;
|
||||
let reconnectCount = 0;
|
||||
|
||||
async function reconnect() {
|
||||
if (reconnecting) return;
|
||||
reconnecting = true;
|
||||
try {
|
||||
reconnectCount++;
|
||||
await new Promise(r => setTimeout(r, 10));
|
||||
} finally {
|
||||
reconnecting = false;
|
||||
}
|
||||
}
|
||||
|
||||
// Fire 3 concurrent reconnects — only 1 should run
|
||||
await Promise.all([reconnect(), reconnect(), reconnect()]);
|
||||
expect(reconnectCount).toBe(1);
|
||||
});
|
||||
|
||||
it('executeRaw retry does not infinite-loop on persistent connection failure', async () => {
|
||||
// Simulate: first call fails (connection error), reconnect succeeds,
|
||||
// but retry also fails with a NON-connection error
|
||||
|
||||
@@ -49,46 +49,42 @@ describe.skipIf(skip)('v0.41.25.0 db-singleton shared-recovery regressions (#157
|
||||
tmpAuditDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-1570-e2e-'));
|
||||
});
|
||||
|
||||
test('CASE 1: shared singleton survives mid-operation disconnect via retry reconnect', async () => {
|
||||
// Reproduce the dream-cycle scenario: caller A is mid-batch, caller B
|
||||
// disconnects the module singleton, caller A's NEXT attempt enters
|
||||
// retry and the reconnect callback rebuilds the singleton before the
|
||||
// retry's fn fires. This is the symptom-fix contract we ship.
|
||||
await db.connect({ database_url: DATABASE_URL! });
|
||||
test('CASE 1: a borrower disconnect leaves the shared singleton ALIVE — no reconnect needed (#1471 ownership fix)', async () => {
|
||||
// The dream-cycle scenario: caller A is mid-batch, caller B (a probe engine
|
||||
// that BORROWED the singleton) disconnects. Pre-#1471, B's disconnect
|
||||
// cascaded to db.disconnect() and nulled the singleton for A, so A's next
|
||||
// call threw "connect() has not been called" and only batchRetry's reconnect
|
||||
// could recover (and sync/synthesize, which never enter batchRetry, stayed
|
||||
// broken). Post-#1471, B is a borrower (it joined the singleton beforeAll
|
||||
// created) and its disconnect is a no-op — the singleton survives WITHOUT
|
||||
// any reconnect, which is what protects the non-batch phases.
|
||||
await db.connect({ database_url: DATABASE_URL! }); // already up from beforeAll → no-op
|
||||
|
||||
const engineA = new PostgresEngine();
|
||||
await engineA.connect({ database_url: DATABASE_URL! });
|
||||
await engineA.connect({ database_url: DATABASE_URL! }); // borrows
|
||||
const engineB = new PostgresEngine();
|
||||
await engineB.connect({ database_url: DATABASE_URL! });
|
||||
await engineB.connect({ database_url: DATABASE_URL! }); // borrows
|
||||
|
||||
// Sanity: both engines share the live singleton.
|
||||
expect((await engineA.sql`SELECT 1 as ok`)[0].ok).toBe(1);
|
||||
expect((await engineB.sql`SELECT 1 as ok`)[0].ok).toBe(1);
|
||||
|
||||
// Engine B disconnects mid-operation (the "offending caller" scenario).
|
||||
// This nulls the module singleton for engine A too.
|
||||
// Engine B (a borrower) disconnects mid-operation. The bug fix: this MUST
|
||||
// NOT null the singleton engine A is still using.
|
||||
await engineB.disconnect();
|
||||
|
||||
// Engine A's direct unsafe call will throw — proving the bug class
|
||||
// exists at the engine.sql layer.
|
||||
let directThrew = false;
|
||||
try {
|
||||
await engineA.sql`SELECT 1`;
|
||||
} catch {
|
||||
directThrew = true;
|
||||
}
|
||||
expect(directThrew).toBe(true);
|
||||
// Engine A's direct call now SUCCEEDS (pre-fix it threw). This is the
|
||||
// inverted assertion — the path that used to "prove the bug exists" now
|
||||
// proves the bug is gone. No reconnect, no retry: just works.
|
||||
const afterBorrowerDisconnect = await engineA.sql`SELECT 1 as ok`;
|
||||
expect(afterBorrowerDisconnect[0].ok).toBe(1);
|
||||
|
||||
// The retry layer's reconnect callback recovers. We exercise it via
|
||||
// engine.reconnect() directly (which is what batchRetry's injected
|
||||
// reconnect callback calls). After reconnect, engine A's next call
|
||||
// succeeds.
|
||||
// Defense-in-depth: reconnect() still works on a borrower (re-borrows the
|
||||
// still-live singleton) — the genuine-transient-drop recovery path is intact.
|
||||
await engineA.reconnect();
|
||||
const afterRecovery = await engineA.sql`SELECT 1 as ok`;
|
||||
expect(afterRecovery[0].ok).toBe(1);
|
||||
expect((await engineA.sql`SELECT 1 as ok`)[0].ok).toBe(1);
|
||||
|
||||
// Cleanup
|
||||
await engineA.disconnect();
|
||||
await engineA.disconnect(); // borrower no-op; singleton torn down by afterAll
|
||||
});
|
||||
|
||||
test('CASE 2: diagnostic audit records every mid-process disconnect call', async () => {
|
||||
|
||||
@@ -23,7 +23,7 @@
|
||||
* (second call no-ops).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { describe, test, expect, beforeEach, afterAll } from 'bun:test';
|
||||
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
||||
import * as db from '../../src/core/db.ts';
|
||||
|
||||
@@ -35,12 +35,13 @@ if (skip) {
|
||||
console.log('Skipping postgres-engine-disconnect-idempotency E2E (DATABASE_URL not set)');
|
||||
}
|
||||
|
||||
describe.skipIf(skip)('PostgresEngine.disconnect idempotency', () => {
|
||||
beforeAll(async () => {
|
||||
// Establish the module-level connection so we can verify it survives
|
||||
// the instance-pool engine's double-disconnect.
|
||||
const ok1 = (rows: unknown[]) => (rows[0] as { ok: number }).ok;
|
||||
|
||||
describe.skipIf(skip)('PostgresEngine.disconnect idempotency + module-singleton ownership (#1471)', () => {
|
||||
// Every test builds its own module-singleton scenario from a clean slate, so
|
||||
// order is irrelevant and the ownership cases don't leak state into each other.
|
||||
beforeEach(async () => {
|
||||
await db.disconnect();
|
||||
await db.connect({ database_url: DATABASE_URL! });
|
||||
}, 30_000);
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -48,40 +49,85 @@ describe.skipIf(skip)('PostgresEngine.disconnect idempotency', () => {
|
||||
});
|
||||
|
||||
test('instance-pool engine: second disconnect() does NOT clobber module singleton', async () => {
|
||||
const engine = new PostgresEngine();
|
||||
await engine.connect({ database_url: DATABASE_URL!, poolSize: 2 });
|
||||
|
||||
// First disconnect — closes the engine's own pool.
|
||||
await engine.disconnect();
|
||||
|
||||
// Sanity: module-level connection still alive (this is what
|
||||
// helpers.ts's getConn() returns).
|
||||
const before = await db.getConnection().unsafe('SELECT 1 as ok');
|
||||
expect((before[0] as unknown as { ok: number }).ok).toBe(1);
|
||||
|
||||
// Second disconnect — pre-fix, this fell through to db.disconnect()
|
||||
// and cleared the module-level singleton. Post-fix, it's a no-op.
|
||||
await engine.disconnect();
|
||||
|
||||
// Module-level connection MUST still be alive.
|
||||
const after = await db.getConnection().unsafe('SELECT 1 as ok');
|
||||
expect((after[0] as unknown as { ok: number }).ok).toBe(1);
|
||||
});
|
||||
|
||||
test('module-singleton engine: second disconnect() is a no-op', async () => {
|
||||
// Re-establish module-level connection (idempotent; no-op if still
|
||||
// connected from beforeAll).
|
||||
// Establish a module-level baseline (the cycle's singleton).
|
||||
await db.connect({ database_url: DATABASE_URL! });
|
||||
|
||||
const engine = new PostgresEngine();
|
||||
// No poolSize → uses the module-level singleton.
|
||||
await engine.connect({ database_url: DATABASE_URL! });
|
||||
await engine.connect({ database_url: DATABASE_URL!, poolSize: 2 });
|
||||
|
||||
// First disconnect closes module-level singleton (this engine owned it).
|
||||
await engine.disconnect();
|
||||
expect(ok1(await db.getConnection().unsafe('SELECT 1 as ok'))).toBe(1);
|
||||
|
||||
// Second disconnect must NOT throw — should be a no-op since
|
||||
// _connectionStyle was reset to null.
|
||||
// Second disconnect — pre-fix this fell through to db.disconnect() and
|
||||
// cleared the module-level singleton. Post-fix it's a no-op (instance style).
|
||||
await engine.disconnect();
|
||||
expect(ok1(await db.getConnection().unsafe('SELECT 1 as ok'))).toBe(1);
|
||||
});
|
||||
|
||||
test('owner-first / borrower-second: a borrower disconnect must NOT null the owner singleton', async () => {
|
||||
// The exact dream-cycle bug: the owner (cycle engine) creates the singleton,
|
||||
// a probe (lint/doctor config-lift) borrows it, the probe disconnects, and
|
||||
// pre-fix that nulled the singleton the owner was still using.
|
||||
const owner = new PostgresEngine();
|
||||
await owner.connect({ database_url: DATABASE_URL! }); // module branch → creates → owns
|
||||
const borrower = new PostgresEngine();
|
||||
await borrower.connect({ database_url: DATABASE_URL! }); // singleton exists → borrows
|
||||
|
||||
await borrower.disconnect(); // pre-fix: db.disconnect() → singleton null
|
||||
|
||||
// The owner must still be able to run DB work (this is sync/synthesize).
|
||||
expect(ok1(await owner.executeRaw('SELECT 1 as ok'))).toBe(1);
|
||||
expect(() => db.getConnection()).not.toThrow();
|
||||
|
||||
// And the owner — the true creator — DOES tear it down.
|
||||
await owner.disconnect();
|
||||
expect(() => db.getConnection()).toThrow(/No database connection/);
|
||||
});
|
||||
|
||||
test('ownership tracks CREATION, not role: a probe that creates the singleton owns it', async () => {
|
||||
// codex #2: ownership is not "the cycle engine" by name — it is whoever
|
||||
// atomically created the pool. If a probe creates first, it owns; a later
|
||||
// joiner is the borrower. (Safe in gbrain because the CLI engine is always
|
||||
// the first creator and last to disconnect — the dominance invariant.)
|
||||
const firstCreator = new PostgresEngine();
|
||||
await firstCreator.connect({ database_url: DATABASE_URL! }); // creates → owns
|
||||
const joiner = new PostgresEngine();
|
||||
await joiner.connect({ database_url: DATABASE_URL! }); // joins → borrows
|
||||
|
||||
await joiner.disconnect(); // no-op on the singleton
|
||||
expect(() => db.getConnection()).not.toThrow();
|
||||
await firstCreator.disconnect(); // creator tears down
|
||||
expect(() => db.getConnection()).toThrow(/No database connection/);
|
||||
});
|
||||
|
||||
test('symmetric CLI-exit: a sole owner connect+disconnect tears the singleton down (no hang regression)', async () => {
|
||||
const engine = new PostgresEngine();
|
||||
await engine.connect({ database_url: DATABASE_URL! });
|
||||
await engine.disconnect();
|
||||
// Pool must be CLOSED so the CLI event loop drains and `gbrain init` /
|
||||
// op-dispatch exit cleanly (the failure mode refcount would risk).
|
||||
expect(() => db.getConnection()).toThrow(/No database connection/);
|
||||
// Idempotent second disconnect.
|
||||
await expect(engine.disconnect()).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
test('owner reconnect with a live borrower: borrower resolves the rebuilt singleton', async () => {
|
||||
const owner = new PostgresEngine();
|
||||
await owner.connect({ database_url: DATABASE_URL! }); // owns
|
||||
const borrower = new PostgresEngine();
|
||||
await borrower.connect({ database_url: DATABASE_URL! }); // borrows
|
||||
|
||||
// Owner reconnect (the batchRetry path): tears down the old singleton and
|
||||
// builds a fresh one, re-acquiring ownership via the atomic db.connect() token.
|
||||
await owner.reconnect();
|
||||
|
||||
// The borrower's normal query path (this.sql → db.getConnection()) resolves
|
||||
// the NEW singleton. (codex #4: the borrower's cached connectionManager pool
|
||||
// is stale — that ddl()-only edge is a filed TODO, not this normal path.)
|
||||
expect(ok1(await borrower.sql.unsafe('SELECT 1 as ok'))).toBe(1);
|
||||
expect(ok1(await owner.executeRaw('SELECT 1 as ok'))).toBe(1);
|
||||
|
||||
await owner.disconnect();
|
||||
expect(() => db.getConnection()).toThrow(/No database connection/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -40,4 +40,28 @@ describe('PostgresEngine.sql getter self-heal (issue #1678)', () => {
|
||||
(e as unknown as { _sql: unknown })._sql = fakeSql;
|
||||
expect(e.sql as unknown).toBe(fakeSql);
|
||||
});
|
||||
|
||||
// #1471: the getter self-heal is INTENTIONALLY gated to instance style. A
|
||||
// module-style engine with a null _sql must fall through to the loud legacy
|
||||
// db.getConnection() error, NOT the instance "reaped pool" self-heal. Post
|
||||
// ownership-fix, the module singleton never goes null via a borrower
|
||||
// disconnect, so a null module singleton signals a genuine bug we want loud
|
||||
// (never connected, or a real owner-side teardown) rather than papered over.
|
||||
it('module-style + null _sql falls through to the loud legacy error, not the instance self-heal', () => {
|
||||
const e = new PostgresEngine();
|
||||
(e as unknown as { _connectionStyle: string })._connectionStyle = 'module';
|
||||
(e as unknown as { _sql: unknown })._sql = null;
|
||||
|
||||
let thrown: unknown;
|
||||
try {
|
||||
void e.sql; // delegates to db.getConnection(), which throws (no module connect)
|
||||
} catch (err) {
|
||||
thrown = err;
|
||||
}
|
||||
expect(thrown).toBeDefined();
|
||||
const msg = (thrown as Error).message;
|
||||
// The intentional asymmetry: module style keeps the legacy loud message.
|
||||
expect(msg).toContain('No database connection');
|
||||
expect(msg).not.toContain('instance connection pool');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
/**
|
||||
* postgres-engine.ts module-singleton ownership guardrails (#1471).
|
||||
*
|
||||
* Root cause: when an engine connects via the module-singleton path (no
|
||||
* poolSize), BOTH the engine that creates the shared db.ts `sql` singleton
|
||||
* (the owner, e.g. the CLI cycle engine) and any engine constructed while
|
||||
* that singleton already exists (a borrower, e.g. the probe engine in
|
||||
* resolveLintContentSanity / doctor) got `_connectionStyle = 'module'`. The
|
||||
* old disconnect() then called `db.disconnect()` for BOTH, so a short-lived
|
||||
* borrower's teardown nulled the shared `sql` the owner was still using —
|
||||
* every later cycle phase then threw "No database connection: connect() has
|
||||
* not been called" and stranded the cycle lock.
|
||||
*
|
||||
* Fix: track ownership via a token returned atomically by db.connect(). Only
|
||||
* the engine whose connect() actually created the singleton may db.disconnect()
|
||||
* it; borrowers clear their own marker without touching the shared connection.
|
||||
*
|
||||
* Source-level, DB-free guardrails — matching the existing
|
||||
* postgres-engine.test.ts convention (runtime mocking of postgres.js's
|
||||
* tagged-template interface is painful under bun ESM; live behaviour is
|
||||
* exercised by test/e2e/postgres-engine-disconnect-idempotency.test.ts).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
const ENGINE_SRC = readFileSync(
|
||||
join(import.meta.dir, '..', 'src', 'core', 'postgres-engine.ts'),
|
||||
'utf-8',
|
||||
);
|
||||
const DB_SRC = readFileSync(
|
||||
join(import.meta.dir, '..', 'src', 'core', 'db.ts'),
|
||||
'utf-8',
|
||||
);
|
||||
|
||||
describe('postgres-engine / module-singleton ownership (#1471)', () => {
|
||||
// Boolean assertions on purpose: matching a regex against the 4500-line
|
||||
// source dumps the whole file into the failure message. `.test()` -> bool
|
||||
// keeps RED output (and CI logs) readable.
|
||||
|
||||
test('db.connect() returns the ownership token (Promise<boolean>), not void', () => {
|
||||
const sig = /export async function connect\(config: EngineConfig\): Promise<boolean>/.test(DB_SRC);
|
||||
expect(sig).toBe(true);
|
||||
});
|
||||
|
||||
test('db.connect() borrower path returns false; creator path returns true', () => {
|
||||
const connect = stripComments(extractFn(DB_SRC, 'connect'));
|
||||
// Borrower (singleton already exists) → return false.
|
||||
expect(/if\s*\(\s*sql\s*\)\s*\{[\s\S]*?return false/.test(connect)).toBe(true);
|
||||
// Creator (built + validated the pool) → return true.
|
||||
expect(/return true/.test(connect)).toBe(true);
|
||||
});
|
||||
|
||||
test('PostgresEngine tracks module-singleton ownership with a dedicated flag', () => {
|
||||
expect(ENGINE_SRC.includes('_ownsModuleSingleton')).toBe(true);
|
||||
});
|
||||
|
||||
test('connect() stores the ownership token from db.connect() — no separate pre-sample (TOCTOU guard)', () => {
|
||||
const connect = stripComments(extractMethod(ENGINE_SRC, 'connect'));
|
||||
// Ownership is the RETURN of db.connect(), assigned to the flag.
|
||||
expect(/_ownsModuleSingleton\s*=\s*await\s+db\.connect\s*\(/.test(connect)).toBe(true);
|
||||
// Regression guard against the original TOCTOU shape: no separate
|
||||
// db.isConnected() probe sampled before db.connect().
|
||||
expect(/db\.isConnected\s*\(/.test(connect)).toBe(false);
|
||||
});
|
||||
|
||||
test('disconnect() calls db.disconnect() ONLY when this engine owns the singleton', () => {
|
||||
const disconnect = stripComments(extractMethod(ENGINE_SRC, 'disconnect'));
|
||||
// The shared-singleton teardown must be guarded by the ownership flag — a
|
||||
// borrower clears its marker without nulling the owner's connection.
|
||||
const guarded = /if\s*\(\s*this\._ownsModuleSingleton\s*\)\s*\{[\s\S]*?db\.disconnect\s*\(\s*\)/.test(disconnect);
|
||||
expect(guarded).toBe(true);
|
||||
// And the only db.disconnect() in the method is the guarded one (no
|
||||
// unconditional clobber survives).
|
||||
const calls = [...disconnect.matchAll(/db\.disconnect\s*\(\s*\)/g)];
|
||||
expect(calls.length).toBe(1);
|
||||
});
|
||||
|
||||
test('db.disconnect() snapshots + nulls the singleton BEFORE awaiting end() (codex #6)', () => {
|
||||
const disconnect = stripComments(extractFn(DB_SRC, 'disconnect'));
|
||||
const snapshotIdx = disconnect.search(/const\s+s\s*=\s*sql/);
|
||||
const nullIdx = disconnect.search(/\bsql\s*=\s*null/);
|
||||
const endIdx = disconnect.search(/\bawait\s+s\.end\s*\(/);
|
||||
expect(snapshotIdx).toBeGreaterThanOrEqual(0);
|
||||
expect(nullIdx).toBeGreaterThanOrEqual(0);
|
||||
expect(endIdx).toBeGreaterThanOrEqual(0);
|
||||
// null the module ref BEFORE the await, so a concurrent connect() can't
|
||||
// join a pool that's already closing.
|
||||
expect(nullIdx < endIdx).toBe(true);
|
||||
});
|
||||
|
||||
test('reconnect() is connection-style aware: module-singleton path never tears down the shared pool (#1745)', () => {
|
||||
const reconnect = stripComments(extractMethod(ENGINE_SRC, 'reconnect'));
|
||||
// Module-singleton engines must NOT route through this.disconnect()/db.disconnect()
|
||||
// on reconnect — they recover idempotently via db.connect() + setReadPool, so a
|
||||
// transient blip can't null the shared singleton other phases are using.
|
||||
expect(/this\._connectionStyle\s*!==\s*'instance'/.test(reconnect)).toBe(true);
|
||||
expect(/db\.connect\(this\._savedConfig\)/.test(reconnect)).toBe(true);
|
||||
expect(/setReadPool\(db\.getConnection\(\)\)/.test(reconnect)).toBe(true);
|
||||
// The instance path keeps the `_reconnecting` re-entrancy guard.
|
||||
expect(/this\._reconnecting\s*=\s*true/.test(reconnect)).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
function stripComments(s: string): string {
|
||||
return s
|
||||
.replace(/\/\*[\s\S]*?\*\//g, '')
|
||||
.replace(/(^|\s)\/\/[^\n]*/g, '$1');
|
||||
}
|
||||
|
||||
// Balance a `{...}` body starting at the first `{` after `headerIdx`, where
|
||||
// headerIdx points at (or before) the signature's parameter list. Skips the
|
||||
// parameter-list parens so a `{` inside a param type isn't mistaken for the body.
|
||||
function balanceBodyFrom(source: string, headerIdx: number, what: string): string {
|
||||
let i = source.indexOf('(', headerIdx);
|
||||
let pdepth = 0;
|
||||
for (; i < source.length; i++) {
|
||||
if (source[i] === '(') pdepth++;
|
||||
else if (source[i] === ')') {
|
||||
pdepth--;
|
||||
if (pdepth === 0) { i++; break; }
|
||||
}
|
||||
}
|
||||
i = source.indexOf('{', i);
|
||||
if (i < 0) throw new Error(`no body brace for ${what}`);
|
||||
const start = i;
|
||||
let depth = 0;
|
||||
for (; i < source.length; i++) {
|
||||
if (source[i] === '{') depth++;
|
||||
else if (source[i] === '}') {
|
||||
depth--;
|
||||
if (depth === 0) return source.slice(start, i + 1);
|
||||
}
|
||||
}
|
||||
throw new Error(`unbalanced body for ${what}`);
|
||||
}
|
||||
|
||||
// Class method body by name (async or not).
|
||||
function extractMethod(source: string, name: string): string {
|
||||
const openRe = new RegExp(`^\\s+(?:async\\s+)?${name}\\s*\\(`, 'm');
|
||||
const match = openRe.exec(source);
|
||||
if (!match) throw new Error(`method ${name} not found`);
|
||||
return balanceBodyFrom(source, match.index, name);
|
||||
}
|
||||
|
||||
// Top-level exported function body by name.
|
||||
function extractFn(source: string, name: string): string {
|
||||
const openRe = new RegExp(`export async function ${name}\\s*\\(`);
|
||||
const match = openRe.exec(source);
|
||||
if (!match) throw new Error(`function ${name} not found`);
|
||||
return balanceBodyFrom(source, match.index, name);
|
||||
}
|
||||
Reference in New Issue
Block a user