Files
gbrain/test/e2e/postgres-reconnect-singleton.test.ts
T
ec5fed2921 v0.42.20.0 fix: reliability wave — PGLite capture lock-pin + Postgres reconnect race + search embed-hang (#1762 #1745 #1775) (#1810)
* fix(core): drain fire-and-forget sinks before disconnect via a background-work registry (#1762)

New src/core/background-work.ts registry (Map<name,drainer>, ordered drain,
awaited abort). facts-queue (order 0, abort=shutdown), last-retrieved (1), and
eval-capture (3, now self-tracked) register as sinks. Both CLI exit paths
(op-dispatch finally + handleCliOnly finally) drain the registry before
engine.disconnect() so a PGLite db.close() can't race in-flight work into the
re-pump busy-loop that pinned the single-writer lock. Op-dispatch error path
converts process.exit(1) to exitCode+return so the finally still drains.

* fix(ai): bound every outbound AI call so a stalled provider can't hang (#1762/#1775)

withDefaultTimeout composes a per-touchpoint default deadline (chat 300s,
embed/multimodal 60s) with any caller signal via AbortSignal.any. Applied at the
SDK call layer (chat generateText, expand generateObject, OCR, per-sub-batch
embed) — covers native-anthropic + retries — plus per-request multimodal fetch.
embedQuery forwards abortSignal. Env: GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS.

* fix(postgres): module-mode reconnect preserves the shared singleton (#1745)

reconnect() branches on connection style. Module-singleton engines re-establish
idempotently via db.connect() (no-op when alive) + refresh the ConnectionManager
read pool, never db.disconnect() — so a transient blip no longer nulls the shared
sql out from under concurrent ops (which threw 'connect() has not been called').
Fail-loud on real connect failure. Instance pools keep teardown+recreate.

* fix(search): bound the query-time embed so a stall falls back to keyword (#1775)

search/query default to cheap-hybrid (embeds the query); a stalled provider made
the embed never settle, so the keyword fallback never engaged and the command
force-exited with no output. One shared QueryEmbedDeadline (6s, floored 2s per
embed) covers both the cache-lookup and inner embeds via embedQueryBounded
(abortSignal + Promise.race) → existing keyword fallback engages. Also registers
the search-cache background-work drainer (now bounded). Env: GBRAIN_QUERY_EMBED_TIMEOUT_MS.

* test+chore: reliability wave tests + v0.42.11.0 (#1762 #1745 #1775)

New: background-work registry unit, query-embed deadline unit, eval-capture
drain unit, postgres reconnect E2E (#1745), gbrain capture exit-clean case in
the PGLite serial test. Updated fix-wave-structural assertions to the registry
shape. VERSION/package.json/CHANGELOG -> 0.42.11.0; TODOS retrofit marked done.

Incorporates + hardens PR #1763 (drain-before-disconnect + embed fetch timeout);
the residual hung-Haiku hole is closed by the facts shutdown() abort belt.

Co-Authored-By: ElliotDrel <noreply@github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: document background-work registry + v0.42.11.0 reliability wave in CLAUDE.md (regen llms)

* chore: bump release version 0.42.11.0 -> 0.42.20.0

Rename the reliability-wave release version per request. Trio
(VERSION / package.json / CHANGELOG) reconciled; in-code version-tag
comments and test fixtures updated; llms regenerated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: ElliotDrel <noreply@github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 08:42:25 -07:00

99 lines
4.0 KiB
TypeScript

/**
* E2E for #1745 (v0.42.20.0): module-mode reconnect() must NOT tear down the
* shared module singleton.
*
* The bug: a transient blip triggers withRetry's reconnect callback →
* PostgresEngine.reconnect() → this.disconnect() → (module mode) db.disconnect()
* → `sql.end(); sql = null`. Concurrent ops (other dream-cycle phases, the
* minion-queue promoteDelayed loop) read db.getConnection() during that null
* window and throw "No database connection: connect() has not been called".
*
* The fix: module-mode reconnect() never calls db.disconnect(); it idempotently
* re-establishes via db.connect() (a no-op when the singleton is alive, which is
* the common case) and refreshes the ConnectionManager read pool. postgres.js
* auto-heals dead sockets, so a transient blip recovers without a teardown.
*
* Instance-pool engines (worker / `jobs work`) keep the teardown+recreate path.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
import * as db from '../../src/core/db.ts';
const DATABASE_URL = process.env.DATABASE_URL;
const skip = !DATABASE_URL;
if (skip) {
// eslint-disable-next-line no-console
console.log('Skipping postgres-reconnect-singleton E2E (DATABASE_URL not set)');
}
describe.skipIf(skip)('#1745 — module-mode reconnect preserves the shared singleton', () => {
beforeAll(async () => {
await db.disconnect();
await db.connect({ database_url: DATABASE_URL! });
}, 30_000);
afterAll(async () => {
await db.disconnect();
});
test('module reconnect() keeps db.getConnection() live (no null window)', async () => {
const engine = new PostgresEngine();
await engine.connect({ database_url: DATABASE_URL! }); // module singleton (no poolSize)
// Capture the singleton reference BEFORE reconnect — pre-fix, reconnect's
// disconnect would sql.end() + null this out.
const sqlBefore = db.getConnection();
await engine.reconnect();
// Post-fix: the singleton is untouched (db.connect() is a no-op when alive),
// so getConnection() still returns a usable client and queries still work.
const sqlAfter = db.getConnection();
const rows = await sqlAfter.unsafe('SELECT 1 as ok');
expect((rows[0] as unknown as { ok: number }).ok).toBe(1);
// Same live pool object (no teardown/recreate in module mode).
expect(sqlAfter).toBe(sqlBefore);
}, 30_000);
test('concurrent reader during reconnect never sees "connect() has not been called"', async () => {
const engine = new PostgresEngine();
await engine.connect({ database_url: DATABASE_URL! });
// Fire a burst of reads through the module singleton WHILE reconnecting.
// Pre-fix, the disconnect→connect window nulled the singleton and the
// concurrent readers threw. Post-fix there is no null window.
const readers = Array.from({ length: 20 }, async () => {
const r = await db.getConnection().unsafe('SELECT 1 as ok');
return (r[0] as unknown as { ok: number }).ok;
});
const [reconnectErr, ...results] = await Promise.all([
engine.reconnect().then(() => null).catch((e) => e),
...readers,
]);
expect(reconnectErr).toBeNull();
for (const ok of results) expect(ok).toBe(1);
}, 30_000);
test('instance-pool reconnect() still teardown+recreates its own pool', async () => {
const engine = new PostgresEngine();
await engine.connect({ database_url: DATABASE_URL!, poolSize: 2 });
// Works before.
const before = await engine.executeRaw<{ ok: number }>('SELECT 1 as ok');
expect(before[0].ok).toBe(1);
await engine.reconnect(); // instance path: tears down + rebuilds _sql
// Works after (fresh pool).
const after = await engine.executeRaw<{ ok: number }>('SELECT 1 as ok');
expect(after[0].ok).toBe(1);
await engine.disconnect();
// Module singleton untouched by the instance engine's reconnect/disconnect.
const mod = await db.getConnection().unsafe('SELECT 1 as ok');
expect((mod[0] as unknown as { ok: number }).ok).toBe(1);
}, 30_000);
});