mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* fix(core): drain fire-and-forget sinks before disconnect via a background-work registry (#1762) New src/core/background-work.ts registry (Map<name,drainer>, ordered drain, awaited abort). facts-queue (order 0, abort=shutdown), last-retrieved (1), and eval-capture (3, now self-tracked) register as sinks. Both CLI exit paths (op-dispatch finally + handleCliOnly finally) drain the registry before engine.disconnect() so a PGLite db.close() can't race in-flight work into the re-pump busy-loop that pinned the single-writer lock. Op-dispatch error path converts process.exit(1) to exitCode+return so the finally still drains. * fix(ai): bound every outbound AI call so a stalled provider can't hang (#1762/#1775) withDefaultTimeout composes a per-touchpoint default deadline (chat 300s, embed/multimodal 60s) with any caller signal via AbortSignal.any. Applied at the SDK call layer (chat generateText, expand generateObject, OCR, per-sub-batch embed) — covers native-anthropic + retries — plus per-request multimodal fetch. embedQuery forwards abortSignal. Env: GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS. * fix(postgres): module-mode reconnect preserves the shared singleton (#1745) reconnect() branches on connection style. Module-singleton engines re-establish idempotently via db.connect() (no-op when alive) + refresh the ConnectionManager read pool, never db.disconnect() — so a transient blip no longer nulls the shared sql out from under concurrent ops (which threw 'connect() has not been called'). Fail-loud on real connect failure. Instance pools keep teardown+recreate. * fix(search): bound the query-time embed so a stall falls back to keyword (#1775) search/query default to cheap-hybrid (embeds the query); a stalled provider made the embed never settle, so the keyword fallback never engaged and the command force-exited with no output. One shared QueryEmbedDeadline (6s, floored 2s per embed) covers both the cache-lookup and inner embeds via embedQueryBounded (abortSignal + Promise.race) → existing keyword fallback engages. Also registers the search-cache background-work drainer (now bounded). Env: GBRAIN_QUERY_EMBED_TIMEOUT_MS. * test+chore: reliability wave tests + v0.42.11.0 (#1762 #1745 #1775) New: background-work registry unit, query-embed deadline unit, eval-capture drain unit, postgres reconnect E2E (#1745), gbrain capture exit-clean case in the PGLite serial test. Updated fix-wave-structural assertions to the registry shape. VERSION/package.json/CHANGELOG -> 0.42.11.0; TODOS retrofit marked done. Incorporates + hardens PR #1763 (drain-before-disconnect + embed fetch timeout); the residual hung-Haiku hole is closed by the facts shutdown() abort belt. Co-Authored-By: ElliotDrel <noreply@github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document background-work registry + v0.42.11.0 reliability wave in CLAUDE.md (regen llms) * chore: bump release version 0.42.11.0 -> 0.42.20.0 Rename the reliability-wave release version per request. Trio (VERSION / package.json / CHANGELOG) reconciled; in-code version-tag comments and test fixtures updated; llms regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: ElliotDrel <noreply@github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
99 lines
4.0 KiB
TypeScript
99 lines
4.0 KiB
TypeScript
/**
|
|
* E2E for #1745 (v0.42.20.0): module-mode reconnect() must NOT tear down the
|
|
* shared module singleton.
|
|
*
|
|
* The bug: a transient blip triggers withRetry's reconnect callback →
|
|
* PostgresEngine.reconnect() → this.disconnect() → (module mode) db.disconnect()
|
|
* → `sql.end(); sql = null`. Concurrent ops (other dream-cycle phases, the
|
|
* minion-queue promoteDelayed loop) read db.getConnection() during that null
|
|
* window and throw "No database connection: connect() has not been called".
|
|
*
|
|
* The fix: module-mode reconnect() never calls db.disconnect(); it idempotently
|
|
* re-establishes via db.connect() (a no-op when the singleton is alive, which is
|
|
* the common case) and refreshes the ConnectionManager read pool. postgres.js
|
|
* auto-heals dead sockets, so a transient blip recovers without a teardown.
|
|
*
|
|
* Instance-pool engines (worker / `jobs work`) keep the teardown+recreate path.
|
|
*/
|
|
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
|
import * as db from '../../src/core/db.ts';
|
|
|
|
const DATABASE_URL = process.env.DATABASE_URL;
|
|
const skip = !DATABASE_URL;
|
|
|
|
if (skip) {
|
|
// eslint-disable-next-line no-console
|
|
console.log('Skipping postgres-reconnect-singleton E2E (DATABASE_URL not set)');
|
|
}
|
|
|
|
describe.skipIf(skip)('#1745 — module-mode reconnect preserves the shared singleton', () => {
|
|
beforeAll(async () => {
|
|
await db.disconnect();
|
|
await db.connect({ database_url: DATABASE_URL! });
|
|
}, 30_000);
|
|
|
|
afterAll(async () => {
|
|
await db.disconnect();
|
|
});
|
|
|
|
test('module reconnect() keeps db.getConnection() live (no null window)', async () => {
|
|
const engine = new PostgresEngine();
|
|
await engine.connect({ database_url: DATABASE_URL! }); // module singleton (no poolSize)
|
|
|
|
// Capture the singleton reference BEFORE reconnect — pre-fix, reconnect's
|
|
// disconnect would sql.end() + null this out.
|
|
const sqlBefore = db.getConnection();
|
|
|
|
await engine.reconnect();
|
|
|
|
// Post-fix: the singleton is untouched (db.connect() is a no-op when alive),
|
|
// so getConnection() still returns a usable client and queries still work.
|
|
const sqlAfter = db.getConnection();
|
|
const rows = await sqlAfter.unsafe('SELECT 1 as ok');
|
|
expect((rows[0] as unknown as { ok: number }).ok).toBe(1);
|
|
// Same live pool object (no teardown/recreate in module mode).
|
|
expect(sqlAfter).toBe(sqlBefore);
|
|
}, 30_000);
|
|
|
|
test('concurrent reader during reconnect never sees "connect() has not been called"', async () => {
|
|
const engine = new PostgresEngine();
|
|
await engine.connect({ database_url: DATABASE_URL! });
|
|
|
|
// Fire a burst of reads through the module singleton WHILE reconnecting.
|
|
// Pre-fix, the disconnect→connect window nulled the singleton and the
|
|
// concurrent readers threw. Post-fix there is no null window.
|
|
const readers = Array.from({ length: 20 }, async () => {
|
|
const r = await db.getConnection().unsafe('SELECT 1 as ok');
|
|
return (r[0] as unknown as { ok: number }).ok;
|
|
});
|
|
const [reconnectErr, ...results] = await Promise.all([
|
|
engine.reconnect().then(() => null).catch((e) => e),
|
|
...readers,
|
|
]);
|
|
expect(reconnectErr).toBeNull();
|
|
for (const ok of results) expect(ok).toBe(1);
|
|
}, 30_000);
|
|
|
|
test('instance-pool reconnect() still teardown+recreates its own pool', async () => {
|
|
const engine = new PostgresEngine();
|
|
await engine.connect({ database_url: DATABASE_URL!, poolSize: 2 });
|
|
|
|
// Works before.
|
|
const before = await engine.executeRaw<{ ok: number }>('SELECT 1 as ok');
|
|
expect(before[0].ok).toBe(1);
|
|
|
|
await engine.reconnect(); // instance path: tears down + rebuilds _sql
|
|
|
|
// Works after (fresh pool).
|
|
const after = await engine.executeRaw<{ ok: number }>('SELECT 1 as ok');
|
|
expect(after[0].ok).toBe(1);
|
|
|
|
await engine.disconnect();
|
|
|
|
// Module singleton untouched by the instance engine's reconnect/disconnect.
|
|
const mod = await db.getConnection().unsafe('SELECT 1 as ok');
|
|
expect((mod[0] as unknown as { ok: number }).ok).toBe(1);
|
|
}, 30_000);
|
|
});
|