mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.42.42.0 fix(cli): bounded teardown + explicit exit — kill the 10s force-exit tax on txn-mode poolers (#2084) (#2141)
* feat(core): finishCliTeardown + flushThenExit — bounded teardown, owned exit verdict (#2084) cli-force-exit.ts becomes the single owner of one-shot CLI exit + teardown: - finishCliTeardown: bounded sink drain -> bounded disconnect under a backstop whose deadline is COMPUTED from the bounds it guards (floor 10s; GBRAIN_TEARDOWN_DEADLINE_MS env override). Arms at teardown start, never before the op handler. - flushThenExit: stdio write-fence (unref'd guard, EPIPE-safe) + REF'D aliveness grace for non-TTY stdio — Bun only delivers queued pipe writes while the process is alive (no flush API reaches the native queue). - setCliExitVerdict/currentExitCode: the exit verdict lives in a gbrain-owned channel, never read back from process.exitCode (PGLite's Emscripten runtime scribbles its own status there mid-run). - background-work.ts exports backgroundWorkSinkCount() for the deadline formula. Unit tests + a spawned-Bun harness proving byte-complete piped output. * fix(cli): route all nine disconnect sites through finishCliTeardown; one exit seam (#2084) Deletes the pre-handler 10s force-exit timer (it measured handler + teardown combined: PgBouncer txn-mode deployments paid a flat 10s banner tax on every query, and any >10s op was killed mid-run with exit 0 and truncated output). Sweeps op-dispatch, CLI_ONLY fall-through, search dashboard, read-only timeout path, dream, doctor x3 (fixing a pre-existing pool leak when DB checks throw), and ze-switch. The ONE process exit lives in main().then/catch via flushThenExit(currentExitCode()), gated by shouldForceExitAfterMain(). Exit-code writers (op-dispatch catch, reindex, transcripts, brainstorm, autopilot, frontmatter) now set the verdict through setCliExitVerdict. * fix(pglite): contain Emscripten's process.exitCode writes at PGlite.create (#2084) PGLite's WASM runtime writes its own status into process.exitCode (99 at create; in-memory brains run initdb whose status lands on a later tick; the exit status at close) — on PGLite every error exit was silently clobbered. preservingProcessExitCode wraps create() to keep the global tidy; db.close() stays unwrapped (its 0-write is baseline behavior test runners depend on). The CLI verdict itself is immune: it lives in the owned channel. * test: e2e + structural pins for the #2084 teardown contract E2E: failed op exits 1; every swept command spawned (brain-copy isolation for mutators, no-network); slow-handler regression via the deadline env knob; piped --json parses complete; teardown banner absent on every happy path; daemon survival untouched. Structural: no bare awaited engine disconnects in cli.ts; DISCONNECT_HARD_DEADLINE_MS gone; >=9 helper call sites; verdict channel + create-wrap pins. * test: fix R1 env-isolation violations in retrieval-reflex tests Pre-existing on master: both files mutated GBRAIN_RETRIEVAL_REFLEX directly, failing scripts/check-test-isolation.sh (bun run verify). Converted to the canonical withEnv() pattern; the reflex describe's beforeEach also never restored the flag, leaking it across the shard. * docs: KEY_FILES entries for the teardown contract; close + file TODOS (#2084) KEY_FILES.md: current-state entry for cli-force-exit.ts (helper + central exit seam pair, verdict channel, cli.ts-scoped claim); background-work.ts and pglite-engine.ts entries updated. TODOS.md: the drain-before-owner-disconnect P3 (filed from #1972) is done by this wave; files the trigger-gated GBRAIN_COMMAND_DEADLINE_MS follow-up (eng-review D2/D14). * fix: pre-landing review fixes (#2084) Review army (testing/maintainability/security/performance, 0 critical): - drain defense-in-depth: a throwing drain warns and still disconnects (cannot escape a caller's finally or skip the engine teardown) - behavioral tests for preservingProcessExitCode (connect pins 0; create-throw restores the pre-call verdict) - D9 widening test (live-registry sink count feeds the deadline formula), env 0/negative boundary cases, verdict mirror-write assertion - stale comments: header diagram backstop line, structural-test 'both lifecycle calls' contradiction, KEY_FILES 10s-force-exit clauses, e2e D11 falsification story corrected - named the formula's pool-end literals * fix: adversarial-review hardening — daemon-safe command resolution, flush knob, ref'd backstop (#2084) Cross-model adversarial review (Claude subagent + Codex, both P1'd it): - shouldForceExitAfterMain now resolves the command through parseGlobalFlags — the old first-non-dash heuristic read `gbrain --timeout 30s serve` as command "30s" and the new exit seam would have killed the daemon ~250ms after boot with exit 0 (unit-pinned) - GBRAIN_FLUSH_GRACE_MS env override for the non-TTY aliveness grace (batch consumers piping large payloads to slow readers can raise it; agent loops can lower it) - backstop timer is now REF'D: a hung teardown on an otherwise-empty event loop previously exited naturally — skipping the flush and surfacing PGLite's scribbled process.exitCode - flushThenExit: real process.exit latched once per process - doctor-site comment corrected; in-command process.exit teardown-bypass class (pre-existing) filed as a P2 TODO * chore: bump version and changelog (v0.42.42.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: move #2084 exitCode-containment lifecycle tests to the serial quarantine (R3) * docs: update project documentation for v0.42.42.0 - docs/TESTING.md: replace the stale 4-file serial-quarantine enumeration with a current-state description (the quarantine is glob-discovered, now several dozen files incl. the #2084 exitCode-containment suite); add unit inventory entries for test/cli-finish-teardown.test.ts and test/flush-then-exit-harness.test.ts. - docs/architecture/KEY_FILES.md: rephrase the pglite-engine exitCode containment note to current-state wording (clears the check-key-files-current-state prose-history warning). llms bundles regenerated (byte-identical: both docs are link-only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: apply cross-model doc-review findings for v0.42.42.0 Codex review of docs-vs-shipped-code found 9 gaps; all verified against the code before fixing: - CHANGELOG.md (0.42.42.0 entry, precision narrowing only — no entries touched): "every CLI exit path" -> "every cli.ts disconnect site"; "on every path" -> "on every routed exit path"; dream/doctor/ze-switch claim scoped to dispatcher teardown (command-internal process.exit sites are tracked in TODOS as the open P2). - docs/architecture/KEY_FILES.md: the teardown backstop is REF'D, not unref'd (matches the F3 adversarial-review decision in the code). - src/core/cli-force-exit.ts: header diagram comment had the same stale unref'd claim + `process.exitCode ?? 0`; now matches the implementation (ref'd timer, `currentExitCode()`). Comment-only change. - docs/TESTING.md: verify is the 30-check parallel battery via run-verify-parallel.sh (was described as 4 checks); CI is 10 weighted LPT shards + dedicated verify/serial/slow jobs (was "4-way FNV on shard 1"); test:serial runs one bun process per file (not --max-concurrency=1); dead "cap: 10" line rewritten as debt guidance; inventory entries added for test/cli-should-force-exit.test.ts and test/e2e/pglite-cli-exit.serial.test.ts. bun run verify green (30/30); #2084 test files green; llms bundles regenerated (byte-identical — reference docs are link-only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route v0.42.41.0's raw exitCode writers through the verdict channel; reconcile merged structural pins (#2084) CI fallout from merging the v0.42.41.0 triage wave into the #2084 exit-seam design — both waves fixed the same timer-placement bug independently: - doctor.ts + extract.ts set failure exit codes via raw `process.exitCode =` writes (v0.42.41.0's process.exit -> exitCode conversion); the #2084 exit seam reads only the gbrain-owned verdict channel, so doctor FAILs exited 0 (Tier 1 RLS e2e + half-migrated-Minions tests). Converted to setCliExitVerdict, same as the wallclock-124 site in the merge commit. - cli-force-exit-teardown-arming.test.ts pinned v0.42.41.0's inline finally-armed timer, which the merge replaced with finishCliTeardown; rewritten to pin the merged invariant (no pre-try arming in cli.ts; the backstop arms inside the helper before the drain). - eval-capture drain timing bound 1s -> 2s: flaked at 1023ms under CI shard load after the new test files shifted LPT shard packing (13x budget slack still proves bounded-not-hung). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Fable 5
Claude Fable 5
parent
7c27fa129b
commit
4ee530f3c5
@@ -0,0 +1,510 @@
|
||||
/**
|
||||
* #2084 — unit tests for the one-shot CLI teardown + exit contract in
|
||||
* src/core/cli-force-exit.ts: finishCliTeardown (teardown-only, computed
|
||||
* backstop deadline), flushThenExit (write-fence + guard + EPIPE + once-latch),
|
||||
* and computeTeardownDeadlineMs (formula / floor / env override).
|
||||
*
|
||||
* Real short timers, no fake clocks. Every test that touches process.exitCode
|
||||
* or GBRAIN_TEARDOWN_DEADLINE_MS restores it in a finally so the suite stays
|
||||
* order-independent.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
finishCliTeardown,
|
||||
flushThenExit,
|
||||
computeTeardownDeadlineMs,
|
||||
TEARDOWN_DEADLINE_FLOOR_MS,
|
||||
setCliExitVerdict,
|
||||
currentExitCode,
|
||||
_resetCliExitVerdictForTests,
|
||||
type MinimalWritable,
|
||||
} from '../src/core/cli-force-exit.ts';
|
||||
import { POOL_END_TIMEOUT_SECONDS } from '../src/core/db.ts';
|
||||
import {
|
||||
backgroundWorkSinkCount,
|
||||
__registerDrainerForTest,
|
||||
} from '../src/core/background-work.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
function fakeStream(): MinimalWritable & { writes: string[] } {
|
||||
const writes: string[] = [];
|
||||
return {
|
||||
writes,
|
||||
write(chunk: string, cb?: (err?: Error | null) => void) {
|
||||
writes.push(chunk);
|
||||
if (cb) queueMicrotask(() => cb());
|
||||
return true;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('computeTeardownDeadlineMs', () => {
|
||||
test('formula: sinks × drain + facts grace + 2 × pool bound + slack', () => {
|
||||
const poolEndBoundMs = POOL_END_TIMEOUT_SECONDS * 1000 + 500;
|
||||
// 4 sinks × 2000 + 2000 + 2×poolEnd + 2000 — the Site B worst case that
|
||||
// falsified the old static 10s (eng-review D9).
|
||||
const got = computeTeardownDeadlineMs({ sinkCount: 4, drainTimeoutMs: 2000 });
|
||||
expect(got).toBe(4 * 2000 + 2000 + 2 * poolEndBoundMs + 2000);
|
||||
expect(got).toBeGreaterThan(10_000); // the codex-found arithmetic bug, pinned
|
||||
});
|
||||
|
||||
test('floors at TEARDOWN_DEADLINE_FLOOR_MS for small budgets', () => {
|
||||
const got = computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 });
|
||||
expect(got).toBe(TEARDOWN_DEADLINE_FLOOR_MS);
|
||||
});
|
||||
|
||||
test('GBRAIN_TEARDOWN_DEADLINE_MS env override wins over the formula', async () => {
|
||||
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: '1234' }, async () => {
|
||||
expect(computeTeardownDeadlineMs({ sinkCount: 4, drainTimeoutMs: 2000 })).toBe(1234);
|
||||
});
|
||||
});
|
||||
|
||||
test('garbage env values fall back to the formula', async () => {
|
||||
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: 'banana' }, async () => {
|
||||
expect(
|
||||
computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 }),
|
||||
).toBe(TEARDOWN_DEADLINE_FLOOR_MS);
|
||||
});
|
||||
});
|
||||
|
||||
test('zero and negative env values fall back to the formula (not "fire immediately")', async () => {
|
||||
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: '0' }, async () => {
|
||||
expect(computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 })).toBe(
|
||||
TEARDOWN_DEADLINE_FLOOR_MS,
|
||||
);
|
||||
});
|
||||
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: '-5' }, async () => {
|
||||
expect(computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 })).toBe(
|
||||
TEARDOWN_DEADLINE_FLOOR_MS,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
test('a newly registered sink widens the computed deadline (D9: formula reads the live registry)', () => {
|
||||
// Register two sinks and compare between them: in a bare unit-test process
|
||||
// no production sinks are loaded, so the zero-sink baseline sits below the
|
||||
// 10s floor and would mask the first sink's delta.
|
||||
const mkSink = (name: string) =>
|
||||
__registerDrainerForTest({ name, order: 99, drain: async () => ({ unfinished: 0 }) });
|
||||
const un1 = mkSink('test-2084-sink-a');
|
||||
try {
|
||||
const withOne = computeTeardownDeadlineMs({
|
||||
sinkCount: backgroundWorkSinkCount(),
|
||||
drainTimeoutMs: 5000,
|
||||
});
|
||||
const un2 = mkSink('test-2084-sink-b');
|
||||
try {
|
||||
const withTwo = computeTeardownDeadlineMs({
|
||||
sinkCount: backgroundWorkSinkCount(),
|
||||
drainTimeoutMs: 5000,
|
||||
});
|
||||
expect(withOne).toBeGreaterThan(TEARDOWN_DEADLINE_FLOOR_MS); // above the floor — delta is visible
|
||||
expect(withTwo).toBe(withOne + 5000);
|
||||
} finally {
|
||||
un2();
|
||||
}
|
||||
} finally {
|
||||
un1();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('finishCliTeardown — clean path', () => {
|
||||
test('drains with the injected budget, disconnects, returns; no exit, no warn', async () => {
|
||||
const calls: string[] = [];
|
||||
let drainBudget = -1;
|
||||
const exits: number[] = [];
|
||||
const warns: string[] = [];
|
||||
await finishCliTeardown({
|
||||
engine: { disconnect: async () => void calls.push('disconnect') },
|
||||
drainTimeoutMs: 777,
|
||||
deadlineMs: 250,
|
||||
drain: async ({ timeoutMs }) => {
|
||||
drainBudget = timeoutMs;
|
||||
calls.push('drain');
|
||||
},
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: (m) => void warns.push(m),
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
});
|
||||
// Past the 250ms deadline: a leaked backstop would fire here.
|
||||
await sleep(400);
|
||||
expect(calls).toEqual(['drain', 'disconnect']);
|
||||
expect(drainBudget).toBe(777);
|
||||
expect(exits).toEqual([]);
|
||||
expect(warns).toEqual([]);
|
||||
});
|
||||
|
||||
test('drain runs BEFORE disconnect (live-engine window for sinks)', async () => {
|
||||
const order: string[] = [];
|
||||
await finishCliTeardown({
|
||||
engine: { disconnect: async () => void order.push('disconnect') },
|
||||
deadlineMs: 1000,
|
||||
drain: async () => {
|
||||
await sleep(20);
|
||||
order.push('drain');
|
||||
},
|
||||
exit: () => {},
|
||||
warn: () => {},
|
||||
});
|
||||
expect(order).toEqual(['drain', 'disconnect']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('finishCliTeardown — backstop on hung teardown', () => {
|
||||
test('hung disconnect fires the banner and exits with current exitCode', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
_resetCliExitVerdictForTests(); // no verdict set ⇒ currentExitCode() === 0
|
||||
const exits: number[] = [];
|
||||
const warns: string[] = [];
|
||||
let resolveHang!: () => void;
|
||||
const teardown = finishCliTeardown({
|
||||
engine: { disconnect: () => new Promise<void>((r) => (resolveHang = r)) },
|
||||
deadlineMs: 100,
|
||||
drain: async () => {},
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: (m) => void warns.push(m),
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(300);
|
||||
expect(warns.length).toBe(1);
|
||||
expect(warns[0]).toContain('did not return within');
|
||||
expect(warns[0]).toContain('100ms');
|
||||
expect(exits).toEqual([0]);
|
||||
resolveHang(); // unhang so the promise settles
|
||||
await teardown;
|
||||
} finally {
|
||||
_resetCliExitVerdictForTests();
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('backstop honors an exit code the errored op already set', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
setCliExitVerdict(1); // what the op-dispatch catch does
|
||||
const exits: number[] = [];
|
||||
let resolveHang!: () => void;
|
||||
const teardown = finishCliTeardown({
|
||||
engine: { disconnect: () => new Promise<void>((r) => (resolveHang = r)) },
|
||||
deadlineMs: 100,
|
||||
drain: async () => {},
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: () => {},
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(300);
|
||||
expect(exits).toEqual([1]);
|
||||
resolveHang();
|
||||
await teardown;
|
||||
} finally {
|
||||
_resetCliExitVerdictForTests();
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('hung DRAIN (not just disconnect) also trips the backstop', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
_resetCliExitVerdictForTests();
|
||||
const exits: number[] = [];
|
||||
const warns: string[] = [];
|
||||
let resolveHang!: () => void;
|
||||
const teardown = finishCliTeardown({
|
||||
engine: { disconnect: async () => {} },
|
||||
deadlineMs: 100,
|
||||
drain: () => new Promise<void>((r) => (resolveHang = r)),
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: (m) => void warns.push(m),
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(300);
|
||||
expect(warns.length).toBe(1);
|
||||
expect(exits).toEqual([0]);
|
||||
resolveHang();
|
||||
await teardown;
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('verdict channel — immune to PGLite WASM process.exitCode writes', () => {
|
||||
test('engine teardown that rewrites process.exitCode does not change the verdict', async () => {
|
||||
// PGLite's Emscripten runtime writes its own status into process.exitCode
|
||||
// at arbitrary points (99 at create, initdb status on a later tick for
|
||||
// in-memory brains, 0 at close) — pre-#2084 this clobbered an errored
|
||||
// op's exit 1 back to 0 on every PGLite error path. The verdict lives in
|
||||
// the gbrain-owned channel and never reads the global back.
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
setCliExitVerdict(1); // the op errored
|
||||
await finishCliTeardown({
|
||||
engine: {
|
||||
disconnect: async () => {
|
||||
process.exitCode = 0; // what PGLite's WASM shutdown does
|
||||
},
|
||||
},
|
||||
deadlineMs: 1000,
|
||||
drain: async () => {},
|
||||
exit: () => {},
|
||||
warn: () => {},
|
||||
});
|
||||
expect(currentExitCode()).toBe(1);
|
||||
} finally {
|
||||
_resetCliExitVerdictForTests();
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('mid-run WASM write (in-memory initdb status) cannot fake a verdict', () => {
|
||||
_resetCliExitVerdictForTests();
|
||||
try {
|
||||
process.exitCode = 100; // what in-memory PGLite's initdb does mid-run
|
||||
expect(currentExitCode()).toBe(0); // no gbrain verdict was ever set
|
||||
setCliExitVerdict(2);
|
||||
expect(currentExitCode()).toBe(2);
|
||||
// The mirror write exists for EXTERNAL readers of the global.
|
||||
expect(process.exitCode).toBe(2);
|
||||
} finally {
|
||||
_resetCliExitVerdictForTests();
|
||||
process.exitCode = 0;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('finishCliTeardown — disconnect failure (D3: exit code reports the op)', () => {
|
||||
test('a throwing drain is warned, disconnect still runs, helper resolves', async () => {
|
||||
// The registry is contractually non-throwing; this pins the defense-in-depth
|
||||
// guard — a drain rejection must not skip disconnect or escape the caller's
|
||||
// finally (it would replace a successful op's completion).
|
||||
const calls: string[] = [];
|
||||
const warns: string[] = [];
|
||||
await finishCliTeardown({
|
||||
engine: { disconnect: async () => void calls.push('disconnect') },
|
||||
deadlineMs: 1000,
|
||||
drain: async () => {
|
||||
throw new Error('sink registry blew up');
|
||||
},
|
||||
exit: () => {},
|
||||
warn: (m) => void warns.push(m),
|
||||
});
|
||||
expect(calls).toEqual(['disconnect']);
|
||||
expect(warns.length).toBe(1);
|
||||
expect(warns[0]).toContain('sink registry blew up');
|
||||
});
|
||||
|
||||
test('disconnect throw is warned and swallowed; helper resolves', async () => {
|
||||
const warns: string[] = [];
|
||||
const exits: number[] = [];
|
||||
await finishCliTeardown({
|
||||
engine: {
|
||||
disconnect: async () => {
|
||||
throw new Error('pool already dead');
|
||||
},
|
||||
},
|
||||
deadlineMs: 1000,
|
||||
drain: async () => {},
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: (m) => void warns.push(m),
|
||||
});
|
||||
expect(warns.length).toBe(1);
|
||||
expect(warns[0]).toContain('pool already dead');
|
||||
expect(exits).toEqual([]); // helper never exits on the non-backstop path
|
||||
});
|
||||
});
|
||||
|
||||
describe('flushThenExit', () => {
|
||||
test('exits after BOTH stream callbacks fire, exactly once, with the code', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const events: string[] = [];
|
||||
const exits: number[] = [];
|
||||
const slowStream = (name: string): MinimalWritable => ({
|
||||
write(_c: string, cb?: (err?: Error | null) => void) {
|
||||
setTimeout(() => {
|
||||
events.push(`${name}-flushed`);
|
||||
cb?.();
|
||||
}, 50);
|
||||
return true;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
});
|
||||
flushThenExit(3, {
|
||||
exit: (c) => {
|
||||
events.push('exit');
|
||||
exits.push(c);
|
||||
},
|
||||
stdout: slowStream('stdout'),
|
||||
stderr: slowStream('stderr'),
|
||||
guardMs: 2000,
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(200);
|
||||
expect(events).toEqual(['stdout-flushed', 'stderr-flushed', 'exit']);
|
||||
expect(exits).toEqual([3]);
|
||||
expect(process.exitCode).toBe(3); // belt-and-braces for natural exit
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('non-TTY default: exit waits the aliveness grace AFTER the fence', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const exits: number[] = [];
|
||||
const t0 = Date.now();
|
||||
let fencedAt = -1;
|
||||
const stream: MinimalWritable = {
|
||||
write(_c: string, cb?: (err?: Error | null) => void) {
|
||||
fencedAt = Date.now() - t0;
|
||||
if (cb) queueMicrotask(() => cb());
|
||||
return true;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: stream,
|
||||
stderr: stream,
|
||||
guardMs: 2000,
|
||||
graceMs: 120, // fakes are non-TTY; explicit grace keeps the test tight
|
||||
});
|
||||
await sleep(60);
|
||||
expect(exits).toEqual([]); // fence done, still inside the grace window
|
||||
await sleep(150);
|
||||
expect(exits).toEqual([0]);
|
||||
expect(fencedAt).toBeGreaterThanOrEqual(0);
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('guard fires when a callback never arrives (blocked pipe)', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const exits: number[] = [];
|
||||
const blockedStream: MinimalWritable = {
|
||||
write() {
|
||||
return false; // never calls cb — reader stopped consuming
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
const t0 = Date.now();
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: blockedStream,
|
||||
stderr: blockedStream,
|
||||
guardMs: 100,
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(300);
|
||||
expect(exits).toEqual([0]);
|
||||
expect(Date.now() - t0).toBeGreaterThanOrEqual(100);
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('sync write throw (EPIPE) still exits', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const exits: number[] = [];
|
||||
const epipeStream: MinimalWritable = {
|
||||
write() {
|
||||
throw Object.assign(new Error('write EPIPE'), { code: 'EPIPE' });
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: epipeStream,
|
||||
stderr: epipeStream,
|
||||
guardMs: 2000,
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(50);
|
||||
expect(exits).toEqual([0]);
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('GBRAIN_FLUSH_GRACE_MS env override is honored (batch/incident knob)', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
await withEnv({ GBRAIN_FLUSH_GRACE_MS: '0' }, async () => {
|
||||
const exits: number[] = [];
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
guardMs: 2000,
|
||||
// no graceMs → resolves through the env override (fakes are non-TTY)
|
||||
});
|
||||
await sleep(60);
|
||||
expect(exits).toEqual([0]); // grace 0: exit right after the fence
|
||||
});
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('once-latch: guard + late callbacks cannot double-exit', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const exits: number[] = [];
|
||||
// stdout flushes late (after the guard), stderr never — both race finish().
|
||||
const lateStream: MinimalWritable = {
|
||||
write(_c: string, cb?: (err?: Error | null) => void) {
|
||||
setTimeout(() => cb?.(), 150);
|
||||
return true;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
const neverStream: MinimalWritable = {
|
||||
write() {
|
||||
return false;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: lateStream,
|
||||
stderr: neverStream,
|
||||
guardMs: 80,
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(400);
|
||||
expect(exits).toEqual([0]); // exactly one exit despite guard + late cb
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1,56 +1,55 @@
|
||||
/**
|
||||
* Structural regression — the DISCONNECT_HARD_DEADLINE_MS force-exit timer in
|
||||
* cli.ts main() must be armed at TEARDOWN ENTRY (inside the finally, before
|
||||
* the drain + disconnect), never before the op-dispatch try block.
|
||||
* Structural regression — the teardown hard-deadline must be armed at
|
||||
* TEARDOWN ENTRY, never before the op-dispatch body.
|
||||
*
|
||||
* Pre-fix bug: the 10s unref'd setTimeout was armed BEFORE the try, so any op
|
||||
* whose handler ran past 10s wall-clock was killed mid-flight with
|
||||
* process.exit(0) and ZERO stdout — an empty "success" indistinguishable from
|
||||
* no results (a healthy `gbrain search` on a slow Postgres pooler hit this on
|
||||
* every run). Armed in the finally, the timer still bounds a hung
|
||||
* drain/disconnect (the C13 contract) but can no longer kill a
|
||||
* slow-but-progressing op body.
|
||||
* Pre-fix bug (closed independently by v0.42.41.0 and the #2084 wave, merged):
|
||||
* a 10s unref'd setTimeout armed BEFORE the try killed any op whose handler
|
||||
* ran past 10s wall-clock with process.exit(0) and ZERO stdout — an empty
|
||||
* "success" indistinguishable from no results.
|
||||
*
|
||||
* Source-grep is the right tool here (same rationale as
|
||||
* fix-wave-structural.test.ts): the rule is "this arming must stay at this
|
||||
* location". A behavioral test would need >10s of real wall-clock plus a
|
||||
* deliberately slow op handler in a spawned CLI — slow and flaky by
|
||||
* construction.
|
||||
* Post-merge shape (#2084): the deadline lives inside `finishCliTeardown`
|
||||
* (src/core/cli-force-exit.ts), armed as the helper's first act — i.e. at
|
||||
* teardown entry, because every cli.ts call site invokes the helper from a
|
||||
* `finally`. The op body's wallclock is bounded separately by the read-scope
|
||||
* withTimeout wrap (v0.42.41.0). Source-grep is the right tool here (same
|
||||
* rationale as fix-wave-structural.test.ts): a behavioral test would need
|
||||
* >10s of real wall-clock in a spawned CLI.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
describe('cli.ts — disconnect hard-deadline armed at teardown entry, not before the op body', () => {
|
||||
test('forceExitTimer setTimeout lives inside the finally, gated on the daemon guard, before the drain', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
test('no timer arming exists between op-dispatch setup and the try; the deadline arms inside finishCliTeardown before the drain', () => {
|
||||
const cli = readFileSync('src/cli.ts', 'utf8');
|
||||
|
||||
const decl = src.indexOf('const DISCONNECT_HARD_DEADLINE_MS');
|
||||
expect(decl).toBeGreaterThan(-1);
|
||||
const tryIdx = src.indexOf('try {', decl);
|
||||
// The old pre-try arming constant must stay gone (its return is the
|
||||
// kill-slow-ops-with-exit-0 regression).
|
||||
expect(cli).not.toContain('DISCONNECT_HARD_DEADLINE_MS');
|
||||
|
||||
// Between the op-dispatch engine connect and the try there is no
|
||||
// setTimeout call site (`setTimeout(` matches calls only; the
|
||||
// ReturnType<typeof setTimeout> annotation stays allowed).
|
||||
const connectIdx = cli.indexOf('// Local engine path (unchanged behavior for local installs).');
|
||||
expect(connectIdx).toBeGreaterThan(-1);
|
||||
const tryIdx = cli.indexOf('try {', connectIdx);
|
||||
expect(tryIdx).toBeGreaterThan(-1);
|
||||
const finallyIdx = src.indexOf('} finally {', tryIdx);
|
||||
expect(cli.slice(connectIdx, tryIdx)).not.toContain('setTimeout(');
|
||||
|
||||
// The op-dispatch finally routes through the shared teardown helper.
|
||||
const finallyIdx = cli.indexOf('} finally {', tryIdx);
|
||||
expect(finallyIdx).toBeGreaterThan(-1);
|
||||
const armIdx = src.indexOf('forceExitTimer = setTimeout', decl);
|
||||
const teardownCallIdx = cli.indexOf('finishCliTeardown({ engine, drainTimeoutMs: 1000 })', finallyIdx);
|
||||
expect(teardownCallIdx).toBeGreaterThan(finallyIdx);
|
||||
|
||||
// Inside the helper, the backstop arms BEFORE the drain runs — teardown
|
||||
// entry, bounding drain + disconnect and nothing else.
|
||||
const helper = readFileSync('src/core/cli-force-exit.ts', 'utf8');
|
||||
const armIdx = helper.indexOf('const backstop = setTimeout(');
|
||||
expect(armIdx).toBeGreaterThan(-1);
|
||||
const drainIdx = src.indexOf('drainAllBackgroundWorkForCliExit', finallyIdx);
|
||||
expect(drainIdx).toBeGreaterThan(-1);
|
||||
|
||||
// NO arming between the deadline declaration and the op-body try — a
|
||||
// pre-try timer kills slow-but-progressing op handlers mid-flight with
|
||||
// exit 0 and empty stdout. (`setTimeout(` matches only a call site; the
|
||||
// `ReturnType<typeof setTimeout>` type annotation stays allowed.)
|
||||
expect(src.slice(decl, tryIdx)).not.toContain('setTimeout(');
|
||||
|
||||
// The arming sits AFTER the finally opens (teardown entry) and BEFORE the
|
||||
// drain + disconnect it exists to bound.
|
||||
expect(armIdx).toBeGreaterThan(finallyIdx);
|
||||
expect(armIdx).toBeLessThan(drainIdx);
|
||||
|
||||
// Still gated on the daemon-survival guard so `serve` stays alive, and
|
||||
// still unref'd + cleared on clean teardown.
|
||||
expect(src.slice(finallyIdx, drainIdx)).toMatch(/if \(shouldForceExitAfterMain\(\)\)/);
|
||||
expect(src.slice(finallyIdx, drainIdx)).toContain('forceExitTimer.unref?.()');
|
||||
expect(src.slice(drainIdx)).toContain('if (forceExitTimer) clearTimeout(forceExitTimer)');
|
||||
const drainIdx = helper.indexOf('await drain({ timeoutMs: drainTimeoutMs })', armIdx);
|
||||
expect(drainIdx).toBeGreaterThan(armIdx);
|
||||
// Cleared on clean teardown.
|
||||
expect(helper.indexOf('clearTimeout(backstop)', drainIdx)).toBeGreaterThan(drainIdx);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -38,6 +38,18 @@ describe('shouldForceExitAfterMain — daemon survival gate', () => {
|
||||
expect(shouldForceExitAfterMain(['get', 'people/alice'])).toBe(true);
|
||||
});
|
||||
|
||||
test('#2084 cross-model finding: space-separated global flag values cannot fake a command', () => {
|
||||
// `--timeout 30s serve` — the old first-non-dash heuristic resolved the
|
||||
// command as `30s` → true → the central exit seam would process.exit the
|
||||
// freshly started daemon ~250ms after boot, exit 0, no error. The gate now
|
||||
// resolves the command through parseGlobalFlags, matching main()'s dispatch.
|
||||
expect(shouldForceExitAfterMain(['--timeout', '30s', 'serve'])).toBe(false);
|
||||
expect(shouldForceExitAfterMain(['--timeout', '30s', 'serve', '--http'])).toBe(false);
|
||||
expect(shouldForceExitAfterMain(['--progress-interval', '500', 'serve'])).toBe(false);
|
||||
// ...and the same shape before a one-shot command still force-exits.
|
||||
expect(shouldForceExitAfterMain(['--timeout', '30s', 'query', 'x'])).toBe(true);
|
||||
});
|
||||
|
||||
test('returns true for non-daemon CLI commands', () => {
|
||||
expect(shouldForceExitAfterMain(['stats'])).toBe(true);
|
||||
expect(shouldForceExitAfterMain(['doctor'])).toBe(true);
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { spawn, spawnSync } from 'child_process';
|
||||
import {
|
||||
cpSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
rmSync,
|
||||
@@ -152,12 +153,13 @@ afterAll(() => {
|
||||
function runWithTimeout(
|
||||
args: string[],
|
||||
timeoutMs: number,
|
||||
envOverride?: Record<string, string>,
|
||||
): Promise<{ code: number | null; stdout: string; stderr: string; durationMs: number }> {
|
||||
return new Promise((resolveOut) => {
|
||||
const t0 = Date.now();
|
||||
const child = spawn(SHIM_PATH, args, {
|
||||
cwd: REPO_ROOT,
|
||||
env: runEnv,
|
||||
env: envOverride ? { ...runEnv, ...envOverride } : runEnv,
|
||||
});
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
@@ -173,6 +175,13 @@ function runWithTimeout(
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* #2084: the teardown backstop banner must NEVER appear on a healthy run —
|
||||
* it now means a teardown component violated its own bound, not "this
|
||||
* command was slower than 10s end-to-end" (the pre-#2084 misfire).
|
||||
*/
|
||||
const TEARDOWN_BANNER = 'did not return within';
|
||||
|
||||
describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290)', () => {
|
||||
test('gbrain search "foxtrot" exits 0 within 15s', async () => {
|
||||
const { code, stdout, stderr, durationMs } = await runWithTimeout(
|
||||
@@ -185,6 +194,7 @@ describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290
|
||||
`STDOUT:\n${stdout}\nSTDERR:\n${stderr}`,
|
||||
);
|
||||
}
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).toBe(0);
|
||||
// Must have actually returned a hit — else bumpLastRetrievedAt
|
||||
// would have early-returned on empty pageIds and the bug wouldn't
|
||||
@@ -203,6 +213,7 @@ describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290
|
||||
`STDOUT:\n${stdout}\nSTDERR:\n${stderr}`,
|
||||
);
|
||||
}
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).toBe(0);
|
||||
expect(stdout).toContain('foxtrot');
|
||||
}, 30_000);
|
||||
@@ -258,6 +269,7 @@ describe('v0.42.20.0 — gbrain capture (CLI_ONLY) exits cleanly + frees the loc
|
||||
|
||||
// The real lock-pin symptom: the NEXT command times out waiting for the
|
||||
// PGLite lock. Assert a subsequent read runs cleanly and quickly.
|
||||
expect(cap.stderr).not.toContain(TEARDOWN_BANNER);
|
||||
const next = await runWithTimeout(['get', 'meetings/capture-test'], 15_000);
|
||||
expect(next.durationMs).toBeLessThan(15_000);
|
||||
expect(next.stderr).not.toContain('Timed out waiting for PGLite lock');
|
||||
@@ -267,6 +279,136 @@ describe('v0.42.20.0 — gbrain capture (CLI_ONLY) exits cleanly + frees the loc
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describe('#2084 — explicit-exit teardown: every swept site exits clean, exit codes report the op', () => {
|
||||
// D6C hardening: mutating commands run against a throwaway COPY of the
|
||||
// seeded GBRAIN_HOME so a remediation/dream pass can't contaminate the
|
||||
// brain the other tests share.
|
||||
function copyBrainHome(label: string): string {
|
||||
const copy = mkdtempSync(join(tmpdir(), `gbrain-2084-${label}-`));
|
||||
cpSync(tmpHome, copy, { recursive: true });
|
||||
return copy;
|
||||
}
|
||||
|
||||
test('D5: failed op exits 1 with the error on stderr (exit code = op outcome)', async () => {
|
||||
const { code, stderr, durationMs } = await runWithTimeout(
|
||||
['get', 'nonexistent-slug-2084'],
|
||||
15_000,
|
||||
);
|
||||
expect(durationMs).toBeLessThan(15_000);
|
||||
expect(code).toBe(1);
|
||||
expect(stderr.length).toBeGreaterThan(0);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
}, 30_000);
|
||||
|
||||
test('search stats (dashboard path, Site C) exits 0, no banner', async () => {
|
||||
const { code, stdout, stderr } = await runWithTimeout(['search', 'stats'], 20_000);
|
||||
expect(code).toBe(0);
|
||||
expect(stdout.length).toBeGreaterThan(0);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
}, 30_000);
|
||||
|
||||
test('sources list (read-only timeout path, Site D) exits 0, no banner', async () => {
|
||||
const { code, stderr } = await runWithTimeout(['sources', 'list'], 20_000);
|
||||
expect(code).toBe(0);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
}, 30_000);
|
||||
|
||||
test('doctor (Site G — leak-fix shape) exits without hanging, no banner', async () => {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(['doctor'], 45_000);
|
||||
expect(durationMs).toBeLessThan(45_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
// Keyless CI may surface advisory findings; doctor's exit code reflects
|
||||
// brain health, not teardown health. The pin is: exits, no banner.
|
||||
expect(code).not.toBeNull();
|
||||
}, 60_000);
|
||||
|
||||
test('doctor --remediation-plan (Site F) exits without hanging, no banner', async () => {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(
|
||||
['doctor', '--remediation-plan'],
|
||||
30_000,
|
||||
);
|
||||
expect(durationMs).toBeLessThan(30_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).not.toBeNull();
|
||||
}, 45_000);
|
||||
|
||||
test('doctor --remediate (Site F, mutating) runs on a brain copy, exits, no banner', async () => {
|
||||
const copy = copyBrainHome('remediate');
|
||||
try {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(
|
||||
['doctor', '--remediate'],
|
||||
45_000,
|
||||
{ GBRAIN_HOME: copy },
|
||||
);
|
||||
expect(durationMs).toBeLessThan(45_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).not.toBeNull();
|
||||
} finally {
|
||||
rmSync(copy, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test('dream --dry-run (Site E — the overnight-cron TODO site) exits, no banner', async () => {
|
||||
const copy = copyBrainHome('dream');
|
||||
try {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(
|
||||
['dream', '--dry-run'],
|
||||
60_000,
|
||||
{ GBRAIN_HOME: copy },
|
||||
);
|
||||
// Keyless CI: LLM-dependent phases degrade; the IRON rule is exits + no
|
||||
// banner. A hang here is the silent-overnight-zombie regression.
|
||||
expect(durationMs).toBeLessThan(60_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).not.toBeNull();
|
||||
} finally {
|
||||
rmSync(copy, { recursive: true, force: true });
|
||||
}
|
||||
}, 90_000);
|
||||
|
||||
test('ze-switch --dry-run (Site H) exits without hanging, no banner', async () => {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(
|
||||
['ze-switch', '--dry-run'],
|
||||
30_000,
|
||||
);
|
||||
expect(durationMs).toBeLessThan(30_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).not.toBeNull();
|
||||
}, 45_000);
|
||||
|
||||
test('D11: teardown deadline does NOT cover handler time (slow-handler regression)', async () => {
|
||||
// Post-fix the deadline arms at teardown start, so a 500ms deadline cannot
|
||||
// touch the handler: results print in full regardless. NOTE the falsification
|
||||
// story is forward-looking, not historical — pre-#2084 code had no env knob
|
||||
// (a static 10s constant), so this spawn would pass there too; the guard
|
||||
// against re-hoisting the timer above the handler is the structural pin on
|
||||
// DISCONNECT_HARD_DEADLINE_MS absence in fix-wave-structural.test.ts. This
|
||||
// test pins that the env override is honored AND output survives a deadline
|
||||
// far smaller than handler time.
|
||||
const { code, stdout, durationMs } = await runWithTimeout(
|
||||
['search', 'foxtrot', '--limit', '3'],
|
||||
15_000,
|
||||
{ GBRAIN_TEARDOWN_DEADLINE_MS: '500' },
|
||||
);
|
||||
expect(durationMs).toBeLessThan(15_000);
|
||||
expect(code).toBe(0);
|
||||
expect(stdout.length).toBeGreaterThan(0); // output intact = handler wasn't killed
|
||||
}, 30_000);
|
||||
|
||||
test('D10: piped --json output parses complete (no exit truncation)', async () => {
|
||||
// `search stats --json` emits a pure JSON document (the shared-op search
|
||||
// path renders human format regardless of --json). A truncated-by-exit
|
||||
// pipe fails to parse — the #1959 class, end-to-end.
|
||||
const { code, stdout, stderr } = await runWithTimeout(
|
||||
['search', 'stats', '--json'],
|
||||
20_000,
|
||||
);
|
||||
expect(code).toBe(0);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(() => JSON.parse(stdout)).not.toThrow();
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
describe('v0.41.8.0 — daemon survival (regression guard for narrow force-exit)', () => {
|
||||
test('gbrain serve --http stays alive past the timeout window', async () => {
|
||||
// Pick a likely-free ephemeral port. We're testing "still alive
|
||||
|
||||
@@ -72,6 +72,9 @@ describe('awaitPendingEvalCaptures', () => {
|
||||
const r = await awaitPendingEvalCaptures(150);
|
||||
const elapsed = Date.now() - start;
|
||||
expect(r.unfinished).toBe(1);
|
||||
expect(elapsed).toBeLessThan(1000);
|
||||
// Bound proves "bounded, not a hang" — the alternative is infinite. 2s
|
||||
// (13x the 150ms budget) absorbs CI shard-load timer jitter; the old 1s
|
||||
// bound flaked at 1023ms on a loaded GitHub runner.
|
||||
expect(elapsed).toBeLessThan(2000);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -125,15 +125,15 @@ describe('v0.36.1.x #1124 — query --no-expand actually negates expand', () =>
|
||||
|
||||
describe('v0.42.20.0 — background-work registry drains every sink before disconnect', () => {
|
||||
// Supersedes the v0.41.8.0 #1247/#1269/#1290 per-call last-retrieved drain:
|
||||
// last-retrieved is now one of four registry sinks; cli.ts drains the whole
|
||||
// registry (drainAllBackgroundWorkForCliExit) before disconnect on BOTH the
|
||||
// op-dispatch path AND the CLI_ONLY path (the latter closes #1762 for capture).
|
||||
test('cli.ts imports + uses drainAllBackgroundWorkForCliExit', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
expect(src).toMatch(/import\s+\{\s*drainAllBackgroundWorkForCliExit\s*\}\s*from\s+['"]\.\/core\/background-work\.ts['"]/);
|
||||
// Two call sites: op-dispatch finally + handleCliOnly finally.
|
||||
const calls = src.match(/await\s+drainAllBackgroundWorkForCliExit\s*\(/g) ?? [];
|
||||
expect(calls.length).toBeGreaterThanOrEqual(2);
|
||||
// last-retrieved is one of four registry sinks. #2084 moved the registry
|
||||
// drain out of cli.ts's inline finallys into finishCliTeardown
|
||||
// (cli-force-exit.ts), which every cli.ts teardown site routes through —
|
||||
// the drain-before-disconnect invariant is pinned there (and behaviorally
|
||||
// by test/cli-finish-teardown.test.ts).
|
||||
test('cli-force-exit.ts imports + drains the registry inside finishCliTeardown', () => {
|
||||
const src = readFileSync('src/core/cli-force-exit.ts', 'utf8');
|
||||
expect(src).toMatch(/import\s+\{\s*drainAllBackgroundWorkForCliExit[\s\S]*?\}\s*from\s+['"]\.\/background-work\.ts['"]/);
|
||||
expect(src).toMatch(/export async function finishCliTeardown/);
|
||||
});
|
||||
|
||||
test('last-retrieved.ts still exports the bounded drain + registers a drainer', () => {
|
||||
@@ -155,17 +155,17 @@ describe('v0.42.20.0 — background-work registry drains every sink before disco
|
||||
.toMatch(/name:\s*'eval-capture'/);
|
||||
});
|
||||
|
||||
test('cli.ts behavioral positioning: registry drain appears BEFORE engine.disconnect (op-dispatch)', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
const localPath = src.match(/\/\/ Local engine path \(unchanged behavior[\s\S]+?^\}/m);
|
||||
expect(localPath).not.toBeNull();
|
||||
const block = localPath![0];
|
||||
const drainCallRe = /await\s+drainAllBackgroundWorkForCliExit\s*\(/;
|
||||
const disconnectCallRe = /await\s+engine\.disconnect\s*\(/;
|
||||
expect(block).toMatch(drainCallRe);
|
||||
expect(block).toMatch(disconnectCallRe);
|
||||
const drainIdx = block.indexOf(block.match(drainCallRe)![0]);
|
||||
const disconnectIdx = block.indexOf(block.match(disconnectCallRe)![0]);
|
||||
test('finishCliTeardown positioning: registry drain appears BEFORE engine disconnect', () => {
|
||||
// #2084: the invariant moved from cli.ts's inline finallys into the shared
|
||||
// helper. The drain must run against a live engine (facts abort-path
|
||||
// logIngest, #1762) before disconnect tears the pools down.
|
||||
const src = readFileSync('src/core/cli-force-exit.ts', 'utf8');
|
||||
const drainCallRe = /await\s+drain\s*\(\s*\{\s*timeoutMs:\s*drainTimeoutMs\s*\}\s*\)/;
|
||||
const disconnectCallRe = /await\s+opts\.engine\.disconnect\s*\(/;
|
||||
expect(src).toMatch(drainCallRe);
|
||||
expect(src).toMatch(disconnectCallRe);
|
||||
const drainIdx = src.indexOf(src.match(drainCallRe)![0]);
|
||||
const disconnectIdx = src.indexOf(src.match(disconnectCallRe)![0]);
|
||||
expect(drainIdx).toBeLessThan(disconnectIdx);
|
||||
});
|
||||
|
||||
@@ -184,6 +184,56 @@ describe('v0.42.20.0 — background-work registry drains every sink before disco
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2084 — cli.ts owns process-exit teardown via finishCliTeardown', () => {
|
||||
test('no bare awaited engine disconnects remain in cli.ts', () => {
|
||||
// The awaited forms are the call-site contract (comments never use the
|
||||
// awaited literal, so this is comment-proof — eng-review D13.2). A bare
|
||||
// disconnect skips the bounded drain + computed-deadline backstop and
|
||||
// reopens the lingering-socket hang class.
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
expect(src).not.toContain('await engine.disconnect()');
|
||||
expect(src).not.toContain('await eng.disconnect()');
|
||||
});
|
||||
|
||||
test('the pre-handler hard-deadline timer is gone (handler time is not teardown budget)', () => {
|
||||
// Pre-#2084 the op-dispatch timer armed BEFORE the op handler, so any op
|
||||
// slower than 10s was force-killed mid-run with exit 0 and truncated
|
||||
// output. The deadline now arms inside finishCliTeardown, at teardown
|
||||
// start only.
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
expect(src).not.toContain('DISCONNECT_HARD_DEADLINE_MS');
|
||||
});
|
||||
|
||||
test('all nine swept sites route through finishCliTeardown; one exit seam', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
const calls = src.match(/await finishCliTeardown\(/g) ?? [];
|
||||
expect(calls.length).toBeGreaterThanOrEqual(9);
|
||||
// The single process-exit seam: flushThenExit in the import.meta.main
|
||||
// block, fed by currentExitCode().
|
||||
expect(src).toMatch(/import\.meta\.main/);
|
||||
expect(src).toMatch(/flushThenExit\(currentExitCode\(\)\)/);
|
||||
});
|
||||
|
||||
test('pglite-engine contains the Emscripten process.exitCode hijack', () => {
|
||||
// PGLite's WASM runtime writes its own status into process.exitCode (99
|
||||
// alive / exit status on close) and ignores `undefined` assignment. The
|
||||
// create call runs inside preservingProcessExitCode to keep the global
|
||||
// tidy; close is deliberately unwrapped (see below) — the CLI's verdict
|
||||
// is immune either way via the owned channel.
|
||||
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
|
||||
expect(src).toMatch(/preservingProcessExitCode\(\(\)\s*=>\s*\n?\s*PGlite\.create/);
|
||||
// close stays UNWRAPPED by design: its status write is baseline behavior
|
||||
// test runners depend on; the CLI's verdict is immune because it lives in
|
||||
// the gbrain-owned channel, never read back from process.exitCode.
|
||||
const helper = readFileSync('src/core/cli-force-exit.ts', 'utf8');
|
||||
expect(helper).toMatch(/let cliVerdict: number \| null = null/);
|
||||
expect(helper).toMatch(/return cliVerdict \?\? 0/);
|
||||
// The op-dispatch catch must set the verdict through the owned channel.
|
||||
const cli = readFileSync('src/cli.ts', 'utf8');
|
||||
expect(cli).toMatch(/setCliExitVerdict\(1\);/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => {
|
||||
test('pglite-engine.ts exports classifyPgliteInitError + buildPgliteInitErrorMessage', () => {
|
||||
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
|
||||
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
/**
|
||||
* #2084 (D10) — spawned harness proving flushThenExit against REAL Bun pipe
|
||||
* semantics: writes HARNESS_BYTES of 'x' to stdout, then flushThenExit with
|
||||
* HARNESS_EXIT_CODE. The parent test pipes stdout to a slow-attaching reader
|
||||
* and asserts byte-complete output + the exit code — the exact scenario the
|
||||
* pre-#2084 force-exit truncated (#1959).
|
||||
*/
|
||||
|
||||
import { flushThenExit } from '../../src/core/cli-force-exit.ts';
|
||||
|
||||
const size = Number(process.env.HARNESS_BYTES ?? 4_000_000);
|
||||
const code = Number(process.env.HARNESS_EXIT_CODE ?? 7);
|
||||
const guardMs = Number(process.env.HARNESS_GUARD_MS ?? 2_000);
|
||||
const graceEnv = process.env.HARNESS_GRACE_MS;
|
||||
|
||||
const chunk = 'x'.repeat(65_536);
|
||||
let written = 0;
|
||||
while (written < size) {
|
||||
const n = Math.min(chunk.length, size - written);
|
||||
process.stdout.write(n === chunk.length ? chunk : chunk.slice(0, n));
|
||||
written += n;
|
||||
}
|
||||
|
||||
flushThenExit(code, {
|
||||
guardMs,
|
||||
...(graceEnv !== undefined ? { graceMs: Number(graceEnv) } : {}),
|
||||
});
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* #2084 (D10) — flushThenExit proven on a real spawned Bun process.
|
||||
*
|
||||
* The unit tests in cli-finish-teardown.test.ts inject fake streams; they
|
||||
* prove the helper's logic but not Bun's actual pipe behavior (does an empty
|
||||
* write('', cb) really fence all prior buffered chunks?). These tests spawn
|
||||
* test/fixtures/flush-then-exit-harness.ts and assert:
|
||||
* 1. multi-MB piped stdout arrives byte-complete with the right exit code
|
||||
* even when the reader attaches late (#1959 truncation regression pin);
|
||||
* 2. with empty buffers the fence resolves promptly — the process does NOT
|
||||
* sit out the flush guard (canary for Bun eliding empty-write callbacks).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { spawn } from 'child_process';
|
||||
import { resolve } from 'path';
|
||||
|
||||
const HARNESS = resolve(import.meta.dir, 'fixtures', 'flush-then-exit-harness.ts');
|
||||
|
||||
function runHarness(env: Record<string, string>, readerDelayMs: number): Promise<{
|
||||
bytes: number;
|
||||
code: number | null;
|
||||
durationMs: number;
|
||||
}> {
|
||||
return new Promise((resolveOut, reject) => {
|
||||
const t0 = Date.now();
|
||||
const child = spawn('bun', ['run', HARNESS], {
|
||||
env: { ...process.env, ...env },
|
||||
stdio: ['ignore', 'pipe', 'inherit'],
|
||||
});
|
||||
let bytes = 0;
|
||||
child.stdout.pause();
|
||||
setTimeout(() => {
|
||||
child.stdout.on('data', (d: Buffer) => (bytes += d.length));
|
||||
child.stdout.resume();
|
||||
}, readerDelayMs);
|
||||
const killer = setTimeout(() => {
|
||||
child.kill('SIGKILL');
|
||||
reject(new Error(`harness did not exit (bytes so far: ${bytes})`));
|
||||
}, 30_000);
|
||||
child.on('close', (code) => {
|
||||
clearTimeout(killer);
|
||||
resolveOut({ bytes, code, durationMs: Date.now() - t0 });
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
describe('flushThenExit on a real Bun process (D10)', () => {
|
||||
test('4MB piped stdout arrives byte-complete with the exit code, late reader', async () => {
|
||||
// Bun delivers queued pipe writes only while the process is alive (see the
|
||||
// cli-force-exit.ts module header). The reader attaches 200ms late; the
|
||||
// 1500ms aliveness grace must cover attach + transfer. Pre-#2084 (immediate
|
||||
// process.exit, no grace) this received 0 of the 4MB — verified during
|
||||
// implementation; that is the #1959 truncation class.
|
||||
const SIZE = 4_000_000;
|
||||
const { bytes, code } = await runHarness(
|
||||
{
|
||||
HARNESS_BYTES: String(SIZE),
|
||||
HARNESS_EXIT_CODE: '7',
|
||||
HARNESS_GUARD_MS: '2000',
|
||||
HARNESS_GRACE_MS: '1500',
|
||||
},
|
||||
200,
|
||||
);
|
||||
expect(bytes).toBe(SIZE);
|
||||
expect(code).toBe(7);
|
||||
}, 40_000);
|
||||
|
||||
test('default grace: small output survives exit with a concurrent reader', async () => {
|
||||
// No HARNESS_GRACE_MS → production default (non-TTY grace). Immediate
|
||||
// reader, 100-byte output: must arrive complete. Pre-#2084 even this case
|
||||
// lost ALL bytes when exit fired before a loop turn.
|
||||
const { bytes, code } = await runHarness(
|
||||
{ HARNESS_BYTES: '100', HARNESS_EXIT_CODE: '0', HARNESS_GUARD_MS: '2000' },
|
||||
0,
|
||||
);
|
||||
expect(bytes).toBe(100);
|
||||
expect(code).toBe(0);
|
||||
}, 40_000);
|
||||
|
||||
test('fence resolves promptly — wall time well under guard + grace ceiling', async () => {
|
||||
// Guard 8s, grace 250ms: if Bun ever elides the empty-write callback, the
|
||||
// process sits out the full guard and wall time exceeds it. A working
|
||||
// fence exits in startup time + grace (~1-2s).
|
||||
const { code, durationMs } = await runHarness(
|
||||
{
|
||||
HARNESS_BYTES: '100',
|
||||
HARNESS_EXIT_CODE: '0',
|
||||
HARNESS_GUARD_MS: '8000',
|
||||
HARNESS_GRACE_MS: '250',
|
||||
},
|
||||
0,
|
||||
);
|
||||
expect(code).toBe(0);
|
||||
expect(durationMs).toBeLessThan(6_000);
|
||||
}, 40_000);
|
||||
});
|
||||
@@ -219,3 +219,38 @@ describe('PGLiteEngine.disconnect() — v0.41.8.0 lifecycle invariants', () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// #2084 — preservingProcessExitCode behavioral containment
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
describe('PGLiteEngine: Emscripten process.exitCode containment (#2084)', () => {
|
||||
test('connect() leaves process.exitCode pinned at 0, not the Emscripten 99', async () => {
|
||||
const prev = process.exitCode;
|
||||
const eng = new PGLiteEngine();
|
||||
try {
|
||||
await eng.connect({ engine: 'pglite' });
|
||||
// Emscripten writes 99 during create; the wrapper pins explicit 0 when
|
||||
// nothing was set before (undefined cannot be restored — the accessor
|
||||
// falls back to the WASM status).
|
||||
expect(Number(process.exitCode)).toBe(0);
|
||||
} finally {
|
||||
await eng.disconnect();
|
||||
process.exitCode = prev;
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test('a pre-call verdict survives the create-throw path (finally restores)', async () => {
|
||||
const prev = process.exitCode;
|
||||
const eng = new PGLiteEngine();
|
||||
try {
|
||||
process.exitCode = 3;
|
||||
// A dataDir under a regular FILE cannot be created — PGlite.create rejects.
|
||||
await expect(
|
||||
eng.connect({ engine: 'pglite', database_path: '/dev/null/nope/brain' }),
|
||||
).rejects.toThrow();
|
||||
expect(Number(process.exitCode)).toBe(3);
|
||||
} finally {
|
||||
process.exitCode = prev;
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
@@ -1311,7 +1311,10 @@ describe('PGLiteEngine: v0.13.1 error-wrap on connect() (#223)', () => {
|
||||
// issue and suggest gbrain doctor. Must NOT suggest "missing migrations"
|
||||
// as a cause (that was conflating #218 and #223 — migrations run AFTER
|
||||
// create()).
|
||||
expect(src).toContain('this._db = await PGlite.create');
|
||||
// #2084 wrapped the create call in preservingProcessExitCode (Emscripten
|
||||
// exitCode containment); the try/catch + error wrap around it is unchanged.
|
||||
expect(src).toContain('this._db = await preservingProcessExitCode(() =>');
|
||||
expect(src).toContain('PGlite.create({');
|
||||
expect(src).toContain('https://github.com/garrytan/gbrain/issues/223');
|
||||
expect(src).toContain('gbrain doctor');
|
||||
expect(src).toContain('Original error:');
|
||||
|
||||
Reference in New Issue
Block a user