Merge remote-tracking branch 'origin/master' into garrytan/newest-agents-filed-issues

# Conflicts:
#	TODOS.md
#	docs/architecture/KEY_FILES.md
#	src/cli.ts
#	src/commands/autopilot.ts
#	src/commands/brainstorm.ts
#	src/commands/doctor.ts
#	src/commands/extract.ts
#	src/commands/frontmatter.ts
#	src/commands/reindex.ts
#	src/commands/transcripts.ts
#	src/core/cli-force-exit.ts
#	test/cli-force-exit-teardown-arming.test.ts
#	test/fix-wave-structural.test.ts
This commit is contained in:
Garry Tan
2026-06-12 09:44:22 -07:00
33 changed files with 1524 additions and 728 deletions
+24 -4
View File
@@ -2,6 +2,28 @@
All notable changes to GBrain will be documented in this file.
## [0.42.42.0] - 2026-06-12
**`gbrain query` no longer pays a flat 10-second exit tax on managed Postgres behind a transaction-mode pooler — and CLI exit codes finally tell the truth on PGLite.** On deployments where the pooler holds sockets open past the bounded pool drain (gbrain#2084, a residual of gbrain#1972), every query printed its results and then sat for 10 seconds until the force-exit banner fired. The cause was two-layered: the hard-deadline timer was armed *before* the operation handler, so a multi-second search on a large brain burned the teardown budget (and any operation slower than 10 seconds was silently killed mid-run with exit 0 and truncated output); and the CLI never exited explicitly on success — it waited for Bun's event loop to drain, which a stuck pooler socket can hold open forever.
The teardown contract now lives in one place: every cli.ts disconnect site runs a bounded background-work drain and a bounded disconnect under a backstop whose deadline is computed from the bounds it guards (so it fires only when something violated its own bound), then the process exits explicitly — after fencing stdout/stderr and holding a short aliveness window so piped output is delivered (Bun queues pipe writes in a native buffer that only drains while the process is alive). The most-used command in the CLI now exits in milliseconds-to-a-couple-seconds instead of ten.
Along the way the wave fixed a deeper, silent bug: PGLite's WASM runtime writes its own status into `process.exitCode` at arbitrary points mid-run, which meant **every error exit on PGLite-engine brains has been reporting success (exit 0)** — scripts and agents keying on exit codes never saw failures. The CLI verdict now lives in a gbrain-owned channel that the WASM runtime cannot touch.
### Fixed
- **The flat 10s teardown tax + force-exit banner on transaction-mode poolers (gbrain#2084).** Queries exit promptly; the banner now appears only when a teardown component genuinely violated its own bound.
- **Slow operations are no longer killed mid-run with a false success.** The teardown deadline starts at teardown, never before the operation handler — a 30-second sync or a deep query runs to completion.
- **Error exits on PGLite report exit 1, not 0.** Failed operations (e.g. `gbrain get <missing-page>`) now exit non-zero on every engine; the exit code reports the operation, not the cleanup.
- **Piped output survives the exit.** Output is fenced and given a delivery window before the process exits, on every routed exit path including the backstop (the truncation class from gbrain#1959).
- **`gbrain doctor` no longer leaks its connection pool when DB checks throw**, and `dream`, `doctor`, `ze-switch`, and the search dashboards route their dispatcher teardown through the same bounded path (closing a long-standing drain gap on the overnight-cron path).
- **Daemon safety with space-separated global flags.** `gbrain --timeout 30s serve` is recognized as the daemon it is — the exit gate resolves the command exactly the way dispatch does.
### Added
- **`GBRAIN_TEARDOWN_DEADLINE_MS`** — env override for the teardown backstop deadline (incident escape hatch; the default is computed from the drain and pool bounds).
- **`GBRAIN_FLUSH_GRACE_MS`** — env override for the pre-exit output-delivery window (default 250ms on pipes, 0 on TTYs). Raise it when piping very large payloads into slow consumers; lower it for high-frequency scripted invocations that capture to files.
### To take advantage of v0.42.42.0
`gbrain upgrade`. No configuration needed. If your `gbrain query` has been printing results and then hanging ~10 seconds before a `force-exiting` banner, this release removes both the wait and the banner. If your scripts check gbrain exit codes on a PGLite brain, they will start seeing real failures — previously masked as exit 0 — so a wrapper that suddenly reports errors is the fix working, not a regression.
## [0.42.41.0] - 2026-06-11
**A correctness-and-reliability wave: your conversation facts survive a cycle, write-through stops polluting other repos, autopilot rides out a DB blip instead of crash-looping, and concurrent PGLite processes stop corrupting each other.** A triage of open reports surfaced six bugs with no fix yet plus a batch of community PRs; this ships them together, each with a regression test.
@@ -16422,8 +16444,7 @@ The OAuth provider in `src/core/oauth-provider.ts` got a parallel hardening pass
Smaller hardening: admin cookies set `Secure` when behind HTTPS or a public-URL proxy (F9), magic-link nonces are bounded by an LRU cap (F10), `/mcp` wraps `transport.handleRequest` in try/catch so SDK throws hit a JSON-RPC 500 instead of express's default HTML error page (F14), and OperationError + unexpected exceptions both route through the unified `buildError`/`serializeError` envelope (F15). DCR disable became a constructor option on the provider rather than a serve-http monkey-patch (F12 — cleanup, not security).
To take advantage of v0.26.9
============================
=====================
`gbrain upgrade` is a one-step upgrade. There is no migration; all changes are application-layer.
1. **Upgrade.** `gbrain upgrade`. Confirm `gbrain --version` shows `0.26.9`.
@@ -16557,8 +16578,7 @@ Both run at `--max-concurrency=1` after the parallel pass, same as the existing
Wallclock observed: 74s on a Mac dev box (running `bun run test` with the new quarantines). Already at the v0.26.9 informational target. The full intra-file marker flip (with codemod + per-file `test.concurrent()`) lands in v0.26.9 and aims for the same ≤60s with pinned config.
To take advantage of v0.26.7
============================
=====================
`gbrain upgrade` does nothing functional in this release — it ships test infrastructure, not user-facing code. But if you contribute tests:
1. **Run `bun run verify` before pushing.** The new `check-test-isolation.sh` runs alongside the privacy + jsonb + progress checks. Catches new env-mutation, mock.module, and PGLite-pattern violations before CI does.
+34 -5
View File
@@ -265,11 +265,40 @@ GSTACK REVIEW REPORT at
cleanup. Do this BEFORE introducing any concurrent module-engine connect path.
- [x] **P3 — `dream` + CLI_ONLY fall-through paths don't drain the facts /
last-retrieved queues before the owner disconnect.** DONE in the #2084 wave:
`drainThenDisconnect(engine)` in `src/cli.ts` is the shared helper, applied at
ALL eight owner-disconnect sites (op-dispatch, CLI_ONLY fall-through, search
dashboard, doctor remediation x3, ze-switch, dream, read-only-timeout path),
with its own bounded hard-deadline. Pinned by `test/cli-drain-then-disconnect.test.ts`.
last-retrieved queues before the owner disconnect.** DONE in the #2084 fix:
`finishCliTeardown` (`src/core/cli-force-exit.ts`) is exactly the shared
drain-before-disconnect helper this item asked for, and ALL NINE cli.ts
disconnect sites route through it (op-dispatch, fall-through, dream, doctor
×3, ze-switch, search dashboard, read-only timeout path). Structural guard:
no bare `await engine.disconnect()` remains in cli.ts
(`test/fix-wave-structural.test.ts` `#2084` describe).
- [ ] **P2 — command-module `process.exit` sites bypass the #2084 teardown
contract.** Several CLI_ONLY command modules exit directly on their normal
paths (`doctor.ts` ~10 sites incl. its verdict exit, `dream.ts` ~23,
`ze-switch.ts` ~9, plus friction/claw-test/eval verdict exits in cli.ts) —
those exits preempt the call-site `finally`, so the background-work drain,
bounded disconnect, and `flushThenExit` grace are all skipped on those paths
(pre-existing class, NOT introduced by #2084; pre-fix the same exits skipped
the inline drains too). Consequences: `gbrain doctor --json | <slow reader>`
keeps the #1959 truncation exposure; a dream path that exits mid-cycle
discards in-flight facts/search-cache writes. Fix shape: convert in-command
`process.exit(n)` to `setCliExitVerdict(n)` + return (the central seam
exits), or route them through a shared `exitCommand(n)` helper that runs
teardown first. Surfaced by the #2084 cross-model adversarial review (F2).
- [ ] **P3 — opt-in whole-command wallclock cap (`GBRAIN_COMMAND_DEADLINE_MS`),
build ONLY on a real wedged-handler incident.** The #2084 fix deliberately
removed the blanket pre-handler 10s force-exit (it killed slow-legit ops with
exit 0 and truncated output); per-op deadlines (query-embed deadline,
`withTimeout` on read-only commands) own handler wallclock now, and
`connectEngine` hangs — the historically observed zombie class — were never
covered by the old timer anyway. If production ever shows a genuinely wedged
handler (trigger: a non-`serve` command alive >30min with no progress
output), add an opt-in env cap that exits NON-ZERO with a truthful banner.
Attach point: the `GBRAIN_TEARDOWN_DEADLINE_MS` / `computeTeardownDeadlineMs`
plumbing in `src/core/cli-force-exit.ts`. Do not build speculatively —
follow-up from the #2084 eng review (decision D2/D14).
## v0.42.x AI SDK v6 tool-schema fix follow-ups (#1782/#1764)
Surfaced by the codex outside-voice pass during `/plan-eng-review` and
+1 -1
View File
@@ -1 +1 @@
0.42.41.0
0.42.42.0
+9 -5
View File
@@ -10,16 +10,16 @@ Seven test command tiers, each with a clear scope:
| Command | What it runs | Wallclock | When to use |
|---|---|---|---|
| `bun run test` | Parallel unit-test fast loop. 8-shard fan-out via `scripts/run-unit-parallel.sh`, then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | ~85s on a Mac dev box (3650+ tests) | Inner edit loop. Default. |
| `bun run verify` | CI's authoritative pre-test gate set: `check:privacy && check:jsonb && check:progress && check:wasm && bun run typecheck`. The 4 checks `.github/workflows/test.yml` runs on shard 1 + typecheck. Single source of truth — CI literally calls `bun run verify`. | ~12s (wasm-compile dominates) | Before pushing; before `/ship`. |
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (~30 checks — privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; runs at `--max-concurrency=1`). | ~1s per quarantined file | Debugging a specific quarantined file. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. |
### CI vs local: intentionally divergent file sets
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` 4-way, which uses FNV-1a hash bucketing and INCLUDES `*.slow.test.ts`. CI EXCLUDES `*.serial.test.ts` from the hash buckets and runs them on shard 1 via `bun run test:serial` at `--max-concurrency=1` — keeping serial files out of the hash buckets is what preserves the `mock.module` quarantine (top-level mocks in serial files would otherwise leak into the parallel files they share a shard process with). CI is the ground truth for "did everything pass."
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include.
@@ -39,7 +39,7 @@ If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the
- `*.test.ts` → fast loop (parallel 8-shard fan-out).
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; uses `--max-concurrency=1`. Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Currently: `test/brain-registry.serial.test.ts`, `test/reconcile-links.serial.test.ts`, `test/core/cycle.serial.test.ts`, `test/embed.serial.test.ts` (the latter two use `mock.module(...)` which leaks across files in the shard process). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
@@ -111,7 +111,7 @@ Rename to `*.serial.test.ts` when:
- The file is genuinely env-coupled (e.g. `gbrain-home-isolation.test.ts`, `claw-test-cli.test.ts`) — module-load env readers + ESM caching defeat dynamic-import-after-env tricks.
- The file's tests intentionally share state across `it()` boundaries.
Quarantine count cap: 10 (informational). Beyond that, push back on the design.
The quarantine has grown to dozens of files — treat it as debt: every addition needs a reason from the list above, and prefer fixing the contention root cause when one exists.
### Unit test inventory
@@ -123,6 +123,9 @@ Unit tests and what they cover:
- `test/chunkers/recursive.test.ts` — chunking.
- `test/parity.test.ts` — operations contract parity.
- `test/cli.test.ts` — CLI structure.
- `test/cli-finish-teardown.test.ts` — the #2084 teardown contract: `computeTeardownDeadlineMs` formula/floor/live-registry scaling + `GBRAIN_TEARDOWN_DEADLINE_MS` override (garbage/zero/negative values fall back to the formula); `finishCliTeardown` clean path (drain BEFORE disconnect, no exit, no warn), backstop on hung drain or disconnect (honors an errored op's exit code), throwing drain/disconnect warned + swallowed; the gbrain-owned verdict channel is immune to PGLite WASM `process.exitCode` writes; `flushThenExit` unit coverage with mocked streams (exits once after both stream callbacks, non-TTY aliveness grace, blocked-pipe guard, EPIPE-safe, `GBRAIN_FLUSH_GRACE_MS` override).
- `test/flush-then-exit-harness.test.ts` — real spawned-Bun pipe semantics for `flushThenExit` (fixture: `test/fixtures/flush-then-exit-harness.ts`): a 4MB piped stdout payload arrives byte-complete with the exit code even with a late reader, small output survives exit with a concurrent reader, and the fence resolves promptly (wall time well under the guard + grace ceiling).
- `test/cli-should-force-exit.test.ts``shouldForceExitAfterMain` daemon-survival gate: `serve` (stdio and `--http`) never force-exits, including with preceding global flags; op commands / empty / flag-only argv do; the #2084 case that space-separated global-flag VALUES can't fake a command (`--timeout 30s serve` resolves to the `serve` daemon, not a `30s` command).
- `test/config.test.ts` — config redaction.
- `test/files.test.ts` — MIME/hash.
- `test/import-file.test.ts` — import pipeline.
@@ -219,6 +222,7 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
- `test/e2e/sync.test.ts``--skip-failed` failure-loop test alongside happy-path tests: broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format.
- `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required).
- `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use.
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
- `test/e2e/openclaw-reference-compat.test.ts``check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
- `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `<fork>/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
- `test/e2e/search-exclude.test.ts``test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths.
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -143,5 +143,5 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.41.0"
"version": "0.42.42.0"
}
+93 -166
View File
@@ -25,8 +25,7 @@ import type { AIGatewayConfig } from './core/ai/types.ts';
import type { BrainEngine } from './core/engine.ts';
import { operations, OperationError } from './core/operations.ts';
import type { Operation, OperationContext } from './core/operations.ts';
import { drainAllBackgroundWorkForCliExit } from './core/background-work.ts';
import { shouldForceExitAfterMain, flushStdoutThenExit, setCliExitCode, getCliExitCode } from './core/cli-force-exit.ts';
import { shouldForceExitAfterMain, finishCliTeardown, flushThenExit, currentExitCode, setCliExitVerdict } from './core/cli-force-exit.ts';
import { serializeMarkdown } from './core/markdown.ts';
import { parseGlobalFlags, setCliOptions, getCliOptions } from './core/cli-options.ts';
import type { CliOptions } from './core/cli-options.ts';
@@ -262,9 +261,11 @@ async function main() {
await withTimeout(runSearch(engine, subArgs), timeoutMs, label);
}
} finally {
// search stats/tune read the query_cache — a pending cache write must
// drain, not get killed by the deliberate exit (#2084 drain hoist).
await drainThenDisconnect(engine);
// #2084: `search diagnose` runs real hybrid retrieval (arms search-cache
// writes) — route through the shared bounded teardown like every other
// one-shot path. The connect-timeout process.exit(124) above is reviewed
// and intentionally unchanged: no engine exists at that point.
await finishCliTeardown({ engine });
}
return;
}
@@ -354,42 +355,26 @@ async function main() {
// Local engine path (unchanged behavior for local installs).
const engine = await connectEngine();
// v0.41.8.0 (#1247, #1269, #1290): the search / query / get_page
// op handlers fire-and-forget `bumpLastRetrievedAt` after returning
// results. On PGLite that IIFE keeps Bun's event loop alive past
// engine.disconnect(), hanging the CLI at ~95-98% CPU until SIGKILL.
// Drain the fire-and-forget set BEFORE disconnect; force-exit only
// if the drain itself times out (preserves stderr diagnostic signal
// AND guarantees the CLI doesn't re-hang at the disconnect layer).
//
// Defense-in-depth (adversarial-review C13): `engine.disconnect()` itself
// can hang on PGLite (db.close() or releaseLock racing OS-level FS state).
// The unref'd hard-exit fallback is armed inside drainThenDisconnect (called
// from the `finally` below), so it bounds ONLY the teardown phase (drain +
// disconnect) — the same helper every owner-disconnect site uses. It used to
// be armed HERE, before the try, which silently killed any op whose BODY ran
// past the deadline: on a slow Postgres pooler (6-10s per fresh connection)
// a healthy `gbrain search` was force-exited mid-handler with code 0 and
// ZERO stdout — an empty "success" indistinguishable from no results. The
// exitCode honor (v0.42.20.0) can't help there: a mid-op kill fires before
// any error path sets exitCode. Op-body wallclock bounds are the read-scope
// withTimeout wrap inside the try below, not this teardown backstop.
// Daemons (`serve`) are excluded so they stay alive.
// Wallclock bound for READ-scope op handlers. With the hard-deadline timer
// correctly scoped to teardown, a genuinely WEDGED read handler (hung pooler
// connection mid-query) would otherwise hang the CLI forever — the #1633
// zombie class the old (buggy) pre-try timer accidentally bounded at 10s.
// 180s sits far above any healthy slow-pooler run (6-10s/connection);
// --timeout=Ns overrides. Writes/admin stay unbounded: a long import/embed
// must never be killed by a default deadline.
// #2084: the teardown contract (bounded drain of every background-work sink,
// bounded disconnect, computed-deadline backstop) lives in finishCliTeardown
// — see src/core/cli-force-exit.ts for the full design. The hard-deadline
// timer arms at TEARDOWN start inside the helper, never before the handler:
// the pre-#2084 placement here measured handler + teardown combined, so a
// slow-but-healthy query burned the teardown budget (the flat-10s-banner
// bug) and any >10s op was force-killed mid-run with exit 0. The explicit
// process exit happens once, in the import.meta.main seam at the bottom of
// this file — NOT here.
// v0.42.41.0 (merged): wallclock bound for READ-scope op handlers. With the
// teardown backstop correctly scoped to teardown, a genuinely WEDGED read
// handler (hung pooler connection mid-query) would otherwise hang the CLI
// forever — the #1633 zombie class the old pre-try timer accidentally
// bounded at 10s. 180s sits far above any healthy slow-pooler run
// (6-10s/connection); --timeout=Ns overrides. Writes/admin stay unbounded:
// a long import/embed must never be killed by a default deadline. On
// timeout the abandoned handler may hold ref'd sockets — harmless here,
// because the import.meta.main seam exits explicitly on every one-shot path.
const READ_OP_TIMEOUT_MS = 180_000;
// Set when a wallclock bound fired. The abandoned (timed-out but still
// running) handler can hold ref'd sockets/timers that keep Bun's event loop
// alive after main() returns — so the finally must hard-exit after teardown
// on this path, or the timeout print is followed by an immortal process:
// the same zombie class, resurrected through the timeout door (adversarial
// review finding).
let wallclockTimedOut = false;
try {
const { withTimeout, OperationTimeoutError } = await import('./core/timeout.ts');
@@ -399,8 +384,10 @@ async function main() {
? ''
: ` (default ${e.ms}ms; pass --timeout=Ns to override)`;
console.error(`${e.label} timed out${hint}.`);
setCliExitCode(124);
wallclockTimedOut = true;
// 124 = timeout convention (matches the read-only dispatch path). Set
// through the verdict channel — a raw process.exitCode write is invisible
// to the exit seam and PGLite's WASM runtime can scribble over it.
setCliExitVerdict(124);
};
// Context build does DB I/O (resolveSourceId) and runs for EVERY op —
@@ -416,7 +403,7 @@ async function main() {
} catch (e: unknown) {
if (e instanceof OperationTimeoutError) {
onWallclockTimeout(e);
return; // the finally below still drains + disconnects, then exits
return; // the finally drains + disconnects; the import.meta.main seam exits
}
throw e;
}
@@ -432,7 +419,7 @@ async function main() {
} catch (e: unknown) {
if (e instanceof OperationTimeoutError) {
onWallclockTimeout(e);
return; // the finally below still drains + disconnects, then exits
return; // the finally drains + disconnects; the import.meta.main seam exits
}
throw e;
}
@@ -459,76 +446,12 @@ async function main() {
} else {
console.error(e instanceof Error ? e.message : String(e));
}
setCliExitCode(1);
setCliExitVerdict(1);
} finally {
// 1s per-sink drain timeout: read paths with no pending work pay the
// ~0ms fast path; capture/import that DO enqueue pay up to 1s (+ facts
// shutdown grace) while in-flight Haiku finishes.
await drainThenDisconnect(engine, { drainTimeoutMs: 1000 });
// Wallclock-timeout path (master v0.42.41.0): the ABANDONED handler
// (withTimeout races, it does not cancel) can hold ref'd sockets / SDK
// retry timers that keep Bun's event loop alive. The entrypoint
// flush-exit fires when main() resolves and covers this; the explicit
// call here is belt-and-braces in case a future caller invokes this op
// path outside the guarded entrypoint.
if (wallclockTimedOut) {
void flushStdoutThenExit(getCliExitCode());
}
}
}
/**
* v0.43 (#2084, closes the TODOS P3 drain-hoist) — THE owner-disconnect for
* every CLI exit path. One helper, all sites, so a future teardown bug has
* one place to be wrong in.
*
* drainThenDisconnect(engine)
* ├── arm unref'd 10s hard-deadline (HUNG-teardown backstop — PGLite
* │ db.close()/releaseLock can hang on OS-level FS state; the
* │ entrypoint flush-exit can never fire if main() never resolves)
* ├── drain background-work registry (facts → last-retrieved →
* │ search-cache → eval-capture → volunteer-events), so a PGLite
* │ db.close() can't race in-flight work into the re-pump
* │ busy-loop (#1762)
* └── engine.disconnect() (best-effort), then clear the deadline
*
* The 10s timer is armed HERE — around the teardown window only — never
* before the op handler (master's v0.42.41.0 triage wave fixed the same
* pre-armed-timer bug independently: it force-killed any op slower than
* 10s). On the happy path the timer never fires: main() resolves and the
* entrypoint's flushStdoutThenExit ends the process deliberately (#2084's
* fix for lingering embedding/PgBouncer sockets riding the backstop).
*/
const DISCONNECT_HARD_DEADLINE_MS = 10_000;
export async function drainThenDisconnect(
engine: BrainEngine,
opts?: { drainTimeoutMs?: number },
): Promise<void> {
let deadlineTimer: ReturnType<typeof setTimeout> | undefined;
if (shouldForceExitAfterMain()) {
deadlineTimer = setTimeout(() => {
console.warn(
`[cli] engine.disconnect() did not return within ${DISCONNECT_HARD_DEADLINE_MS}ms — force-exiting`,
);
// Honor an exit code an errored op already set — a bare process.exit(0)
// here would mask a failed op as success if the drain/disconnect hangs.
process.exit(getCliExitCode());
}, DISCONNECT_HARD_DEADLINE_MS);
// unref so the timer itself doesn't keep the event loop alive — only
// the actual pending work (PGLite WASM handle) does.
deadlineTimer.unref?.();
}
try {
await drainAllBackgroundWorkForCliExit(
opts?.drainTimeoutMs !== undefined ? { timeoutMs: opts.drainTimeoutMs } : undefined,
);
try {
await engine.disconnect();
} catch {
/* best-effort — kernel reclaims sockets on exit (timeout.ts doctrine) */
}
} finally {
if (deadlineTimer) clearTimeout(deadlineTimer);
// 1s per-sink drain budget: read paths with no pending work pay the ~0ms
// fast path; capture/import that DO enqueue pay up to 1s (+ facts shutdown
// grace) while in-flight Haiku finishes (#1762 drain-before-disconnect).
await finishCliTeardown({ engine, drainTimeoutMs: 1000 });
}
}
@@ -1260,15 +1183,13 @@ async function handleCliOnly(command: string, args: string[]) {
}
if (command === 'friction') {
const { runFriction } = await import('./commands/friction.ts');
// v0.43 (#2084 inner-exit sweep): exitCode + return instead of a
// mid-handler process.exit — flows through the entrypoint flush-exit
// so buffered stdout is never truncated.
setCliExitCode(runFriction(args));
// #2084 inner-exit sweep: verdict + return so teardown + the flush seam run.
setCliExitVerdict(runFriction(args));
return;
}
if (command === 'claw-test') {
const { runClawTest } = await import('./commands/claw-test.ts');
setCliExitCode(await runClawTest(args));
setCliExitVerdict(await runClawTest(args));
return;
}
if (command === 'report') {
@@ -1315,13 +1236,13 @@ async function handleCliOnly(command: string, args: string[]) {
if (args.includes('--remediation-plan')) {
const { runRemediationPlan } = await import('./commands/doctor.ts');
const eng = await connectEngine();
try { await runRemediationPlan(eng, args); } finally { await drainThenDisconnect(eng); }
try { await runRemediationPlan(eng, args); } finally { await finishCliTeardown({ engine: eng }); }
return;
}
if (args.includes('--remediate')) {
const { runRemediate } = await import('./commands/doctor.ts');
const eng = await connectEngine();
try { await runRemediate(eng, args); } finally { await drainThenDisconnect(eng); }
try { await runRemediate(eng, args); } finally { await finishCliTeardown({ engine: eng }); }
return;
}
@@ -1334,13 +1255,21 @@ async function handleCliOnly(command: string, args: string[]) {
// "user chose --fast while config is present".
await runDoctor(null, args, getDbUrlSource());
} else {
// #2084: both failure kinds (connect throw, runDoctor(eng) throw) still
// fall back to filesystem-only checks — identical to the prior shape.
// The finally closes the gap where a runDoctor(eng) throw used to skip
// the in-try disconnect. NOTE: runDoctor normally calls process.exit
// itself, which preempts this finally — in-command exit sites bypassing
// teardown are a pre-existing class, tracked as a TODOS.md follow-up.
let eng: BrainEngine | null = null;
try {
const eng = await connectEngine();
eng = await connectEngine();
await runDoctor(eng, args);
await drainThenDisconnect(eng);
} catch {
// DB unavailable — still run filesystem checks
await runDoctor(null, args, getDbUrlSource());
} finally {
if (eng) await finishCliTeardown({ engine: eng });
}
}
return;
@@ -1354,7 +1283,7 @@ async function handleCliOnly(command: string, args: string[]) {
try {
await runZeSwitch(args, eng);
} finally {
await drainThenDisconnect(eng);
await finishCliTeardown({ engine: eng });
}
return;
}
@@ -1370,7 +1299,7 @@ async function handleCliOnly(command: string, args: string[]) {
execSync(`bash "${scriptPath}"`, { stdio: 'inherit', env: { ...process.env } });
} catch (e: any) {
// Non-zero exit = some tests failed (exit code = failure count)
setCliExitCode(e.status ?? 1);
setCliExitVerdict(e.status ?? 1);
}
return;
}
@@ -1399,14 +1328,15 @@ async function handleCliOnly(command: string, args: string[]) {
await runDream(eng, args);
} finally {
// #1471 invariant tripwire (the dream-cycle owner): `eng` created the
// module singleton (first module connector) and is disconnected LAST,
// module singleton (first module connector) and is torn down LAST,
// here, after the whole cycle. The ownership fix relies on this owner's
// lifetime strictly dominating every borrower (lint/doctor probe engines
// created mid-cycle). Do NOT disconnect `eng` before runDream returns, or
// created mid-cycle). Do NOT tear down `eng` before runDream returns, or
// a borrower could outlive the owner and lose the shared singleton.
// #2084 drain hoist: dream's cycle enqueues facts/last-retrieved
// writes — drain them against the live owner engine before teardown.
if (eng) await drainThenDisconnect(eng);
// #2084: routed through the shared bounded teardown — dream runs as an
// overnight cron, where a lingering-socket hang is a silent zombie
// (closes the TODOS.md drain-before-owner-disconnect item).
if (eng) await finishCliTeardown({ engine: eng });
}
return;
}
@@ -1418,7 +1348,7 @@ async function handleCliOnly(command: string, args: string[]) {
// The handler self-configures the AI gateway from loadConfig() + process.env.
if (command === 'eval' && args[0] === 'cross-modal') {
const { runEvalCrossModal } = await import('./commands/eval-cross-modal.ts');
setCliExitCode(await runEvalCrossModal(args.slice(1)));
setCliExitVerdict(await runEvalCrossModal(args.slice(1)));
return;
}
@@ -1430,7 +1360,7 @@ async function handleCliOnly(command: string, args: string[]) {
// engine-required path below.
if (command === 'eval' && args[0] === 'takes-quality' && args[1] === 'replay') {
const { runReplayNoBrain } = await import('./commands/eval-takes-quality.ts');
setCliExitCode(await runReplayNoBrain(args.slice(2)));
setCliExitVerdict(await runReplayNoBrain(args.slice(2)));
return;
}
@@ -1463,7 +1393,7 @@ async function handleCliOnly(command: string, args: string[]) {
// gate runs on machines with no `~/.gbrain/config.json`.
if (command === 'eval' && args[0] === 'conversation-parser') {
const { runEvalConversationParser } = await import('./commands/eval-conversation-parser.ts');
setCliExitCode(await runEvalConversationParser(args.slice(1)));
setCliExitVerdict(await runEvalConversationParser(args.slice(1)));
return;
}
@@ -1492,7 +1422,7 @@ async function handleCliOnly(command: string, args: string[]) {
const cfgPre = loadConfig();
if (isThinClient(cfgPre)) {
const { runEvalWhoknows } = await import('./commands/eval-whoknows.ts');
setCliExitCode(await runEvalWhoknows(null, args.slice(1)));
setCliExitVerdict(await runEvalWhoknows(null, args.slice(1)));
return;
}
}
@@ -1507,7 +1437,7 @@ async function handleCliOnly(command: string, args: string[]) {
if (cfgPre && isThinClient(cfgPre)) {
const { runStatus } = await import('./commands/status.ts');
const result = await runStatus(null, args);
setCliExitCode(result.exitCode);
setCliExitVerdict(result.exitCode);
return;
}
}
@@ -1587,9 +1517,7 @@ async function handleCliOnly(command: string, args: string[]) {
}
throw e;
} finally {
// #2084 drain hoist: `gbrain search` writes the query cache + bumps
// last_retrieved_at fire-and-forget — drain before disconnect.
await drainThenDisconnect(engine);
await finishCliTeardown({ engine });
}
return;
}
@@ -1641,7 +1569,7 @@ async function handleCliOnly(command: string, args: string[]) {
// so wrappers (sync, CI scripts, `&& gbrain doctor`) propagate.
const importResult = await runImport(engine, args);
if (importResult.errors > 0) {
setCliExitCode(1);
setCliExitVerdict(1);
}
break;
}
@@ -1823,10 +1751,8 @@ async function handleCliOnly(command: string, args: string[]) {
case 'status': {
const { runStatus } = await import('./commands/status.ts');
const result = await runStatus(engine, args);
// v0.43 (#2084 inner-exit sweep): a mid-switch process.exit skipped
// the finally's drain + disconnect entirely. exitCode + break flows
// through both, then the entrypoint flush-exit ends the process.
setCliExitCode(result.exitCode);
// #2084 inner-exit sweep: a mid-switch exit skips the finally teardown.
setCliExitVerdict(result.exitCode);
break;
}
// v0.38 — Capture: single human-facing entrypoint for ingestion.
@@ -1999,7 +1925,8 @@ async function handleCliOnly(command: string, args: string[]) {
case 'watch': {
// v0.43 (#2095): push-based context transport. Blocks in the stdin
// iteration (interactive stays alive; piped exits at EOF), then the
// finally below drains the volunteer-events sink with everything else.
// finally below runs finishCliTeardown (volunteer events drain with
// every other sink) and the import.meta.main seam flush-exits.
const { runWatch } = await import('./commands/watch.ts');
await runWatch(engine, args);
break;
@@ -2074,20 +2001,16 @@ async function handleCliOnly(command: string, args: string[]) {
}
} finally {
syncWatchdog?.dispose(); // #1633: tear down the hard-deadline watchdog on clean exit
// v0.42.20.0 (#1762) — the CLI_ONLY path (which owns `gbrain capture`)
// lacked the op-dispatch drain-before-disconnect contract. `put_page` fires
// a fire-and-forget facts:absorb job AFTER printing the receipt; on a
// multi-chunk page that job is in flight when this finally tears the engine
// down, and `engine.disconnect()` nulling PGLite's _db mid-job spins
// db.close() into a 100%-CPU busy-loop that pins the single-writer lock.
// Drain every background-work sink first (facts shutdown() abort cancels a
// hung Haiku), THEN disconnect. The drain-before-disconnect is the causal
// fix; the force-exit defense below is secondary (it CANNOT preempt a WASM
// busy-loop on a pinned JS thread — that's exactly why the drain matters).
// #1471: this is also the fall-through OWNER-disconnect — the owner is torn
// down LAST (after the drain), so module-singleton borrowers never outlive it.
// #2084 — the CLI_ONLY fall-through teardown (drain every background-work
// sink, THEN disconnect, under a computed-deadline backstop) lives in
// finishCliTeardown. `gbrain capture`'s fire-and-forget facts:absorb job
// gets its drain window before PGLite's db.close() can race it into the
// re-pump busy-loop (#1762). #1471: this is also the fall-through
// OWNER-disconnect — the owner is torn down LAST (after the drain), so
// module-singleton borrowers never outlive it. `serve` skips teardown
// entirely: the daemon owns its lifecycle.
if (command !== 'serve') {
await drainThenDisconnect(engine);
await finishCliTeardown({ engine });
}
}
}
@@ -2401,21 +2324,25 @@ Run gbrain <command> --help for command-specific help.
// Only auto-run when invoked as the entry point (the compiled binary or
// `bun src/cli.ts`). Guarded so tests can import cliAliases / printOpHelp
// without triggering argv parsing + main(). v114 (#1941).
//
// #2084 — the ONE process-exit seam for one-shot commands. Every teardown site
// routes through finishCliTeardown (which returns); the exit itself happens
// here, after main() settles, so the CLI never waits on Bun's event loop to
// drain (stuck PgBouncer sockets kept it alive — endPoolBounded races PAST a
// stuck pool.end() by design). flushThenExit fences stdout/stderr and holds a
// short aliveness grace so piped output is delivered before exit (#1959).
// Daemons (`serve`) are excluded by shouldForceExitAfterMain and keep the
// pre-#2084 behavior: main() resolves and the server's own work keeps the
// process alive. A fatal error still exits 1 for every command, daemons
// included (matches the prior unconditional process.exit(1) on rejection).
if (import.meta.main) {
// v0.43 (#2084): exit DELIBERATELY when main() resolves. Lingering sockets
// (embedding fetch keep-alive, PgBouncer txn-mode sockets endPoolBounded
// raced past) keep Bun's event loop alive after bounded teardown resolves —
// pre-fix, every `gbrain query` paid a flat 10s tax riding the hard-deadline
// backstop. flushStdoutThenExit drains stdout/stderr first (incident #1959:
// a force-exit truncated piped stdout). Daemons (serve; interactive watch)
// are excluded by shouldForceExitAfterMain and keep the event loop.
main().then(
() => {
if (shouldForceExitAfterMain()) void flushStdoutThenExit(getCliExitCode());
if (shouldForceExitAfterMain()) flushThenExit(currentExitCode());
},
(e) => {
console.error(e instanceof Error ? (e.message || String(e)) : String(e));
void flushStdoutThenExit(1);
console.error(e.message || e);
flushThenExit(1);
},
);
}
+3 -3
View File
@@ -18,6 +18,7 @@
*/
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync } from 'fs';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { join } from 'path';
import { execSync } from 'child_process';
import type { BrainEngine } from '../core/engine.ts';
@@ -37,7 +38,6 @@ import { logSelfUpgrade } from '../core/audit/self-upgrade-audit.ts';
import { detectInstallMethod } from './upgrade.ts';
import { evaluateQuietHours } from '../core/minions/quiet-hours.ts';
import { inspectLock } from '../core/db-lock.ts';
import { setCliExitCode } from '../core/cli-force-exit.ts';
/**
* v0.37.7.0 #1162 — classify autopilot reconnect-loop errors.
@@ -548,7 +548,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
`Exiting so launchd ThrottleInterval can apply backoff.`,
);
stopping = true;
setCliExitCode(1);
setCliExitVerdict(1);
break;
}
if (autopilotReconnectFails >= AUTOPILOT_MAX_RECONNECT_FAILS) {
@@ -557,7 +557,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
`Last error: ${(e as Error).message ?? 'unknown'}. Exiting.`,
);
stopping = true;
setCliExitCode(1);
setCliExitVerdict(1);
break;
}
}
+2 -2
View File
@@ -13,6 +13,7 @@
*/
import type { BrainEngine } from '../core/engine.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import {
runBrainstorm,
formatBrainstormMarkdown,
@@ -27,7 +28,6 @@ import { serializeMarkdown } from '../core/markdown.ts';
import { importFromContent } from '../core/import-file.ts';
import { writePageThrough, type WriteThroughResult } from '../core/write-through.ts';
import { randomBytes } from 'crypto';
import { setCliExitCode } from '../core/cli-force-exit.ts';
export interface BrainstormCliArgs {
question?: string;
@@ -323,7 +323,7 @@ async function runBrainstormCli(
const msg = formatSaveOutcome(outcome, { profileLabel: profile.label, slug });
if (msg.stdout) console.log(msg.stdout);
for (const line of msg.stderr) console.error(line);
if (msg.exitCode) setCliExitCode(msg.exitCode);
if (msg.exitCode) setCliExitVerdict(msg.exitCode);
}
}
+5 -7
View File
@@ -1,6 +1,6 @@
import type { BrainEngine } from '../core/engine.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import * as db from '../core/db.ts';
import { setCliExitCode } from '../core/cli-force-exit.ts';
import { LATEST_VERSION, getIdleBlockers } from '../core/migrate.ts';
import { checkResolvable } from '../core/check-resolvable.ts';
import { autoFixDryViolations, type AutoFixReport, type FixOutcome } from '../core/dry-fix.ts';
@@ -7226,12 +7226,10 @@ export async function runDoctor(
} catch { /* best-effort */ }
}
// Use the owned exit verdict instead of process.exit() so cleanup handlers
// (e.g. Bun unload events, open database connections) still run before the
// process terminates — and so the deliberate flush-exit (#2084) reports
// doctor's verdict: a RAW process.exitCode write is silently zeroed by
// getCliExitCode() at the entrypoint (the PGLite-pollution defense).
setCliExitCode(hasFail ? 1 : 0);
// Use process.exitCode instead of process.exit() so cleanup handlers
// (e.g. Bun unload events, open database connections) still run before
// the process terminates. process.exit() is a hard kill that bypasses them.
setCliExitVerdict(hasFail ? 1 : 0);
}
// ---------------------------------------------------------------------------
+2 -2
View File
@@ -28,8 +28,8 @@
* pagination). FS-source preserves the original v0.10.1 walker behavior.
*/
import { setCliExitCode } from '../core/cli-force-exit.ts';
import { readFileSync, readdirSync, lstatSync, existsSync } from 'fs';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { join, relative, dirname } from 'path';
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/engine.ts';
import type { PageType } from '../core/types.ts';
@@ -865,7 +865,7 @@ Status (v0.42):
(r.first_batch_error ? ` (first error: ${r.first_batch_error})` : '') +
` — timeline is incomplete.`,
);
setCliExitCode(1);
setCliExitVerdict(1);
}
} else if (byMention || ner) {
// v0.41.18.0 (T7): combined --by-mention + --ner walk shares one
+6 -6
View File
@@ -16,6 +16,7 @@
*/
import { readFileSync, writeFileSync, existsSync, lstatSync, readdirSync } from 'fs';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { join, relative, resolve } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { loadConfig, toEngineConfig } from '../core/config.ts';
@@ -30,7 +31,6 @@ import {
type AuditFix,
} from '../core/brain-writer.ts';
import { isSyncable, pruneDir, slugifyPath } from '../core/sync.ts';
import { setCliExitCode } from '../core/cli-force-exit.ts';
export async function runFrontmatter(args: string[]): Promise<void> {
const sub = args[0];
@@ -64,7 +64,7 @@ export async function runFrontmatter(args: string[]): Promise<void> {
}
console.error(`Unknown frontmatter subcommand: ${sub}\n`);
printHelp();
setCliExitCode(1);
setCliExitVerdict(1);
}
async function connectEngineForAudit(): Promise<BrainEngine> {
@@ -165,14 +165,14 @@ async function runValidate(rest: string[]): Promise<void> {
}
if (!target) {
console.error('error: gbrain frontmatter validate requires a <path> argument');
setCliExitCode(1);
setCliExitVerdict(1);
return;
}
const resolved = resolve(target);
if (!existsSync(resolved)) {
console.error(`error: path not found: ${target}`);
setCliExitCode(1);
setCliExitVerdict(1);
return;
}
@@ -243,7 +243,7 @@ async function runValidate(rest: string[]): Promise<void> {
}
}
setCliExitCode(totalErrors > 0 && !flags.fix ? 1 : 0);
setCliExitVerdict(totalErrors > 0 && !flags.fix ? 1 : 0);
}
/**
@@ -379,7 +379,7 @@ async function runGenerate(args: string[]): Promise<void> {
if (!targetPath) {
console.error('error: gbrain frontmatter generate requires a <path> argument');
console.error('usage: gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]');
setCliExitCode(1);
setCliExitVerdict(1);
return;
}
+2 -2
View File
@@ -22,6 +22,7 @@
*/
import type { BrainEngine } from '../core/engine.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
import { MARKDOWN_CHUNKER_VERSION } from '../core/chunkers/recursive.ts';
import { importFromContent, importFromFile } from '../core/import-file.ts';
import { serializeMarkdown } from '../core/markdown.ts';
@@ -29,7 +30,6 @@ import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
import { existsSync } from 'fs';
import { resolve } from 'path';
import { setCliExitCode } from '../core/cli-force-exit.ts';
// v0.41.15.0 (T10, D9): per-batch parallel workers.
import { runSlidingPool } from '../core/worker-pool.ts';
import { resolveWorkersWithClamp } from '../core/sync-concurrency.ts';
@@ -151,7 +151,7 @@ export async function runReindex(engine: BrainEngine, args: string[]): Promise<R
} else {
process.stderr.write('Usage: gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--repo PATH]\n');
}
setCliExitCode(2);
setCliExitVerdict(2);
return { pending: 0, reindexed: 0, skipped: 0, failed: 0, dryRun: !!opts.dryRun, chunkerVersion: MARKDOWN_CHUNKER_VERSION };
}
+2 -2
View File
@@ -14,7 +14,7 @@
*/
import type { BrainEngine } from '../core/engine.ts';
import { setCliExitCode } from '../core/cli-force-exit.ts';
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
interface RunOpts {
days?: number;
@@ -64,7 +64,7 @@ export async function runTranscripts(engine: BrainEngine, args: string[]): Promi
const sub = args[0];
if (sub !== 'recent') {
console.log(HELP);
if (sub && sub !== '--help' && sub !== '-h') setCliExitCode(2);
if (sub && sub !== '--help' && sub !== '-h') setCliExitVerdict(2);
return;
}
+1 -1
View File
@@ -10,7 +10,7 @@
* Lifecycle: BLOCKS in the stdin iteration (like `gbrain jobs work`), so an
* interactive TTY session stays alive until Ctrl-C / Ctrl-D and piped input
* exits at EOF. Either way the handler RETURNS, the CLI_ONLY finally runs
* drainThenDisconnect (volunteer events bank before teardown), and the
* finishCliTeardown (volunteer events bank before teardown), and the
* entrypoint flush-exit ends the process deliberately — which is exactly why
* `watch` is NOT in DAEMON_COMMANDS: it never returns from main() while work
* is still running. SIGINT closes the stream and flows through the same
+10
View File
@@ -64,6 +64,16 @@ export function registerBackgroundWorkDrainer(d: BackgroundWorkDrainer): void {
drainers.set(d.name, d);
}
/**
* Number of registered sinks. Used by `finishCliTeardown` (cli-force-exit.ts)
* to COMPUTE its backstop deadline from the bounds it guards — a 5th sink
* registering automatically widens the deadline instead of silently making
* the worst-case bounded drain exceed a static number (#2084 eng-review D9).
*/
export function backgroundWorkSinkCount(): number {
return drainers.size;
}
/**
* Test seam — registers a drainer and returns an unregister handle. Preferred
* over a blunt reset: real sink modules register at import time and won't re-run
+302 -101
View File
@@ -1,131 +1,332 @@
/**
* v0.41.8.0 — narrow force-exit gate for the cli.ts op-dispatch finally.
* One-shot CLI exit + teardown contract (#2084, supersedes the narrower
* v0.41.8.0 drain-timeout-only force-exit).
*
* The cli.ts caller fires `process.exit(0)` ONLY when:
* 1. The op-dispatch drain timed out (drainResult.outcome === 'timeout')
* 2. AND this function returns true (i.e. the command is NOT a daemon)
* The CLI must never rely on Bun's event loop draining to exit: on PgBouncer
* transaction-mode, `endPoolBounded` (db.ts) deliberately races PAST a stuck
* `pool.end()`, so the promise resolves while the stuck sockets stay open and
* keep the loop alive (#2084's flat 10s teardown tax). Per the doctrine in
* timeout.ts, `process.exit` is the real resource-release mechanism for
* one-shot commands — the kernel reclaims sockets.
*
* The function lives in its own module — not inline in cli.ts — so tests
* can import + drive it without triggering cli.ts's top-level main() side
* effect (cli.ts is a script entrypoint). Mirrors PR #1337's
* `shouldForceExitAfterMain` guard, but narrower in scope: this wave
* only force-exits after the drain timed out, NOT unconditionally for
* every non-serve command.
* The contract is a PAIR (documented together in KEY_FILES.md):
*
* Daemon list is just `serve` (stdio + HTTP): it RETURNS from its handler
* while the event loop carries the server. Every other long-runner —
* `jobs work`, `autopilot`, and v0.43's `gbrain watch` (#2095) — BLOCKS
* inside its awaited handler until done (watch blocks in the stdin
* iteration: interactive stays alive until Ctrl-C/Ctrl-D, piped input ends
* at EOF), so when main() resolves the work is over and the deliberate
* flush-exit is correct. Add a command here ONLY if it returns early and
* leaves the event loop holding the daemon.
* op handler returns / throws (catch sets the verdict: setCliExitVerdict(1))
*
* ▼ (per call site, in its finally — nine sites in cli.ts)
* finishCliTeardown({ engine, drainTimeoutMs? }) ← teardown ONLY, never exits*
*
* ├─ arm ref'd backstop timer; deadline COMPUTED from the bounds
* │ it guards (sinks × drainTimeoutMs + facts-abort grace
* │ + 2 × pool-end bound + slack, floor 10s). The backstop fires
* │ ONLY if a component violated its own bound; on fire it prints
* │ a truthful banner and *flushThenExit(currentExitCode()).
* │ GBRAIN_TEARDOWN_DEADLINE_MS overrides (incident escape hatch).
* ▼
* drain background sinks (bounded per-sink; CLI-exit-only contract)
* ▼
* engine.disconnect() — a throw is warned + swallowed: the exit code
* │ reports the OPERATION, not the cleanup
* ▼
* clear backstop, RETURN to caller
* │
* ▼ (exactly ONE place: cli.ts import.meta.main main().then/catch)
* shouldForceExitAfterMain() && flushThenExit(currentExitCode())
* — fence stdout+stderr (write-fence raced with an unref'd guard,
* EPIPE-safe), hold a short REF'D aliveness grace for non-TTY stdio
* (Bun only delivers queued pipe writes while alive), then
* process.exit. Stuck sockets become irrelevant.
*
* The hard-deadline timer is armed at TEARDOWN start, never before the op
* handler — a slow-but-healthy handler must not erode the teardown budget
* (the pre-#2084 bug force-killed any >10s op mid-run with exit 0 and
* truncated output).
*
* Daemons: `serve` is excluded at both layers — its command never reaches a
* finishCliTeardown call site, and the central exit is gated by
* `shouldForceExitAfterMain`. The helper itself has NO daemon flag: the drain
* it runs is CLI-exit-only (it can permanently shut down process-level sinks),
* so a long-lived process must simply never call it.
*
* This module stays importable without cli.ts side effects so tests can drive
* every path directly (cli.ts is a script entrypoint).
*/
import { drainAllBackgroundWorkForCliExit, backgroundWorkSinkCount } from './background-work.ts';
import { POOL_END_TIMEOUT_SECONDS } from './db.ts';
import { parseGlobalFlags } from './cli-options.ts';
const DAEMON_COMMANDS: ReadonlySet<string> = new Set(['serve']);
export function shouldForceExitAfterMain(
argv: string[] = process.argv.slice(2),
): boolean {
const command = argv.find((arg) => !arg.startsWith('-'));
// Resolve the command the same way main() does — parseGlobalFlags strips
// global flags INCLUDING space-separated values (`--timeout 30s`), so the
// command here always matches the dispatched one. The old first-non-dash
// heuristic saw `30s` as the command for `gbrain --timeout 30s serve` and
// (post-#2084, where this gates an unconditional process.exit) would have
// killed the daemon ~250ms after boot. Cross-model adversarial finding.
let command: string | undefined;
try {
command = parseGlobalFlags(argv).rest[0];
} catch {
command = argv.find((arg) => !arg.startsWith('-'));
}
if (!command) return true;
return !DAEMON_COMMANDS.has(command);
}
/** Floor for the computed backstop deadline (the historical hard deadline). */
export const TEARDOWN_DEADLINE_FLOOR_MS = 10_000;
/** Allowance for the facts sink's awaited abort() (shutdown of an in-flight job). */
const FACTS_ABORT_GRACE_MS = 2_000;
/** Headroom over the sum of the guarded bounds so timer jitter can't false-fire. */
const TEARDOWN_SLACK_MS = 2_000;
/** Max wait for the stdio flush fence before exiting anyway (blocked pipe). */
const FLUSH_GUARD_MS = 2_000;
/**
* v0.43 (#2084) — gbrain owns its exit verdict.
*
* `process.exitCode` is NOT trustworthy in this process: PGLite's Emscripten
* runtime writes the WASM backend's proc_exit status into it (initdb at
* create-time, the postmaster at close-time — see `exitCode=status` in
* pglite's dist), and those writes land asynchronously outside any
* snapshot/restore window. Pre-#2084 the success path never read
* process.exitCode so the pollution was invisible; the deliberate flush-exit
* MUST NOT propagate it (a clean `gbrain apply-migrations` was exiting 99).
*
* So gbrain records its own verdict here: every gbrain-owned exit-code
* assignment routes through `setCliExitCode()` (which also mirrors to
* process.exitCode for anything else that reads it), and the exit paths read
* `getCliExitCode()` — never ambient process.exitCode.
* Aliveness grace between the fence and process.exit when stdio is NOT a TTY.
* Empirically verified (#2084 probes): Bun's process.stdout queues pipe writes
* in a native writer that only pushes to the fd on event-loop turns WHILE THE
* PROCESS IS ALIVE — process.exit discards the queue, natural event-loop exit
* discards it too, and no API reaches it (write callbacks fire on accept, not
* delivery; writableLength/bytesWritten read 0 throughout;
* Bun.stdout.writer().flush() is a different writer; fs.writeSync(1) is also
* queued). Staying alive briefly is the ONLY flush. TTY writes are synchronous
* — no grace needed there.
*/
let _cliExitCode: number | undefined;
const FLUSH_GRACE_PIPE_MS = 250;
export function setCliExitCode(code: number): void {
_cliExitCode = code;
process.exitCode = code;
/**
* Resolve the non-TTY aliveness grace: `GBRAIN_FLUSH_GRACE_MS` env override
* (incident/batch escape hatch, same env-only pattern as
* GBRAIN_TEARDOWN_DEADLINE_MS) over the 250ms default. Consumers piping LARGE
* payloads into slow readers (a reader that attaches later than the grace
* loses the tail — Bun gives no delivery signal to wait on) can raise it;
* high-frequency agent loops capturing to files can lower it.
*/
function resolveFlushGraceMs(): number {
const env = Number(process.env.GBRAIN_FLUSH_GRACE_MS);
if (Number.isFinite(env) && env >= 0) return env;
return FLUSH_GRACE_PIPE_MS;
}
/** Default per-sink drain budget (matches drainAllBackgroundWorkForCliExit). */
const DEFAULT_DRAIN_TIMEOUT_MS = 2_000;
export function getCliExitCode(): number {
return _cliExitCode ?? 0;
}
/** Test seam — reset the recorded verdict between cases. */
export function _resetCliExitCodeForTests(): void {
_cliExitCode = undefined;
/**
* Backstop deadline for drain + disconnect COMBINED, computed from the bounds
* it guards so it fires only when a component violated its own bound (#2084
* eng-review D9 — a static 10s fired on healthy-but-slow bounded teardown:
* 4 sinks × 2s + facts grace + 2 × ~2.5s pool ends ≈ 13s).
* `GBRAIN_TEARDOWN_DEADLINE_MS` overrides the formula (incident escape hatch,
* same env-only pattern as the GBRAIN_SYNC_* knobs).
*/
export function computeTeardownDeadlineMs(opts: {
sinkCount: number;
drainTimeoutMs: number;
}): number {
const env = Number(process.env.GBRAIN_TEARDOWN_DEADLINE_MS);
if (Number.isFinite(env) && env > 0) return env;
// +500 mirrors endPoolBounded's slack over the postgres.js hint (db.ts);
// ×2 budgets the worst case of two sequential pool ends (direct + read).
const poolEndBoundMs = POOL_END_TIMEOUT_SECONDS * 1000 + 500;
const computed =
opts.sinkCount * opts.drainTimeoutMs +
FACTS_ABORT_GRACE_MS +
2 * poolEndBoundMs +
TEARDOWN_SLACK_MS;
return Math.max(TEARDOWN_DEADLINE_FLOOR_MS, computed);
}
/**
* v0.43 (#2084) — deliberate exit after bounded teardown.
*
* Lingering sockets (embedding-provider fetch keep-alive, PgBouncer txn-mode
* sockets `endPoolBounded` raced past) keep Bun's event loop alive after
* teardown RESOLVES, so the CLI used to ride the 10s hard-deadline backstop
* on every `gbrain query`. The fix is to exit on purpose the moment main()
* resolves — but only after stdout/stderr have actually drained: incident
* #1959 (see src/core/db.ts) was a force-exit truncating piped stdout
* mid-payload.
*
* Flush contract: a stream is drained when `writableLength === 0` — bytes
* queued in the JS-side buffer are the only ones `process.exit()` can lose
* (the kernel owns anything already written to the fd). `writableNeedDrain`
* is NOT sufficient (it only says "below high-water mark"). We wake on
* 'drain' when it fires, and poll on a short tick because 'drain' is only
* emitted after a write() returned false — a small queued chunk can flush
* without ever signalling. A blocked pipe (reader stopped consuming) is
* bounded by `guardMs`: partial output to a stalled reader is unavoidable,
* hanging the process is not.
* Minimal writable surface for the flush fence — process.stdout/stderr satisfy
* it; tests inject fakes.
*/
export interface FlushableStream {
writableLength?: number;
once(event: 'drain', listener: () => void): unknown;
off?(event: 'drain', listener: () => void): unknown;
removeListener?(event: 'drain', listener: () => void): unknown;
export interface MinimalWritable {
write(chunk: string, cb?: (err?: Error | null) => void): boolean;
once?(event: string, listener: (...args: unknown[]) => void): unknown;
}
export async function flushStdoutThenExit(
code: number,
deps?: {
streams?: FlushableStream[];
exit?: (code: number) => void;
guardMs?: number;
},
): Promise<void> {
const streams = deps?.streams ?? [
process.stdout as unknown as FlushableStream,
process.stderr as unknown as FlushableStream,
/**
* #2084 — the CLI's exit verdict lives in a gbrain-OWNED variable, never read
* back from `process.exitCode`. PGLite's Emscripten runtime writes its own
* status into `process.exitCode` at arbitrary points DURING a run (99 at
* create; in-memory brains run initdb whose exit status, e.g. 100, lands on a
* later event-loop turn — after any point-in-time snapshot), so the global is
* unreadable as a verdict channel on PGLite. Writers call `setCliExitVerdict`
* (which mirrors into `process.exitCode` for anything external that reads the
* global); the exit seam reads `currentExitCode()`, which trusts only the
* owned variable. No verdict set ⇒ 0.
*/
let cliVerdict: number | null = null;
export function setCliExitVerdict(code: number): void {
cliVerdict = code;
process.exitCode = code; // best-effort mirror; never read back
}
export function currentExitCode(): number {
return cliVerdict ?? 0;
}
/** Test seam — clears the verdict so each test starts clean. */
export function _resetCliExitVerdictForTests(): void {
cliVerdict = null;
}
export interface FlushThenExitOpts {
exit?: (code: number) => void;
stdout?: MinimalWritable;
stderr?: MinimalWritable;
guardMs?: number;
/**
* Aliveness window between the fence and exit. Default: 0 when BOTH stdio
* streams are TTYs (synchronous writes), FLUSH_GRACE_PIPE_MS otherwise.
* The grace timer is deliberately ref'd — keeping the loop alive is the
* only thing that delivers Bun's queued pipe writes (see module constant).
*/
graceMs?: number;
}
/**
* Flush stdout + stderr, then exit with `code` — exactly once.
*
* Two stages, both bounded:
* 1. Fence: an empty `write('', cb)` per stream serializes behind the accept
* queue; an unref'd guard bounds a stream whose callback never fires.
* (In Bun the callback fires on ACCEPT, not delivery — the fence alone is
* NOT sufficient; verified in the #2084 probes.)
* 2. Aliveness grace: a REF'D timer keeps the process alive `graceMs` so
* Bun's native writer can push the queued bytes to the fd / a consuming
* reader (#1959 truncation class). TTY stdio skips this (sync writes).
*
* A reader that consumes nothing for longer than guard+grace loses the tail —
* unavoidable without waiting forever; strictly better than the pre-#2084
* behavior (immediate process.exit discarded everything still queued).
*
* `process.exitCode` is set up front so that even a stubbed `exit` (tests) or
* a natural event-loop exit keeps the right code.
*/
/** Process-level guard: the REAL process.exit fires at most once even if both
* the backstop and the central seam reach flushThenExit (test-injected exit
* fns are exempt so unit tests stay independent). */
let realExitInitiated = false;
export function flushThenExit(code: number, opts: FlushThenExitOpts = {}): void {
if (!opts.exit) {
if (realExitInitiated) return;
realExitInitiated = true;
}
const exit = opts.exit ?? ((c: number) => process.exit(c));
const streams: MinimalWritable[] = [
opts.stdout ?? process.stdout,
opts.stderr ?? process.stderr,
];
const exit = deps?.exit ?? ((c: number) => process.exit(c));
const guardMs = deps?.guardMs ?? 2000;
const deadline = Date.now() + guardMs;
for (const stream of streams) {
while ((stream.writableLength ?? 0) > 0 && Date.now() < deadline) {
await new Promise<void>((resolve) => {
const onDrain = () => {
clearTimeout(tick);
resolve();
};
// Poll tick: 'drain' only fires after a backpressured write, so a
// buffer that empties without one needs the re-check. unref'd so the
// wait itself never holds the loop open.
const tick = setTimeout(() => {
(stream.off ?? stream.removeListener)?.call(stream, 'drain', onDrain);
resolve();
}, 25);
(tick as { unref?: () => void }).unref?.();
stream.once('drain', onDrain);
});
const guardMs = opts.guardMs ?? FLUSH_GUARD_MS;
const bothTty = streams.every((s) => (s as { isTTY?: boolean }).isTTY === true);
const graceMs = opts.graceMs ?? (bothTty ? 0 : resolveFlushGraceMs());
process.exitCode = code;
let fenced = false;
let guard: ReturnType<typeof setTimeout> | undefined;
const finish = () => {
if (fenced) return;
fenced = true;
if (guard) clearTimeout(guard);
if (graceMs <= 0) {
exit(code);
return;
}
// Ref'd on purpose: aliveness IS the flush (Bun pipe-write semantics).
setTimeout(() => exit(code), graceMs);
};
let pending = streams.length;
const done = () => {
pending -= 1;
if (pending <= 0) finish();
};
guard = setTimeout(finish, guardMs);
guard.unref?.();
for (const s of streams) {
try {
// EPIPE on a closed pipe surfaces as an async 'error' event; swallow it —
// the guard or the other stream's callback still drives the exit.
s.once?.('error', () => {});
s.write('', () => done());
} catch {
done(); // sync EPIPE / destroyed stream
}
}
exit(code);
}
export interface FinishCliTeardownOpts {
/** Engine to disconnect. A disconnect throw is warned + swallowed (D3). */
engine: { disconnect(): Promise<void> };
/** Per-sink drain budget. Default 2000 (the registry default). */
drainTimeoutMs?: number;
/** Test seam — wins over the env override and the computed formula. */
deadlineMs?: number;
/** Forwarded to flushThenExit on the backstop path (test seam). */
graceMs?: number;
// ---- test seams (default to the real thing) ----
exit?: (code: number) => void;
warn?: (msg: string) => void;
drain?: (opts: { timeoutMs: number }) => Promise<void>;
stdout?: MinimalWritable;
stderr?: MinimalWritable;
}
/**
* CLI-EXIT-ONLY teardown: bounded drain of every background-work sink, then
* bounded engine disconnect, under a computed-deadline backstop. Returns to
* the caller — the explicit process exit happens once, in cli.ts's
* import.meta.main seam (see module header). The backstop timer is the ONLY
* exit in here, and it means a component violated its own bound.
*/
export async function finishCliTeardown(opts: FinishCliTeardownOpts): Promise<void> {
const drainTimeoutMs = opts.drainTimeoutMs ?? DEFAULT_DRAIN_TIMEOUT_MS;
const warn = opts.warn ?? ((m: string) => console.warn(m));
const drain = opts.drain ?? drainAllBackgroundWorkForCliExit;
const deadlineMs =
opts.deadlineMs ??
computeTeardownDeadlineMs({ sinkCount: backgroundWorkSinkCount(), drainTimeoutMs });
const backstop = setTimeout(() => {
warn(
`[cli] teardown (background-work drain + engine.disconnect()) did not return within ${deadlineMs}ms — force-exiting`,
);
// currentExitCode() reads the gbrain-owned verdict channel — an errored
// op's setCliExitVerdict(1) is honored even when PGLite has scribbled over
// process.exitCode; a bare exit(0) would mask the failure.
flushThenExit(currentExitCode(), opts);
}, deadlineMs);
// Deliberately REF'D (adversarial F3): if teardown hangs while nothing else
// keeps Bun's loop alive, an unref'd timer would let the process exit
// NATURALLY — skipping the flush and exiting with whatever PGLite scribbled
// into process.exitCode. The ref'd timer costs nothing on the clean path
// (cleared in the finally as soon as teardown returns).
try {
try {
await drain({ timeoutMs: drainTimeoutMs });
} catch (e) {
// The registry is contractually non-throwing, but a throw here must not
// skip the disconnect or escape a caller's finally (it would replace a
// successful op's completion). Same D3 posture as the disconnect guard.
warn(
`[cli] background-work drain failed during teardown: ${e instanceof Error ? e.message : String(e)} — continuing to disconnect`,
);
}
try {
await opts.engine.disconnect();
} catch (e) {
// D3: the exit code reports the operation, not the cleanup. Matches the
// non-throwing posture of endPoolBounded (db.ts).
warn(
`[cli] engine.disconnect() failed during teardown: ${e instanceof Error ? e.message : String(e)} — continuing to exit`,
);
}
} finally {
clearTimeout(backstop);
}
}
+1 -1
View File
@@ -73,7 +73,7 @@ export async function insertVolunteerEvents(
// ── Fire-and-forget sink (eng-review D4) ─────────────────────────────────
// Mirrors src/core/last-retrieved.ts: track every dangling INSERT promise in
// a module Set, register a drainer so drainThenDisconnect settles them
// a module Set, register a drainer so finishCliTeardown settles them
// against a live engine before teardown on EVERY CLI exit path (the commit-1
// drain hoist). Logging failure never fails the caller.
+40 -6
View File
@@ -195,6 +195,32 @@ export function buildPgliteInitErrorMessage(
return `${header}\n${hint}\n Original error: ${original}`;
}
/**
* #2084 — PGLite's Emscripten runtime hijacks `process.exitCode` as ITS status
* channel: instantiation REPLACES the property with an accessor whose getter
* falls back to the WASM runtime status (99 while alive, the exit status after
* close) whenever no explicit value was assigned — and assigning `undefined`
* resets to that fallback, so "unset" cannot be restored. Pre-fix, every clean
* PGLite run carried a bogus 99 until close zeroed it, and an errored op's
* exit 1 survived only by accident of write ordering.
*
* Containment: around PGlite.create(), snapshot the pre-call value and restore
* it — pinning an explicit 0 when nothing was set, because restoring
* `undefined` would surface the WASM fallback instead. This keeps the GLOBAL
* tidy for external readers; the CLI's own verdict never reads it (it lives in
* the owned channel: setCliExitVerdict/currentExitCode, cli-force-exit.ts —
* in-memory brains run initdb whose status lands on a later tick, past any
* snapshot). db.close() stays unwrapped (see the comment at the close site).
*/
async function preservingProcessExitCode<T>(fn: () => Promise<T>): Promise<T> {
const pre = process.exitCode;
try {
return await fn();
} finally {
process.exitCode = typeof pre === 'number' || typeof pre === 'string' ? pre : 0;
}
}
export class PGLiteEngine implements BrainEngine {
readonly kind = 'pglite' as const;
private _db: PGLiteDB | null = null;
@@ -243,13 +269,15 @@ export class PGLiteEngine implements BrainEngine {
// the postmaster at close-time), and the writes land asynchronously —
// a snapshot/restore around these awaits does NOT contain them. That is
// why the CLI's exit paths read gbrain's own verdict
// (cli-force-exit.ts getCliExitCode), never ambient process.exitCode.
// (cli-force-exit.ts currentExitCode), never ambient process.exitCode.
try {
this._db = await PGlite.create({
dataDir,
loadDataDir,
extensions: { vector, pg_trgm },
});
this._db = await preservingProcessExitCode(() =>
PGlite.create({
dataDir,
loadDataDir,
extensions: { vector, pg_trgm },
}),
);
} catch (err) {
// v0.13.1: any PGLite.create() failure becomes actionable. v0.41.8.0
// (#1340): the previous error hint hardcoded the macOS 26.3 link, but
@@ -289,6 +317,12 @@ export class PGLiteEngine implements BrainEngine {
this._lock = null;
try {
if (db) {
// Deliberately NOT wrapped in preservingProcessExitCode: close's
// status write (0) is long-standing baseline behavior that test-runner
// processes depend on (wrapping it flipped bun test's own exit code —
// #2084 implementation note), and the CLI's exit verdict doesn't read
// process.exitCode at all — it lives in the gbrain-owned channel
// (setCliExitVerdict/currentExitCode in cli-force-exit.ts).
await db.close();
}
} finally {
-99
View File
@@ -1,99 +0,0 @@
/**
* v0.43 (#2084 drain hoist, closes TODOS P3) — drainThenDisconnect unit tests.
*
* One helper owns every CLI owner-disconnect: drain the background-work
* registry FIRST, then disconnect (best-effort), bounded by the unref'd
* hard-deadline. Pre-hoist, six bare sites (search dashboard, doctor
* remediation, ze-switch, dream, read-only timeout path) skipped the drain —
* an in-flight fire-and-forget write was killed at disconnect/exit.
*/
import { describe, test, expect } from 'bun:test';
import { drainThenDisconnect } from '../src/cli.ts';
import { __registerDrainerForTest } from '../src/core/background-work.ts';
import type { BrainEngine } from '../src/core/engine.ts';
function makeEngine(events: string[], opts?: { throwOnDisconnect?: boolean }): BrainEngine {
return {
disconnect: async () => {
events.push('disconnect');
if (opts?.throwOnDisconnect) throw new Error('disconnect blew up');
},
} as unknown as BrainEngine;
}
describe('drainThenDisconnect — drain registry, then disconnect, bounded', () => {
test('drains registered sinks BEFORE engine.disconnect()', async () => {
const events: string[] = [];
const unregister = __registerDrainerForTest({
name: 'test-sink-order',
order: 99,
drain: async () => {
events.push('drain');
return { unfinished: 0 };
},
});
try {
await drainThenDisconnect(makeEngine(events));
} finally {
unregister();
}
expect(events.indexOf('drain')).toBeGreaterThanOrEqual(0);
expect(events.indexOf('disconnect')).toBeGreaterThan(events.indexOf('drain'));
});
test('a pending fire-and-forget write survives (drained, not killed)', async () => {
// Simulates the search-stats path: a cache write is in flight when the
// command finishes. The hoisted drain must let it settle before teardown.
const events: string[] = [];
let settled = false;
const pending = new Promise<void>((r) =>
setTimeout(() => {
settled = true;
events.push('write-settled');
r();
}, 30),
);
const unregister = __registerDrainerForTest({
name: 'test-sink-pending-write',
order: 99,
drain: async () => {
await pending;
return { unfinished: 0 };
},
});
try {
await drainThenDisconnect(makeEngine(events));
} finally {
unregister();
}
expect(settled).toBe(true);
expect(events).toEqual(['write-settled', 'disconnect']);
});
test('disconnect failure is swallowed (best-effort; kernel reclaims on exit)', async () => {
const events: string[] = [];
await expect(
drainThenDisconnect(makeEngine(events, { throwOnDisconnect: true })),
).resolves.toBeUndefined();
expect(events).toEqual(['disconnect']);
});
test('honors the per-site drain timeout passthrough', async () => {
const seen: number[] = [];
const unregister = __registerDrainerForTest({
name: 'test-sink-timeout',
order: 99,
drain: async (timeoutMs: number) => {
seen.push(timeoutMs);
return { unfinished: 0 };
},
});
try {
await drainThenDisconnect(makeEngine([]), { drainTimeoutMs: 1000 });
} finally {
unregister();
}
expect(seen).toEqual([1000]);
});
});
+31
View File
@@ -0,0 +1,31 @@
/**
* #2084 class pin — every exit-code write in src/ routes through
* setCliExitVerdict.
*
* A RAW `process.exitCode = N` is silently ZEROED by the deliberate
* flush-exit: currentExitCode() reads only gbrain's owned verdict (the
* PGLite-Emscripten-pollution defense), so a command that bypasses the
* setter reports success on failure. Caught live twice in one day: doctor's
* FAIL path exited 0 after a merge introduced a raw write. Runtime variants
* are pinned in test/cli-finish-teardown.test.ts; this is the structural
* guard that catches the NEXT raw write at review time.
*/
import { describe, test, expect } from 'bun:test';
import { execSync } from 'child_process';
describe('exit-verdict ownership — no raw process.exitCode assignments', () => {
test('every exit-code write in src/ routes through setCliExitVerdict', () => {
// Two legitimate writers are exempt:
// - cli-force-exit.ts: setCliExitVerdict's own mirror-write.
// - pglite-engine.ts preservingProcessExitCode: #2141's containment
// RESTORE around PGlite.create() — it keeps the GLOBAL tidy for
// external readers and is explicitly not a verdict write (the owned
// channel never reads process.exitCode).
const hits = execSync(
`grep -rn "process.exitCode = " src --include='*.ts' | grep -v "core/cli-force-exit.ts" | grep -v "core/pglite-engine.ts" || true`,
{ encoding: 'utf-8', cwd: new URL('..', import.meta.url).pathname },
).trim();
expect(hits).toBe('');
});
});
+510
View File
@@ -0,0 +1,510 @@
/**
* #2084 — unit tests for the one-shot CLI teardown + exit contract in
* src/core/cli-force-exit.ts: finishCliTeardown (teardown-only, computed
* backstop deadline), flushThenExit (write-fence + guard + EPIPE + once-latch),
* and computeTeardownDeadlineMs (formula / floor / env override).
*
* Real short timers, no fake clocks. Every test that touches process.exitCode
* or GBRAIN_TEARDOWN_DEADLINE_MS restores it in a finally so the suite stays
* order-independent.
*/
import { describe, test, expect } from 'bun:test';
import {
finishCliTeardown,
flushThenExit,
computeTeardownDeadlineMs,
TEARDOWN_DEADLINE_FLOOR_MS,
setCliExitVerdict,
currentExitCode,
_resetCliExitVerdictForTests,
type MinimalWritable,
} from '../src/core/cli-force-exit.ts';
import { POOL_END_TIMEOUT_SECONDS } from '../src/core/db.ts';
import {
backgroundWorkSinkCount,
__registerDrainerForTest,
} from '../src/core/background-work.ts';
import { withEnv } from './helpers/with-env.ts';
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
function fakeStream(): MinimalWritable & { writes: string[] } {
const writes: string[] = [];
return {
writes,
write(chunk: string, cb?: (err?: Error | null) => void) {
writes.push(chunk);
if (cb) queueMicrotask(() => cb());
return true;
},
once() {
return this;
},
};
}
describe('computeTeardownDeadlineMs', () => {
test('formula: sinks × drain + facts grace + 2 × pool bound + slack', () => {
const poolEndBoundMs = POOL_END_TIMEOUT_SECONDS * 1000 + 500;
// 4 sinks × 2000 + 2000 + 2×poolEnd + 2000 — the Site B worst case that
// falsified the old static 10s (eng-review D9).
const got = computeTeardownDeadlineMs({ sinkCount: 4, drainTimeoutMs: 2000 });
expect(got).toBe(4 * 2000 + 2000 + 2 * poolEndBoundMs + 2000);
expect(got).toBeGreaterThan(10_000); // the codex-found arithmetic bug, pinned
});
test('floors at TEARDOWN_DEADLINE_FLOOR_MS for small budgets', () => {
const got = computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 });
expect(got).toBe(TEARDOWN_DEADLINE_FLOOR_MS);
});
test('GBRAIN_TEARDOWN_DEADLINE_MS env override wins over the formula', async () => {
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: '1234' }, async () => {
expect(computeTeardownDeadlineMs({ sinkCount: 4, drainTimeoutMs: 2000 })).toBe(1234);
});
});
test('garbage env values fall back to the formula', async () => {
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: 'banana' }, async () => {
expect(
computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 }),
).toBe(TEARDOWN_DEADLINE_FLOOR_MS);
});
});
test('zero and negative env values fall back to the formula (not "fire immediately")', async () => {
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: '0' }, async () => {
expect(computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 })).toBe(
TEARDOWN_DEADLINE_FLOOR_MS,
);
});
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: '-5' }, async () => {
expect(computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 })).toBe(
TEARDOWN_DEADLINE_FLOOR_MS,
);
});
});
test('a newly registered sink widens the computed deadline (D9: formula reads the live registry)', () => {
// Register two sinks and compare between them: in a bare unit-test process
// no production sinks are loaded, so the zero-sink baseline sits below the
// 10s floor and would mask the first sink's delta.
const mkSink = (name: string) =>
__registerDrainerForTest({ name, order: 99, drain: async () => ({ unfinished: 0 }) });
const un1 = mkSink('test-2084-sink-a');
try {
const withOne = computeTeardownDeadlineMs({
sinkCount: backgroundWorkSinkCount(),
drainTimeoutMs: 5000,
});
const un2 = mkSink('test-2084-sink-b');
try {
const withTwo = computeTeardownDeadlineMs({
sinkCount: backgroundWorkSinkCount(),
drainTimeoutMs: 5000,
});
expect(withOne).toBeGreaterThan(TEARDOWN_DEADLINE_FLOOR_MS); // above the floor — delta is visible
expect(withTwo).toBe(withOne + 5000);
} finally {
un2();
}
} finally {
un1();
}
});
});
describe('finishCliTeardown — clean path', () => {
test('drains with the injected budget, disconnects, returns; no exit, no warn', async () => {
const calls: string[] = [];
let drainBudget = -1;
const exits: number[] = [];
const warns: string[] = [];
await finishCliTeardown({
engine: { disconnect: async () => void calls.push('disconnect') },
drainTimeoutMs: 777,
deadlineMs: 250,
drain: async ({ timeoutMs }) => {
drainBudget = timeoutMs;
calls.push('drain');
},
exit: (c) => void exits.push(c),
warn: (m) => void warns.push(m),
stdout: fakeStream(),
stderr: fakeStream(),
});
// Past the 250ms deadline: a leaked backstop would fire here.
await sleep(400);
expect(calls).toEqual(['drain', 'disconnect']);
expect(drainBudget).toBe(777);
expect(exits).toEqual([]);
expect(warns).toEqual([]);
});
test('drain runs BEFORE disconnect (live-engine window for sinks)', async () => {
const order: string[] = [];
await finishCliTeardown({
engine: { disconnect: async () => void order.push('disconnect') },
deadlineMs: 1000,
drain: async () => {
await sleep(20);
order.push('drain');
},
exit: () => {},
warn: () => {},
});
expect(order).toEqual(['drain', 'disconnect']);
});
});
describe('finishCliTeardown — backstop on hung teardown', () => {
test('hung disconnect fires the banner and exits with current exitCode', async () => {
const prevCode = process.exitCode;
try {
_resetCliExitVerdictForTests(); // no verdict set ⇒ currentExitCode() === 0
const exits: number[] = [];
const warns: string[] = [];
let resolveHang!: () => void;
const teardown = finishCliTeardown({
engine: { disconnect: () => new Promise<void>((r) => (resolveHang = r)) },
deadlineMs: 100,
drain: async () => {},
exit: (c) => void exits.push(c),
warn: (m) => void warns.push(m),
stdout: fakeStream(),
stderr: fakeStream(),
graceMs: 0,
});
await sleep(300);
expect(warns.length).toBe(1);
expect(warns[0]).toContain('did not return within');
expect(warns[0]).toContain('100ms');
expect(exits).toEqual([0]);
resolveHang(); // unhang so the promise settles
await teardown;
} finally {
_resetCliExitVerdictForTests();
process.exitCode = prevCode;
}
});
test('backstop honors an exit code the errored op already set', async () => {
const prevCode = process.exitCode;
try {
setCliExitVerdict(1); // what the op-dispatch catch does
const exits: number[] = [];
let resolveHang!: () => void;
const teardown = finishCliTeardown({
engine: { disconnect: () => new Promise<void>((r) => (resolveHang = r)) },
deadlineMs: 100,
drain: async () => {},
exit: (c) => void exits.push(c),
warn: () => {},
stdout: fakeStream(),
stderr: fakeStream(),
graceMs: 0,
});
await sleep(300);
expect(exits).toEqual([1]);
resolveHang();
await teardown;
} finally {
_resetCliExitVerdictForTests();
process.exitCode = prevCode;
}
});
test('hung DRAIN (not just disconnect) also trips the backstop', async () => {
const prevCode = process.exitCode;
try {
_resetCliExitVerdictForTests();
const exits: number[] = [];
const warns: string[] = [];
let resolveHang!: () => void;
const teardown = finishCliTeardown({
engine: { disconnect: async () => {} },
deadlineMs: 100,
drain: () => new Promise<void>((r) => (resolveHang = r)),
exit: (c) => void exits.push(c),
warn: (m) => void warns.push(m),
stdout: fakeStream(),
stderr: fakeStream(),
graceMs: 0,
});
await sleep(300);
expect(warns.length).toBe(1);
expect(exits).toEqual([0]);
resolveHang();
await teardown;
} finally {
process.exitCode = prevCode;
}
});
});
describe('verdict channel — immune to PGLite WASM process.exitCode writes', () => {
test('engine teardown that rewrites process.exitCode does not change the verdict', async () => {
// PGLite's Emscripten runtime writes its own status into process.exitCode
// at arbitrary points (99 at create, initdb status on a later tick for
// in-memory brains, 0 at close) — pre-#2084 this clobbered an errored
// op's exit 1 back to 0 on every PGLite error path. The verdict lives in
// the gbrain-owned channel and never reads the global back.
const prevCode = process.exitCode;
try {
setCliExitVerdict(1); // the op errored
await finishCliTeardown({
engine: {
disconnect: async () => {
process.exitCode = 0; // what PGLite's WASM shutdown does
},
},
deadlineMs: 1000,
drain: async () => {},
exit: () => {},
warn: () => {},
});
expect(currentExitCode()).toBe(1);
} finally {
_resetCliExitVerdictForTests();
process.exitCode = prevCode;
}
});
test('mid-run WASM write (in-memory initdb status) cannot fake a verdict', () => {
_resetCliExitVerdictForTests();
try {
process.exitCode = 100; // what in-memory PGLite's initdb does mid-run
expect(currentExitCode()).toBe(0); // no gbrain verdict was ever set
setCliExitVerdict(2);
expect(currentExitCode()).toBe(2);
// The mirror write exists for EXTERNAL readers of the global.
expect(process.exitCode).toBe(2);
} finally {
_resetCliExitVerdictForTests();
process.exitCode = 0;
}
});
});
describe('finishCliTeardown — disconnect failure (D3: exit code reports the op)', () => {
test('a throwing drain is warned, disconnect still runs, helper resolves', async () => {
// The registry is contractually non-throwing; this pins the defense-in-depth
// guard — a drain rejection must not skip disconnect or escape the caller's
// finally (it would replace a successful op's completion).
const calls: string[] = [];
const warns: string[] = [];
await finishCliTeardown({
engine: { disconnect: async () => void calls.push('disconnect') },
deadlineMs: 1000,
drain: async () => {
throw new Error('sink registry blew up');
},
exit: () => {},
warn: (m) => void warns.push(m),
});
expect(calls).toEqual(['disconnect']);
expect(warns.length).toBe(1);
expect(warns[0]).toContain('sink registry blew up');
});
test('disconnect throw is warned and swallowed; helper resolves', async () => {
const warns: string[] = [];
const exits: number[] = [];
await finishCliTeardown({
engine: {
disconnect: async () => {
throw new Error('pool already dead');
},
},
deadlineMs: 1000,
drain: async () => {},
exit: (c) => void exits.push(c),
warn: (m) => void warns.push(m),
});
expect(warns.length).toBe(1);
expect(warns[0]).toContain('pool already dead');
expect(exits).toEqual([]); // helper never exits on the non-backstop path
});
});
describe('flushThenExit', () => {
test('exits after BOTH stream callbacks fire, exactly once, with the code', async () => {
const prevCode = process.exitCode;
try {
const events: string[] = [];
const exits: number[] = [];
const slowStream = (name: string): MinimalWritable => ({
write(_c: string, cb?: (err?: Error | null) => void) {
setTimeout(() => {
events.push(`${name}-flushed`);
cb?.();
}, 50);
return true;
},
once() {
return this;
},
});
flushThenExit(3, {
exit: (c) => {
events.push('exit');
exits.push(c);
},
stdout: slowStream('stdout'),
stderr: slowStream('stderr'),
guardMs: 2000,
graceMs: 0,
});
await sleep(200);
expect(events).toEqual(['stdout-flushed', 'stderr-flushed', 'exit']);
expect(exits).toEqual([3]);
expect(process.exitCode).toBe(3); // belt-and-braces for natural exit
} finally {
process.exitCode = prevCode;
}
});
test('non-TTY default: exit waits the aliveness grace AFTER the fence', async () => {
const prevCode = process.exitCode;
try {
const exits: number[] = [];
const t0 = Date.now();
let fencedAt = -1;
const stream: MinimalWritable = {
write(_c: string, cb?: (err?: Error | null) => void) {
fencedAt = Date.now() - t0;
if (cb) queueMicrotask(() => cb());
return true;
},
once() {
return this;
},
};
flushThenExit(0, {
exit: (c) => void exits.push(c),
stdout: stream,
stderr: stream,
guardMs: 2000,
graceMs: 120, // fakes are non-TTY; explicit grace keeps the test tight
});
await sleep(60);
expect(exits).toEqual([]); // fence done, still inside the grace window
await sleep(150);
expect(exits).toEqual([0]);
expect(fencedAt).toBeGreaterThanOrEqual(0);
} finally {
process.exitCode = prevCode;
}
});
test('guard fires when a callback never arrives (blocked pipe)', async () => {
const prevCode = process.exitCode;
try {
const exits: number[] = [];
const blockedStream: MinimalWritable = {
write() {
return false; // never calls cb — reader stopped consuming
},
once() {
return this;
},
};
const t0 = Date.now();
flushThenExit(0, {
exit: (c) => void exits.push(c),
stdout: blockedStream,
stderr: blockedStream,
guardMs: 100,
graceMs: 0,
});
await sleep(300);
expect(exits).toEqual([0]);
expect(Date.now() - t0).toBeGreaterThanOrEqual(100);
} finally {
process.exitCode = prevCode;
}
});
test('sync write throw (EPIPE) still exits', async () => {
const prevCode = process.exitCode;
try {
const exits: number[] = [];
const epipeStream: MinimalWritable = {
write() {
throw Object.assign(new Error('write EPIPE'), { code: 'EPIPE' });
},
once() {
return this;
},
};
flushThenExit(0, {
exit: (c) => void exits.push(c),
stdout: epipeStream,
stderr: epipeStream,
guardMs: 2000,
graceMs: 0,
});
await sleep(50);
expect(exits).toEqual([0]);
} finally {
process.exitCode = prevCode;
}
});
test('GBRAIN_FLUSH_GRACE_MS env override is honored (batch/incident knob)', async () => {
const prevCode = process.exitCode;
try {
await withEnv({ GBRAIN_FLUSH_GRACE_MS: '0' }, async () => {
const exits: number[] = [];
flushThenExit(0, {
exit: (c) => void exits.push(c),
stdout: fakeStream(),
stderr: fakeStream(),
guardMs: 2000,
// no graceMs → resolves through the env override (fakes are non-TTY)
});
await sleep(60);
expect(exits).toEqual([0]); // grace 0: exit right after the fence
});
} finally {
process.exitCode = prevCode;
}
});
test('once-latch: guard + late callbacks cannot double-exit', async () => {
const prevCode = process.exitCode;
try {
const exits: number[] = [];
// stdout flushes late (after the guard), stderr never — both race finish().
const lateStream: MinimalWritable = {
write(_c: string, cb?: (err?: Error | null) => void) {
setTimeout(() => cb?.(), 150);
return true;
},
once() {
return this;
},
};
const neverStream: MinimalWritable = {
write() {
return false;
},
once() {
return this;
},
};
flushThenExit(0, {
exit: (c) => void exits.push(c),
stdout: lateStream,
stderr: neverStream,
guardMs: 80,
graceMs: 0,
});
await sleep(400);
expect(exits).toEqual([0]); // exactly one exit despite guard + late cb
} finally {
process.exitCode = prevCode;
}
});
});
-155
View File
@@ -1,155 +0,0 @@
/**
* v0.43 (#2084) — flushStdoutThenExit unit tests.
*
* The deliberate-exit path must never truncate buffered output (incident
* #1959: a force-exit cut piped stdout mid-payload) and must never hang on
* a blocked pipe. Streams + exit are injected so the tests drive both
* properties without killing the test process.
*/
import { describe, test, expect } from 'bun:test';
import { flushStdoutThenExit, type FlushableStream } from '../src/core/cli-force-exit.ts';
/** Minimal fake of the stdout/stderr surface flushStdoutThenExit touches. */
function makeStream(initialLength: number): FlushableStream & {
setLength(n: number): void;
emitDrain(): void;
drainListeners(): number;
} {
let length = initialLength;
const listeners: Array<() => void> = [];
return {
get writableLength() {
return length;
},
once(_event: 'drain', listener: () => void) {
listeners.push(listener);
return this;
},
off(_event: 'drain', listener: () => void) {
const i = listeners.indexOf(listener);
if (i >= 0) listeners.splice(i, 1);
return this;
},
setLength(n: number) {
length = n;
},
emitDrain() {
const current = listeners.splice(0, listeners.length);
for (const l of current) l();
},
drainListeners() {
return listeners.length;
},
};
}
describe('flushStdoutThenExit — deliberate exit after output drains', () => {
test('exits immediately with the given code when both streams are drained', async () => {
const calls: number[] = [];
await flushStdoutThenExit(0, {
streams: [makeStream(0), makeStream(0)],
exit: (c) => calls.push(c),
});
expect(calls).toEqual([0]);
});
test('honors a non-zero exit code (errored op sets process.exitCode=1)', async () => {
const calls: number[] = [];
await flushStdoutThenExit(1, {
streams: [makeStream(0)],
exit: (c) => calls.push(c),
});
expect(calls).toEqual([1]);
});
test('waits for a pending buffer to drain before exiting', async () => {
const stream = makeStream(4096);
const calls: number[] = [];
const done = flushStdoutThenExit(0, {
streams: [stream],
exit: (c) => calls.push(c),
guardMs: 5000,
});
// Not exited while bytes are queued.
await new Promise((r) => setTimeout(r, 10));
expect(calls).toEqual([]);
stream.setLength(0);
stream.emitDrain();
await done;
expect(calls).toEqual([0]);
});
test('drains stdout AND stderr — stderr alone can hold the exit', async () => {
const stdout = makeStream(0);
const stderr = makeStream(2048);
const calls: number[] = [];
const done = flushStdoutThenExit(1, {
streams: [stdout, stderr],
exit: (c) => calls.push(c),
guardMs: 5000,
});
await new Promise((r) => setTimeout(r, 10));
expect(calls).toEqual([]);
stderr.setLength(0);
stderr.emitDrain();
await done;
expect(calls).toEqual([1]);
});
test('poll tick drains without a drain event ("drain" only fires after backpressure)', async () => {
const stream = makeStream(512);
const calls: number[] = [];
const done = flushStdoutThenExit(0, {
streams: [stream],
exit: (c) => calls.push(c),
guardMs: 5000,
});
// Buffer empties WITHOUT an emitted drain event — the 25ms poll must see it.
setTimeout(() => stream.setLength(0), 30);
await done;
expect(calls).toEqual([0]);
});
test('blocked pipe: the guard deadline exits anyway (never hangs)', async () => {
const stream = makeStream(65536); // reader stopped consuming; never drains
const calls: number[] = [];
const t0 = Date.now();
await flushStdoutThenExit(0, {
streams: [stream],
exit: (c) => calls.push(c),
guardMs: 120,
});
expect(calls).toEqual([0]);
expect(Date.now() - t0).toBeLessThan(2000);
});
test('removes its drain listener when the poll tick wins (no listener leak)', async () => {
const stream = makeStream(256);
const calls: number[] = [];
const done = flushStdoutThenExit(0, {
streams: [stream],
exit: (c) => calls.push(c),
guardMs: 5000,
});
setTimeout(() => stream.setLength(0), 30);
await done;
expect(stream.drainListeners()).toBe(0);
});
});
describe('exit-verdict ownership — no raw process.exitCode assignments (#2084 class pin)', () => {
test('every exit-code write in src/ routes through setCliExitCode', async () => {
// A RAW `process.exitCode = N` is silently ZEROED by the deliberate
// flush-exit: getCliExitCode() reads only gbrain's owned verdict (the
// PGLite-Emscripten-pollution defense), so a command that bypasses the
// setter reports success on failure. Caught live: doctor's FAIL path
// exited 0 after the v0.42.41.0 merge introduced a raw write.
const { execSync } = await import('child_process');
const hits = execSync(
`grep -rn "process.exitCode = " src --include='*.ts' | grep -v "core/cli-force-exit.ts" || true`,
{ encoding: 'utf-8', cwd: new URL('..', import.meta.url).pathname },
).trim();
expect(hits).toBe('');
});
});
+39 -48
View File
@@ -1,64 +1,55 @@
/**
* Structural regression — the DISCONNECT_HARD_DEADLINE_MS force-exit timer
* must be armed at TEARDOWN ENTRY, never before the op-dispatch try block.
* Structural regression — the teardown hard-deadline must be armed at
* TEARDOWN ENTRY, never before the op-dispatch body.
*
* Pre-fix bug (fixed independently by master's v0.42.41.0 triage wave AND
* the #2084 wave): the 10s unref'd setTimeout was armed BEFORE the try, so
* any op whose handler ran past 10s wall-clock was killed mid-flight with
* process.exit(0) and ZERO stdout — an empty "success" indistinguishable
* from no results (a healthy `gbrain search` on a slow Postgres pooler hit
* this on every run).
* Pre-fix bug (closed independently by v0.42.41.0 and the #2084 wave, merged):
* a 10s unref'd setTimeout armed BEFORE the try killed any op whose handler
* ran past 10s wall-clock with process.exit(0) and ZERO stdout — an empty
* "success" indistinguishable from no results.
*
* Post-#2084 the arming lives INSIDE the shared `drainThenDisconnect`
* helper — the single owner-disconnect every CLI exit path calls from its
* finally — so it structurally bounds ONLY the teardown window (drain +
* disconnect) at every site, not just the op-dispatch path.
*
* Source-grep is the right tool here (same rationale as
* fix-wave-structural.test.ts): the rule is "this arming must stay at this
* location". A behavioral test would need >10s of real wall-clock plus a
* deliberately slow op handler in a spawned CLI — slow and flaky by
* construction.
* Post-merge shape (#2084): the deadline lives inside `finishCliTeardown`
* (src/core/cli-force-exit.ts), armed as the helper's first act — i.e. at
* teardown entry, because every cli.ts call site invokes the helper from a
* `finally`. The op body's wallclock is bounded separately by the read-scope
* withTimeout wrap (v0.42.41.0). Source-grep is the right tool here (same
* rationale as fix-wave-structural.test.ts): a behavioral test would need
* >10s of real wall-clock in a spawned CLI.
*/
import { describe, test, expect } from 'bun:test';
import { readFileSync } from 'fs';
describe('cli.ts — disconnect hard-deadline armed at teardown entry, not before the op body', () => {
test('no timer arming between the op-dispatch entry and its try block', () => {
const src = readFileSync('src/cli.ts', 'utf8');
// The op-dispatch local-engine path: from connectEngine to its try, no
// setTimeout call may be armed (a pre-try timer kills slow-but-progressing
// op handlers mid-flight with exit 0 and empty stdout). `setTimeout(`
// matches only a call site; ReturnType<typeof setTimeout> stays allowed.
const entry = src.indexOf('// Local engine path (unchanged behavior');
expect(entry).toBeGreaterThan(-1);
const tryIdx = src.indexOf('try {', entry);
test('no timer arming exists between op-dispatch setup and the try; the deadline arms inside finishCliTeardown before the drain', () => {
const cli = readFileSync('src/cli.ts', 'utf8');
// The old pre-try arming constant must stay gone (its return is the
// kill-slow-ops-with-exit-0 regression).
expect(cli).not.toContain('DISCONNECT_HARD_DEADLINE_MS');
// Between the op-dispatch engine connect and the try there is no
// setTimeout call site (`setTimeout(` matches calls only; the
// ReturnType<typeof setTimeout> annotation stays allowed).
const connectIdx = cli.indexOf('// Local engine path (unchanged behavior for local installs).');
expect(connectIdx).toBeGreaterThan(-1);
const tryIdx = cli.indexOf('try {', connectIdx);
expect(tryIdx).toBeGreaterThan(-1);
expect(src.slice(entry, tryIdx)).not.toContain('setTimeout(');
});
expect(cli.slice(connectIdx, tryIdx)).not.toContain('setTimeout(');
test('the deadline arming lives inside drainThenDisconnect: gated, unref\'d, before the drain, cleared after', () => {
const src = readFileSync('src/cli.ts', 'utf8');
const helper = src.match(/export async function drainThenDisconnect[\s\S]+?^\}/m);
expect(helper).not.toBeNull();
const block = helper![0];
// The op-dispatch finally routes through the shared teardown helper.
const finallyIdx = cli.indexOf('} finally {', tryIdx);
expect(finallyIdx).toBeGreaterThan(-1);
const teardownCallIdx = cli.indexOf('finishCliTeardown({ engine, drainTimeoutMs: 1000 })', finallyIdx);
expect(teardownCallIdx).toBeGreaterThan(finallyIdx);
const armIdx = block.indexOf('deadlineTimer = setTimeout');
const drainIdx = block.indexOf('drainAllBackgroundWorkForCliExit');
// Inside the helper, the backstop arms BEFORE the drain runs — teardown
// entry, bounding drain + disconnect and nothing else.
const helper = readFileSync('src/core/cli-force-exit.ts', 'utf8');
const armIdx = helper.indexOf('const backstop = setTimeout(');
expect(armIdx).toBeGreaterThan(-1);
expect(drainIdx).toBeGreaterThan(-1);
// Armed at teardown entry, before the drain + disconnect it bounds.
expect(armIdx).toBeLessThan(drainIdx);
// Gated on the daemon-survival guard so `serve` stays alive.
expect(block.slice(0, armIdx)).toMatch(/if \(shouldForceExitAfterMain\(\)\)/);
// Unref'd so the timer itself never keeps the event loop alive.
expect(block).toContain('deadlineTimer.unref?.()');
const drainIdx = helper.indexOf('await drain({ timeoutMs: drainTimeoutMs })', armIdx);
expect(drainIdx).toBeGreaterThan(armIdx);
// Cleared on clean teardown.
expect(block).toContain('if (deadlineTimer) clearTimeout(deadlineTimer)');
// The DISCONNECT_HARD_DEADLINE_MS declaration sits with the helper.
const decl = src.indexOf('const DISCONNECT_HARD_DEADLINE_MS');
expect(decl).toBeGreaterThan(-1);
expect(src.indexOf('export async function drainThenDisconnect')).toBeGreaterThan(decl);
expect(helper.indexOf('clearTimeout(backstop)', drainIdx)).toBeGreaterThan(drainIdx);
});
});
+8 -72
View File
@@ -1,85 +1,21 @@
/**
* v0.43 (#2084) — CRITICAL regression: piped stdout is never truncated by the
* deliberate flush-exit (eng-review D3; the incident #1959 class).
* v0.43 (#2084) — real-CLI pipe completeness pin (the incident #1959 class).
*
* Two layers:
* 1. A subprocess that writes 256KB to a REAL pipe (4x the 64KB kernel pipe
* buffer, so write() backpressures and bytes sit in the JS-side stream
* buffer) and then calls the production `flushStdoutThenExit`. If the
* flush gate is wrong, the tail of the payload is sheared off — this is
* exactly how incident #1959 presented ("relational query came back
* empty" — output truncated by a force-exit).
* 2. The real CLI (`bun src/cli.ts --tools-json`, engine-free) over a pipe:
* parseable, byte-stable across runs, and exits deliberately (well under
* the 10s backstop).
* The synthetic flush-mechanism coverage lives in
* test/flush-then-exit-harness.test.ts (4MB late-reader byte-complete pin).
* This file keeps the IMPLEMENTATION-AGNOSTIC check: the actual CLI, run the
* way agents run it (piped stdout), produces complete, parseable, byte-stable
* output and exits deliberately — well under the teardown backstop.
*/
import { describe, test, expect } from 'bun:test';
import { spawnSync } from 'child_process';
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
import { join, resolve } from 'path';
import { tmpdir } from 'os';
const REPO = resolve(import.meta.dir, '..');
const CLI = join(REPO, 'src', 'cli.ts');
const PAYLOAD_BYTES = 256 * 1024;
describe('cli pipe truncation — deliberate exit flushes piped stdout (#2084)', () => {
test('256KB through a real pipe survives flushStdoutThenExit byte-exactly', () => {
const dir = mkdtempSync(join(tmpdir(), 'gbrain-flush-'));
const script = join(dir, 'flush-child.ts');
try {
writeFileSync(
script,
[
`import { flushStdoutThenExit } from ${JSON.stringify(join(REPO, 'src', 'core', 'cli-force-exit.ts'))};`,
`const payload = 'x'.repeat(${PAYLOAD_BYTES}) + 'END\\n';`,
// No await — mirrors the cli.ts entrypoint, which fires the flush
// exit as a dangling promise after main() resolves.
`process.stdout.write(payload);`,
`void flushStdoutThenExit(0);`,
].join('\n'),
);
const res = spawnSync('bun', [script], {
stdio: ['ignore', 'pipe', 'pipe'],
encoding: 'utf-8',
timeout: 30_000,
maxBuffer: 16 * 1024 * 1024,
});
expect(res.status).toBe(0);
const out = res.stdout ?? '';
expect(Buffer.byteLength(out, 'utf-8')).toBe(PAYLOAD_BYTES + 4);
expect(out.endsWith('END\n')).toBe(true);
} finally {
rmSync(dir, { recursive: true, force: true });
}
}, 60_000);
test('exit code survives the flush (errored command stays non-zero)', () => {
const dir = mkdtempSync(join(tmpdir(), 'gbrain-flush-code-'));
const script = join(dir, 'flush-code-child.ts');
try {
writeFileSync(
script,
[
`import { flushStdoutThenExit } from ${JSON.stringify(join(REPO, 'src', 'core', 'cli-force-exit.ts'))};`,
`process.stdout.write('partial output before failure\\n');`,
`void flushStdoutThenExit(3);`,
].join('\n'),
);
const res = spawnSync('bun', [script], {
stdio: ['ignore', 'pipe', 'pipe'],
encoding: 'utf-8',
timeout: 30_000,
});
expect(res.status).toBe(3);
expect(res.stdout).toBe('partial output before failure\n');
} finally {
rmSync(dir, { recursive: true, force: true });
}
}, 60_000);
describe('cli pipe completeness — deliberate exit never truncates piped stdout (#2084)', () => {
test('real CLI: --tools-json over a pipe is complete, parseable, byte-stable, and prompt', () => {
const run = () => {
const t0 = Date.now();
@@ -98,7 +34,7 @@ describe('cli pipe truncation — deliberate exit flushes piped stdout (#2084)',
// Truncated JSON does not parse — the strongest single-run completeness check.
const parsed = JSON.parse(first.stdout);
expect(Array.isArray(parsed)).toBe(true);
// Deliberate exit, not the 10s hard-deadline backstop.
// Deliberate exit, not the teardown backstop.
expect(first.ms).toBeLessThan(9_000);
const second = run();
+12
View File
@@ -38,6 +38,18 @@ describe('shouldForceExitAfterMain — daemon survival gate', () => {
expect(shouldForceExitAfterMain(['get', 'people/alice'])).toBe(true);
});
test('#2084 cross-model finding: space-separated global flag values cannot fake a command', () => {
// `--timeout 30s serve` — the old first-non-dash heuristic resolved the
// command as `30s` → true → the central exit seam would process.exit the
// freshly started daemon ~250ms after boot, exit 0, no error. The gate now
// resolves the command through parseGlobalFlags, matching main()'s dispatch.
expect(shouldForceExitAfterMain(['--timeout', '30s', 'serve'])).toBe(false);
expect(shouldForceExitAfterMain(['--timeout', '30s', 'serve', '--http'])).toBe(false);
expect(shouldForceExitAfterMain(['--progress-interval', '500', 'serve'])).toBe(false);
// ...and the same shape before a one-shot command still force-exits.
expect(shouldForceExitAfterMain(['--timeout', '30s', 'query', 'x'])).toBe(true);
});
test('returns true for non-daemon CLI commands', () => {
expect(shouldForceExitAfterMain(['stats'])).toBe(true);
expect(shouldForceExitAfterMain(['doctor'])).toBe(true);
+143 -1
View File
@@ -34,6 +34,7 @@
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { spawn, spawnSync } from 'child_process';
import {
cpSync,
mkdirSync,
mkdtempSync,
rmSync,
@@ -152,12 +153,13 @@ afterAll(() => {
function runWithTimeout(
args: string[],
timeoutMs: number,
envOverride?: Record<string, string>,
): Promise<{ code: number | null; stdout: string; stderr: string; durationMs: number }> {
return new Promise((resolveOut) => {
const t0 = Date.now();
const child = spawn(SHIM_PATH, args, {
cwd: REPO_ROOT,
env: runEnv,
env: envOverride ? { ...runEnv, ...envOverride } : runEnv,
});
let stdout = '';
let stderr = '';
@@ -173,6 +175,13 @@ function runWithTimeout(
});
}
/**
* #2084: the teardown backstop banner must NEVER appear on a healthy run —
* it now means a teardown component violated its own bound, not "this
* command was slower than 10s end-to-end" (the pre-#2084 misfire).
*/
const TEARDOWN_BANNER = 'did not return within';
describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290)', () => {
test('gbrain search "foxtrot" exits 0 within 15s', async () => {
const { code, stdout, stderr, durationMs } = await runWithTimeout(
@@ -185,6 +194,7 @@ describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290
`STDOUT:\n${stdout}\nSTDERR:\n${stderr}`,
);
}
expect(stderr).not.toContain(TEARDOWN_BANNER);
expect(code).toBe(0);
// Must have actually returned a hit — else bumpLastRetrievedAt
// would have early-returned on empty pageIds and the bug wouldn't
@@ -203,6 +213,7 @@ describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290
`STDOUT:\n${stdout}\nSTDERR:\n${stderr}`,
);
}
expect(stderr).not.toContain(TEARDOWN_BANNER);
expect(code).toBe(0);
expect(stdout).toContain('foxtrot');
}, 30_000);
@@ -258,6 +269,7 @@ describe('v0.42.20.0 — gbrain capture (CLI_ONLY) exits cleanly + frees the loc
// The real lock-pin symptom: the NEXT command times out waiting for the
// PGLite lock. Assert a subsequent read runs cleanly and quickly.
expect(cap.stderr).not.toContain(TEARDOWN_BANNER);
const next = await runWithTimeout(['get', 'meetings/capture-test'], 15_000);
expect(next.durationMs).toBeLessThan(15_000);
expect(next.stderr).not.toContain('Timed out waiting for PGLite lock');
@@ -267,6 +279,136 @@ describe('v0.42.20.0 — gbrain capture (CLI_ONLY) exits cleanly + frees the loc
}, 60_000);
});
describe('#2084 — explicit-exit teardown: every swept site exits clean, exit codes report the op', () => {
// D6C hardening: mutating commands run against a throwaway COPY of the
// seeded GBRAIN_HOME so a remediation/dream pass can't contaminate the
// brain the other tests share.
function copyBrainHome(label: string): string {
const copy = mkdtempSync(join(tmpdir(), `gbrain-2084-${label}-`));
cpSync(tmpHome, copy, { recursive: true });
return copy;
}
test('D5: failed op exits 1 with the error on stderr (exit code = op outcome)', async () => {
const { code, stderr, durationMs } = await runWithTimeout(
['get', 'nonexistent-slug-2084'],
15_000,
);
expect(durationMs).toBeLessThan(15_000);
expect(code).toBe(1);
expect(stderr.length).toBeGreaterThan(0);
expect(stderr).not.toContain(TEARDOWN_BANNER);
}, 30_000);
test('search stats (dashboard path, Site C) exits 0, no banner', async () => {
const { code, stdout, stderr } = await runWithTimeout(['search', 'stats'], 20_000);
expect(code).toBe(0);
expect(stdout.length).toBeGreaterThan(0);
expect(stderr).not.toContain(TEARDOWN_BANNER);
}, 30_000);
test('sources list (read-only timeout path, Site D) exits 0, no banner', async () => {
const { code, stderr } = await runWithTimeout(['sources', 'list'], 20_000);
expect(code).toBe(0);
expect(stderr).not.toContain(TEARDOWN_BANNER);
}, 30_000);
test('doctor (Site G — leak-fix shape) exits without hanging, no banner', async () => {
const { code, durationMs, stderr } = await runWithTimeout(['doctor'], 45_000);
expect(durationMs).toBeLessThan(45_000);
expect(stderr).not.toContain(TEARDOWN_BANNER);
// Keyless CI may surface advisory findings; doctor's exit code reflects
// brain health, not teardown health. The pin is: exits, no banner.
expect(code).not.toBeNull();
}, 60_000);
test('doctor --remediation-plan (Site F) exits without hanging, no banner', async () => {
const { code, durationMs, stderr } = await runWithTimeout(
['doctor', '--remediation-plan'],
30_000,
);
expect(durationMs).toBeLessThan(30_000);
expect(stderr).not.toContain(TEARDOWN_BANNER);
expect(code).not.toBeNull();
}, 45_000);
test('doctor --remediate (Site F, mutating) runs on a brain copy, exits, no banner', async () => {
const copy = copyBrainHome('remediate');
try {
const { code, durationMs, stderr } = await runWithTimeout(
['doctor', '--remediate'],
45_000,
{ GBRAIN_HOME: copy },
);
expect(durationMs).toBeLessThan(45_000);
expect(stderr).not.toContain(TEARDOWN_BANNER);
expect(code).not.toBeNull();
} finally {
rmSync(copy, { recursive: true, force: true });
}
}, 60_000);
test('dream --dry-run (Site E — the overnight-cron TODO site) exits, no banner', async () => {
const copy = copyBrainHome('dream');
try {
const { code, durationMs, stderr } = await runWithTimeout(
['dream', '--dry-run'],
60_000,
{ GBRAIN_HOME: copy },
);
// Keyless CI: LLM-dependent phases degrade; the IRON rule is exits + no
// banner. A hang here is the silent-overnight-zombie regression.
expect(durationMs).toBeLessThan(60_000);
expect(stderr).not.toContain(TEARDOWN_BANNER);
expect(code).not.toBeNull();
} finally {
rmSync(copy, { recursive: true, force: true });
}
}, 90_000);
test('ze-switch --dry-run (Site H) exits without hanging, no banner', async () => {
const { code, durationMs, stderr } = await runWithTimeout(
['ze-switch', '--dry-run'],
30_000,
);
expect(durationMs).toBeLessThan(30_000);
expect(stderr).not.toContain(TEARDOWN_BANNER);
expect(code).not.toBeNull();
}, 45_000);
test('D11: teardown deadline does NOT cover handler time (slow-handler regression)', async () => {
// Post-fix the deadline arms at teardown start, so a 500ms deadline cannot
// touch the handler: results print in full regardless. NOTE the falsification
// story is forward-looking, not historical — pre-#2084 code had no env knob
// (a static 10s constant), so this spawn would pass there too; the guard
// against re-hoisting the timer above the handler is the structural pin on
// DISCONNECT_HARD_DEADLINE_MS absence in fix-wave-structural.test.ts. This
// test pins that the env override is honored AND output survives a deadline
// far smaller than handler time.
const { code, stdout, durationMs } = await runWithTimeout(
['search', 'foxtrot', '--limit', '3'],
15_000,
{ GBRAIN_TEARDOWN_DEADLINE_MS: '500' },
);
expect(durationMs).toBeLessThan(15_000);
expect(code).toBe(0);
expect(stdout.length).toBeGreaterThan(0); // output intact = handler wasn't killed
}, 30_000);
test('D10: piped --json output parses complete (no exit truncation)', async () => {
// `search stats --json` emits a pure JSON document (the shared-op search
// path renders human format regardless of --json). A truncated-by-exit
// pipe fails to parse — the #1959 class, end-to-end.
const { code, stdout, stderr } = await runWithTimeout(
['search', 'stats', '--json'],
20_000,
);
expect(code).toBe(0);
expect(stderr).not.toContain(TEARDOWN_BANNER);
expect(() => JSON.parse(stdout)).not.toThrow();
}, 30_000);
});
describe('v0.41.8.0 — daemon survival (regression guard for narrow force-exit)', () => {
test('gbrain serve --http stays alive past the timeout window', async () => {
// Pick a likely-free ephemeral port. We're testing "still alive
+4 -1
View File
@@ -72,6 +72,9 @@ describe('awaitPendingEvalCaptures', () => {
const r = await awaitPendingEvalCaptures(150);
const elapsed = Date.now() - start;
expect(r.unfinished).toBe(1);
expect(elapsed).toBeLessThan(1000);
// Bound proves "bounded, not a hang" — the alternative is infinite. 2s
// (13x the 150ms budget) absorbs CI shard-load timer jitter; the old 1s
// bound flaked at 1023ms on a loaded GitHub runner.
expect(elapsed).toBeLessThan(2000);
});
});
+70 -31
View File
@@ -125,20 +125,15 @@ describe('v0.36.1.x #1124 — query --no-expand actually negates expand', () =>
describe('v0.42.20.0 — background-work registry drains every sink before disconnect', () => {
// Supersedes the v0.41.8.0 #1247/#1269/#1290 per-call last-retrieved drain:
// last-retrieved is one of the registry sinks. Since the v0.43 #2084 drain
// hoist, cli.ts routes EVERY owner-disconnect through drainThenDisconnect
// (one helper, all 8 sites) instead of two inline drain+disconnect pairs.
test('cli.ts imports the registry drain + routes owner-disconnects through drainThenDisconnect', () => {
const src = readFileSync('src/cli.ts', 'utf8');
expect(src).toMatch(/import\s+\{\s*drainAllBackgroundWorkForCliExit\s*\}\s*from\s+['"]\.\/core\/background-work\.ts['"]/);
// The single registry-drain call site lives inside the shared helper...
expect(src).toMatch(/export async function drainThenDisconnect/);
expect(src).toMatch(/await\s+drainAllBackgroundWorkForCliExit\s*\(/);
// ...and the helper covers every owner-disconnect (op-dispatch, CLI_ONLY
// fall-through, search dashboard, doctor x3, ze-switch, dream,
// read-only-timeout path).
const calls = src.match(/await\s+drainThenDisconnect\s*\(/g) ?? [];
expect(calls.length).toBeGreaterThanOrEqual(8);
// last-retrieved is one of four registry sinks. #2084 moved the registry
// drain out of cli.ts's inline finallys into finishCliTeardown
// (cli-force-exit.ts), which every cli.ts teardown site routes through —
// the drain-before-disconnect invariant is pinned there (and behaviorally
// by test/cli-finish-teardown.test.ts).
test('cli-force-exit.ts imports + drains the registry inside finishCliTeardown', () => {
const src = readFileSync('src/core/cli-force-exit.ts', 'utf8');
expect(src).toMatch(/import\s+\{\s*drainAllBackgroundWorkForCliExit[\s\S]*?\}\s*from\s+['"]\.\/background-work\.ts['"]/);
expect(src).toMatch(/export async function finishCliTeardown/);
});
test('last-retrieved.ts still exports the bounded drain + registers a drainer', () => {
@@ -160,24 +155,18 @@ describe('v0.42.20.0 — background-work registry drains every sink before disco
.toMatch(/name:\s*'eval-capture'/);
});
test('cli.ts behavioral positioning: drainThenDisconnect drains the registry BEFORE engine.disconnect', () => {
const src = readFileSync('src/cli.ts', 'utf8');
// The ordering invariant lives ONCE, inside the shared helper (v0.43
// #2084 drain hoist): registry drain, THEN best-effort disconnect.
const helper = src.match(/export async function drainThenDisconnect[\s\S]+?^\}/m);
expect(helper).not.toBeNull();
const block = helper![0];
const drainCallRe = /await\s+drainAllBackgroundWorkForCliExit\s*\(/;
const disconnectCallRe = /await\s+engine\.disconnect\s*\(/;
expect(block).toMatch(drainCallRe);
expect(block).toMatch(disconnectCallRe);
const drainIdx = block.indexOf(block.match(drainCallRe)![0]);
const disconnectIdx = block.indexOf(block.match(disconnectCallRe)![0]);
test('finishCliTeardown positioning: registry drain appears BEFORE engine disconnect', () => {
// #2084: the invariant moved from cli.ts's inline finallys into the shared
// helper. The drain must run against a live engine (facts abort-path
// logIngest, #1762) before disconnect tears the pools down.
const src = readFileSync('src/core/cli-force-exit.ts', 'utf8');
const drainCallRe = /await\s+drain\s*\(\s*\{\s*timeoutMs:\s*drainTimeoutMs\s*\}\s*\)/;
const disconnectCallRe = /await\s+opts\.engine\.disconnect\s*\(/;
expect(src).toMatch(drainCallRe);
expect(src).toMatch(disconnectCallRe);
const drainIdx = src.indexOf(src.match(drainCallRe)![0]);
const disconnectIdx = src.indexOf(src.match(disconnectCallRe)![0]);
expect(drainIdx).toBeLessThan(disconnectIdx);
// And the op-dispatch finally routes through the helper.
const localPath = src.match(/\/\/ Local engine path \(unchanged behavior[\s\S]+?\n \}\n\}/m);
expect(localPath).not.toBeNull();
expect(localPath![0]).toMatch(/await\s+drainThenDisconnect\s*\(/);
});
test('background-work.ts: Map registry, ordered drain, awaited abort, test seam', () => {
@@ -195,6 +184,56 @@ describe('v0.42.20.0 — background-work registry drains every sink before disco
});
});
describe('#2084 — cli.ts owns process-exit teardown via finishCliTeardown', () => {
test('no bare awaited engine disconnects remain in cli.ts', () => {
// The awaited forms are the call-site contract (comments never use the
// awaited literal, so this is comment-proof — eng-review D13.2). A bare
// disconnect skips the bounded drain + computed-deadline backstop and
// reopens the lingering-socket hang class.
const src = readFileSync('src/cli.ts', 'utf8');
expect(src).not.toContain('await engine.disconnect()');
expect(src).not.toContain('await eng.disconnect()');
});
test('the pre-handler hard-deadline timer is gone (handler time is not teardown budget)', () => {
// Pre-#2084 the op-dispatch timer armed BEFORE the op handler, so any op
// slower than 10s was force-killed mid-run with exit 0 and truncated
// output. The deadline now arms inside finishCliTeardown, at teardown
// start only.
const src = readFileSync('src/cli.ts', 'utf8');
expect(src).not.toContain('DISCONNECT_HARD_DEADLINE_MS');
});
test('all nine swept sites route through finishCliTeardown; one exit seam', () => {
const src = readFileSync('src/cli.ts', 'utf8');
const calls = src.match(/await finishCliTeardown\(/g) ?? [];
expect(calls.length).toBeGreaterThanOrEqual(9);
// The single process-exit seam: flushThenExit in the import.meta.main
// block, fed by currentExitCode().
expect(src).toMatch(/import\.meta\.main/);
expect(src).toMatch(/flushThenExit\(currentExitCode\(\)\)/);
});
test('pglite-engine contains the Emscripten process.exitCode hijack', () => {
// PGLite's WASM runtime writes its own status into process.exitCode (99
// alive / exit status on close) and ignores `undefined` assignment. The
// create call runs inside preservingProcessExitCode to keep the global
// tidy; close is deliberately unwrapped (see below) — the CLI's verdict
// is immune either way via the owned channel.
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
expect(src).toMatch(/preservingProcessExitCode\(\(\)\s*=>\s*\n?\s*PGlite\.create/);
// close stays UNWRAPPED by design: its status write is baseline behavior
// test runners depend on; the CLI's verdict is immune because it lives in
// the gbrain-owned channel, never read back from process.exitCode.
const helper = readFileSync('src/core/cli-force-exit.ts', 'utf8');
expect(helper).toMatch(/let cliVerdict: number \| null = null/);
expect(helper).toMatch(/return cliVerdict \?\? 0/);
// The op-dispatch catch must set the verdict through the owned channel.
const cli = readFileSync('src/cli.ts', 'utf8');
expect(cli).toMatch(/setCliExitVerdict\(1\);/);
});
});
describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => {
test('pglite-engine.ts exports classifyPgliteInitError + buildPgliteInitErrorMessage', () => {
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
+27
View File
@@ -0,0 +1,27 @@
/**
* #2084 (D10) — spawned harness proving flushThenExit against REAL Bun pipe
* semantics: writes HARNESS_BYTES of 'x' to stdout, then flushThenExit with
* HARNESS_EXIT_CODE. The parent test pipes stdout to a slow-attaching reader
* and asserts byte-complete output + the exit code — the exact scenario the
* pre-#2084 force-exit truncated (#1959).
*/
import { flushThenExit } from '../../src/core/cli-force-exit.ts';
const size = Number(process.env.HARNESS_BYTES ?? 4_000_000);
const code = Number(process.env.HARNESS_EXIT_CODE ?? 7);
const guardMs = Number(process.env.HARNESS_GUARD_MS ?? 2_000);
const graceEnv = process.env.HARNESS_GRACE_MS;
const chunk = 'x'.repeat(65_536);
let written = 0;
while (written < size) {
const n = Math.min(chunk.length, size - written);
process.stdout.write(n === chunk.length ? chunk : chunk.slice(0, n));
written += n;
}
flushThenExit(code, {
guardMs,
...(graceEnv !== undefined ? { graceMs: Number(graceEnv) } : {}),
});
+97
View File
@@ -0,0 +1,97 @@
/**
* #2084 (D10) — flushThenExit proven on a real spawned Bun process.
*
* The unit tests in cli-finish-teardown.test.ts inject fake streams; they
* prove the helper's logic but not Bun's actual pipe behavior (does an empty
* write('', cb) really fence all prior buffered chunks?). These tests spawn
* test/fixtures/flush-then-exit-harness.ts and assert:
* 1. multi-MB piped stdout arrives byte-complete with the right exit code
* even when the reader attaches late (#1959 truncation regression pin);
* 2. with empty buffers the fence resolves promptly — the process does NOT
* sit out the flush guard (canary for Bun eliding empty-write callbacks).
*/
import { describe, test, expect } from 'bun:test';
import { spawn } from 'child_process';
import { resolve } from 'path';
const HARNESS = resolve(import.meta.dir, 'fixtures', 'flush-then-exit-harness.ts');
function runHarness(env: Record<string, string>, readerDelayMs: number): Promise<{
bytes: number;
code: number | null;
durationMs: number;
}> {
return new Promise((resolveOut, reject) => {
const t0 = Date.now();
const child = spawn('bun', ['run', HARNESS], {
env: { ...process.env, ...env },
stdio: ['ignore', 'pipe', 'inherit'],
});
let bytes = 0;
child.stdout.pause();
setTimeout(() => {
child.stdout.on('data', (d: Buffer) => (bytes += d.length));
child.stdout.resume();
}, readerDelayMs);
const killer = setTimeout(() => {
child.kill('SIGKILL');
reject(new Error(`harness did not exit (bytes so far: ${bytes})`));
}, 30_000);
child.on('close', (code) => {
clearTimeout(killer);
resolveOut({ bytes, code, durationMs: Date.now() - t0 });
});
});
}
describe('flushThenExit on a real Bun process (D10)', () => {
test('4MB piped stdout arrives byte-complete with the exit code, late reader', async () => {
// Bun delivers queued pipe writes only while the process is alive (see the
// cli-force-exit.ts module header). The reader attaches 200ms late; the
// 1500ms aliveness grace must cover attach + transfer. Pre-#2084 (immediate
// process.exit, no grace) this received 0 of the 4MB — verified during
// implementation; that is the #1959 truncation class.
const SIZE = 4_000_000;
const { bytes, code } = await runHarness(
{
HARNESS_BYTES: String(SIZE),
HARNESS_EXIT_CODE: '7',
HARNESS_GUARD_MS: '2000',
HARNESS_GRACE_MS: '1500',
},
200,
);
expect(bytes).toBe(SIZE);
expect(code).toBe(7);
}, 40_000);
test('default grace: small output survives exit with a concurrent reader', async () => {
// No HARNESS_GRACE_MS → production default (non-TTY grace). Immediate
// reader, 100-byte output: must arrive complete. Pre-#2084 even this case
// lost ALL bytes when exit fired before a loop turn.
const { bytes, code } = await runHarness(
{ HARNESS_BYTES: '100', HARNESS_EXIT_CODE: '0', HARNESS_GUARD_MS: '2000' },
0,
);
expect(bytes).toBe(100);
expect(code).toBe(0);
}, 40_000);
test('fence resolves promptly — wall time well under guard + grace ceiling', async () => {
// Guard 8s, grace 250ms: if Bun ever elides the empty-write callback, the
// process sits out the full guard and wall time exceeds it. A working
// fence exits in startup time + grace (~1-2s).
const { code, durationMs } = await runHarness(
{
HARNESS_BYTES: '100',
HARNESS_EXIT_CODE: '0',
HARNESS_GUARD_MS: '8000',
HARNESS_GRACE_MS: '250',
},
0,
);
expect(code).toBe(0);
expect(durationMs).toBeLessThan(6_000);
}, 40_000);
});
@@ -219,3 +219,38 @@ describe('PGLiteEngine.disconnect() — v0.41.8.0 lifecycle invariants', () => {
}
});
});
// ─────────────────────────────────────────────────────────────────
// #2084 — preservingProcessExitCode behavioral containment
// ─────────────────────────────────────────────────────────────────
describe('PGLiteEngine: Emscripten process.exitCode containment (#2084)', () => {
test('connect() leaves process.exitCode pinned at 0, not the Emscripten 99', async () => {
const prev = process.exitCode;
const eng = new PGLiteEngine();
try {
await eng.connect({ engine: 'pglite' });
// Emscripten writes 99 during create; the wrapper pins explicit 0 when
// nothing was set before (undefined cannot be restored — the accessor
// falls back to the WASM status).
expect(Number(process.exitCode)).toBe(0);
} finally {
await eng.disconnect();
process.exitCode = prev;
}
}, 60_000);
test('a pre-call verdict survives the create-throw path (finally restores)', async () => {
const prev = process.exitCode;
const eng = new PGLiteEngine();
try {
process.exitCode = 3;
// A dataDir under a regular FILE cannot be created — PGlite.create rejects.
await expect(
eng.connect({ engine: 'pglite', database_path: '/dev/null/nope/brain' }),
).rejects.toThrow();
expect(Number(process.exitCode)).toBe(3);
} finally {
process.exitCode = prev;
}
}, 60_000);
});
+4 -1
View File
@@ -1311,7 +1311,10 @@ describe('PGLiteEngine: v0.13.1 error-wrap on connect() (#223)', () => {
// issue and suggest gbrain doctor. Must NOT suggest "missing migrations"
// as a cause (that was conflating #218 and #223 — migrations run AFTER
// create()).
expect(src).toContain('this._db = await PGlite.create');
// #2084 wrapped the create call in preservingProcessExitCode (Emscripten
// exitCode containment); the try/catch + error wrap around it is unchanged.
expect(src).toContain('this._db = await preservingProcessExitCode(() =>');
expect(src).toContain('PGlite.create({');
expect(src).toContain('https://github.com/garrytan/gbrain/issues/223');
expect(src).toContain('gbrain doctor');
expect(src).toContain('Original error:');