mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
Merge remote-tracking branch 'origin/master' into garrytan/newest-agents-filed-issues
# Conflicts: # TODOS.md # docs/architecture/KEY_FILES.md # src/cli.ts # src/commands/autopilot.ts # src/commands/brainstorm.ts # src/commands/doctor.ts # src/commands/extract.ts # src/commands/frontmatter.ts # src/commands/reindex.ts # src/commands/transcripts.ts # src/core/cli-force-exit.ts # test/cli-force-exit-teardown-arming.test.ts # test/fix-wave-structural.test.ts
This commit is contained in:
+24
-4
@@ -2,6 +2,28 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.42.0] - 2026-06-12
|
||||
|
||||
**`gbrain query` no longer pays a flat 10-second exit tax on managed Postgres behind a transaction-mode pooler — and CLI exit codes finally tell the truth on PGLite.** On deployments where the pooler holds sockets open past the bounded pool drain (gbrain#2084, a residual of gbrain#1972), every query printed its results and then sat for 10 seconds until the force-exit banner fired. The cause was two-layered: the hard-deadline timer was armed *before* the operation handler, so a multi-second search on a large brain burned the teardown budget (and any operation slower than 10 seconds was silently killed mid-run with exit 0 and truncated output); and the CLI never exited explicitly on success — it waited for Bun's event loop to drain, which a stuck pooler socket can hold open forever.
|
||||
|
||||
The teardown contract now lives in one place: every cli.ts disconnect site runs a bounded background-work drain and a bounded disconnect under a backstop whose deadline is computed from the bounds it guards (so it fires only when something violated its own bound), then the process exits explicitly — after fencing stdout/stderr and holding a short aliveness window so piped output is delivered (Bun queues pipe writes in a native buffer that only drains while the process is alive). The most-used command in the CLI now exits in milliseconds-to-a-couple-seconds instead of ten.
|
||||
|
||||
Along the way the wave fixed a deeper, silent bug: PGLite's WASM runtime writes its own status into `process.exitCode` at arbitrary points mid-run, which meant **every error exit on PGLite-engine brains has been reporting success (exit 0)** — scripts and agents keying on exit codes never saw failures. The CLI verdict now lives in a gbrain-owned channel that the WASM runtime cannot touch.
|
||||
|
||||
### Fixed
|
||||
- **The flat 10s teardown tax + force-exit banner on transaction-mode poolers (gbrain#2084).** Queries exit promptly; the banner now appears only when a teardown component genuinely violated its own bound.
|
||||
- **Slow operations are no longer killed mid-run with a false success.** The teardown deadline starts at teardown, never before the operation handler — a 30-second sync or a deep query runs to completion.
|
||||
- **Error exits on PGLite report exit 1, not 0.** Failed operations (e.g. `gbrain get <missing-page>`) now exit non-zero on every engine; the exit code reports the operation, not the cleanup.
|
||||
- **Piped output survives the exit.** Output is fenced and given a delivery window before the process exits, on every routed exit path including the backstop (the truncation class from gbrain#1959).
|
||||
- **`gbrain doctor` no longer leaks its connection pool when DB checks throw**, and `dream`, `doctor`, `ze-switch`, and the search dashboards route their dispatcher teardown through the same bounded path (closing a long-standing drain gap on the overnight-cron path).
|
||||
- **Daemon safety with space-separated global flags.** `gbrain --timeout 30s serve` is recognized as the daemon it is — the exit gate resolves the command exactly the way dispatch does.
|
||||
|
||||
### Added
|
||||
- **`GBRAIN_TEARDOWN_DEADLINE_MS`** — env override for the teardown backstop deadline (incident escape hatch; the default is computed from the drain and pool bounds).
|
||||
- **`GBRAIN_FLUSH_GRACE_MS`** — env override for the pre-exit output-delivery window (default 250ms on pipes, 0 on TTYs). Raise it when piping very large payloads into slow consumers; lower it for high-frequency scripted invocations that capture to files.
|
||||
|
||||
### To take advantage of v0.42.42.0
|
||||
`gbrain upgrade`. No configuration needed. If your `gbrain query` has been printing results and then hanging ~10 seconds before a `force-exiting` banner, this release removes both the wait and the banner. If your scripts check gbrain exit codes on a PGLite brain, they will start seeing real failures — previously masked as exit 0 — so a wrapper that suddenly reports errors is the fix working, not a regression.
|
||||
## [0.42.41.0] - 2026-06-11
|
||||
|
||||
**A correctness-and-reliability wave: your conversation facts survive a cycle, write-through stops polluting other repos, autopilot rides out a DB blip instead of crash-looping, and concurrent PGLite processes stop corrupting each other.** A triage of open reports surfaced six bugs with no fix yet plus a batch of community PRs; this ships them together, each with a regression test.
|
||||
@@ -16422,8 +16444,7 @@ The OAuth provider in `src/core/oauth-provider.ts` got a parallel hardening pass
|
||||
Smaller hardening: admin cookies set `Secure` when behind HTTPS or a public-URL proxy (F9), magic-link nonces are bounded by an LRU cap (F10), `/mcp` wraps `transport.handleRequest` in try/catch so SDK throws hit a JSON-RPC 500 instead of express's default HTML error page (F14), and OperationError + unexpected exceptions both route through the unified `buildError`/`serializeError` envelope (F15). DCR disable became a constructor option on the provider rather than a serve-http monkey-patch (F12 — cleanup, not security).
|
||||
|
||||
To take advantage of v0.26.9
|
||||
============================
|
||||
|
||||
=====================
|
||||
`gbrain upgrade` is a one-step upgrade. There is no migration; all changes are application-layer.
|
||||
|
||||
1. **Upgrade.** `gbrain upgrade`. Confirm `gbrain --version` shows `0.26.9`.
|
||||
@@ -16557,8 +16578,7 @@ Both run at `--max-concurrency=1` after the parallel pass, same as the existing
|
||||
Wallclock observed: 74s on a Mac dev box (running `bun run test` with the new quarantines). Already at the v0.26.9 informational target. The full intra-file marker flip (with codemod + per-file `test.concurrent()`) lands in v0.26.9 and aims for the same ≤60s with pinned config.
|
||||
|
||||
To take advantage of v0.26.7
|
||||
============================
|
||||
|
||||
=====================
|
||||
`gbrain upgrade` does nothing functional in this release — it ships test infrastructure, not user-facing code. But if you contribute tests:
|
||||
|
||||
1. **Run `bun run verify` before pushing.** The new `check-test-isolation.sh` runs alongside the privacy + jsonb + progress checks. Catches new env-mutation, mock.module, and PGLite-pattern violations before CI does.
|
||||
|
||||
@@ -265,11 +265,40 @@ GSTACK REVIEW REPORT at
|
||||
cleanup. Do this BEFORE introducing any concurrent module-engine connect path.
|
||||
|
||||
- [x] **P3 — `dream` + CLI_ONLY fall-through paths don't drain the facts /
|
||||
last-retrieved queues before the owner disconnect.** DONE in the #2084 wave:
|
||||
`drainThenDisconnect(engine)` in `src/cli.ts` is the shared helper, applied at
|
||||
ALL eight owner-disconnect sites (op-dispatch, CLI_ONLY fall-through, search
|
||||
dashboard, doctor remediation x3, ze-switch, dream, read-only-timeout path),
|
||||
with its own bounded hard-deadline. Pinned by `test/cli-drain-then-disconnect.test.ts`.
|
||||
last-retrieved queues before the owner disconnect.** DONE in the #2084 fix:
|
||||
`finishCliTeardown` (`src/core/cli-force-exit.ts`) is exactly the shared
|
||||
drain-before-disconnect helper this item asked for, and ALL NINE cli.ts
|
||||
disconnect sites route through it (op-dispatch, fall-through, dream, doctor
|
||||
×3, ze-switch, search dashboard, read-only timeout path). Structural guard:
|
||||
no bare `await engine.disconnect()` remains in cli.ts
|
||||
(`test/fix-wave-structural.test.ts` `#2084` describe).
|
||||
|
||||
- [ ] **P2 — command-module `process.exit` sites bypass the #2084 teardown
|
||||
contract.** Several CLI_ONLY command modules exit directly on their normal
|
||||
paths (`doctor.ts` ~10 sites incl. its verdict exit, `dream.ts` ~23,
|
||||
`ze-switch.ts` ~9, plus friction/claw-test/eval verdict exits in cli.ts) —
|
||||
those exits preempt the call-site `finally`, so the background-work drain,
|
||||
bounded disconnect, and `flushThenExit` grace are all skipped on those paths
|
||||
(pre-existing class, NOT introduced by #2084; pre-fix the same exits skipped
|
||||
the inline drains too). Consequences: `gbrain doctor --json | <slow reader>`
|
||||
keeps the #1959 truncation exposure; a dream path that exits mid-cycle
|
||||
discards in-flight facts/search-cache writes. Fix shape: convert in-command
|
||||
`process.exit(n)` to `setCliExitVerdict(n)` + return (the central seam
|
||||
exits), or route them through a shared `exitCommand(n)` helper that runs
|
||||
teardown first. Surfaced by the #2084 cross-model adversarial review (F2).
|
||||
|
||||
- [ ] **P3 — opt-in whole-command wallclock cap (`GBRAIN_COMMAND_DEADLINE_MS`),
|
||||
build ONLY on a real wedged-handler incident.** The #2084 fix deliberately
|
||||
removed the blanket pre-handler 10s force-exit (it killed slow-legit ops with
|
||||
exit 0 and truncated output); per-op deadlines (query-embed deadline,
|
||||
`withTimeout` on read-only commands) own handler wallclock now, and
|
||||
`connectEngine` hangs — the historically observed zombie class — were never
|
||||
covered by the old timer anyway. If production ever shows a genuinely wedged
|
||||
handler (trigger: a non-`serve` command alive >30min with no progress
|
||||
output), add an opt-in env cap that exits NON-ZERO with a truthful banner.
|
||||
Attach point: the `GBRAIN_TEARDOWN_DEADLINE_MS` / `computeTeardownDeadlineMs`
|
||||
plumbing in `src/core/cli-force-exit.ts`. Do not build speculatively —
|
||||
follow-up from the #2084 eng review (decision D2/D14).
|
||||
## v0.42.x AI SDK v6 tool-schema fix follow-ups (#1782/#1764)
|
||||
|
||||
Surfaced by the codex outside-voice pass during `/plan-eng-review` and
|
||||
|
||||
+9
-5
@@ -10,16 +10,16 @@ Seven test command tiers, each with a clear scope:
|
||||
| Command | What it runs | Wallclock | When to use |
|
||||
|---|---|---|---|
|
||||
| `bun run test` | Parallel unit-test fast loop. 8-shard fan-out via `scripts/run-unit-parallel.sh`, then a serial pass over `*.serial.test.ts`. Excludes `*.slow.test.ts` and `test/e2e/*`. No pre-checks, no typecheck. | ~85s on a Mac dev box (3650+ tests) | Inner edit loop. Default. |
|
||||
| `bun run verify` | CI's authoritative pre-test gate set: `check:privacy && check:jsonb && check:progress && check:wasm && bun run typecheck`. The 4 checks `.github/workflows/test.yml` runs on shard 1 + typecheck. Single source of truth — CI literally calls `bun run verify`. | ~12s (wasm-compile dominates) | Before pushing; before `/ship`. |
|
||||
| `bun run verify` | CI's authoritative pre-test gate set, fanned out in parallel by `scripts/run-verify-parallel.sh`: the full `check:*` battery (~30 checks — privacy, jsonb, progress, source-id, test-isolation, wasm, …) plus `bun run typecheck`. The `CHECKS` array in that script is the single source of truth — CI literally calls `bun run verify` in a dedicated job. | ~16s (parallel; typecheck dominates) | Before pushing; before `/ship`. |
|
||||
| `bun run test:full` | `verify && bun run test && bun run test:slow && [smart e2e]`. The local equivalent of "everything CI runs." Smart e2e: runs e2e only when `DATABASE_URL` is set; else loud skip notice to stderr. | ~3-5min depending on slow + e2e | Pre-merge sanity, before opening a PR. |
|
||||
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
|
||||
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; runs at `--max-concurrency=1`). | ~1s per quarantined file | Debugging a specific quarantined file. |
|
||||
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
|
||||
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
|
||||
| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. |
|
||||
|
||||
### CI vs local: intentionally divergent file sets
|
||||
|
||||
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` 4-way, which uses FNV-1a hash bucketing and INCLUDES `*.slow.test.ts`. CI EXCLUDES `*.serial.test.ts` from the hash buckets and runs them on shard 1 via `bun run test:serial` at `--max-concurrency=1` — keeping serial files out of the hash buckets is what preserves the `mock.module` quarantine (top-level mocks in serial files would otherwise leak into the parallel files they share a shard process with). CI is the ground truth for "did everything pass."
|
||||
- **CI matrix** (`.github/workflows/test.yml`) runs `scripts/test-shard.sh` across 10 matrix shards partitioned by weight-aware LPT bin-packing (`scripts/sharding.ts`) and INCLUDES `*.slow.test.ts` (the two outlier slow files run as dedicated jobs alongside the matrix). CI EXCLUDES `*.serial.test.ts` from the shards and runs them in a dedicated job via `bun run test:serial`, one bun process per file — keeping serial files out of the shard processes is what preserves the `mock.module` quarantine (a top-level mock in one file leaks into every other file sharing its process). `bun run verify` gets its own job too. CI is the ground truth for "did everything pass."
|
||||
- **Local fast loop** (`scripts/run-unit-shard.sh` via the parallel wrapper) uses round-robin-by-index sharding and EXCLUDES `*.slow.test.ts` AND `*.serial.test.ts`. Local trades coverage for inner-loop speed; CI catches what local skips.
|
||||
|
||||
This divergence is intentional. Don't try to make them equal — the two scripts deliberately solve different problems. The regression test at `test/scripts/run-unit-shard.test.ts` pins what the local fast loop should and shouldn't include.
|
||||
@@ -39,7 +39,7 @@ If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the
|
||||
|
||||
- `*.test.ts` → fast loop (parallel 8-shard fan-out).
|
||||
- `*.slow.test.ts` → run via `bun run test:slow` only (intentional cold-path tests; would dominate the fast loop's wallclock).
|
||||
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; uses `--max-concurrency=1`. Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Currently: `test/brain-registry.serial.test.ts`, `test/reconcile-links.serial.test.ts`, `test/core/cycle.serial.test.ts`, `test/embed.serial.test.ts` (the latter two use `mock.module(...)` which leaks across files in the shard process). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
|
||||
- `*.serial.test.ts` → run via `bun run test:serial` after the parallel pass completes; one bun process per file (`--max-concurrency=1` within a shared process is not enough — the module registry still leaks `mock.module`). Quarantine for tests that share file-wide state and race when run alongside other files in the same `bun test` process. Several dozen files, discovered by the `*.serial.test.ts` glob — no list to maintain. Typical residents: `mock.module(...)` users (top-level mocks leak across files in a shard process, e.g. `test/embed.serial.test.ts`), env-coupled files (e.g. `test/brain-registry.serial.test.ts`), and process-lifecycle suites that assert on `process.exitCode` (e.g. `test/pglite-engine-disconnect.serial.test.ts`). **Do not put the parallelism back on a serial file unless you've fixed the contention root cause** (it just re-introduces the flake).
|
||||
- `test/e2e/*.test.ts` → real-Postgres E2E. Skipped when `DATABASE_URL` is unset.
|
||||
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
|
||||
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
|
||||
@@ -111,7 +111,7 @@ Rename to `*.serial.test.ts` when:
|
||||
- The file is genuinely env-coupled (e.g. `gbrain-home-isolation.test.ts`, `claw-test-cli.test.ts`) — module-load env readers + ESM caching defeat dynamic-import-after-env tricks.
|
||||
- The file's tests intentionally share state across `it()` boundaries.
|
||||
|
||||
Quarantine count cap: 10 (informational). Beyond that, push back on the design.
|
||||
The quarantine has grown to dozens of files — treat it as debt: every addition needs a reason from the list above, and prefer fixing the contention root cause when one exists.
|
||||
|
||||
### Unit test inventory
|
||||
|
||||
@@ -123,6 +123,9 @@ Unit tests and what they cover:
|
||||
- `test/chunkers/recursive.test.ts` — chunking.
|
||||
- `test/parity.test.ts` — operations contract parity.
|
||||
- `test/cli.test.ts` — CLI structure.
|
||||
- `test/cli-finish-teardown.test.ts` — the #2084 teardown contract: `computeTeardownDeadlineMs` formula/floor/live-registry scaling + `GBRAIN_TEARDOWN_DEADLINE_MS` override (garbage/zero/negative values fall back to the formula); `finishCliTeardown` clean path (drain BEFORE disconnect, no exit, no warn), backstop on hung drain or disconnect (honors an errored op's exit code), throwing drain/disconnect warned + swallowed; the gbrain-owned verdict channel is immune to PGLite WASM `process.exitCode` writes; `flushThenExit` unit coverage with mocked streams (exits once after both stream callbacks, non-TTY aliveness grace, blocked-pipe guard, EPIPE-safe, `GBRAIN_FLUSH_GRACE_MS` override).
|
||||
- `test/flush-then-exit-harness.test.ts` — real spawned-Bun pipe semantics for `flushThenExit` (fixture: `test/fixtures/flush-then-exit-harness.ts`): a 4MB piped stdout payload arrives byte-complete with the exit code even with a late reader, small output survives exit with a concurrent reader, and the fence resolves promptly (wall time well under the guard + grace ceiling).
|
||||
- `test/cli-should-force-exit.test.ts` — `shouldForceExitAfterMain` daemon-survival gate: `serve` (stdio and `--http`) never force-exits, including with preceding global flags; op commands / empty / flag-only argv do; the #2084 case that space-separated global-flag VALUES can't fake a command (`--timeout 30s serve` resolves to the `serve` daemon, not a `30s` command).
|
||||
- `test/config.test.ts` — config redaction.
|
||||
- `test/files.test.ts` — MIME/hash.
|
||||
- `test/import-file.test.ts` — import pipeline.
|
||||
@@ -219,6 +222,7 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
|
||||
- `test/e2e/sync.test.ts` — `--skip-failed` failure-loop test alongside happy-path tests: broken file → `performSync` returns `blocked_by_failures` with grouped breakdown → `performSync({skipFailed: true})` advances bookmark and returns `AcknowledgeResult` with code summary → second broken file → second cycle. Saves and restores the user's real `~/.gbrain/sync-failures.jsonl` so the test is hermetic. Asserts bookmark gating, JSONL state, dedup across paths, summary aggregation, and the literal doctor-rendering string format.
|
||||
- `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required).
|
||||
- `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use.
|
||||
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` — `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
|
||||
- `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `<fork>/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
|
||||
- `test/e2e/search-exclude.test.ts` — `test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths.
|
||||
|
||||
File diff suppressed because one or more lines are too long
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.41.0"
|
||||
"version": "0.42.42.0"
|
||||
}
|
||||
|
||||
+93
-166
@@ -25,8 +25,7 @@ import type { AIGatewayConfig } from './core/ai/types.ts';
|
||||
import type { BrainEngine } from './core/engine.ts';
|
||||
import { operations, OperationError } from './core/operations.ts';
|
||||
import type { Operation, OperationContext } from './core/operations.ts';
|
||||
import { drainAllBackgroundWorkForCliExit } from './core/background-work.ts';
|
||||
import { shouldForceExitAfterMain, flushStdoutThenExit, setCliExitCode, getCliExitCode } from './core/cli-force-exit.ts';
|
||||
import { shouldForceExitAfterMain, finishCliTeardown, flushThenExit, currentExitCode, setCliExitVerdict } from './core/cli-force-exit.ts';
|
||||
import { serializeMarkdown } from './core/markdown.ts';
|
||||
import { parseGlobalFlags, setCliOptions, getCliOptions } from './core/cli-options.ts';
|
||||
import type { CliOptions } from './core/cli-options.ts';
|
||||
@@ -262,9 +261,11 @@ async function main() {
|
||||
await withTimeout(runSearch(engine, subArgs), timeoutMs, label);
|
||||
}
|
||||
} finally {
|
||||
// search stats/tune read the query_cache — a pending cache write must
|
||||
// drain, not get killed by the deliberate exit (#2084 drain hoist).
|
||||
await drainThenDisconnect(engine);
|
||||
// #2084: `search diagnose` runs real hybrid retrieval (arms search-cache
|
||||
// writes) — route through the shared bounded teardown like every other
|
||||
// one-shot path. The connect-timeout process.exit(124) above is reviewed
|
||||
// and intentionally unchanged: no engine exists at that point.
|
||||
await finishCliTeardown({ engine });
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -354,42 +355,26 @@ async function main() {
|
||||
|
||||
// Local engine path (unchanged behavior for local installs).
|
||||
const engine = await connectEngine();
|
||||
// v0.41.8.0 (#1247, #1269, #1290): the search / query / get_page
|
||||
// op handlers fire-and-forget `bumpLastRetrievedAt` after returning
|
||||
// results. On PGLite that IIFE keeps Bun's event loop alive past
|
||||
// engine.disconnect(), hanging the CLI at ~95-98% CPU until SIGKILL.
|
||||
// Drain the fire-and-forget set BEFORE disconnect; force-exit only
|
||||
// if the drain itself times out (preserves stderr diagnostic signal
|
||||
// AND guarantees the CLI doesn't re-hang at the disconnect layer).
|
||||
//
|
||||
// Defense-in-depth (adversarial-review C13): `engine.disconnect()` itself
|
||||
// can hang on PGLite (db.close() or releaseLock racing OS-level FS state).
|
||||
// The unref'd hard-exit fallback is armed inside drainThenDisconnect (called
|
||||
// from the `finally` below), so it bounds ONLY the teardown phase (drain +
|
||||
// disconnect) — the same helper every owner-disconnect site uses. It used to
|
||||
// be armed HERE, before the try, which silently killed any op whose BODY ran
|
||||
// past the deadline: on a slow Postgres pooler (6-10s per fresh connection)
|
||||
// a healthy `gbrain search` was force-exited mid-handler with code 0 and
|
||||
// ZERO stdout — an empty "success" indistinguishable from no results. The
|
||||
// exitCode honor (v0.42.20.0) can't help there: a mid-op kill fires before
|
||||
// any error path sets exitCode. Op-body wallclock bounds are the read-scope
|
||||
// withTimeout wrap inside the try below, not this teardown backstop.
|
||||
// Daemons (`serve`) are excluded so they stay alive.
|
||||
// Wallclock bound for READ-scope op handlers. With the hard-deadline timer
|
||||
// correctly scoped to teardown, a genuinely WEDGED read handler (hung pooler
|
||||
// connection mid-query) would otherwise hang the CLI forever — the #1633
|
||||
// zombie class the old (buggy) pre-try timer accidentally bounded at 10s.
|
||||
// 180s sits far above any healthy slow-pooler run (6-10s/connection);
|
||||
// --timeout=Ns overrides. Writes/admin stay unbounded: a long import/embed
|
||||
// must never be killed by a default deadline.
|
||||
// #2084: the teardown contract (bounded drain of every background-work sink,
|
||||
// bounded disconnect, computed-deadline backstop) lives in finishCliTeardown
|
||||
// — see src/core/cli-force-exit.ts for the full design. The hard-deadline
|
||||
// timer arms at TEARDOWN start inside the helper, never before the handler:
|
||||
// the pre-#2084 placement here measured handler + teardown combined, so a
|
||||
// slow-but-healthy query burned the teardown budget (the flat-10s-banner
|
||||
// bug) and any >10s op was force-killed mid-run with exit 0. The explicit
|
||||
// process exit happens once, in the import.meta.main seam at the bottom of
|
||||
// this file — NOT here.
|
||||
|
||||
// v0.42.41.0 (merged): wallclock bound for READ-scope op handlers. With the
|
||||
// teardown backstop correctly scoped to teardown, a genuinely WEDGED read
|
||||
// handler (hung pooler connection mid-query) would otherwise hang the CLI
|
||||
// forever — the #1633 zombie class the old pre-try timer accidentally
|
||||
// bounded at 10s. 180s sits far above any healthy slow-pooler run
|
||||
// (6-10s/connection); --timeout=Ns overrides. Writes/admin stay unbounded:
|
||||
// a long import/embed must never be killed by a default deadline. On
|
||||
// timeout the abandoned handler may hold ref'd sockets — harmless here,
|
||||
// because the import.meta.main seam exits explicitly on every one-shot path.
|
||||
const READ_OP_TIMEOUT_MS = 180_000;
|
||||
// Set when a wallclock bound fired. The abandoned (timed-out but still
|
||||
// running) handler can hold ref'd sockets/timers that keep Bun's event loop
|
||||
// alive after main() returns — so the finally must hard-exit after teardown
|
||||
// on this path, or the timeout print is followed by an immortal process:
|
||||
// the same zombie class, resurrected through the timeout door (adversarial
|
||||
// review finding).
|
||||
let wallclockTimedOut = false;
|
||||
|
||||
try {
|
||||
const { withTimeout, OperationTimeoutError } = await import('./core/timeout.ts');
|
||||
@@ -399,8 +384,10 @@ async function main() {
|
||||
? ''
|
||||
: ` (default ${e.ms}ms; pass --timeout=Ns to override)`;
|
||||
console.error(`${e.label} timed out${hint}.`);
|
||||
setCliExitCode(124);
|
||||
wallclockTimedOut = true;
|
||||
// 124 = timeout convention (matches the read-only dispatch path). Set
|
||||
// through the verdict channel — a raw process.exitCode write is invisible
|
||||
// to the exit seam and PGLite's WASM runtime can scribble over it.
|
||||
setCliExitVerdict(124);
|
||||
};
|
||||
|
||||
// Context build does DB I/O (resolveSourceId) and runs for EVERY op —
|
||||
@@ -416,7 +403,7 @@ async function main() {
|
||||
} catch (e: unknown) {
|
||||
if (e instanceof OperationTimeoutError) {
|
||||
onWallclockTimeout(e);
|
||||
return; // the finally below still drains + disconnects, then exits
|
||||
return; // the finally drains + disconnects; the import.meta.main seam exits
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
@@ -432,7 +419,7 @@ async function main() {
|
||||
} catch (e: unknown) {
|
||||
if (e instanceof OperationTimeoutError) {
|
||||
onWallclockTimeout(e);
|
||||
return; // the finally below still drains + disconnects, then exits
|
||||
return; // the finally drains + disconnects; the import.meta.main seam exits
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
@@ -459,76 +446,12 @@ async function main() {
|
||||
} else {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
}
|
||||
setCliExitCode(1);
|
||||
setCliExitVerdict(1);
|
||||
} finally {
|
||||
// 1s per-sink drain timeout: read paths with no pending work pay the
|
||||
// ~0ms fast path; capture/import that DO enqueue pay up to 1s (+ facts
|
||||
// shutdown grace) while in-flight Haiku finishes.
|
||||
await drainThenDisconnect(engine, { drainTimeoutMs: 1000 });
|
||||
// Wallclock-timeout path (master v0.42.41.0): the ABANDONED handler
|
||||
// (withTimeout races, it does not cancel) can hold ref'd sockets / SDK
|
||||
// retry timers that keep Bun's event loop alive. The entrypoint
|
||||
// flush-exit fires when main() resolves and covers this; the explicit
|
||||
// call here is belt-and-braces in case a future caller invokes this op
|
||||
// path outside the guarded entrypoint.
|
||||
if (wallclockTimedOut) {
|
||||
void flushStdoutThenExit(getCliExitCode());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.43 (#2084, closes the TODOS P3 drain-hoist) — THE owner-disconnect for
|
||||
* every CLI exit path. One helper, all sites, so a future teardown bug has
|
||||
* one place to be wrong in.
|
||||
*
|
||||
* drainThenDisconnect(engine)
|
||||
* ├── arm unref'd 10s hard-deadline (HUNG-teardown backstop — PGLite
|
||||
* │ db.close()/releaseLock can hang on OS-level FS state; the
|
||||
* │ entrypoint flush-exit can never fire if main() never resolves)
|
||||
* ├── drain background-work registry (facts → last-retrieved →
|
||||
* │ search-cache → eval-capture → volunteer-events), so a PGLite
|
||||
* │ db.close() can't race in-flight work into the re-pump
|
||||
* │ busy-loop (#1762)
|
||||
* └── engine.disconnect() (best-effort), then clear the deadline
|
||||
*
|
||||
* The 10s timer is armed HERE — around the teardown window only — never
|
||||
* before the op handler (master's v0.42.41.0 triage wave fixed the same
|
||||
* pre-armed-timer bug independently: it force-killed any op slower than
|
||||
* 10s). On the happy path the timer never fires: main() resolves and the
|
||||
* entrypoint's flushStdoutThenExit ends the process deliberately (#2084's
|
||||
* fix for lingering embedding/PgBouncer sockets riding the backstop).
|
||||
*/
|
||||
const DISCONNECT_HARD_DEADLINE_MS = 10_000;
|
||||
export async function drainThenDisconnect(
|
||||
engine: BrainEngine,
|
||||
opts?: { drainTimeoutMs?: number },
|
||||
): Promise<void> {
|
||||
let deadlineTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
if (shouldForceExitAfterMain()) {
|
||||
deadlineTimer = setTimeout(() => {
|
||||
console.warn(
|
||||
`[cli] engine.disconnect() did not return within ${DISCONNECT_HARD_DEADLINE_MS}ms — force-exiting`,
|
||||
);
|
||||
// Honor an exit code an errored op already set — a bare process.exit(0)
|
||||
// here would mask a failed op as success if the drain/disconnect hangs.
|
||||
process.exit(getCliExitCode());
|
||||
}, DISCONNECT_HARD_DEADLINE_MS);
|
||||
// unref so the timer itself doesn't keep the event loop alive — only
|
||||
// the actual pending work (PGLite WASM handle) does.
|
||||
deadlineTimer.unref?.();
|
||||
}
|
||||
try {
|
||||
await drainAllBackgroundWorkForCliExit(
|
||||
opts?.drainTimeoutMs !== undefined ? { timeoutMs: opts.drainTimeoutMs } : undefined,
|
||||
);
|
||||
try {
|
||||
await engine.disconnect();
|
||||
} catch {
|
||||
/* best-effort — kernel reclaims sockets on exit (timeout.ts doctrine) */
|
||||
}
|
||||
} finally {
|
||||
if (deadlineTimer) clearTimeout(deadlineTimer);
|
||||
// 1s per-sink drain budget: read paths with no pending work pay the ~0ms
|
||||
// fast path; capture/import that DO enqueue pay up to 1s (+ facts shutdown
|
||||
// grace) while in-flight Haiku finishes (#1762 drain-before-disconnect).
|
||||
await finishCliTeardown({ engine, drainTimeoutMs: 1000 });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1260,15 +1183,13 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
if (command === 'friction') {
|
||||
const { runFriction } = await import('./commands/friction.ts');
|
||||
// v0.43 (#2084 inner-exit sweep): exitCode + return instead of a
|
||||
// mid-handler process.exit — flows through the entrypoint flush-exit
|
||||
// so buffered stdout is never truncated.
|
||||
setCliExitCode(runFriction(args));
|
||||
// #2084 inner-exit sweep: verdict + return so teardown + the flush seam run.
|
||||
setCliExitVerdict(runFriction(args));
|
||||
return;
|
||||
}
|
||||
if (command === 'claw-test') {
|
||||
const { runClawTest } = await import('./commands/claw-test.ts');
|
||||
setCliExitCode(await runClawTest(args));
|
||||
setCliExitVerdict(await runClawTest(args));
|
||||
return;
|
||||
}
|
||||
if (command === 'report') {
|
||||
@@ -1315,13 +1236,13 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
if (args.includes('--remediation-plan')) {
|
||||
const { runRemediationPlan } = await import('./commands/doctor.ts');
|
||||
const eng = await connectEngine();
|
||||
try { await runRemediationPlan(eng, args); } finally { await drainThenDisconnect(eng); }
|
||||
try { await runRemediationPlan(eng, args); } finally { await finishCliTeardown({ engine: eng }); }
|
||||
return;
|
||||
}
|
||||
if (args.includes('--remediate')) {
|
||||
const { runRemediate } = await import('./commands/doctor.ts');
|
||||
const eng = await connectEngine();
|
||||
try { await runRemediate(eng, args); } finally { await drainThenDisconnect(eng); }
|
||||
try { await runRemediate(eng, args); } finally { await finishCliTeardown({ engine: eng }); }
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1334,13 +1255,21 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
// "user chose --fast while config is present".
|
||||
await runDoctor(null, args, getDbUrlSource());
|
||||
} else {
|
||||
// #2084: both failure kinds (connect throw, runDoctor(eng) throw) still
|
||||
// fall back to filesystem-only checks — identical to the prior shape.
|
||||
// The finally closes the gap where a runDoctor(eng) throw used to skip
|
||||
// the in-try disconnect. NOTE: runDoctor normally calls process.exit
|
||||
// itself, which preempts this finally — in-command exit sites bypassing
|
||||
// teardown are a pre-existing class, tracked as a TODOS.md follow-up.
|
||||
let eng: BrainEngine | null = null;
|
||||
try {
|
||||
const eng = await connectEngine();
|
||||
eng = await connectEngine();
|
||||
await runDoctor(eng, args);
|
||||
await drainThenDisconnect(eng);
|
||||
} catch {
|
||||
// DB unavailable — still run filesystem checks
|
||||
await runDoctor(null, args, getDbUrlSource());
|
||||
} finally {
|
||||
if (eng) await finishCliTeardown({ engine: eng });
|
||||
}
|
||||
}
|
||||
return;
|
||||
@@ -1354,7 +1283,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
try {
|
||||
await runZeSwitch(args, eng);
|
||||
} finally {
|
||||
await drainThenDisconnect(eng);
|
||||
await finishCliTeardown({ engine: eng });
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -1370,7 +1299,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
execSync(`bash "${scriptPath}"`, { stdio: 'inherit', env: { ...process.env } });
|
||||
} catch (e: any) {
|
||||
// Non-zero exit = some tests failed (exit code = failure count)
|
||||
setCliExitCode(e.status ?? 1);
|
||||
setCliExitVerdict(e.status ?? 1);
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -1399,14 +1328,15 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runDream(eng, args);
|
||||
} finally {
|
||||
// #1471 invariant tripwire (the dream-cycle owner): `eng` created the
|
||||
// module singleton (first module connector) and is disconnected LAST,
|
||||
// module singleton (first module connector) and is torn down LAST,
|
||||
// here, after the whole cycle. The ownership fix relies on this owner's
|
||||
// lifetime strictly dominating every borrower (lint/doctor probe engines
|
||||
// created mid-cycle). Do NOT disconnect `eng` before runDream returns, or
|
||||
// created mid-cycle). Do NOT tear down `eng` before runDream returns, or
|
||||
// a borrower could outlive the owner and lose the shared singleton.
|
||||
// #2084 drain hoist: dream's cycle enqueues facts/last-retrieved
|
||||
// writes — drain them against the live owner engine before teardown.
|
||||
if (eng) await drainThenDisconnect(eng);
|
||||
// #2084: routed through the shared bounded teardown — dream runs as an
|
||||
// overnight cron, where a lingering-socket hang is a silent zombie
|
||||
// (closes the TODOS.md drain-before-owner-disconnect item).
|
||||
if (eng) await finishCliTeardown({ engine: eng });
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -1418,7 +1348,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
// The handler self-configures the AI gateway from loadConfig() + process.env.
|
||||
if (command === 'eval' && args[0] === 'cross-modal') {
|
||||
const { runEvalCrossModal } = await import('./commands/eval-cross-modal.ts');
|
||||
setCliExitCode(await runEvalCrossModal(args.slice(1)));
|
||||
setCliExitVerdict(await runEvalCrossModal(args.slice(1)));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1430,7 +1360,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
// engine-required path below.
|
||||
if (command === 'eval' && args[0] === 'takes-quality' && args[1] === 'replay') {
|
||||
const { runReplayNoBrain } = await import('./commands/eval-takes-quality.ts');
|
||||
setCliExitCode(await runReplayNoBrain(args.slice(2)));
|
||||
setCliExitVerdict(await runReplayNoBrain(args.slice(2)));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1463,7 +1393,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
// gate runs on machines with no `~/.gbrain/config.json`.
|
||||
if (command === 'eval' && args[0] === 'conversation-parser') {
|
||||
const { runEvalConversationParser } = await import('./commands/eval-conversation-parser.ts');
|
||||
setCliExitCode(await runEvalConversationParser(args.slice(1)));
|
||||
setCliExitVerdict(await runEvalConversationParser(args.slice(1)));
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -1492,7 +1422,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
const cfgPre = loadConfig();
|
||||
if (isThinClient(cfgPre)) {
|
||||
const { runEvalWhoknows } = await import('./commands/eval-whoknows.ts');
|
||||
setCliExitCode(await runEvalWhoknows(null, args.slice(1)));
|
||||
setCliExitVerdict(await runEvalWhoknows(null, args.slice(1)));
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -1507,7 +1437,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
if (cfgPre && isThinClient(cfgPre)) {
|
||||
const { runStatus } = await import('./commands/status.ts');
|
||||
const result = await runStatus(null, args);
|
||||
setCliExitCode(result.exitCode);
|
||||
setCliExitVerdict(result.exitCode);
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -1587,9 +1517,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
throw e;
|
||||
} finally {
|
||||
// #2084 drain hoist: `gbrain search` writes the query cache + bumps
|
||||
// last_retrieved_at fire-and-forget — drain before disconnect.
|
||||
await drainThenDisconnect(engine);
|
||||
await finishCliTeardown({ engine });
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -1641,7 +1569,7 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
// so wrappers (sync, CI scripts, `&& gbrain doctor`) propagate.
|
||||
const importResult = await runImport(engine, args);
|
||||
if (importResult.errors > 0) {
|
||||
setCliExitCode(1);
|
||||
setCliExitVerdict(1);
|
||||
}
|
||||
break;
|
||||
}
|
||||
@@ -1823,10 +1751,8 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
case 'status': {
|
||||
const { runStatus } = await import('./commands/status.ts');
|
||||
const result = await runStatus(engine, args);
|
||||
// v0.43 (#2084 inner-exit sweep): a mid-switch process.exit skipped
|
||||
// the finally's drain + disconnect entirely. exitCode + break flows
|
||||
// through both, then the entrypoint flush-exit ends the process.
|
||||
setCliExitCode(result.exitCode);
|
||||
// #2084 inner-exit sweep: a mid-switch exit skips the finally teardown.
|
||||
setCliExitVerdict(result.exitCode);
|
||||
break;
|
||||
}
|
||||
// v0.38 — Capture: single human-facing entrypoint for ingestion.
|
||||
@@ -1999,7 +1925,8 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
case 'watch': {
|
||||
// v0.43 (#2095): push-based context transport. Blocks in the stdin
|
||||
// iteration (interactive stays alive; piped exits at EOF), then the
|
||||
// finally below drains the volunteer-events sink with everything else.
|
||||
// finally below runs finishCliTeardown (volunteer events drain with
|
||||
// every other sink) and the import.meta.main seam flush-exits.
|
||||
const { runWatch } = await import('./commands/watch.ts');
|
||||
await runWatch(engine, args);
|
||||
break;
|
||||
@@ -2074,20 +2001,16 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
}
|
||||
} finally {
|
||||
syncWatchdog?.dispose(); // #1633: tear down the hard-deadline watchdog on clean exit
|
||||
// v0.42.20.0 (#1762) — the CLI_ONLY path (which owns `gbrain capture`)
|
||||
// lacked the op-dispatch drain-before-disconnect contract. `put_page` fires
|
||||
// a fire-and-forget facts:absorb job AFTER printing the receipt; on a
|
||||
// multi-chunk page that job is in flight when this finally tears the engine
|
||||
// down, and `engine.disconnect()` nulling PGLite's _db mid-job spins
|
||||
// db.close() into a 100%-CPU busy-loop that pins the single-writer lock.
|
||||
// Drain every background-work sink first (facts shutdown() abort cancels a
|
||||
// hung Haiku), THEN disconnect. The drain-before-disconnect is the causal
|
||||
// fix; the force-exit defense below is secondary (it CANNOT preempt a WASM
|
||||
// busy-loop on a pinned JS thread — that's exactly why the drain matters).
|
||||
// #1471: this is also the fall-through OWNER-disconnect — the owner is torn
|
||||
// down LAST (after the drain), so module-singleton borrowers never outlive it.
|
||||
// #2084 — the CLI_ONLY fall-through teardown (drain every background-work
|
||||
// sink, THEN disconnect, under a computed-deadline backstop) lives in
|
||||
// finishCliTeardown. `gbrain capture`'s fire-and-forget facts:absorb job
|
||||
// gets its drain window before PGLite's db.close() can race it into the
|
||||
// re-pump busy-loop (#1762). #1471: this is also the fall-through
|
||||
// OWNER-disconnect — the owner is torn down LAST (after the drain), so
|
||||
// module-singleton borrowers never outlive it. `serve` skips teardown
|
||||
// entirely: the daemon owns its lifecycle.
|
||||
if (command !== 'serve') {
|
||||
await drainThenDisconnect(engine);
|
||||
await finishCliTeardown({ engine });
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2401,21 +2324,25 @@ Run gbrain <command> --help for command-specific help.
|
||||
// Only auto-run when invoked as the entry point (the compiled binary or
|
||||
// `bun src/cli.ts`). Guarded so tests can import cliAliases / printOpHelp
|
||||
// without triggering argv parsing + main(). v114 (#1941).
|
||||
//
|
||||
// #2084 — the ONE process-exit seam for one-shot commands. Every teardown site
|
||||
// routes through finishCliTeardown (which returns); the exit itself happens
|
||||
// here, after main() settles, so the CLI never waits on Bun's event loop to
|
||||
// drain (stuck PgBouncer sockets kept it alive — endPoolBounded races PAST a
|
||||
// stuck pool.end() by design). flushThenExit fences stdout/stderr and holds a
|
||||
// short aliveness grace so piped output is delivered before exit (#1959).
|
||||
// Daemons (`serve`) are excluded by shouldForceExitAfterMain and keep the
|
||||
// pre-#2084 behavior: main() resolves and the server's own work keeps the
|
||||
// process alive. A fatal error still exits 1 for every command, daemons
|
||||
// included (matches the prior unconditional process.exit(1) on rejection).
|
||||
if (import.meta.main) {
|
||||
// v0.43 (#2084): exit DELIBERATELY when main() resolves. Lingering sockets
|
||||
// (embedding fetch keep-alive, PgBouncer txn-mode sockets endPoolBounded
|
||||
// raced past) keep Bun's event loop alive after bounded teardown resolves —
|
||||
// pre-fix, every `gbrain query` paid a flat 10s tax riding the hard-deadline
|
||||
// backstop. flushStdoutThenExit drains stdout/stderr first (incident #1959:
|
||||
// a force-exit truncated piped stdout). Daemons (serve; interactive watch)
|
||||
// are excluded by shouldForceExitAfterMain and keep the event loop.
|
||||
main().then(
|
||||
() => {
|
||||
if (shouldForceExitAfterMain()) void flushStdoutThenExit(getCliExitCode());
|
||||
if (shouldForceExitAfterMain()) flushThenExit(currentExitCode());
|
||||
},
|
||||
(e) => {
|
||||
console.error(e instanceof Error ? (e.message || String(e)) : String(e));
|
||||
void flushStdoutThenExit(1);
|
||||
console.error(e.message || e);
|
||||
flushThenExit(1);
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, writeFileSync, mkdirSync, appendFileSync, utimesSync, unlinkSync } from 'fs';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import { join } from 'path';
|
||||
import { execSync } from 'child_process';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
@@ -37,7 +38,6 @@ import { logSelfUpgrade } from '../core/audit/self-upgrade-audit.ts';
|
||||
import { detectInstallMethod } from './upgrade.ts';
|
||||
import { evaluateQuietHours } from '../core/minions/quiet-hours.ts';
|
||||
import { inspectLock } from '../core/db-lock.ts';
|
||||
import { setCliExitCode } from '../core/cli-force-exit.ts';
|
||||
|
||||
/**
|
||||
* v0.37.7.0 #1162 — classify autopilot reconnect-loop errors.
|
||||
@@ -548,7 +548,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
`Exiting so launchd ThrottleInterval can apply backoff.`,
|
||||
);
|
||||
stopping = true;
|
||||
setCliExitCode(1);
|
||||
setCliExitVerdict(1);
|
||||
break;
|
||||
}
|
||||
if (autopilotReconnectFails >= AUTOPILOT_MAX_RECONNECT_FAILS) {
|
||||
@@ -557,7 +557,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
|
||||
`Last error: ${(e as Error).message ?? 'unknown'}. Exiting.`,
|
||||
);
|
||||
stopping = true;
|
||||
setCliExitCode(1);
|
||||
setCliExitVerdict(1);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import {
|
||||
runBrainstorm,
|
||||
formatBrainstormMarkdown,
|
||||
@@ -27,7 +28,6 @@ import { serializeMarkdown } from '../core/markdown.ts';
|
||||
import { importFromContent } from '../core/import-file.ts';
|
||||
import { writePageThrough, type WriteThroughResult } from '../core/write-through.ts';
|
||||
import { randomBytes } from 'crypto';
|
||||
import { setCliExitCode } from '../core/cli-force-exit.ts';
|
||||
|
||||
export interface BrainstormCliArgs {
|
||||
question?: string;
|
||||
@@ -323,7 +323,7 @@ async function runBrainstormCli(
|
||||
const msg = formatSaveOutcome(outcome, { profileLabel: profile.label, slug });
|
||||
if (msg.stdout) console.log(msg.stdout);
|
||||
for (const line of msg.stderr) console.error(line);
|
||||
if (msg.exitCode) setCliExitCode(msg.exitCode);
|
||||
if (msg.exitCode) setCliExitVerdict(msg.exitCode);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import * as db from '../core/db.ts';
|
||||
import { setCliExitCode } from '../core/cli-force-exit.ts';
|
||||
import { LATEST_VERSION, getIdleBlockers } from '../core/migrate.ts';
|
||||
import { checkResolvable } from '../core/check-resolvable.ts';
|
||||
import { autoFixDryViolations, type AutoFixReport, type FixOutcome } from '../core/dry-fix.ts';
|
||||
@@ -7226,12 +7226,10 @@ export async function runDoctor(
|
||||
} catch { /* best-effort */ }
|
||||
}
|
||||
|
||||
// Use the owned exit verdict instead of process.exit() so cleanup handlers
|
||||
// (e.g. Bun unload events, open database connections) still run before the
|
||||
// process terminates — and so the deliberate flush-exit (#2084) reports
|
||||
// doctor's verdict: a RAW process.exitCode write is silently zeroed by
|
||||
// getCliExitCode() at the entrypoint (the PGLite-pollution defense).
|
||||
setCliExitCode(hasFail ? 1 : 0);
|
||||
// Use process.exitCode instead of process.exit() so cleanup handlers
|
||||
// (e.g. Bun unload events, open database connections) still run before
|
||||
// the process terminates. process.exit() is a hard kill that bypasses them.
|
||||
setCliExitVerdict(hasFail ? 1 : 0);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -28,8 +28,8 @@
|
||||
* pagination). FS-source preserves the original v0.10.1 walker behavior.
|
||||
*/
|
||||
|
||||
import { setCliExitCode } from '../core/cli-force-exit.ts';
|
||||
import { readFileSync, readdirSync, lstatSync, existsSync } from 'fs';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import { join, relative, dirname } from 'path';
|
||||
import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from '../core/engine.ts';
|
||||
import type { PageType } from '../core/types.ts';
|
||||
@@ -865,7 +865,7 @@ Status (v0.42):
|
||||
(r.first_batch_error ? ` (first error: ${r.first_batch_error})` : '') +
|
||||
` — timeline is incomplete.`,
|
||||
);
|
||||
setCliExitCode(1);
|
||||
setCliExitVerdict(1);
|
||||
}
|
||||
} else if (byMention || ner) {
|
||||
// v0.41.18.0 (T7): combined --by-mention + --ner walk shares one
|
||||
|
||||
@@ -16,6 +16,7 @@
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync, lstatSync, readdirSync } from 'fs';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import { join, relative, resolve } from 'path';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { loadConfig, toEngineConfig } from '../core/config.ts';
|
||||
@@ -30,7 +31,6 @@ import {
|
||||
type AuditFix,
|
||||
} from '../core/brain-writer.ts';
|
||||
import { isSyncable, pruneDir, slugifyPath } from '../core/sync.ts';
|
||||
import { setCliExitCode } from '../core/cli-force-exit.ts';
|
||||
|
||||
export async function runFrontmatter(args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
@@ -64,7 +64,7 @@ export async function runFrontmatter(args: string[]): Promise<void> {
|
||||
}
|
||||
console.error(`Unknown frontmatter subcommand: ${sub}\n`);
|
||||
printHelp();
|
||||
setCliExitCode(1);
|
||||
setCliExitVerdict(1);
|
||||
}
|
||||
|
||||
async function connectEngineForAudit(): Promise<BrainEngine> {
|
||||
@@ -165,14 +165,14 @@ async function runValidate(rest: string[]): Promise<void> {
|
||||
}
|
||||
if (!target) {
|
||||
console.error('error: gbrain frontmatter validate requires a <path> argument');
|
||||
setCliExitCode(1);
|
||||
setCliExitVerdict(1);
|
||||
return;
|
||||
}
|
||||
|
||||
const resolved = resolve(target);
|
||||
if (!existsSync(resolved)) {
|
||||
console.error(`error: path not found: ${target}`);
|
||||
setCliExitCode(1);
|
||||
setCliExitVerdict(1);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -243,7 +243,7 @@ async function runValidate(rest: string[]): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
setCliExitCode(totalErrors > 0 && !flags.fix ? 1 : 0);
|
||||
setCliExitVerdict(totalErrors > 0 && !flags.fix ? 1 : 0);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -379,7 +379,7 @@ async function runGenerate(args: string[]): Promise<void> {
|
||||
if (!targetPath) {
|
||||
console.error('error: gbrain frontmatter generate requires a <path> argument');
|
||||
console.error('usage: gbrain frontmatter generate <path> [--fix] [--dry-run] [--json]');
|
||||
setCliExitCode(1);
|
||||
setCliExitVerdict(1);
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
import { MARKDOWN_CHUNKER_VERSION } from '../core/chunkers/recursive.ts';
|
||||
import { importFromContent, importFromFile } from '../core/import-file.ts';
|
||||
import { serializeMarkdown } from '../core/markdown.ts';
|
||||
@@ -29,7 +30,6 @@ import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { existsSync } from 'fs';
|
||||
import { resolve } from 'path';
|
||||
import { setCliExitCode } from '../core/cli-force-exit.ts';
|
||||
// v0.41.15.0 (T10, D9): per-batch parallel workers.
|
||||
import { runSlidingPool } from '../core/worker-pool.ts';
|
||||
import { resolveWorkersWithClamp } from '../core/sync-concurrency.ts';
|
||||
@@ -151,7 +151,7 @@ export async function runReindex(engine: BrainEngine, args: string[]): Promise<R
|
||||
} else {
|
||||
process.stderr.write('Usage: gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--repo PATH]\n');
|
||||
}
|
||||
setCliExitCode(2);
|
||||
setCliExitVerdict(2);
|
||||
return { pending: 0, reindexed: 0, skipped: 0, failed: 0, dryRun: !!opts.dryRun, chunkerVersion: MARKDOWN_CHUNKER_VERSION };
|
||||
}
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import { setCliExitCode } from '../core/cli-force-exit.ts';
|
||||
import { setCliExitVerdict } from '../core/cli-force-exit.ts';
|
||||
|
||||
interface RunOpts {
|
||||
days?: number;
|
||||
@@ -64,7 +64,7 @@ export async function runTranscripts(engine: BrainEngine, args: string[]): Promi
|
||||
const sub = args[0];
|
||||
if (sub !== 'recent') {
|
||||
console.log(HELP);
|
||||
if (sub && sub !== '--help' && sub !== '-h') setCliExitCode(2);
|
||||
if (sub && sub !== '--help' && sub !== '-h') setCliExitVerdict(2);
|
||||
return;
|
||||
}
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
* Lifecycle: BLOCKS in the stdin iteration (like `gbrain jobs work`), so an
|
||||
* interactive TTY session stays alive until Ctrl-C / Ctrl-D and piped input
|
||||
* exits at EOF. Either way the handler RETURNS, the CLI_ONLY finally runs
|
||||
* drainThenDisconnect (volunteer events bank before teardown), and the
|
||||
* finishCliTeardown (volunteer events bank before teardown), and the
|
||||
* entrypoint flush-exit ends the process deliberately — which is exactly why
|
||||
* `watch` is NOT in DAEMON_COMMANDS: it never returns from main() while work
|
||||
* is still running. SIGINT closes the stream and flows through the same
|
||||
|
||||
@@ -64,6 +64,16 @@ export function registerBackgroundWorkDrainer(d: BackgroundWorkDrainer): void {
|
||||
drainers.set(d.name, d);
|
||||
}
|
||||
|
||||
/**
|
||||
* Number of registered sinks. Used by `finishCliTeardown` (cli-force-exit.ts)
|
||||
* to COMPUTE its backstop deadline from the bounds it guards — a 5th sink
|
||||
* registering automatically widens the deadline instead of silently making
|
||||
* the worst-case bounded drain exceed a static number (#2084 eng-review D9).
|
||||
*/
|
||||
export function backgroundWorkSinkCount(): number {
|
||||
return drainers.size;
|
||||
}
|
||||
|
||||
/**
|
||||
* Test seam — registers a drainer and returns an unregister handle. Preferred
|
||||
* over a blunt reset: real sink modules register at import time and won't re-run
|
||||
|
||||
+302
-101
@@ -1,131 +1,332 @@
|
||||
/**
|
||||
* v0.41.8.0 — narrow force-exit gate for the cli.ts op-dispatch finally.
|
||||
* One-shot CLI exit + teardown contract (#2084, supersedes the narrower
|
||||
* v0.41.8.0 drain-timeout-only force-exit).
|
||||
*
|
||||
* The cli.ts caller fires `process.exit(0)` ONLY when:
|
||||
* 1. The op-dispatch drain timed out (drainResult.outcome === 'timeout')
|
||||
* 2. AND this function returns true (i.e. the command is NOT a daemon)
|
||||
* The CLI must never rely on Bun's event loop draining to exit: on PgBouncer
|
||||
* transaction-mode, `endPoolBounded` (db.ts) deliberately races PAST a stuck
|
||||
* `pool.end()`, so the promise resolves while the stuck sockets stay open and
|
||||
* keep the loop alive (#2084's flat 10s teardown tax). Per the doctrine in
|
||||
* timeout.ts, `process.exit` is the real resource-release mechanism for
|
||||
* one-shot commands — the kernel reclaims sockets.
|
||||
*
|
||||
* The function lives in its own module — not inline in cli.ts — so tests
|
||||
* can import + drive it without triggering cli.ts's top-level main() side
|
||||
* effect (cli.ts is a script entrypoint). Mirrors PR #1337's
|
||||
* `shouldForceExitAfterMain` guard, but narrower in scope: this wave
|
||||
* only force-exits after the drain timed out, NOT unconditionally for
|
||||
* every non-serve command.
|
||||
* The contract is a PAIR (documented together in KEY_FILES.md):
|
||||
*
|
||||
* Daemon list is just `serve` (stdio + HTTP): it RETURNS from its handler
|
||||
* while the event loop carries the server. Every other long-runner —
|
||||
* `jobs work`, `autopilot`, and v0.43's `gbrain watch` (#2095) — BLOCKS
|
||||
* inside its awaited handler until done (watch blocks in the stdin
|
||||
* iteration: interactive stays alive until Ctrl-C/Ctrl-D, piped input ends
|
||||
* at EOF), so when main() resolves the work is over and the deliberate
|
||||
* flush-exit is correct. Add a command here ONLY if it returns early and
|
||||
* leaves the event loop holding the daemon.
|
||||
* op handler returns / throws (catch sets the verdict: setCliExitVerdict(1))
|
||||
* │
|
||||
* ▼ (per call site, in its finally — nine sites in cli.ts)
|
||||
* finishCliTeardown({ engine, drainTimeoutMs? }) ← teardown ONLY, never exits*
|
||||
* │
|
||||
* ├─ arm ref'd backstop timer; deadline COMPUTED from the bounds
|
||||
* │ it guards (sinks × drainTimeoutMs + facts-abort grace
|
||||
* │ + 2 × pool-end bound + slack, floor 10s). The backstop fires
|
||||
* │ ONLY if a component violated its own bound; on fire it prints
|
||||
* │ a truthful banner and *flushThenExit(currentExitCode()).
|
||||
* │ GBRAIN_TEARDOWN_DEADLINE_MS overrides (incident escape hatch).
|
||||
* ▼
|
||||
* drain background sinks (bounded per-sink; CLI-exit-only contract)
|
||||
* ▼
|
||||
* engine.disconnect() — a throw is warned + swallowed: the exit code
|
||||
* │ reports the OPERATION, not the cleanup
|
||||
* ▼
|
||||
* clear backstop, RETURN to caller
|
||||
* │
|
||||
* ▼ (exactly ONE place: cli.ts import.meta.main main().then/catch)
|
||||
* shouldForceExitAfterMain() && flushThenExit(currentExitCode())
|
||||
* — fence stdout+stderr (write-fence raced with an unref'd guard,
|
||||
* EPIPE-safe), hold a short REF'D aliveness grace for non-TTY stdio
|
||||
* (Bun only delivers queued pipe writes while alive), then
|
||||
* process.exit. Stuck sockets become irrelevant.
|
||||
*
|
||||
* The hard-deadline timer is armed at TEARDOWN start, never before the op
|
||||
* handler — a slow-but-healthy handler must not erode the teardown budget
|
||||
* (the pre-#2084 bug force-killed any >10s op mid-run with exit 0 and
|
||||
* truncated output).
|
||||
*
|
||||
* Daemons: `serve` is excluded at both layers — its command never reaches a
|
||||
* finishCliTeardown call site, and the central exit is gated by
|
||||
* `shouldForceExitAfterMain`. The helper itself has NO daemon flag: the drain
|
||||
* it runs is CLI-exit-only (it can permanently shut down process-level sinks),
|
||||
* so a long-lived process must simply never call it.
|
||||
*
|
||||
* This module stays importable without cli.ts side effects so tests can drive
|
||||
* every path directly (cli.ts is a script entrypoint).
|
||||
*/
|
||||
|
||||
import { drainAllBackgroundWorkForCliExit, backgroundWorkSinkCount } from './background-work.ts';
|
||||
import { POOL_END_TIMEOUT_SECONDS } from './db.ts';
|
||||
import { parseGlobalFlags } from './cli-options.ts';
|
||||
|
||||
const DAEMON_COMMANDS: ReadonlySet<string> = new Set(['serve']);
|
||||
|
||||
export function shouldForceExitAfterMain(
|
||||
argv: string[] = process.argv.slice(2),
|
||||
): boolean {
|
||||
const command = argv.find((arg) => !arg.startsWith('-'));
|
||||
// Resolve the command the same way main() does — parseGlobalFlags strips
|
||||
// global flags INCLUDING space-separated values (`--timeout 30s`), so the
|
||||
// command here always matches the dispatched one. The old first-non-dash
|
||||
// heuristic saw `30s` as the command for `gbrain --timeout 30s serve` and
|
||||
// (post-#2084, where this gates an unconditional process.exit) would have
|
||||
// killed the daemon ~250ms after boot. Cross-model adversarial finding.
|
||||
let command: string | undefined;
|
||||
try {
|
||||
command = parseGlobalFlags(argv).rest[0];
|
||||
} catch {
|
||||
command = argv.find((arg) => !arg.startsWith('-'));
|
||||
}
|
||||
if (!command) return true;
|
||||
return !DAEMON_COMMANDS.has(command);
|
||||
}
|
||||
|
||||
/** Floor for the computed backstop deadline (the historical hard deadline). */
|
||||
export const TEARDOWN_DEADLINE_FLOOR_MS = 10_000;
|
||||
/** Allowance for the facts sink's awaited abort() (shutdown of an in-flight job). */
|
||||
const FACTS_ABORT_GRACE_MS = 2_000;
|
||||
/** Headroom over the sum of the guarded bounds so timer jitter can't false-fire. */
|
||||
const TEARDOWN_SLACK_MS = 2_000;
|
||||
/** Max wait for the stdio flush fence before exiting anyway (blocked pipe). */
|
||||
const FLUSH_GUARD_MS = 2_000;
|
||||
/**
|
||||
* v0.43 (#2084) — gbrain owns its exit verdict.
|
||||
*
|
||||
* `process.exitCode` is NOT trustworthy in this process: PGLite's Emscripten
|
||||
* runtime writes the WASM backend's proc_exit status into it (initdb at
|
||||
* create-time, the postmaster at close-time — see `exitCode=status` in
|
||||
* pglite's dist), and those writes land asynchronously outside any
|
||||
* snapshot/restore window. Pre-#2084 the success path never read
|
||||
* process.exitCode so the pollution was invisible; the deliberate flush-exit
|
||||
* MUST NOT propagate it (a clean `gbrain apply-migrations` was exiting 99).
|
||||
*
|
||||
* So gbrain records its own verdict here: every gbrain-owned exit-code
|
||||
* assignment routes through `setCliExitCode()` (which also mirrors to
|
||||
* process.exitCode for anything else that reads it), and the exit paths read
|
||||
* `getCliExitCode()` — never ambient process.exitCode.
|
||||
* Aliveness grace between the fence and process.exit when stdio is NOT a TTY.
|
||||
* Empirically verified (#2084 probes): Bun's process.stdout queues pipe writes
|
||||
* in a native writer that only pushes to the fd on event-loop turns WHILE THE
|
||||
* PROCESS IS ALIVE — process.exit discards the queue, natural event-loop exit
|
||||
* discards it too, and no API reaches it (write callbacks fire on accept, not
|
||||
* delivery; writableLength/bytesWritten read 0 throughout;
|
||||
* Bun.stdout.writer().flush() is a different writer; fs.writeSync(1) is also
|
||||
* queued). Staying alive briefly is the ONLY flush. TTY writes are synchronous
|
||||
* — no grace needed there.
|
||||
*/
|
||||
let _cliExitCode: number | undefined;
|
||||
const FLUSH_GRACE_PIPE_MS = 250;
|
||||
|
||||
export function setCliExitCode(code: number): void {
|
||||
_cliExitCode = code;
|
||||
process.exitCode = code;
|
||||
/**
|
||||
* Resolve the non-TTY aliveness grace: `GBRAIN_FLUSH_GRACE_MS` env override
|
||||
* (incident/batch escape hatch, same env-only pattern as
|
||||
* GBRAIN_TEARDOWN_DEADLINE_MS) over the 250ms default. Consumers piping LARGE
|
||||
* payloads into slow readers (a reader that attaches later than the grace
|
||||
* loses the tail — Bun gives no delivery signal to wait on) can raise it;
|
||||
* high-frequency agent loops capturing to files can lower it.
|
||||
*/
|
||||
function resolveFlushGraceMs(): number {
|
||||
const env = Number(process.env.GBRAIN_FLUSH_GRACE_MS);
|
||||
if (Number.isFinite(env) && env >= 0) return env;
|
||||
return FLUSH_GRACE_PIPE_MS;
|
||||
}
|
||||
/** Default per-sink drain budget (matches drainAllBackgroundWorkForCliExit). */
|
||||
const DEFAULT_DRAIN_TIMEOUT_MS = 2_000;
|
||||
|
||||
export function getCliExitCode(): number {
|
||||
return _cliExitCode ?? 0;
|
||||
}
|
||||
|
||||
/** Test seam — reset the recorded verdict between cases. */
|
||||
export function _resetCliExitCodeForTests(): void {
|
||||
_cliExitCode = undefined;
|
||||
/**
|
||||
* Backstop deadline for drain + disconnect COMBINED, computed from the bounds
|
||||
* it guards so it fires only when a component violated its own bound (#2084
|
||||
* eng-review D9 — a static 10s fired on healthy-but-slow bounded teardown:
|
||||
* 4 sinks × 2s + facts grace + 2 × ~2.5s pool ends ≈ 13s).
|
||||
* `GBRAIN_TEARDOWN_DEADLINE_MS` overrides the formula (incident escape hatch,
|
||||
* same env-only pattern as the GBRAIN_SYNC_* knobs).
|
||||
*/
|
||||
export function computeTeardownDeadlineMs(opts: {
|
||||
sinkCount: number;
|
||||
drainTimeoutMs: number;
|
||||
}): number {
|
||||
const env = Number(process.env.GBRAIN_TEARDOWN_DEADLINE_MS);
|
||||
if (Number.isFinite(env) && env > 0) return env;
|
||||
// +500 mirrors endPoolBounded's slack over the postgres.js hint (db.ts);
|
||||
// ×2 budgets the worst case of two sequential pool ends (direct + read).
|
||||
const poolEndBoundMs = POOL_END_TIMEOUT_SECONDS * 1000 + 500;
|
||||
const computed =
|
||||
opts.sinkCount * opts.drainTimeoutMs +
|
||||
FACTS_ABORT_GRACE_MS +
|
||||
2 * poolEndBoundMs +
|
||||
TEARDOWN_SLACK_MS;
|
||||
return Math.max(TEARDOWN_DEADLINE_FLOOR_MS, computed);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.43 (#2084) — deliberate exit after bounded teardown.
|
||||
*
|
||||
* Lingering sockets (embedding-provider fetch keep-alive, PgBouncer txn-mode
|
||||
* sockets `endPoolBounded` raced past) keep Bun's event loop alive after
|
||||
* teardown RESOLVES, so the CLI used to ride the 10s hard-deadline backstop
|
||||
* on every `gbrain query`. The fix is to exit on purpose the moment main()
|
||||
* resolves — but only after stdout/stderr have actually drained: incident
|
||||
* #1959 (see src/core/db.ts) was a force-exit truncating piped stdout
|
||||
* mid-payload.
|
||||
*
|
||||
* Flush contract: a stream is drained when `writableLength === 0` — bytes
|
||||
* queued in the JS-side buffer are the only ones `process.exit()` can lose
|
||||
* (the kernel owns anything already written to the fd). `writableNeedDrain`
|
||||
* is NOT sufficient (it only says "below high-water mark"). We wake on
|
||||
* 'drain' when it fires, and poll on a short tick because 'drain' is only
|
||||
* emitted after a write() returned false — a small queued chunk can flush
|
||||
* without ever signalling. A blocked pipe (reader stopped consuming) is
|
||||
* bounded by `guardMs`: partial output to a stalled reader is unavoidable,
|
||||
* hanging the process is not.
|
||||
* Minimal writable surface for the flush fence — process.stdout/stderr satisfy
|
||||
* it; tests inject fakes.
|
||||
*/
|
||||
export interface FlushableStream {
|
||||
writableLength?: number;
|
||||
once(event: 'drain', listener: () => void): unknown;
|
||||
off?(event: 'drain', listener: () => void): unknown;
|
||||
removeListener?(event: 'drain', listener: () => void): unknown;
|
||||
export interface MinimalWritable {
|
||||
write(chunk: string, cb?: (err?: Error | null) => void): boolean;
|
||||
once?(event: string, listener: (...args: unknown[]) => void): unknown;
|
||||
}
|
||||
|
||||
export async function flushStdoutThenExit(
|
||||
code: number,
|
||||
deps?: {
|
||||
streams?: FlushableStream[];
|
||||
exit?: (code: number) => void;
|
||||
guardMs?: number;
|
||||
},
|
||||
): Promise<void> {
|
||||
const streams = deps?.streams ?? [
|
||||
process.stdout as unknown as FlushableStream,
|
||||
process.stderr as unknown as FlushableStream,
|
||||
/**
|
||||
* #2084 — the CLI's exit verdict lives in a gbrain-OWNED variable, never read
|
||||
* back from `process.exitCode`. PGLite's Emscripten runtime writes its own
|
||||
* status into `process.exitCode` at arbitrary points DURING a run (99 at
|
||||
* create; in-memory brains run initdb whose exit status, e.g. 100, lands on a
|
||||
* later event-loop turn — after any point-in-time snapshot), so the global is
|
||||
* unreadable as a verdict channel on PGLite. Writers call `setCliExitVerdict`
|
||||
* (which mirrors into `process.exitCode` for anything external that reads the
|
||||
* global); the exit seam reads `currentExitCode()`, which trusts only the
|
||||
* owned variable. No verdict set ⇒ 0.
|
||||
*/
|
||||
let cliVerdict: number | null = null;
|
||||
|
||||
export function setCliExitVerdict(code: number): void {
|
||||
cliVerdict = code;
|
||||
process.exitCode = code; // best-effort mirror; never read back
|
||||
}
|
||||
|
||||
export function currentExitCode(): number {
|
||||
return cliVerdict ?? 0;
|
||||
}
|
||||
|
||||
/** Test seam — clears the verdict so each test starts clean. */
|
||||
export function _resetCliExitVerdictForTests(): void {
|
||||
cliVerdict = null;
|
||||
}
|
||||
|
||||
export interface FlushThenExitOpts {
|
||||
exit?: (code: number) => void;
|
||||
stdout?: MinimalWritable;
|
||||
stderr?: MinimalWritable;
|
||||
guardMs?: number;
|
||||
/**
|
||||
* Aliveness window between the fence and exit. Default: 0 when BOTH stdio
|
||||
* streams are TTYs (synchronous writes), FLUSH_GRACE_PIPE_MS otherwise.
|
||||
* The grace timer is deliberately ref'd — keeping the loop alive is the
|
||||
* only thing that delivers Bun's queued pipe writes (see module constant).
|
||||
*/
|
||||
graceMs?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Flush stdout + stderr, then exit with `code` — exactly once.
|
||||
*
|
||||
* Two stages, both bounded:
|
||||
* 1. Fence: an empty `write('', cb)` per stream serializes behind the accept
|
||||
* queue; an unref'd guard bounds a stream whose callback never fires.
|
||||
* (In Bun the callback fires on ACCEPT, not delivery — the fence alone is
|
||||
* NOT sufficient; verified in the #2084 probes.)
|
||||
* 2. Aliveness grace: a REF'D timer keeps the process alive `graceMs` so
|
||||
* Bun's native writer can push the queued bytes to the fd / a consuming
|
||||
* reader (#1959 truncation class). TTY stdio skips this (sync writes).
|
||||
*
|
||||
* A reader that consumes nothing for longer than guard+grace loses the tail —
|
||||
* unavoidable without waiting forever; strictly better than the pre-#2084
|
||||
* behavior (immediate process.exit discarded everything still queued).
|
||||
*
|
||||
* `process.exitCode` is set up front so that even a stubbed `exit` (tests) or
|
||||
* a natural event-loop exit keeps the right code.
|
||||
*/
|
||||
/** Process-level guard: the REAL process.exit fires at most once even if both
|
||||
* the backstop and the central seam reach flushThenExit (test-injected exit
|
||||
* fns are exempt so unit tests stay independent). */
|
||||
let realExitInitiated = false;
|
||||
|
||||
export function flushThenExit(code: number, opts: FlushThenExitOpts = {}): void {
|
||||
if (!opts.exit) {
|
||||
if (realExitInitiated) return;
|
||||
realExitInitiated = true;
|
||||
}
|
||||
const exit = opts.exit ?? ((c: number) => process.exit(c));
|
||||
const streams: MinimalWritable[] = [
|
||||
opts.stdout ?? process.stdout,
|
||||
opts.stderr ?? process.stderr,
|
||||
];
|
||||
const exit = deps?.exit ?? ((c: number) => process.exit(c));
|
||||
const guardMs = deps?.guardMs ?? 2000;
|
||||
const deadline = Date.now() + guardMs;
|
||||
|
||||
for (const stream of streams) {
|
||||
while ((stream.writableLength ?? 0) > 0 && Date.now() < deadline) {
|
||||
await new Promise<void>((resolve) => {
|
||||
const onDrain = () => {
|
||||
clearTimeout(tick);
|
||||
resolve();
|
||||
};
|
||||
// Poll tick: 'drain' only fires after a backpressured write, so a
|
||||
// buffer that empties without one needs the re-check. unref'd so the
|
||||
// wait itself never holds the loop open.
|
||||
const tick = setTimeout(() => {
|
||||
(stream.off ?? stream.removeListener)?.call(stream, 'drain', onDrain);
|
||||
resolve();
|
||||
}, 25);
|
||||
(tick as { unref?: () => void }).unref?.();
|
||||
stream.once('drain', onDrain);
|
||||
});
|
||||
const guardMs = opts.guardMs ?? FLUSH_GUARD_MS;
|
||||
const bothTty = streams.every((s) => (s as { isTTY?: boolean }).isTTY === true);
|
||||
const graceMs = opts.graceMs ?? (bothTty ? 0 : resolveFlushGraceMs());
|
||||
process.exitCode = code;
|
||||
let fenced = false;
|
||||
let guard: ReturnType<typeof setTimeout> | undefined;
|
||||
const finish = () => {
|
||||
if (fenced) return;
|
||||
fenced = true;
|
||||
if (guard) clearTimeout(guard);
|
||||
if (graceMs <= 0) {
|
||||
exit(code);
|
||||
return;
|
||||
}
|
||||
// Ref'd on purpose: aliveness IS the flush (Bun pipe-write semantics).
|
||||
setTimeout(() => exit(code), graceMs);
|
||||
};
|
||||
let pending = streams.length;
|
||||
const done = () => {
|
||||
pending -= 1;
|
||||
if (pending <= 0) finish();
|
||||
};
|
||||
guard = setTimeout(finish, guardMs);
|
||||
guard.unref?.();
|
||||
for (const s of streams) {
|
||||
try {
|
||||
// EPIPE on a closed pipe surfaces as an async 'error' event; swallow it —
|
||||
// the guard or the other stream's callback still drives the exit.
|
||||
s.once?.('error', () => {});
|
||||
s.write('', () => done());
|
||||
} catch {
|
||||
done(); // sync EPIPE / destroyed stream
|
||||
}
|
||||
}
|
||||
exit(code);
|
||||
}
|
||||
|
||||
export interface FinishCliTeardownOpts {
|
||||
/** Engine to disconnect. A disconnect throw is warned + swallowed (D3). */
|
||||
engine: { disconnect(): Promise<void> };
|
||||
/** Per-sink drain budget. Default 2000 (the registry default). */
|
||||
drainTimeoutMs?: number;
|
||||
/** Test seam — wins over the env override and the computed formula. */
|
||||
deadlineMs?: number;
|
||||
/** Forwarded to flushThenExit on the backstop path (test seam). */
|
||||
graceMs?: number;
|
||||
// ---- test seams (default to the real thing) ----
|
||||
exit?: (code: number) => void;
|
||||
warn?: (msg: string) => void;
|
||||
drain?: (opts: { timeoutMs: number }) => Promise<void>;
|
||||
stdout?: MinimalWritable;
|
||||
stderr?: MinimalWritable;
|
||||
}
|
||||
|
||||
/**
|
||||
* CLI-EXIT-ONLY teardown: bounded drain of every background-work sink, then
|
||||
* bounded engine disconnect, under a computed-deadline backstop. Returns to
|
||||
* the caller — the explicit process exit happens once, in cli.ts's
|
||||
* import.meta.main seam (see module header). The backstop timer is the ONLY
|
||||
* exit in here, and it means a component violated its own bound.
|
||||
*/
|
||||
export async function finishCliTeardown(opts: FinishCliTeardownOpts): Promise<void> {
|
||||
const drainTimeoutMs = opts.drainTimeoutMs ?? DEFAULT_DRAIN_TIMEOUT_MS;
|
||||
const warn = opts.warn ?? ((m: string) => console.warn(m));
|
||||
const drain = opts.drain ?? drainAllBackgroundWorkForCliExit;
|
||||
const deadlineMs =
|
||||
opts.deadlineMs ??
|
||||
computeTeardownDeadlineMs({ sinkCount: backgroundWorkSinkCount(), drainTimeoutMs });
|
||||
|
||||
const backstop = setTimeout(() => {
|
||||
warn(
|
||||
`[cli] teardown (background-work drain + engine.disconnect()) did not return within ${deadlineMs}ms — force-exiting`,
|
||||
);
|
||||
// currentExitCode() reads the gbrain-owned verdict channel — an errored
|
||||
// op's setCliExitVerdict(1) is honored even when PGLite has scribbled over
|
||||
// process.exitCode; a bare exit(0) would mask the failure.
|
||||
flushThenExit(currentExitCode(), opts);
|
||||
}, deadlineMs);
|
||||
// Deliberately REF'D (adversarial F3): if teardown hangs while nothing else
|
||||
// keeps Bun's loop alive, an unref'd timer would let the process exit
|
||||
// NATURALLY — skipping the flush and exiting with whatever PGLite scribbled
|
||||
// into process.exitCode. The ref'd timer costs nothing on the clean path
|
||||
// (cleared in the finally as soon as teardown returns).
|
||||
|
||||
try {
|
||||
try {
|
||||
await drain({ timeoutMs: drainTimeoutMs });
|
||||
} catch (e) {
|
||||
// The registry is contractually non-throwing, but a throw here must not
|
||||
// skip the disconnect or escape a caller's finally (it would replace a
|
||||
// successful op's completion). Same D3 posture as the disconnect guard.
|
||||
warn(
|
||||
`[cli] background-work drain failed during teardown: ${e instanceof Error ? e.message : String(e)} — continuing to disconnect`,
|
||||
);
|
||||
}
|
||||
try {
|
||||
await opts.engine.disconnect();
|
||||
} catch (e) {
|
||||
// D3: the exit code reports the operation, not the cleanup. Matches the
|
||||
// non-throwing posture of endPoolBounded (db.ts).
|
||||
warn(
|
||||
`[cli] engine.disconnect() failed during teardown: ${e instanceof Error ? e.message : String(e)} — continuing to exit`,
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
clearTimeout(backstop);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -73,7 +73,7 @@ export async function insertVolunteerEvents(
|
||||
|
||||
// ── Fire-and-forget sink (eng-review D4) ─────────────────────────────────
|
||||
// Mirrors src/core/last-retrieved.ts: track every dangling INSERT promise in
|
||||
// a module Set, register a drainer so drainThenDisconnect settles them
|
||||
// a module Set, register a drainer so finishCliTeardown settles them
|
||||
// against a live engine before teardown on EVERY CLI exit path (the commit-1
|
||||
// drain hoist). Logging failure never fails the caller.
|
||||
|
||||
|
||||
@@ -195,6 +195,32 @@ export function buildPgliteInitErrorMessage(
|
||||
return `${header}\n${hint}\n Original error: ${original}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* #2084 — PGLite's Emscripten runtime hijacks `process.exitCode` as ITS status
|
||||
* channel: instantiation REPLACES the property with an accessor whose getter
|
||||
* falls back to the WASM runtime status (99 while alive, the exit status after
|
||||
* close) whenever no explicit value was assigned — and assigning `undefined`
|
||||
* resets to that fallback, so "unset" cannot be restored. Pre-fix, every clean
|
||||
* PGLite run carried a bogus 99 until close zeroed it, and an errored op's
|
||||
* exit 1 survived only by accident of write ordering.
|
||||
*
|
||||
* Containment: around PGlite.create(), snapshot the pre-call value and restore
|
||||
* it — pinning an explicit 0 when nothing was set, because restoring
|
||||
* `undefined` would surface the WASM fallback instead. This keeps the GLOBAL
|
||||
* tidy for external readers; the CLI's own verdict never reads it (it lives in
|
||||
* the owned channel: setCliExitVerdict/currentExitCode, cli-force-exit.ts —
|
||||
* in-memory brains run initdb whose status lands on a later tick, past any
|
||||
* snapshot). db.close() stays unwrapped (see the comment at the close site).
|
||||
*/
|
||||
async function preservingProcessExitCode<T>(fn: () => Promise<T>): Promise<T> {
|
||||
const pre = process.exitCode;
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
process.exitCode = typeof pre === 'number' || typeof pre === 'string' ? pre : 0;
|
||||
}
|
||||
}
|
||||
|
||||
export class PGLiteEngine implements BrainEngine {
|
||||
readonly kind = 'pglite' as const;
|
||||
private _db: PGLiteDB | null = null;
|
||||
@@ -243,13 +269,15 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// the postmaster at close-time), and the writes land asynchronously —
|
||||
// a snapshot/restore around these awaits does NOT contain them. That is
|
||||
// why the CLI's exit paths read gbrain's own verdict
|
||||
// (cli-force-exit.ts getCliExitCode), never ambient process.exitCode.
|
||||
// (cli-force-exit.ts currentExitCode), never ambient process.exitCode.
|
||||
try {
|
||||
this._db = await PGlite.create({
|
||||
dataDir,
|
||||
loadDataDir,
|
||||
extensions: { vector, pg_trgm },
|
||||
});
|
||||
this._db = await preservingProcessExitCode(() =>
|
||||
PGlite.create({
|
||||
dataDir,
|
||||
loadDataDir,
|
||||
extensions: { vector, pg_trgm },
|
||||
}),
|
||||
);
|
||||
} catch (err) {
|
||||
// v0.13.1: any PGLite.create() failure becomes actionable. v0.41.8.0
|
||||
// (#1340): the previous error hint hardcoded the macOS 26.3 link, but
|
||||
@@ -289,6 +317,12 @@ export class PGLiteEngine implements BrainEngine {
|
||||
this._lock = null;
|
||||
try {
|
||||
if (db) {
|
||||
// Deliberately NOT wrapped in preservingProcessExitCode: close's
|
||||
// status write (0) is long-standing baseline behavior that test-runner
|
||||
// processes depend on (wrapping it flipped bun test's own exit code —
|
||||
// #2084 implementation note), and the CLI's exit verdict doesn't read
|
||||
// process.exitCode at all — it lives in the gbrain-owned channel
|
||||
// (setCliExitVerdict/currentExitCode in cli-force-exit.ts).
|
||||
await db.close();
|
||||
}
|
||||
} finally {
|
||||
|
||||
@@ -1,99 +0,0 @@
|
||||
/**
|
||||
* v0.43 (#2084 drain hoist, closes TODOS P3) — drainThenDisconnect unit tests.
|
||||
*
|
||||
* One helper owns every CLI owner-disconnect: drain the background-work
|
||||
* registry FIRST, then disconnect (best-effort), bounded by the unref'd
|
||||
* hard-deadline. Pre-hoist, six bare sites (search dashboard, doctor
|
||||
* remediation, ze-switch, dream, read-only timeout path) skipped the drain —
|
||||
* an in-flight fire-and-forget write was killed at disconnect/exit.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { drainThenDisconnect } from '../src/cli.ts';
|
||||
import { __registerDrainerForTest } from '../src/core/background-work.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
function makeEngine(events: string[], opts?: { throwOnDisconnect?: boolean }): BrainEngine {
|
||||
return {
|
||||
disconnect: async () => {
|
||||
events.push('disconnect');
|
||||
if (opts?.throwOnDisconnect) throw new Error('disconnect blew up');
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
describe('drainThenDisconnect — drain registry, then disconnect, bounded', () => {
|
||||
test('drains registered sinks BEFORE engine.disconnect()', async () => {
|
||||
const events: string[] = [];
|
||||
const unregister = __registerDrainerForTest({
|
||||
name: 'test-sink-order',
|
||||
order: 99,
|
||||
drain: async () => {
|
||||
events.push('drain');
|
||||
return { unfinished: 0 };
|
||||
},
|
||||
});
|
||||
try {
|
||||
await drainThenDisconnect(makeEngine(events));
|
||||
} finally {
|
||||
unregister();
|
||||
}
|
||||
expect(events.indexOf('drain')).toBeGreaterThanOrEqual(0);
|
||||
expect(events.indexOf('disconnect')).toBeGreaterThan(events.indexOf('drain'));
|
||||
});
|
||||
|
||||
test('a pending fire-and-forget write survives (drained, not killed)', async () => {
|
||||
// Simulates the search-stats path: a cache write is in flight when the
|
||||
// command finishes. The hoisted drain must let it settle before teardown.
|
||||
const events: string[] = [];
|
||||
let settled = false;
|
||||
const pending = new Promise<void>((r) =>
|
||||
setTimeout(() => {
|
||||
settled = true;
|
||||
events.push('write-settled');
|
||||
r();
|
||||
}, 30),
|
||||
);
|
||||
const unregister = __registerDrainerForTest({
|
||||
name: 'test-sink-pending-write',
|
||||
order: 99,
|
||||
drain: async () => {
|
||||
await pending;
|
||||
return { unfinished: 0 };
|
||||
},
|
||||
});
|
||||
try {
|
||||
await drainThenDisconnect(makeEngine(events));
|
||||
} finally {
|
||||
unregister();
|
||||
}
|
||||
expect(settled).toBe(true);
|
||||
expect(events).toEqual(['write-settled', 'disconnect']);
|
||||
});
|
||||
|
||||
test('disconnect failure is swallowed (best-effort; kernel reclaims on exit)', async () => {
|
||||
const events: string[] = [];
|
||||
await expect(
|
||||
drainThenDisconnect(makeEngine(events, { throwOnDisconnect: true })),
|
||||
).resolves.toBeUndefined();
|
||||
expect(events).toEqual(['disconnect']);
|
||||
});
|
||||
|
||||
test('honors the per-site drain timeout passthrough', async () => {
|
||||
const seen: number[] = [];
|
||||
const unregister = __registerDrainerForTest({
|
||||
name: 'test-sink-timeout',
|
||||
order: 99,
|
||||
drain: async (timeoutMs: number) => {
|
||||
seen.push(timeoutMs);
|
||||
return { unfinished: 0 };
|
||||
},
|
||||
});
|
||||
try {
|
||||
await drainThenDisconnect(makeEngine([]), { drainTimeoutMs: 1000 });
|
||||
} finally {
|
||||
unregister();
|
||||
}
|
||||
expect(seen).toEqual([1000]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,31 @@
|
||||
/**
|
||||
* #2084 class pin — every exit-code write in src/ routes through
|
||||
* setCliExitVerdict.
|
||||
*
|
||||
* A RAW `process.exitCode = N` is silently ZEROED by the deliberate
|
||||
* flush-exit: currentExitCode() reads only gbrain's owned verdict (the
|
||||
* PGLite-Emscripten-pollution defense), so a command that bypasses the
|
||||
* setter reports success on failure. Caught live twice in one day: doctor's
|
||||
* FAIL path exited 0 after a merge introduced a raw write. Runtime variants
|
||||
* are pinned in test/cli-finish-teardown.test.ts; this is the structural
|
||||
* guard that catches the NEXT raw write at review time.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { execSync } from 'child_process';
|
||||
|
||||
describe('exit-verdict ownership — no raw process.exitCode assignments', () => {
|
||||
test('every exit-code write in src/ routes through setCliExitVerdict', () => {
|
||||
// Two legitimate writers are exempt:
|
||||
// - cli-force-exit.ts: setCliExitVerdict's own mirror-write.
|
||||
// - pglite-engine.ts preservingProcessExitCode: #2141's containment
|
||||
// RESTORE around PGlite.create() — it keeps the GLOBAL tidy for
|
||||
// external readers and is explicitly not a verdict write (the owned
|
||||
// channel never reads process.exitCode).
|
||||
const hits = execSync(
|
||||
`grep -rn "process.exitCode = " src --include='*.ts' | grep -v "core/cli-force-exit.ts" | grep -v "core/pglite-engine.ts" || true`,
|
||||
{ encoding: 'utf-8', cwd: new URL('..', import.meta.url).pathname },
|
||||
).trim();
|
||||
expect(hits).toBe('');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,510 @@
|
||||
/**
|
||||
* #2084 — unit tests for the one-shot CLI teardown + exit contract in
|
||||
* src/core/cli-force-exit.ts: finishCliTeardown (teardown-only, computed
|
||||
* backstop deadline), flushThenExit (write-fence + guard + EPIPE + once-latch),
|
||||
* and computeTeardownDeadlineMs (formula / floor / env override).
|
||||
*
|
||||
* Real short timers, no fake clocks. Every test that touches process.exitCode
|
||||
* or GBRAIN_TEARDOWN_DEADLINE_MS restores it in a finally so the suite stays
|
||||
* order-independent.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
finishCliTeardown,
|
||||
flushThenExit,
|
||||
computeTeardownDeadlineMs,
|
||||
TEARDOWN_DEADLINE_FLOOR_MS,
|
||||
setCliExitVerdict,
|
||||
currentExitCode,
|
||||
_resetCliExitVerdictForTests,
|
||||
type MinimalWritable,
|
||||
} from '../src/core/cli-force-exit.ts';
|
||||
import { POOL_END_TIMEOUT_SECONDS } from '../src/core/db.ts';
|
||||
import {
|
||||
backgroundWorkSinkCount,
|
||||
__registerDrainerForTest,
|
||||
} from '../src/core/background-work.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
function fakeStream(): MinimalWritable & { writes: string[] } {
|
||||
const writes: string[] = [];
|
||||
return {
|
||||
writes,
|
||||
write(chunk: string, cb?: (err?: Error | null) => void) {
|
||||
writes.push(chunk);
|
||||
if (cb) queueMicrotask(() => cb());
|
||||
return true;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('computeTeardownDeadlineMs', () => {
|
||||
test('formula: sinks × drain + facts grace + 2 × pool bound + slack', () => {
|
||||
const poolEndBoundMs = POOL_END_TIMEOUT_SECONDS * 1000 + 500;
|
||||
// 4 sinks × 2000 + 2000 + 2×poolEnd + 2000 — the Site B worst case that
|
||||
// falsified the old static 10s (eng-review D9).
|
||||
const got = computeTeardownDeadlineMs({ sinkCount: 4, drainTimeoutMs: 2000 });
|
||||
expect(got).toBe(4 * 2000 + 2000 + 2 * poolEndBoundMs + 2000);
|
||||
expect(got).toBeGreaterThan(10_000); // the codex-found arithmetic bug, pinned
|
||||
});
|
||||
|
||||
test('floors at TEARDOWN_DEADLINE_FLOOR_MS for small budgets', () => {
|
||||
const got = computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 });
|
||||
expect(got).toBe(TEARDOWN_DEADLINE_FLOOR_MS);
|
||||
});
|
||||
|
||||
test('GBRAIN_TEARDOWN_DEADLINE_MS env override wins over the formula', async () => {
|
||||
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: '1234' }, async () => {
|
||||
expect(computeTeardownDeadlineMs({ sinkCount: 4, drainTimeoutMs: 2000 })).toBe(1234);
|
||||
});
|
||||
});
|
||||
|
||||
test('garbage env values fall back to the formula', async () => {
|
||||
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: 'banana' }, async () => {
|
||||
expect(
|
||||
computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 }),
|
||||
).toBe(TEARDOWN_DEADLINE_FLOOR_MS);
|
||||
});
|
||||
});
|
||||
|
||||
test('zero and negative env values fall back to the formula (not "fire immediately")', async () => {
|
||||
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: '0' }, async () => {
|
||||
expect(computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 })).toBe(
|
||||
TEARDOWN_DEADLINE_FLOOR_MS,
|
||||
);
|
||||
});
|
||||
await withEnv({ GBRAIN_TEARDOWN_DEADLINE_MS: '-5' }, async () => {
|
||||
expect(computeTeardownDeadlineMs({ sinkCount: 1, drainTimeoutMs: 100 })).toBe(
|
||||
TEARDOWN_DEADLINE_FLOOR_MS,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
test('a newly registered sink widens the computed deadline (D9: formula reads the live registry)', () => {
|
||||
// Register two sinks and compare between them: in a bare unit-test process
|
||||
// no production sinks are loaded, so the zero-sink baseline sits below the
|
||||
// 10s floor and would mask the first sink's delta.
|
||||
const mkSink = (name: string) =>
|
||||
__registerDrainerForTest({ name, order: 99, drain: async () => ({ unfinished: 0 }) });
|
||||
const un1 = mkSink('test-2084-sink-a');
|
||||
try {
|
||||
const withOne = computeTeardownDeadlineMs({
|
||||
sinkCount: backgroundWorkSinkCount(),
|
||||
drainTimeoutMs: 5000,
|
||||
});
|
||||
const un2 = mkSink('test-2084-sink-b');
|
||||
try {
|
||||
const withTwo = computeTeardownDeadlineMs({
|
||||
sinkCount: backgroundWorkSinkCount(),
|
||||
drainTimeoutMs: 5000,
|
||||
});
|
||||
expect(withOne).toBeGreaterThan(TEARDOWN_DEADLINE_FLOOR_MS); // above the floor — delta is visible
|
||||
expect(withTwo).toBe(withOne + 5000);
|
||||
} finally {
|
||||
un2();
|
||||
}
|
||||
} finally {
|
||||
un1();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('finishCliTeardown — clean path', () => {
|
||||
test('drains with the injected budget, disconnects, returns; no exit, no warn', async () => {
|
||||
const calls: string[] = [];
|
||||
let drainBudget = -1;
|
||||
const exits: number[] = [];
|
||||
const warns: string[] = [];
|
||||
await finishCliTeardown({
|
||||
engine: { disconnect: async () => void calls.push('disconnect') },
|
||||
drainTimeoutMs: 777,
|
||||
deadlineMs: 250,
|
||||
drain: async ({ timeoutMs }) => {
|
||||
drainBudget = timeoutMs;
|
||||
calls.push('drain');
|
||||
},
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: (m) => void warns.push(m),
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
});
|
||||
// Past the 250ms deadline: a leaked backstop would fire here.
|
||||
await sleep(400);
|
||||
expect(calls).toEqual(['drain', 'disconnect']);
|
||||
expect(drainBudget).toBe(777);
|
||||
expect(exits).toEqual([]);
|
||||
expect(warns).toEqual([]);
|
||||
});
|
||||
|
||||
test('drain runs BEFORE disconnect (live-engine window for sinks)', async () => {
|
||||
const order: string[] = [];
|
||||
await finishCliTeardown({
|
||||
engine: { disconnect: async () => void order.push('disconnect') },
|
||||
deadlineMs: 1000,
|
||||
drain: async () => {
|
||||
await sleep(20);
|
||||
order.push('drain');
|
||||
},
|
||||
exit: () => {},
|
||||
warn: () => {},
|
||||
});
|
||||
expect(order).toEqual(['drain', 'disconnect']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('finishCliTeardown — backstop on hung teardown', () => {
|
||||
test('hung disconnect fires the banner and exits with current exitCode', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
_resetCliExitVerdictForTests(); // no verdict set ⇒ currentExitCode() === 0
|
||||
const exits: number[] = [];
|
||||
const warns: string[] = [];
|
||||
let resolveHang!: () => void;
|
||||
const teardown = finishCliTeardown({
|
||||
engine: { disconnect: () => new Promise<void>((r) => (resolveHang = r)) },
|
||||
deadlineMs: 100,
|
||||
drain: async () => {},
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: (m) => void warns.push(m),
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(300);
|
||||
expect(warns.length).toBe(1);
|
||||
expect(warns[0]).toContain('did not return within');
|
||||
expect(warns[0]).toContain('100ms');
|
||||
expect(exits).toEqual([0]);
|
||||
resolveHang(); // unhang so the promise settles
|
||||
await teardown;
|
||||
} finally {
|
||||
_resetCliExitVerdictForTests();
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('backstop honors an exit code the errored op already set', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
setCliExitVerdict(1); // what the op-dispatch catch does
|
||||
const exits: number[] = [];
|
||||
let resolveHang!: () => void;
|
||||
const teardown = finishCliTeardown({
|
||||
engine: { disconnect: () => new Promise<void>((r) => (resolveHang = r)) },
|
||||
deadlineMs: 100,
|
||||
drain: async () => {},
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: () => {},
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(300);
|
||||
expect(exits).toEqual([1]);
|
||||
resolveHang();
|
||||
await teardown;
|
||||
} finally {
|
||||
_resetCliExitVerdictForTests();
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('hung DRAIN (not just disconnect) also trips the backstop', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
_resetCliExitVerdictForTests();
|
||||
const exits: number[] = [];
|
||||
const warns: string[] = [];
|
||||
let resolveHang!: () => void;
|
||||
const teardown = finishCliTeardown({
|
||||
engine: { disconnect: async () => {} },
|
||||
deadlineMs: 100,
|
||||
drain: () => new Promise<void>((r) => (resolveHang = r)),
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: (m) => void warns.push(m),
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(300);
|
||||
expect(warns.length).toBe(1);
|
||||
expect(exits).toEqual([0]);
|
||||
resolveHang();
|
||||
await teardown;
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('verdict channel — immune to PGLite WASM process.exitCode writes', () => {
|
||||
test('engine teardown that rewrites process.exitCode does not change the verdict', async () => {
|
||||
// PGLite's Emscripten runtime writes its own status into process.exitCode
|
||||
// at arbitrary points (99 at create, initdb status on a later tick for
|
||||
// in-memory brains, 0 at close) — pre-#2084 this clobbered an errored
|
||||
// op's exit 1 back to 0 on every PGLite error path. The verdict lives in
|
||||
// the gbrain-owned channel and never reads the global back.
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
setCliExitVerdict(1); // the op errored
|
||||
await finishCliTeardown({
|
||||
engine: {
|
||||
disconnect: async () => {
|
||||
process.exitCode = 0; // what PGLite's WASM shutdown does
|
||||
},
|
||||
},
|
||||
deadlineMs: 1000,
|
||||
drain: async () => {},
|
||||
exit: () => {},
|
||||
warn: () => {},
|
||||
});
|
||||
expect(currentExitCode()).toBe(1);
|
||||
} finally {
|
||||
_resetCliExitVerdictForTests();
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('mid-run WASM write (in-memory initdb status) cannot fake a verdict', () => {
|
||||
_resetCliExitVerdictForTests();
|
||||
try {
|
||||
process.exitCode = 100; // what in-memory PGLite's initdb does mid-run
|
||||
expect(currentExitCode()).toBe(0); // no gbrain verdict was ever set
|
||||
setCliExitVerdict(2);
|
||||
expect(currentExitCode()).toBe(2);
|
||||
// The mirror write exists for EXTERNAL readers of the global.
|
||||
expect(process.exitCode).toBe(2);
|
||||
} finally {
|
||||
_resetCliExitVerdictForTests();
|
||||
process.exitCode = 0;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('finishCliTeardown — disconnect failure (D3: exit code reports the op)', () => {
|
||||
test('a throwing drain is warned, disconnect still runs, helper resolves', async () => {
|
||||
// The registry is contractually non-throwing; this pins the defense-in-depth
|
||||
// guard — a drain rejection must not skip disconnect or escape the caller's
|
||||
// finally (it would replace a successful op's completion).
|
||||
const calls: string[] = [];
|
||||
const warns: string[] = [];
|
||||
await finishCliTeardown({
|
||||
engine: { disconnect: async () => void calls.push('disconnect') },
|
||||
deadlineMs: 1000,
|
||||
drain: async () => {
|
||||
throw new Error('sink registry blew up');
|
||||
},
|
||||
exit: () => {},
|
||||
warn: (m) => void warns.push(m),
|
||||
});
|
||||
expect(calls).toEqual(['disconnect']);
|
||||
expect(warns.length).toBe(1);
|
||||
expect(warns[0]).toContain('sink registry blew up');
|
||||
});
|
||||
|
||||
test('disconnect throw is warned and swallowed; helper resolves', async () => {
|
||||
const warns: string[] = [];
|
||||
const exits: number[] = [];
|
||||
await finishCliTeardown({
|
||||
engine: {
|
||||
disconnect: async () => {
|
||||
throw new Error('pool already dead');
|
||||
},
|
||||
},
|
||||
deadlineMs: 1000,
|
||||
drain: async () => {},
|
||||
exit: (c) => void exits.push(c),
|
||||
warn: (m) => void warns.push(m),
|
||||
});
|
||||
expect(warns.length).toBe(1);
|
||||
expect(warns[0]).toContain('pool already dead');
|
||||
expect(exits).toEqual([]); // helper never exits on the non-backstop path
|
||||
});
|
||||
});
|
||||
|
||||
describe('flushThenExit', () => {
|
||||
test('exits after BOTH stream callbacks fire, exactly once, with the code', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const events: string[] = [];
|
||||
const exits: number[] = [];
|
||||
const slowStream = (name: string): MinimalWritable => ({
|
||||
write(_c: string, cb?: (err?: Error | null) => void) {
|
||||
setTimeout(() => {
|
||||
events.push(`${name}-flushed`);
|
||||
cb?.();
|
||||
}, 50);
|
||||
return true;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
});
|
||||
flushThenExit(3, {
|
||||
exit: (c) => {
|
||||
events.push('exit');
|
||||
exits.push(c);
|
||||
},
|
||||
stdout: slowStream('stdout'),
|
||||
stderr: slowStream('stderr'),
|
||||
guardMs: 2000,
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(200);
|
||||
expect(events).toEqual(['stdout-flushed', 'stderr-flushed', 'exit']);
|
||||
expect(exits).toEqual([3]);
|
||||
expect(process.exitCode).toBe(3); // belt-and-braces for natural exit
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('non-TTY default: exit waits the aliveness grace AFTER the fence', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const exits: number[] = [];
|
||||
const t0 = Date.now();
|
||||
let fencedAt = -1;
|
||||
const stream: MinimalWritable = {
|
||||
write(_c: string, cb?: (err?: Error | null) => void) {
|
||||
fencedAt = Date.now() - t0;
|
||||
if (cb) queueMicrotask(() => cb());
|
||||
return true;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: stream,
|
||||
stderr: stream,
|
||||
guardMs: 2000,
|
||||
graceMs: 120, // fakes are non-TTY; explicit grace keeps the test tight
|
||||
});
|
||||
await sleep(60);
|
||||
expect(exits).toEqual([]); // fence done, still inside the grace window
|
||||
await sleep(150);
|
||||
expect(exits).toEqual([0]);
|
||||
expect(fencedAt).toBeGreaterThanOrEqual(0);
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('guard fires when a callback never arrives (blocked pipe)', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const exits: number[] = [];
|
||||
const blockedStream: MinimalWritable = {
|
||||
write() {
|
||||
return false; // never calls cb — reader stopped consuming
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
const t0 = Date.now();
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: blockedStream,
|
||||
stderr: blockedStream,
|
||||
guardMs: 100,
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(300);
|
||||
expect(exits).toEqual([0]);
|
||||
expect(Date.now() - t0).toBeGreaterThanOrEqual(100);
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('sync write throw (EPIPE) still exits', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const exits: number[] = [];
|
||||
const epipeStream: MinimalWritable = {
|
||||
write() {
|
||||
throw Object.assign(new Error('write EPIPE'), { code: 'EPIPE' });
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: epipeStream,
|
||||
stderr: epipeStream,
|
||||
guardMs: 2000,
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(50);
|
||||
expect(exits).toEqual([0]);
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('GBRAIN_FLUSH_GRACE_MS env override is honored (batch/incident knob)', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
await withEnv({ GBRAIN_FLUSH_GRACE_MS: '0' }, async () => {
|
||||
const exits: number[] = [];
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: fakeStream(),
|
||||
stderr: fakeStream(),
|
||||
guardMs: 2000,
|
||||
// no graceMs → resolves through the env override (fakes are non-TTY)
|
||||
});
|
||||
await sleep(60);
|
||||
expect(exits).toEqual([0]); // grace 0: exit right after the fence
|
||||
});
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
|
||||
test('once-latch: guard + late callbacks cannot double-exit', async () => {
|
||||
const prevCode = process.exitCode;
|
||||
try {
|
||||
const exits: number[] = [];
|
||||
// stdout flushes late (after the guard), stderr never — both race finish().
|
||||
const lateStream: MinimalWritable = {
|
||||
write(_c: string, cb?: (err?: Error | null) => void) {
|
||||
setTimeout(() => cb?.(), 150);
|
||||
return true;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
const neverStream: MinimalWritable = {
|
||||
write() {
|
||||
return false;
|
||||
},
|
||||
once() {
|
||||
return this;
|
||||
},
|
||||
};
|
||||
flushThenExit(0, {
|
||||
exit: (c) => void exits.push(c),
|
||||
stdout: lateStream,
|
||||
stderr: neverStream,
|
||||
guardMs: 80,
|
||||
graceMs: 0,
|
||||
});
|
||||
await sleep(400);
|
||||
expect(exits).toEqual([0]); // exactly one exit despite guard + late cb
|
||||
} finally {
|
||||
process.exitCode = prevCode;
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1,155 +0,0 @@
|
||||
/**
|
||||
* v0.43 (#2084) — flushStdoutThenExit unit tests.
|
||||
*
|
||||
* The deliberate-exit path must never truncate buffered output (incident
|
||||
* #1959: a force-exit cut piped stdout mid-payload) and must never hang on
|
||||
* a blocked pipe. Streams + exit are injected so the tests drive both
|
||||
* properties without killing the test process.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { flushStdoutThenExit, type FlushableStream } from '../src/core/cli-force-exit.ts';
|
||||
|
||||
/** Minimal fake of the stdout/stderr surface flushStdoutThenExit touches. */
|
||||
function makeStream(initialLength: number): FlushableStream & {
|
||||
setLength(n: number): void;
|
||||
emitDrain(): void;
|
||||
drainListeners(): number;
|
||||
} {
|
||||
let length = initialLength;
|
||||
const listeners: Array<() => void> = [];
|
||||
return {
|
||||
get writableLength() {
|
||||
return length;
|
||||
},
|
||||
once(_event: 'drain', listener: () => void) {
|
||||
listeners.push(listener);
|
||||
return this;
|
||||
},
|
||||
off(_event: 'drain', listener: () => void) {
|
||||
const i = listeners.indexOf(listener);
|
||||
if (i >= 0) listeners.splice(i, 1);
|
||||
return this;
|
||||
},
|
||||
setLength(n: number) {
|
||||
length = n;
|
||||
},
|
||||
emitDrain() {
|
||||
const current = listeners.splice(0, listeners.length);
|
||||
for (const l of current) l();
|
||||
},
|
||||
drainListeners() {
|
||||
return listeners.length;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('flushStdoutThenExit — deliberate exit after output drains', () => {
|
||||
test('exits immediately with the given code when both streams are drained', async () => {
|
||||
const calls: number[] = [];
|
||||
await flushStdoutThenExit(0, {
|
||||
streams: [makeStream(0), makeStream(0)],
|
||||
exit: (c) => calls.push(c),
|
||||
});
|
||||
expect(calls).toEqual([0]);
|
||||
});
|
||||
|
||||
test('honors a non-zero exit code (errored op sets process.exitCode=1)', async () => {
|
||||
const calls: number[] = [];
|
||||
await flushStdoutThenExit(1, {
|
||||
streams: [makeStream(0)],
|
||||
exit: (c) => calls.push(c),
|
||||
});
|
||||
expect(calls).toEqual([1]);
|
||||
});
|
||||
|
||||
test('waits for a pending buffer to drain before exiting', async () => {
|
||||
const stream = makeStream(4096);
|
||||
const calls: number[] = [];
|
||||
const done = flushStdoutThenExit(0, {
|
||||
streams: [stream],
|
||||
exit: (c) => calls.push(c),
|
||||
guardMs: 5000,
|
||||
});
|
||||
// Not exited while bytes are queued.
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
expect(calls).toEqual([]);
|
||||
stream.setLength(0);
|
||||
stream.emitDrain();
|
||||
await done;
|
||||
expect(calls).toEqual([0]);
|
||||
});
|
||||
|
||||
test('drains stdout AND stderr — stderr alone can hold the exit', async () => {
|
||||
const stdout = makeStream(0);
|
||||
const stderr = makeStream(2048);
|
||||
const calls: number[] = [];
|
||||
const done = flushStdoutThenExit(1, {
|
||||
streams: [stdout, stderr],
|
||||
exit: (c) => calls.push(c),
|
||||
guardMs: 5000,
|
||||
});
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
expect(calls).toEqual([]);
|
||||
stderr.setLength(0);
|
||||
stderr.emitDrain();
|
||||
await done;
|
||||
expect(calls).toEqual([1]);
|
||||
});
|
||||
|
||||
test('poll tick drains without a drain event ("drain" only fires after backpressure)', async () => {
|
||||
const stream = makeStream(512);
|
||||
const calls: number[] = [];
|
||||
const done = flushStdoutThenExit(0, {
|
||||
streams: [stream],
|
||||
exit: (c) => calls.push(c),
|
||||
guardMs: 5000,
|
||||
});
|
||||
// Buffer empties WITHOUT an emitted drain event — the 25ms poll must see it.
|
||||
setTimeout(() => stream.setLength(0), 30);
|
||||
await done;
|
||||
expect(calls).toEqual([0]);
|
||||
});
|
||||
|
||||
test('blocked pipe: the guard deadline exits anyway (never hangs)', async () => {
|
||||
const stream = makeStream(65536); // reader stopped consuming; never drains
|
||||
const calls: number[] = [];
|
||||
const t0 = Date.now();
|
||||
await flushStdoutThenExit(0, {
|
||||
streams: [stream],
|
||||
exit: (c) => calls.push(c),
|
||||
guardMs: 120,
|
||||
});
|
||||
expect(calls).toEqual([0]);
|
||||
expect(Date.now() - t0).toBeLessThan(2000);
|
||||
});
|
||||
|
||||
test('removes its drain listener when the poll tick wins (no listener leak)', async () => {
|
||||
const stream = makeStream(256);
|
||||
const calls: number[] = [];
|
||||
const done = flushStdoutThenExit(0, {
|
||||
streams: [stream],
|
||||
exit: (c) => calls.push(c),
|
||||
guardMs: 5000,
|
||||
});
|
||||
setTimeout(() => stream.setLength(0), 30);
|
||||
await done;
|
||||
expect(stream.drainListeners()).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('exit-verdict ownership — no raw process.exitCode assignments (#2084 class pin)', () => {
|
||||
test('every exit-code write in src/ routes through setCliExitCode', async () => {
|
||||
// A RAW `process.exitCode = N` is silently ZEROED by the deliberate
|
||||
// flush-exit: getCliExitCode() reads only gbrain's owned verdict (the
|
||||
// PGLite-Emscripten-pollution defense), so a command that bypasses the
|
||||
// setter reports success on failure. Caught live: doctor's FAIL path
|
||||
// exited 0 after the v0.42.41.0 merge introduced a raw write.
|
||||
const { execSync } = await import('child_process');
|
||||
const hits = execSync(
|
||||
`grep -rn "process.exitCode = " src --include='*.ts' | grep -v "core/cli-force-exit.ts" || true`,
|
||||
{ encoding: 'utf-8', cwd: new URL('..', import.meta.url).pathname },
|
||||
).trim();
|
||||
expect(hits).toBe('');
|
||||
});
|
||||
});
|
||||
@@ -1,64 +1,55 @@
|
||||
/**
|
||||
* Structural regression — the DISCONNECT_HARD_DEADLINE_MS force-exit timer
|
||||
* must be armed at TEARDOWN ENTRY, never before the op-dispatch try block.
|
||||
* Structural regression — the teardown hard-deadline must be armed at
|
||||
* TEARDOWN ENTRY, never before the op-dispatch body.
|
||||
*
|
||||
* Pre-fix bug (fixed independently by master's v0.42.41.0 triage wave AND
|
||||
* the #2084 wave): the 10s unref'd setTimeout was armed BEFORE the try, so
|
||||
* any op whose handler ran past 10s wall-clock was killed mid-flight with
|
||||
* process.exit(0) and ZERO stdout — an empty "success" indistinguishable
|
||||
* from no results (a healthy `gbrain search` on a slow Postgres pooler hit
|
||||
* this on every run).
|
||||
* Pre-fix bug (closed independently by v0.42.41.0 and the #2084 wave, merged):
|
||||
* a 10s unref'd setTimeout armed BEFORE the try killed any op whose handler
|
||||
* ran past 10s wall-clock with process.exit(0) and ZERO stdout — an empty
|
||||
* "success" indistinguishable from no results.
|
||||
*
|
||||
* Post-#2084 the arming lives INSIDE the shared `drainThenDisconnect`
|
||||
* helper — the single owner-disconnect every CLI exit path calls from its
|
||||
* finally — so it structurally bounds ONLY the teardown window (drain +
|
||||
* disconnect) at every site, not just the op-dispatch path.
|
||||
*
|
||||
* Source-grep is the right tool here (same rationale as
|
||||
* fix-wave-structural.test.ts): the rule is "this arming must stay at this
|
||||
* location". A behavioral test would need >10s of real wall-clock plus a
|
||||
* deliberately slow op handler in a spawned CLI — slow and flaky by
|
||||
* construction.
|
||||
* Post-merge shape (#2084): the deadline lives inside `finishCliTeardown`
|
||||
* (src/core/cli-force-exit.ts), armed as the helper's first act — i.e. at
|
||||
* teardown entry, because every cli.ts call site invokes the helper from a
|
||||
* `finally`. The op body's wallclock is bounded separately by the read-scope
|
||||
* withTimeout wrap (v0.42.41.0). Source-grep is the right tool here (same
|
||||
* rationale as fix-wave-structural.test.ts): a behavioral test would need
|
||||
* >10s of real wall-clock in a spawned CLI.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
describe('cli.ts — disconnect hard-deadline armed at teardown entry, not before the op body', () => {
|
||||
test('no timer arming between the op-dispatch entry and its try block', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
// The op-dispatch local-engine path: from connectEngine to its try, no
|
||||
// setTimeout call may be armed (a pre-try timer kills slow-but-progressing
|
||||
// op handlers mid-flight with exit 0 and empty stdout). `setTimeout(`
|
||||
// matches only a call site; ReturnType<typeof setTimeout> stays allowed.
|
||||
const entry = src.indexOf('// Local engine path (unchanged behavior');
|
||||
expect(entry).toBeGreaterThan(-1);
|
||||
const tryIdx = src.indexOf('try {', entry);
|
||||
test('no timer arming exists between op-dispatch setup and the try; the deadline arms inside finishCliTeardown before the drain', () => {
|
||||
const cli = readFileSync('src/cli.ts', 'utf8');
|
||||
|
||||
// The old pre-try arming constant must stay gone (its return is the
|
||||
// kill-slow-ops-with-exit-0 regression).
|
||||
expect(cli).not.toContain('DISCONNECT_HARD_DEADLINE_MS');
|
||||
|
||||
// Between the op-dispatch engine connect and the try there is no
|
||||
// setTimeout call site (`setTimeout(` matches calls only; the
|
||||
// ReturnType<typeof setTimeout> annotation stays allowed).
|
||||
const connectIdx = cli.indexOf('// Local engine path (unchanged behavior for local installs).');
|
||||
expect(connectIdx).toBeGreaterThan(-1);
|
||||
const tryIdx = cli.indexOf('try {', connectIdx);
|
||||
expect(tryIdx).toBeGreaterThan(-1);
|
||||
expect(src.slice(entry, tryIdx)).not.toContain('setTimeout(');
|
||||
});
|
||||
expect(cli.slice(connectIdx, tryIdx)).not.toContain('setTimeout(');
|
||||
|
||||
test('the deadline arming lives inside drainThenDisconnect: gated, unref\'d, before the drain, cleared after', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
const helper = src.match(/export async function drainThenDisconnect[\s\S]+?^\}/m);
|
||||
expect(helper).not.toBeNull();
|
||||
const block = helper![0];
|
||||
// The op-dispatch finally routes through the shared teardown helper.
|
||||
const finallyIdx = cli.indexOf('} finally {', tryIdx);
|
||||
expect(finallyIdx).toBeGreaterThan(-1);
|
||||
const teardownCallIdx = cli.indexOf('finishCliTeardown({ engine, drainTimeoutMs: 1000 })', finallyIdx);
|
||||
expect(teardownCallIdx).toBeGreaterThan(finallyIdx);
|
||||
|
||||
const armIdx = block.indexOf('deadlineTimer = setTimeout');
|
||||
const drainIdx = block.indexOf('drainAllBackgroundWorkForCliExit');
|
||||
// Inside the helper, the backstop arms BEFORE the drain runs — teardown
|
||||
// entry, bounding drain + disconnect and nothing else.
|
||||
const helper = readFileSync('src/core/cli-force-exit.ts', 'utf8');
|
||||
const armIdx = helper.indexOf('const backstop = setTimeout(');
|
||||
expect(armIdx).toBeGreaterThan(-1);
|
||||
expect(drainIdx).toBeGreaterThan(-1);
|
||||
// Armed at teardown entry, before the drain + disconnect it bounds.
|
||||
expect(armIdx).toBeLessThan(drainIdx);
|
||||
// Gated on the daemon-survival guard so `serve` stays alive.
|
||||
expect(block.slice(0, armIdx)).toMatch(/if \(shouldForceExitAfterMain\(\)\)/);
|
||||
// Unref'd so the timer itself never keeps the event loop alive.
|
||||
expect(block).toContain('deadlineTimer.unref?.()');
|
||||
const drainIdx = helper.indexOf('await drain({ timeoutMs: drainTimeoutMs })', armIdx);
|
||||
expect(drainIdx).toBeGreaterThan(armIdx);
|
||||
// Cleared on clean teardown.
|
||||
expect(block).toContain('if (deadlineTimer) clearTimeout(deadlineTimer)');
|
||||
// The DISCONNECT_HARD_DEADLINE_MS declaration sits with the helper.
|
||||
const decl = src.indexOf('const DISCONNECT_HARD_DEADLINE_MS');
|
||||
expect(decl).toBeGreaterThan(-1);
|
||||
expect(src.indexOf('export async function drainThenDisconnect')).toBeGreaterThan(decl);
|
||||
expect(helper.indexOf('clearTimeout(backstop)', drainIdx)).toBeGreaterThan(drainIdx);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,85 +1,21 @@
|
||||
/**
|
||||
* v0.43 (#2084) — CRITICAL regression: piped stdout is never truncated by the
|
||||
* deliberate flush-exit (eng-review D3; the incident #1959 class).
|
||||
* v0.43 (#2084) — real-CLI pipe completeness pin (the incident #1959 class).
|
||||
*
|
||||
* Two layers:
|
||||
* 1. A subprocess that writes 256KB to a REAL pipe (4x the 64KB kernel pipe
|
||||
* buffer, so write() backpressures and bytes sit in the JS-side stream
|
||||
* buffer) and then calls the production `flushStdoutThenExit`. If the
|
||||
* flush gate is wrong, the tail of the payload is sheared off — this is
|
||||
* exactly how incident #1959 presented ("relational query came back
|
||||
* empty" — output truncated by a force-exit).
|
||||
* 2. The real CLI (`bun src/cli.ts --tools-json`, engine-free) over a pipe:
|
||||
* parseable, byte-stable across runs, and exits deliberately (well under
|
||||
* the 10s backstop).
|
||||
* The synthetic flush-mechanism coverage lives in
|
||||
* test/flush-then-exit-harness.test.ts (4MB late-reader byte-complete pin).
|
||||
* This file keeps the IMPLEMENTATION-AGNOSTIC check: the actual CLI, run the
|
||||
* way agents run it (piped stdout), produces complete, parseable, byte-stable
|
||||
* output and exits deliberately — well under the teardown backstop.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { spawnSync } from 'child_process';
|
||||
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
|
||||
import { join, resolve } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
|
||||
const REPO = resolve(import.meta.dir, '..');
|
||||
const CLI = join(REPO, 'src', 'cli.ts');
|
||||
|
||||
const PAYLOAD_BYTES = 256 * 1024;
|
||||
|
||||
describe('cli pipe truncation — deliberate exit flushes piped stdout (#2084)', () => {
|
||||
test('256KB through a real pipe survives flushStdoutThenExit byte-exactly', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-flush-'));
|
||||
const script = join(dir, 'flush-child.ts');
|
||||
try {
|
||||
writeFileSync(
|
||||
script,
|
||||
[
|
||||
`import { flushStdoutThenExit } from ${JSON.stringify(join(REPO, 'src', 'core', 'cli-force-exit.ts'))};`,
|
||||
`const payload = 'x'.repeat(${PAYLOAD_BYTES}) + 'END\\n';`,
|
||||
// No await — mirrors the cli.ts entrypoint, which fires the flush
|
||||
// exit as a dangling promise after main() resolves.
|
||||
`process.stdout.write(payload);`,
|
||||
`void flushStdoutThenExit(0);`,
|
||||
].join('\n'),
|
||||
);
|
||||
const res = spawnSync('bun', [script], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
encoding: 'utf-8',
|
||||
timeout: 30_000,
|
||||
maxBuffer: 16 * 1024 * 1024,
|
||||
});
|
||||
expect(res.status).toBe(0);
|
||||
const out = res.stdout ?? '';
|
||||
expect(Buffer.byteLength(out, 'utf-8')).toBe(PAYLOAD_BYTES + 4);
|
||||
expect(out.endsWith('END\n')).toBe(true);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test('exit code survives the flush (errored command stays non-zero)', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'gbrain-flush-code-'));
|
||||
const script = join(dir, 'flush-code-child.ts');
|
||||
try {
|
||||
writeFileSync(
|
||||
script,
|
||||
[
|
||||
`import { flushStdoutThenExit } from ${JSON.stringify(join(REPO, 'src', 'core', 'cli-force-exit.ts'))};`,
|
||||
`process.stdout.write('partial output before failure\\n');`,
|
||||
`void flushStdoutThenExit(3);`,
|
||||
].join('\n'),
|
||||
);
|
||||
const res = spawnSync('bun', [script], {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
encoding: 'utf-8',
|
||||
timeout: 30_000,
|
||||
});
|
||||
expect(res.status).toBe(3);
|
||||
expect(res.stdout).toBe('partial output before failure\n');
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
describe('cli pipe completeness — deliberate exit never truncates piped stdout (#2084)', () => {
|
||||
test('real CLI: --tools-json over a pipe is complete, parseable, byte-stable, and prompt', () => {
|
||||
const run = () => {
|
||||
const t0 = Date.now();
|
||||
@@ -98,7 +34,7 @@ describe('cli pipe truncation — deliberate exit flushes piped stdout (#2084)',
|
||||
// Truncated JSON does not parse — the strongest single-run completeness check.
|
||||
const parsed = JSON.parse(first.stdout);
|
||||
expect(Array.isArray(parsed)).toBe(true);
|
||||
// Deliberate exit, not the 10s hard-deadline backstop.
|
||||
// Deliberate exit, not the teardown backstop.
|
||||
expect(first.ms).toBeLessThan(9_000);
|
||||
|
||||
const second = run();
|
||||
|
||||
@@ -38,6 +38,18 @@ describe('shouldForceExitAfterMain — daemon survival gate', () => {
|
||||
expect(shouldForceExitAfterMain(['get', 'people/alice'])).toBe(true);
|
||||
});
|
||||
|
||||
test('#2084 cross-model finding: space-separated global flag values cannot fake a command', () => {
|
||||
// `--timeout 30s serve` — the old first-non-dash heuristic resolved the
|
||||
// command as `30s` → true → the central exit seam would process.exit the
|
||||
// freshly started daemon ~250ms after boot, exit 0, no error. The gate now
|
||||
// resolves the command through parseGlobalFlags, matching main()'s dispatch.
|
||||
expect(shouldForceExitAfterMain(['--timeout', '30s', 'serve'])).toBe(false);
|
||||
expect(shouldForceExitAfterMain(['--timeout', '30s', 'serve', '--http'])).toBe(false);
|
||||
expect(shouldForceExitAfterMain(['--progress-interval', '500', 'serve'])).toBe(false);
|
||||
// ...and the same shape before a one-shot command still force-exits.
|
||||
expect(shouldForceExitAfterMain(['--timeout', '30s', 'query', 'x'])).toBe(true);
|
||||
});
|
||||
|
||||
test('returns true for non-daemon CLI commands', () => {
|
||||
expect(shouldForceExitAfterMain(['stats'])).toBe(true);
|
||||
expect(shouldForceExitAfterMain(['doctor'])).toBe(true);
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { spawn, spawnSync } from 'child_process';
|
||||
import {
|
||||
cpSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
rmSync,
|
||||
@@ -152,12 +153,13 @@ afterAll(() => {
|
||||
function runWithTimeout(
|
||||
args: string[],
|
||||
timeoutMs: number,
|
||||
envOverride?: Record<string, string>,
|
||||
): Promise<{ code: number | null; stdout: string; stderr: string; durationMs: number }> {
|
||||
return new Promise((resolveOut) => {
|
||||
const t0 = Date.now();
|
||||
const child = spawn(SHIM_PATH, args, {
|
||||
cwd: REPO_ROOT,
|
||||
env: runEnv,
|
||||
env: envOverride ? { ...runEnv, ...envOverride } : runEnv,
|
||||
});
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
@@ -173,6 +175,13 @@ function runWithTimeout(
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* #2084: the teardown backstop banner must NEVER appear on a healthy run —
|
||||
* it now means a teardown component violated its own bound, not "this
|
||||
* command was slower than 10s end-to-end" (the pre-#2084 misfire).
|
||||
*/
|
||||
const TEARDOWN_BANNER = 'did not return within';
|
||||
|
||||
describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290)', () => {
|
||||
test('gbrain search "foxtrot" exits 0 within 15s', async () => {
|
||||
const { code, stdout, stderr, durationMs } = await runWithTimeout(
|
||||
@@ -185,6 +194,7 @@ describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290
|
||||
`STDOUT:\n${stdout}\nSTDERR:\n${stderr}`,
|
||||
);
|
||||
}
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).toBe(0);
|
||||
// Must have actually returned a hit — else bumpLastRetrievedAt
|
||||
// would have early-returned on empty pageIds and the bug wouldn't
|
||||
@@ -203,6 +213,7 @@ describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290
|
||||
`STDOUT:\n${stdout}\nSTDERR:\n${stderr}`,
|
||||
);
|
||||
}
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).toBe(0);
|
||||
expect(stdout).toContain('foxtrot');
|
||||
}, 30_000);
|
||||
@@ -258,6 +269,7 @@ describe('v0.42.20.0 — gbrain capture (CLI_ONLY) exits cleanly + frees the loc
|
||||
|
||||
// The real lock-pin symptom: the NEXT command times out waiting for the
|
||||
// PGLite lock. Assert a subsequent read runs cleanly and quickly.
|
||||
expect(cap.stderr).not.toContain(TEARDOWN_BANNER);
|
||||
const next = await runWithTimeout(['get', 'meetings/capture-test'], 15_000);
|
||||
expect(next.durationMs).toBeLessThan(15_000);
|
||||
expect(next.stderr).not.toContain('Timed out waiting for PGLite lock');
|
||||
@@ -267,6 +279,136 @@ describe('v0.42.20.0 — gbrain capture (CLI_ONLY) exits cleanly + frees the loc
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describe('#2084 — explicit-exit teardown: every swept site exits clean, exit codes report the op', () => {
|
||||
// D6C hardening: mutating commands run against a throwaway COPY of the
|
||||
// seeded GBRAIN_HOME so a remediation/dream pass can't contaminate the
|
||||
// brain the other tests share.
|
||||
function copyBrainHome(label: string): string {
|
||||
const copy = mkdtempSync(join(tmpdir(), `gbrain-2084-${label}-`));
|
||||
cpSync(tmpHome, copy, { recursive: true });
|
||||
return copy;
|
||||
}
|
||||
|
||||
test('D5: failed op exits 1 with the error on stderr (exit code = op outcome)', async () => {
|
||||
const { code, stderr, durationMs } = await runWithTimeout(
|
||||
['get', 'nonexistent-slug-2084'],
|
||||
15_000,
|
||||
);
|
||||
expect(durationMs).toBeLessThan(15_000);
|
||||
expect(code).toBe(1);
|
||||
expect(stderr.length).toBeGreaterThan(0);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
}, 30_000);
|
||||
|
||||
test('search stats (dashboard path, Site C) exits 0, no banner', async () => {
|
||||
const { code, stdout, stderr } = await runWithTimeout(['search', 'stats'], 20_000);
|
||||
expect(code).toBe(0);
|
||||
expect(stdout.length).toBeGreaterThan(0);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
}, 30_000);
|
||||
|
||||
test('sources list (read-only timeout path, Site D) exits 0, no banner', async () => {
|
||||
const { code, stderr } = await runWithTimeout(['sources', 'list'], 20_000);
|
||||
expect(code).toBe(0);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
}, 30_000);
|
||||
|
||||
test('doctor (Site G — leak-fix shape) exits without hanging, no banner', async () => {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(['doctor'], 45_000);
|
||||
expect(durationMs).toBeLessThan(45_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
// Keyless CI may surface advisory findings; doctor's exit code reflects
|
||||
// brain health, not teardown health. The pin is: exits, no banner.
|
||||
expect(code).not.toBeNull();
|
||||
}, 60_000);
|
||||
|
||||
test('doctor --remediation-plan (Site F) exits without hanging, no banner', async () => {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(
|
||||
['doctor', '--remediation-plan'],
|
||||
30_000,
|
||||
);
|
||||
expect(durationMs).toBeLessThan(30_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).not.toBeNull();
|
||||
}, 45_000);
|
||||
|
||||
test('doctor --remediate (Site F, mutating) runs on a brain copy, exits, no banner', async () => {
|
||||
const copy = copyBrainHome('remediate');
|
||||
try {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(
|
||||
['doctor', '--remediate'],
|
||||
45_000,
|
||||
{ GBRAIN_HOME: copy },
|
||||
);
|
||||
expect(durationMs).toBeLessThan(45_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).not.toBeNull();
|
||||
} finally {
|
||||
rmSync(copy, { recursive: true, force: true });
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test('dream --dry-run (Site E — the overnight-cron TODO site) exits, no banner', async () => {
|
||||
const copy = copyBrainHome('dream');
|
||||
try {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(
|
||||
['dream', '--dry-run'],
|
||||
60_000,
|
||||
{ GBRAIN_HOME: copy },
|
||||
);
|
||||
// Keyless CI: LLM-dependent phases degrade; the IRON rule is exits + no
|
||||
// banner. A hang here is the silent-overnight-zombie regression.
|
||||
expect(durationMs).toBeLessThan(60_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).not.toBeNull();
|
||||
} finally {
|
||||
rmSync(copy, { recursive: true, force: true });
|
||||
}
|
||||
}, 90_000);
|
||||
|
||||
test('ze-switch --dry-run (Site H) exits without hanging, no banner', async () => {
|
||||
const { code, durationMs, stderr } = await runWithTimeout(
|
||||
['ze-switch', '--dry-run'],
|
||||
30_000,
|
||||
);
|
||||
expect(durationMs).toBeLessThan(30_000);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(code).not.toBeNull();
|
||||
}, 45_000);
|
||||
|
||||
test('D11: teardown deadline does NOT cover handler time (slow-handler regression)', async () => {
|
||||
// Post-fix the deadline arms at teardown start, so a 500ms deadline cannot
|
||||
// touch the handler: results print in full regardless. NOTE the falsification
|
||||
// story is forward-looking, not historical — pre-#2084 code had no env knob
|
||||
// (a static 10s constant), so this spawn would pass there too; the guard
|
||||
// against re-hoisting the timer above the handler is the structural pin on
|
||||
// DISCONNECT_HARD_DEADLINE_MS absence in fix-wave-structural.test.ts. This
|
||||
// test pins that the env override is honored AND output survives a deadline
|
||||
// far smaller than handler time.
|
||||
const { code, stdout, durationMs } = await runWithTimeout(
|
||||
['search', 'foxtrot', '--limit', '3'],
|
||||
15_000,
|
||||
{ GBRAIN_TEARDOWN_DEADLINE_MS: '500' },
|
||||
);
|
||||
expect(durationMs).toBeLessThan(15_000);
|
||||
expect(code).toBe(0);
|
||||
expect(stdout.length).toBeGreaterThan(0); // output intact = handler wasn't killed
|
||||
}, 30_000);
|
||||
|
||||
test('D10: piped --json output parses complete (no exit truncation)', async () => {
|
||||
// `search stats --json` emits a pure JSON document (the shared-op search
|
||||
// path renders human format regardless of --json). A truncated-by-exit
|
||||
// pipe fails to parse — the #1959 class, end-to-end.
|
||||
const { code, stdout, stderr } = await runWithTimeout(
|
||||
['search', 'stats', '--json'],
|
||||
20_000,
|
||||
);
|
||||
expect(code).toBe(0);
|
||||
expect(stderr).not.toContain(TEARDOWN_BANNER);
|
||||
expect(() => JSON.parse(stdout)).not.toThrow();
|
||||
}, 30_000);
|
||||
});
|
||||
|
||||
describe('v0.41.8.0 — daemon survival (regression guard for narrow force-exit)', () => {
|
||||
test('gbrain serve --http stays alive past the timeout window', async () => {
|
||||
// Pick a likely-free ephemeral port. We're testing "still alive
|
||||
|
||||
@@ -72,6 +72,9 @@ describe('awaitPendingEvalCaptures', () => {
|
||||
const r = await awaitPendingEvalCaptures(150);
|
||||
const elapsed = Date.now() - start;
|
||||
expect(r.unfinished).toBe(1);
|
||||
expect(elapsed).toBeLessThan(1000);
|
||||
// Bound proves "bounded, not a hang" — the alternative is infinite. 2s
|
||||
// (13x the 150ms budget) absorbs CI shard-load timer jitter; the old 1s
|
||||
// bound flaked at 1023ms on a loaded GitHub runner.
|
||||
expect(elapsed).toBeLessThan(2000);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -125,20 +125,15 @@ describe('v0.36.1.x #1124 — query --no-expand actually negates expand', () =>
|
||||
|
||||
describe('v0.42.20.0 — background-work registry drains every sink before disconnect', () => {
|
||||
// Supersedes the v0.41.8.0 #1247/#1269/#1290 per-call last-retrieved drain:
|
||||
// last-retrieved is one of the registry sinks. Since the v0.43 #2084 drain
|
||||
// hoist, cli.ts routes EVERY owner-disconnect through drainThenDisconnect
|
||||
// (one helper, all 8 sites) instead of two inline drain+disconnect pairs.
|
||||
test('cli.ts imports the registry drain + routes owner-disconnects through drainThenDisconnect', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
expect(src).toMatch(/import\s+\{\s*drainAllBackgroundWorkForCliExit\s*\}\s*from\s+['"]\.\/core\/background-work\.ts['"]/);
|
||||
// The single registry-drain call site lives inside the shared helper...
|
||||
expect(src).toMatch(/export async function drainThenDisconnect/);
|
||||
expect(src).toMatch(/await\s+drainAllBackgroundWorkForCliExit\s*\(/);
|
||||
// ...and the helper covers every owner-disconnect (op-dispatch, CLI_ONLY
|
||||
// fall-through, search dashboard, doctor x3, ze-switch, dream,
|
||||
// read-only-timeout path).
|
||||
const calls = src.match(/await\s+drainThenDisconnect\s*\(/g) ?? [];
|
||||
expect(calls.length).toBeGreaterThanOrEqual(8);
|
||||
// last-retrieved is one of four registry sinks. #2084 moved the registry
|
||||
// drain out of cli.ts's inline finallys into finishCliTeardown
|
||||
// (cli-force-exit.ts), which every cli.ts teardown site routes through —
|
||||
// the drain-before-disconnect invariant is pinned there (and behaviorally
|
||||
// by test/cli-finish-teardown.test.ts).
|
||||
test('cli-force-exit.ts imports + drains the registry inside finishCliTeardown', () => {
|
||||
const src = readFileSync('src/core/cli-force-exit.ts', 'utf8');
|
||||
expect(src).toMatch(/import\s+\{\s*drainAllBackgroundWorkForCliExit[\s\S]*?\}\s*from\s+['"]\.\/background-work\.ts['"]/);
|
||||
expect(src).toMatch(/export async function finishCliTeardown/);
|
||||
});
|
||||
|
||||
test('last-retrieved.ts still exports the bounded drain + registers a drainer', () => {
|
||||
@@ -160,24 +155,18 @@ describe('v0.42.20.0 — background-work registry drains every sink before disco
|
||||
.toMatch(/name:\s*'eval-capture'/);
|
||||
});
|
||||
|
||||
test('cli.ts behavioral positioning: drainThenDisconnect drains the registry BEFORE engine.disconnect', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
// The ordering invariant lives ONCE, inside the shared helper (v0.43
|
||||
// #2084 drain hoist): registry drain, THEN best-effort disconnect.
|
||||
const helper = src.match(/export async function drainThenDisconnect[\s\S]+?^\}/m);
|
||||
expect(helper).not.toBeNull();
|
||||
const block = helper![0];
|
||||
const drainCallRe = /await\s+drainAllBackgroundWorkForCliExit\s*\(/;
|
||||
const disconnectCallRe = /await\s+engine\.disconnect\s*\(/;
|
||||
expect(block).toMatch(drainCallRe);
|
||||
expect(block).toMatch(disconnectCallRe);
|
||||
const drainIdx = block.indexOf(block.match(drainCallRe)![0]);
|
||||
const disconnectIdx = block.indexOf(block.match(disconnectCallRe)![0]);
|
||||
test('finishCliTeardown positioning: registry drain appears BEFORE engine disconnect', () => {
|
||||
// #2084: the invariant moved from cli.ts's inline finallys into the shared
|
||||
// helper. The drain must run against a live engine (facts abort-path
|
||||
// logIngest, #1762) before disconnect tears the pools down.
|
||||
const src = readFileSync('src/core/cli-force-exit.ts', 'utf8');
|
||||
const drainCallRe = /await\s+drain\s*\(\s*\{\s*timeoutMs:\s*drainTimeoutMs\s*\}\s*\)/;
|
||||
const disconnectCallRe = /await\s+opts\.engine\.disconnect\s*\(/;
|
||||
expect(src).toMatch(drainCallRe);
|
||||
expect(src).toMatch(disconnectCallRe);
|
||||
const drainIdx = src.indexOf(src.match(drainCallRe)![0]);
|
||||
const disconnectIdx = src.indexOf(src.match(disconnectCallRe)![0]);
|
||||
expect(drainIdx).toBeLessThan(disconnectIdx);
|
||||
// And the op-dispatch finally routes through the helper.
|
||||
const localPath = src.match(/\/\/ Local engine path \(unchanged behavior[\s\S]+?\n \}\n\}/m);
|
||||
expect(localPath).not.toBeNull();
|
||||
expect(localPath![0]).toMatch(/await\s+drainThenDisconnect\s*\(/);
|
||||
});
|
||||
|
||||
test('background-work.ts: Map registry, ordered drain, awaited abort, test seam', () => {
|
||||
@@ -195,6 +184,56 @@ describe('v0.42.20.0 — background-work registry drains every sink before disco
|
||||
});
|
||||
});
|
||||
|
||||
describe('#2084 — cli.ts owns process-exit teardown via finishCliTeardown', () => {
|
||||
test('no bare awaited engine disconnects remain in cli.ts', () => {
|
||||
// The awaited forms are the call-site contract (comments never use the
|
||||
// awaited literal, so this is comment-proof — eng-review D13.2). A bare
|
||||
// disconnect skips the bounded drain + computed-deadline backstop and
|
||||
// reopens the lingering-socket hang class.
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
expect(src).not.toContain('await engine.disconnect()');
|
||||
expect(src).not.toContain('await eng.disconnect()');
|
||||
});
|
||||
|
||||
test('the pre-handler hard-deadline timer is gone (handler time is not teardown budget)', () => {
|
||||
// Pre-#2084 the op-dispatch timer armed BEFORE the op handler, so any op
|
||||
// slower than 10s was force-killed mid-run with exit 0 and truncated
|
||||
// output. The deadline now arms inside finishCliTeardown, at teardown
|
||||
// start only.
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
expect(src).not.toContain('DISCONNECT_HARD_DEADLINE_MS');
|
||||
});
|
||||
|
||||
test('all nine swept sites route through finishCliTeardown; one exit seam', () => {
|
||||
const src = readFileSync('src/cli.ts', 'utf8');
|
||||
const calls = src.match(/await finishCliTeardown\(/g) ?? [];
|
||||
expect(calls.length).toBeGreaterThanOrEqual(9);
|
||||
// The single process-exit seam: flushThenExit in the import.meta.main
|
||||
// block, fed by currentExitCode().
|
||||
expect(src).toMatch(/import\.meta\.main/);
|
||||
expect(src).toMatch(/flushThenExit\(currentExitCode\(\)\)/);
|
||||
});
|
||||
|
||||
test('pglite-engine contains the Emscripten process.exitCode hijack', () => {
|
||||
// PGLite's WASM runtime writes its own status into process.exitCode (99
|
||||
// alive / exit status on close) and ignores `undefined` assignment. The
|
||||
// create call runs inside preservingProcessExitCode to keep the global
|
||||
// tidy; close is deliberately unwrapped (see below) — the CLI's verdict
|
||||
// is immune either way via the owned channel.
|
||||
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
|
||||
expect(src).toMatch(/preservingProcessExitCode\(\(\)\s*=>\s*\n?\s*PGlite\.create/);
|
||||
// close stays UNWRAPPED by design: its status write is baseline behavior
|
||||
// test runners depend on; the CLI's verdict is immune because it lives in
|
||||
// the gbrain-owned channel, never read back from process.exitCode.
|
||||
const helper = readFileSync('src/core/cli-force-exit.ts', 'utf8');
|
||||
expect(helper).toMatch(/let cliVerdict: number \| null = null/);
|
||||
expect(helper).toMatch(/return cliVerdict \?\? 0/);
|
||||
// The op-dispatch catch must set the verdict through the owned channel.
|
||||
const cli = readFileSync('src/cli.ts', 'utf8');
|
||||
expect(cli).toMatch(/setCliExitVerdict\(1\);/);
|
||||
});
|
||||
});
|
||||
|
||||
describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => {
|
||||
test('pglite-engine.ts exports classifyPgliteInitError + buildPgliteInitErrorMessage', () => {
|
||||
const src = readFileSync('src/core/pglite-engine.ts', 'utf8');
|
||||
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
/**
|
||||
* #2084 (D10) — spawned harness proving flushThenExit against REAL Bun pipe
|
||||
* semantics: writes HARNESS_BYTES of 'x' to stdout, then flushThenExit with
|
||||
* HARNESS_EXIT_CODE. The parent test pipes stdout to a slow-attaching reader
|
||||
* and asserts byte-complete output + the exit code — the exact scenario the
|
||||
* pre-#2084 force-exit truncated (#1959).
|
||||
*/
|
||||
|
||||
import { flushThenExit } from '../../src/core/cli-force-exit.ts';
|
||||
|
||||
const size = Number(process.env.HARNESS_BYTES ?? 4_000_000);
|
||||
const code = Number(process.env.HARNESS_EXIT_CODE ?? 7);
|
||||
const guardMs = Number(process.env.HARNESS_GUARD_MS ?? 2_000);
|
||||
const graceEnv = process.env.HARNESS_GRACE_MS;
|
||||
|
||||
const chunk = 'x'.repeat(65_536);
|
||||
let written = 0;
|
||||
while (written < size) {
|
||||
const n = Math.min(chunk.length, size - written);
|
||||
process.stdout.write(n === chunk.length ? chunk : chunk.slice(0, n));
|
||||
written += n;
|
||||
}
|
||||
|
||||
flushThenExit(code, {
|
||||
guardMs,
|
||||
...(graceEnv !== undefined ? { graceMs: Number(graceEnv) } : {}),
|
||||
});
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* #2084 (D10) — flushThenExit proven on a real spawned Bun process.
|
||||
*
|
||||
* The unit tests in cli-finish-teardown.test.ts inject fake streams; they
|
||||
* prove the helper's logic but not Bun's actual pipe behavior (does an empty
|
||||
* write('', cb) really fence all prior buffered chunks?). These tests spawn
|
||||
* test/fixtures/flush-then-exit-harness.ts and assert:
|
||||
* 1. multi-MB piped stdout arrives byte-complete with the right exit code
|
||||
* even when the reader attaches late (#1959 truncation regression pin);
|
||||
* 2. with empty buffers the fence resolves promptly — the process does NOT
|
||||
* sit out the flush guard (canary for Bun eliding empty-write callbacks).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { spawn } from 'child_process';
|
||||
import { resolve } from 'path';
|
||||
|
||||
const HARNESS = resolve(import.meta.dir, 'fixtures', 'flush-then-exit-harness.ts');
|
||||
|
||||
function runHarness(env: Record<string, string>, readerDelayMs: number): Promise<{
|
||||
bytes: number;
|
||||
code: number | null;
|
||||
durationMs: number;
|
||||
}> {
|
||||
return new Promise((resolveOut, reject) => {
|
||||
const t0 = Date.now();
|
||||
const child = spawn('bun', ['run', HARNESS], {
|
||||
env: { ...process.env, ...env },
|
||||
stdio: ['ignore', 'pipe', 'inherit'],
|
||||
});
|
||||
let bytes = 0;
|
||||
child.stdout.pause();
|
||||
setTimeout(() => {
|
||||
child.stdout.on('data', (d: Buffer) => (bytes += d.length));
|
||||
child.stdout.resume();
|
||||
}, readerDelayMs);
|
||||
const killer = setTimeout(() => {
|
||||
child.kill('SIGKILL');
|
||||
reject(new Error(`harness did not exit (bytes so far: ${bytes})`));
|
||||
}, 30_000);
|
||||
child.on('close', (code) => {
|
||||
clearTimeout(killer);
|
||||
resolveOut({ bytes, code, durationMs: Date.now() - t0 });
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
describe('flushThenExit on a real Bun process (D10)', () => {
|
||||
test('4MB piped stdout arrives byte-complete with the exit code, late reader', async () => {
|
||||
// Bun delivers queued pipe writes only while the process is alive (see the
|
||||
// cli-force-exit.ts module header). The reader attaches 200ms late; the
|
||||
// 1500ms aliveness grace must cover attach + transfer. Pre-#2084 (immediate
|
||||
// process.exit, no grace) this received 0 of the 4MB — verified during
|
||||
// implementation; that is the #1959 truncation class.
|
||||
const SIZE = 4_000_000;
|
||||
const { bytes, code } = await runHarness(
|
||||
{
|
||||
HARNESS_BYTES: String(SIZE),
|
||||
HARNESS_EXIT_CODE: '7',
|
||||
HARNESS_GUARD_MS: '2000',
|
||||
HARNESS_GRACE_MS: '1500',
|
||||
},
|
||||
200,
|
||||
);
|
||||
expect(bytes).toBe(SIZE);
|
||||
expect(code).toBe(7);
|
||||
}, 40_000);
|
||||
|
||||
test('default grace: small output survives exit with a concurrent reader', async () => {
|
||||
// No HARNESS_GRACE_MS → production default (non-TTY grace). Immediate
|
||||
// reader, 100-byte output: must arrive complete. Pre-#2084 even this case
|
||||
// lost ALL bytes when exit fired before a loop turn.
|
||||
const { bytes, code } = await runHarness(
|
||||
{ HARNESS_BYTES: '100', HARNESS_EXIT_CODE: '0', HARNESS_GUARD_MS: '2000' },
|
||||
0,
|
||||
);
|
||||
expect(bytes).toBe(100);
|
||||
expect(code).toBe(0);
|
||||
}, 40_000);
|
||||
|
||||
test('fence resolves promptly — wall time well under guard + grace ceiling', async () => {
|
||||
// Guard 8s, grace 250ms: if Bun ever elides the empty-write callback, the
|
||||
// process sits out the full guard and wall time exceeds it. A working
|
||||
// fence exits in startup time + grace (~1-2s).
|
||||
const { code, durationMs } = await runHarness(
|
||||
{
|
||||
HARNESS_BYTES: '100',
|
||||
HARNESS_EXIT_CODE: '0',
|
||||
HARNESS_GUARD_MS: '8000',
|
||||
HARNESS_GRACE_MS: '250',
|
||||
},
|
||||
0,
|
||||
);
|
||||
expect(code).toBe(0);
|
||||
expect(durationMs).toBeLessThan(6_000);
|
||||
}, 40_000);
|
||||
});
|
||||
@@ -219,3 +219,38 @@ describe('PGLiteEngine.disconnect() — v0.41.8.0 lifecycle invariants', () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// #2084 — preservingProcessExitCode behavioral containment
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
describe('PGLiteEngine: Emscripten process.exitCode containment (#2084)', () => {
|
||||
test('connect() leaves process.exitCode pinned at 0, not the Emscripten 99', async () => {
|
||||
const prev = process.exitCode;
|
||||
const eng = new PGLiteEngine();
|
||||
try {
|
||||
await eng.connect({ engine: 'pglite' });
|
||||
// Emscripten writes 99 during create; the wrapper pins explicit 0 when
|
||||
// nothing was set before (undefined cannot be restored — the accessor
|
||||
// falls back to the WASM status).
|
||||
expect(Number(process.exitCode)).toBe(0);
|
||||
} finally {
|
||||
await eng.disconnect();
|
||||
process.exitCode = prev;
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test('a pre-call verdict survives the create-throw path (finally restores)', async () => {
|
||||
const prev = process.exitCode;
|
||||
const eng = new PGLiteEngine();
|
||||
try {
|
||||
process.exitCode = 3;
|
||||
// A dataDir under a regular FILE cannot be created — PGlite.create rejects.
|
||||
await expect(
|
||||
eng.connect({ engine: 'pglite', database_path: '/dev/null/nope/brain' }),
|
||||
).rejects.toThrow();
|
||||
expect(Number(process.exitCode)).toBe(3);
|
||||
} finally {
|
||||
process.exitCode = prev;
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
@@ -1311,7 +1311,10 @@ describe('PGLiteEngine: v0.13.1 error-wrap on connect() (#223)', () => {
|
||||
// issue and suggest gbrain doctor. Must NOT suggest "missing migrations"
|
||||
// as a cause (that was conflating #218 and #223 — migrations run AFTER
|
||||
// create()).
|
||||
expect(src).toContain('this._db = await PGlite.create');
|
||||
// #2084 wrapped the create call in preservingProcessExitCode (Emscripten
|
||||
// exitCode containment); the try/catch + error wrap around it is unchanged.
|
||||
expect(src).toContain('this._db = await preservingProcessExitCode(() =>');
|
||||
expect(src).toContain('PGlite.create({');
|
||||
expect(src).toContain('https://github.com/garrytan/gbrain/issues/223');
|
||||
expect(src).toContain('gbrain doctor');
|
||||
expect(src).toContain('Original error:');
|
||||
|
||||
Reference in New Issue
Block a user