Files
gbrain/test/child-worker-supervisor.test.ts
T
bb2e88c42a v0.42.52.0 fix(reliability): autopilot dead-job storm + supervisor wedge + sync/status/minion reliability (#2194 #2227 #1994 #1737 #1738 #1950 #1984) (#2287)
* test(supervisor): pin LOCK_HELD fence-exit is never counted as a crash (#2227)

A duplicate supervisor loses the queue-scoped DB singleton lock (#1849) and
exits LOCK_HELD before spawning a worker or emitting 'started'. summarizeCrashes
counts only worker_exited, so the fence path is structurally uncountable. Pin it
so a future refactor that logs worker_exited on the fence path fails here instead
of silently re-introducing the crash-budget breaker-trip loop.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autopilot): per-source cycle binds FS phases to source.local_path, not global repo (#2194 #2227)

A per-source autopilot-cycle inherited the global sync.repo_path as brainDir while
stamping DB freshness for source_id — mixed scope. FS phases (sync/lint/extract)
ran against the wrong tree, so the failure-cooldown and freshness gates would
attribute work to the wrong source. Resolve the source's local_path in the handler
(reuse the archive-recheck SELECT) and bind brainDir to it; a pure-DB source gets
null (FS phases skip) instead of falling through to the global checkout. Legacy
no-source dispatch keeps the global repoPath. Prerequisite for the cooldown/split
commits (codex outside-voice #8). Resolves TODOS:634.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(supervisor): detect a live supervisor via the DB lock under split $HOME (#2227)

jobs supervisor status + doctor read the HOME-derived pidfile, so a supervisor
started under a different $HOME (keeper=/root vs ops=/data) read as 'not running'
while healthy — the false signal that drives an operator to spawn a duplicate.
Both surfaces now fall back to the queue-scoped DB singleton lock (#1849), the
HOME-independent authority, when the pidfile shows nothing. New isLockHolderLive
keys on lock freshness (ttl + heartbeat steal-grace), never process.kill, so PID
reuse can't false-positive (pid-liveness-alone-pid-reuse). Status surfaces the
holder host/pid + recorded concurrency/max-rss from the latest started event.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(supervisor): degraded retry instead of permanent give-up on crash storm (#1994 #2227)

max_crashes_exceeded gave up forever, so a transient DB-pooler blip that tripped
the soft budget wedged the queue until a human restart (#2227's breaker-trips tail).
Crossing the soft budget now enters degraded mode: keep respawning with capped
exponential backoff (60s cap — a paced retry, not a hot loop) and emit a loud
crash_budget_degraded health_warn. The existing stable-run reset clears the count
once a respawn survives >5min, so a recovered DB self-heals. Permanent give-up
fires only at a much-higher hard ceiling (maxCrashes × 10), tunable/disablable via
GBRAIN_SUPERVISOR_HARD_STOP_CRASHES (0 = never). Resolves TODOS:92.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autopilot): clamp fan-out to worker concurrency + doctor warning (#2194)

Fan-out resolved to 4 (Postgres) regardless of worker --concurrency, so surplus
cycles queued behind the worker and raced the stalled-sweeper. Two fixes for the
same mismatch:
- resolveEffectiveFanoutMax clamps to max(1, concurrency-1) (reserve a slot),
  gated on a LIVE DB-lock holder so a stale started-audit row can't shrink
  throughput (codex #9/D5); no live holder → unknown → unclamped base. Escape
  hatch autopilot.fanout_clamp_to_concurrency.
- doctor's autopilot_fanout_concurrency check warns when fan-out exceeds
  effective slots — the misconfig was silent before. Advisory (started-event
  concurrency), wired into both doctor surfaces.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autopilot): per-source failure cooldown — break the dead-job storm (#2194)

Only SUCCESS gated dispatch, so a source whose cycle kept failing/timing-out
re-fanned-out every 5-min tick forever (200+ dead jobs/24h). Now a failed source
backs off with bounded exponential cooldown (10→120min). Read at DISPATCH from
minion_jobs dead/failed rows (timeouts/RSS-kills dead-letter via SQL and never
run handler code, so a write-only hook would miss them) AND re-checked at CLAIM
time in the handler (codex #5: already-queued/retrying jobs). A success clears it
(codex #7); null-source rows excluded (codex #6); engine-parity via executeRaw.
Disable with autopilot.failure_cooldown_min=0. Fail-open if config/history reads
error. Surfaced via fanout_cooldown_skipped + the fanout summary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autopilot): split the cycle — per-source phases + one global-maintenance job (#2194 #2227)

N per-source cycles each ran the brain-wide global phases (embed-all/orphans/
purge/…) concurrently, thrashing the same rows and taking the worker 4→10GB in
<60s → RSS-kill → orphaned stalls. Split them: per-source jobs now run only
source-scoped (+ mixed) phases and stamp last_source_cycle_at; a new
autopilot-global-maintenance job runs the global phases ONCE per window
(idempotency_key + maxWaiting:1 = structural single-flight) and stamps
autopilot.last_global_at. This is the codex-endorsed design that replaced the
rejected skip-and-stamp-fresh approach (codex #1/#2): no freshness poisoning, no
starvation — global work always runs as its own job, never marked done when it
wasn't. PHASE_SCOPE is now a runtime partition (GLOBAL ∪ NON_GLOBAL == ALL).
last_full_cycle_at still written for doctor/legacy (no longer a global gate).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(doctor): guard nullable engine in supervisor DB-lock fallback (#2227)

Follow-up to the supervisor-visibility commit: doctor's engine binding is
BrainEngine | null, so the inspectLock fallback must guard on a non-null engine
(tsc TS2345). No behavior change — a null engine simply skips the DB-lock probe
and falls back to the pidfile reading, as before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(doctor): categorize autopilot_fanout_concurrency check as ops (#2194)

Follow-up to the fan-out/concurrency commit: the doctor-categories drift guard
requires every check name in doctor.ts to belong to exactly one category set.
Add the new autopilot_fanout_concurrency check to OPS_CHECK_NAMES (infrastructure
liveness, alongside wedged_queue/supervisor).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: update KEY_FILES for the autopilot cycle split + supervisor degraded-retry (#2194 #2227)

Post-ship document-release: refresh the KEY_FILES current-state entries that
drifted — cycle.ts (GLOBAL/NON_GLOBAL phase split + last_source_cycle_at /
autopilot.last_global_at), jobs.ts (per-source local_path brainDir, claim-time
cooldown, autopilot-global-maintenance handler), supervisor.ts + child-worker
(degraded retry instead of permanent give-up; hard ceiling), db-lock.ts
(isLockHolderLive), handler-timeouts (new handler). Regenerated llms bundle.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minions): handleTimeouts counts the timed-out run as a spent attempt (#1737)

The per-job timeout_at dead-letter (handleTimeouts) set status='dead' without
incrementing attempts_made, unlike the wall-clock and stall dead-letter siblings.
It is the FIRST killer to fire for the long-lane handlers (subagent / embed-backfill
/ autopilot-cycle) because timeout_ms is stamped at submit, so a timed-out long job
reported `attempts: 0/N (started: N)`. Mirror the siblings with attempts_made + 1
(terminal, no retry). Safe against double-count: the worker sweep runs handleStalled
-> handleTimeouts -> handleWallClockTimeouts sequentially and awaited, each guarded on
status='active', so the first to dead-letter excludes the row from the rest.

Regression assertions added (test/minions.test.ts + e2e/minions-resilience.test.ts)
so the increment can't be silently dropped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(agent): recognize trailing switches in `agent run`, keep prompts freeform (#1738)

parseRunFlags() broke flag parsing at the first positional token, so any flag
after the prompt (`gbrain agent run "do X" --detach`) was swallowed into the
prompt string and silently ignored. Now the no-value switches --detach/--follow/
--no-follow are hoisted when they trail the prompt, while everything else stays
verbatim: an unknown --word is treated as prompt text (no "unknown flag" throw),
a --switch mid-prompt is preserved, and `--` suppresses hoisting entirely for a
literal escape. Value-flags now reject a missing or flag-shaped value (and
--max-turns/--timeout-ms a non-number) instead of capturing undefined/NaN.

Contract change: a prompt that starts with or trails an unguarded --word no
longer errors; a literal trailing --detach needs `--`. Help text updated; tests
revised + extended (test/agent-cli.test.ts).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sync): honest live-sync status + progress-aware stall-abort (#1950)

Finishes the #2255 honest-freshness story for two gaps it left.

(a) `gbrain sources status` printed "idle" while a sync proc held the per-source
lock (the reported bug). New shared liveSyncStatus() helper in db-lock.ts reads
the SAME live-lock signal `gbrain doctor` uses; runStatus now shows "running"
(BACKFILL column + a sync_running field in --json) and suppresses the misleading
"never synced" warning while a sync is live. One helper, so the surfaces can't
drift (doctor/status retrofit tracked as a follow-up).

(b) A sync wedged-but-alive kept refreshing its lock heartbeat (it fires on its
own timer) and hadn't hit the wall-clock deadline, so only a manual pkill freed
it. New in-band stall watchdog keys off FORWARD IMPORT PROGRESS (progress.tick),
not the heartbeat: if no file completes for GBRAIN_SYNC_STALL_ABORT_SECONDS
(default 900s), it aborts via a controller composed into opts.signal, so the
drain returns partial() (last_commit unchanged, next run resumes from the
checkpoint) and withRefreshingLock releases the lock. Limits, documented in
code: a single file slower than the window trips it; a fully starved event loop
won't fire the timer (the wall-clock hard deadline is that backstop).

Tests: liveSyncStatus (live/expired/none/per-source) in db-lock-inspect; the
resolveStallAbortSeconds env matrix in sync-hard-deadline.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(status): version field + per-section --deadline-ms budget (#1984)

`gbrain status` had no version in its JSON envelope and could hang on a slow
connection with no way to get a partial answer. Two additions:

- version: the StatusReport JSON now carries the local gbrain CLI version so a
  poller can pin behavior to a build. Thin-client also surfaces remote_version
  (the brain server's version), and the get_status_snapshot MCP op reports its
  version for that parity.
- --deadline-ms=N / --fast: a shared wall-clock budget. Each section is raced
  against the REMAINING budget via Promise.race (NOT process-watchdog, which
  SIGKILLs and can't return partial output), so one slow/hung section can't
  strand the snapshot — it's marked stale and the rest still return. The
  envelope gains partial:true + stale_sections[]; exit code stays 0 (a snapshot
  was produced). Invalid --deadline-ms → exit 2.

Tests: parseDeadlineFlag + withSectionDeadline (hermetic), the usage-error exit,
version presence in the PGLite envelope, and the op's version key.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sync): report stall_timeout distinctly + document in-flight limit (#1950)

Pre-landing review (codex + adversarial): the stall watchdog aborted opts.signal
but the per-iteration abort checks returned partial('timeout'), collapsing a
wedge-reap into a user --timeout/SIGINT so JSON consumers couldn't tell them
apart. Add a 'stall_timeout' reason (set via a stallAborted flag) on the three
import-loop abort sites; deletes/renames-phase and checkpoint sites stay 'timeout'.
Sharpen the watchdog comment: the abort is observed BETWEEN files, so a hang
inside a single importFile is not interrupted until it returns (TODO: thread a
cancellation signal through importFile).

* fix(agent): `--` escape suppresses trailing-switch hoisting anywhere (#1738)

Pre-landing review: the leading-flag loop breaks at the first positional, so the
`escaped` flag only fired for a leading `--`. A `--` placed after a positional
left trailing-switch hoisting active, so `agent run note -- body --detach`
silently detached and dropped the `--` as junk. Suppress hoisting whenever a
literal `--` appears in the prompt. Regression test added.

* fix(status): deadline-ms usage-error + scoped stale_sections + cancel losing remote call (#1984)

Pre-landing review (codex): (1) bare `--deadline-ms` with no value silently fell
through to no-budget/--fast instead of a usage error; (2) thin-client timeout
reported both sync+cycle stale even under `--section sync`, naming a section the
caller excluded (local path was already correct); (3) the section race abandoned
the remote promise locally but didn't cancel the in-flight MCP call — pass the
budget as timeoutMs so the losing side actually cancels. Regression test added.

* v0.42.52.0 fix(reliability): autopilot dead-job storm + supervisor wedge + sync/status/minion reliability (#2194 #2227 #1994 #1737 #1738 #1950 #1984)

Bundles the already-reviewed autopilot/supervisor stabilization (#2194 #2227
#1994: cycle split, per-source failure cooldown, fan-out clamp, degraded
supervisor retry, DB-lock live-supervisor detection) with four operational
fixes: minion timeout attempt-accounting (#1737), agent-run trailing-flag
parsing (#1738), honest live-sync sources status + progress-aware stall
watchdog (#1950), and status version + --deadline-ms partial result (#1984).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: document GBRAIN_SYNC_STALL_ABORT_SECONDS env knob (#1950)

Post-ship doc sync (/document-release): add the sync stall watchdog env var to
the CLAUDE.md sync-tuning table (Five → Six knobs) + regenerate the llms bundle.

* test: quarantine #2249 fanout tests as *.serial (R1 env-isolation) (#2194)

The cherry-picked autopilot-fanout-clamp + doctor-autopilot-fanout-concurrency
tests mutate process.env.GBRAIN_AUDIT_DIR in beforeEach/afterEach, which the
check:test-isolation R1 lint flags (parallel shards load multiple files per
process). Rename to *.serial.test.ts (sanctioned quarantine — they run under
--max-concurrency=1) instead of restructuring the reviewed test bodies. No logic
change; both files stay green (9 tests). Fixes the failing verify CI check.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 06:36:43 -07:00

679 lines
26 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* Tests for the shared spawn-and-respawn core used by MinionSupervisor
* and src/commands/autopilot.ts. Pins the D1 lastExitCode-track behavior
* and the D2 clean-restart-budget gate so future refactors can't silently
* regress the supervisor crash-count incident this wave fixes.
*
* Strategy: each test writes a small shell script to disk that exits with
* a chosen code after an optional sleep. The class spawns the script as
* the "worker" and we assert on the event stream the class emits.
*/
import { describe, it, expect, afterEach } from 'bun:test';
import { chmodSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import {
ChildWorkerSupervisor,
type ChildSupervisorEvent,
} from '../src/core/minions/child-worker-supervisor.ts';
interface Harness {
workerScript: string;
cleanup: () => void;
}
function makeHarness(name: string, body: string): Harness {
const root = join(tmpdir(), `gbrain-cws-test-${name}-${process.pid}-${Date.now()}`);
mkdirSync(root, { recursive: true });
const workerScript = join(root, 'worker.sh');
writeFileSync(workerScript, `#!/bin/sh\n${body}\n`, 'utf8');
chmodSync(workerScript, 0o755);
return {
workerScript,
cleanup: () => {
try {
rmSync(root, { recursive: true, force: true });
} catch {
/* noop */
}
},
};
}
interface RunResult {
events: ChildSupervisorEvent[];
maxCrashesFired: { count: number; max: number } | null;
}
async function runUntilTerminal(
h: Harness,
overrides: Partial<{
maxCrashes: number;
hardStopMaxCrashes: number;
_backoffFloorMs: number;
cleanRestartBudget: number;
cleanRestartWindowMs: number;
cleanRestartBudgetBackoffMs: number;
stableRunResetMs: number;
watchdogLoopBudget: number;
watchdogLoopWindowMs: number;
watchdogBackoffMs: number;
_now: () => number;
stopAfterEvents: number; // safety net so a buggy test can't hang
}>,
): Promise<RunResult> {
const events: ChildSupervisorEvent[] = [];
let stopping = false;
let maxCrashesFired: { count: number; max: number } | null = null;
const stopAfter = overrides.stopAfterEvents ?? 200;
const sup = new ChildWorkerSupervisor({
cliPath: h.workerScript,
args: [],
maxCrashes: overrides.maxCrashes ?? 3,
hardStopMaxCrashes: overrides.hardStopMaxCrashes,
_backoffFloorMs: overrides._backoffFloorMs ?? 5,
cleanRestartBudget: overrides.cleanRestartBudget,
cleanRestartWindowMs: overrides.cleanRestartWindowMs,
cleanRestartBudgetBackoffMs: overrides.cleanRestartBudgetBackoffMs,
stableRunResetMs: overrides.stableRunResetMs,
watchdogLoopBudget: overrides.watchdogLoopBudget,
watchdogLoopWindowMs: overrides.watchdogLoopWindowMs,
watchdogBackoffMs: overrides.watchdogBackoffMs,
_now: overrides._now,
isStopping: () => stopping,
onMaxCrashesExceeded: (count, max) => {
maxCrashesFired = { count, max };
stopping = true;
},
onEvent: (event) => {
events.push(event);
if (events.length >= stopAfter) {
stopping = true;
}
},
});
await sup.run();
return { events, maxCrashesFired };
}
afterEach(() => {
/* per-test harness.cleanup() runs in finally blocks below */
});
describe('ChildWorkerSupervisor', () => {
describe('D1 — code=0 exit classifier', () => {
it('code=0 worker exit does not count as crash; restarts immediately', async () => {
const h = makeHarness('clean-exits', 'exit 0');
try {
const res = await runUntilTerminal(h, {
maxCrashes: 3,
stopAfterEvents: 30, // ~10 spawn/exit/backoff trios
});
expect(res.maxCrashesFired).toBeNull();
const exits = res.events.filter((e) => e.kind === 'worker_exited');
expect(exits.length).toBeGreaterThanOrEqual(3);
for (const e of exits) {
if (e.kind === 'worker_exited') {
expect(e.code).toBe(0);
expect(e.likelyCause).toBe('clean_exit');
// crashCount stays at 0 across every clean exit
expect(e.crashCount).toBe(0);
}
}
const backoffs = res.events.filter((e) => e.kind === 'backoff');
expect(backoffs.length).toBeGreaterThanOrEqual(1);
// Within the default 10-restart budget, all backoffs are ms:0 / clean_exit
for (const e of backoffs) {
if (e.kind === 'backoff') {
// Once we cross the 10-restart budget the reason flips to
// budget_exceeded, but until then they're all clean_exit ms:0.
if (e.reason === 'clean_exit') {
expect(e.ms).toBe(0);
expect(e.crashCount).toBe(0);
}
}
}
} finally {
h.cleanup();
}
});
it('interleaved code=0 and code!=0 exits still trip max_crashes', async () => {
// Worker alternates: each invocation increments a counter file and
// exits 1 on odd hits, 0 on even hits (so exit-sequence is 1,0,1,0,1).
const h = makeHarness(
'interleaved',
`
COUNTER_FILE="$(dirname "$0")/counter"
[ -f "$COUNTER_FILE" ] || echo 0 > "$COUNTER_FILE"
COUNT=$(cat "$COUNTER_FILE")
NEXT=$((COUNT + 1))
echo "$NEXT" > "$COUNTER_FILE"
# Odd-indexed runs (#1, #3, #5...) exit 1; even-indexed exit 0.
if [ $((NEXT % 2)) -eq 1 ]; then exit 1; else exit 0; fi
`,
);
try {
const res = await runUntilTerminal(h, {
maxCrashes: 3,
// issue #1994: the soft budget no longer gives up; pin the hard
// ceiling to 3 so this counting test still fires give-up at 3.
hardStopMaxCrashes: 3,
_backoffFloorMs: 5,
stopAfterEvents: 200,
});
expect(res.maxCrashesFired).not.toBeNull();
// 3 code!=0 exits → hard ceiling=3
expect(res.maxCrashesFired!.count).toBe(3);
const exits = res.events.filter((e) => e.kind === 'worker_exited');
// Should be exactly 5 exits: 1, 0, 1, 0, 1 — then max fires.
const codes = exits
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited')
.map((e) => e.code);
expect(codes).toEqual([1, 0, 1, 0, 1]);
const backoffs = res.events
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'backoff' }> => e.kind === 'backoff');
// Backoffs only fire between iterations 1-4 (not after the 5th, since
// the loop bails out via onMaxCrashesExceeded before applyBackoff).
// Even-index exits (code=0, indices 1+3) → reason='clean_exit'.
// Odd-index exits (code=1, indices 0+2) → reason='crash'.
const reasons = backoffs.map((e) => e.reason);
expect(reasons).toEqual(['crash', 'clean_exit', 'crash', 'clean_exit']);
} finally {
h.cleanup();
}
});
it('code=0 after stable 5min+ run does not reset crashCount', async () => {
// Sequence (4 runs total): exit 1 → exit 0 (6 min, "stable") → exit 1 →
// exit 1. crashCount progression: 1, 1 (unchanged across the long
// clean exit), 2, 3 — last one trips max_crashes=3.
const h = makeHarness(
'stable-clean-no-reset',
`
COUNTER_FILE="$(dirname "$0")/counter"
[ -f "$COUNTER_FILE" ] || echo 0 > "$COUNTER_FILE"
COUNT=$(cat "$COUNTER_FILE")
NEXT=$((COUNT + 1))
echo "$NEXT" > "$COUNTER_FILE"
case $NEXT in
1) exit 1 ;;
2) exit 0 ;;
3) exit 1 ;;
4) exit 1 ;;
*) exit 0 ;;
esac
`,
);
try {
// Fake clock — each spawnOnce reads now() twice (start + exit) and
// applyBackoff may read once more. Run 2 sees a 6-minute duration
// (stable-run reset would fire IF the exit were code!=0 — we assert
// it does NOT fire when the exit is clean).
const SIX_MIN = 6 * 60_000;
const timestamps = [
0, // run 1 start
1_000, // run 1 exit (+1s) → crashCount 1
1_000, // run 2 start
1_000 + SIX_MIN, // run 2 exit (+6min) → code=0, stays at 1
1_000 + SIX_MIN, // run 3 start
1_000 + SIX_MIN + 1_000, // run 3 exit (+1s) → crashCount 2
1_000 + SIX_MIN + 1_000, // run 4 start
1_000 + SIX_MIN + 2_000, // run 4 exit (+1s) → crashCount 3, trips max
];
let idx = 0;
const last = timestamps[timestamps.length - 1];
const fakeNow = () => {
if (idx < timestamps.length) {
return timestamps[idx++];
}
return last + (idx++ - timestamps.length + 1) * 100;
};
const res = await runUntilTerminal(h, {
maxCrashes: 3,
hardStopMaxCrashes: 3, // issue #1994: pin give-up to 3 for this counting test
_backoffFloorMs: 5,
_now: fakeNow,
stopAfterEvents: 200,
});
expect(res.maxCrashesFired).not.toBeNull();
expect(res.maxCrashesFired!.count).toBe(3);
const exits = res.events
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited')
.map((e) => ({ code: e.code, crashCount: e.crashCount, runDurationMs: e.runDurationMs }));
expect(exits.length).toBeGreaterThanOrEqual(4);
expect(exits[0]).toMatchObject({ code: 1, crashCount: 1 });
expect(exits[1]).toMatchObject({ code: 0, crashCount: 1 }); // D1: unchanged
expect(exits[2]).toMatchObject({ code: 1, crashCount: 2 });
expect(exits[3]).toMatchObject({ code: 1, crashCount: 3 });
// Run 2 ran 6min, but because exit code was 0 the stable-run reset
// branch did NOT fire — crashCount stayed at 1. This is the core
// D1 invariant: clean exits never reset crashCount, even stable ones.
expect(exits[1].runDurationMs).toBe(SIX_MIN);
} finally {
h.cleanup();
}
});
});
describe('D2 — clean-restart budget', () => {
it('budget exceeded triggers health_warn + budget_exceeded backoff', async () => {
// Tight budget of 2 so we trip it on the 3rd clean exit.
const h = makeHarness('budget-trip', 'exit 0');
try {
const res = await runUntilTerminal(h, {
maxCrashes: 3, // never trips because code=0 doesn't increment
_backoffFloorMs: 5,
cleanRestartBudget: 2,
cleanRestartWindowMs: 60_000,
cleanRestartBudgetBackoffMs: 10,
stopAfterEvents: 25,
});
const healthWarns = res.events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> => e.kind === 'health_warn',
);
// Once tripped, every subsequent clean exit re-fires health_warn
// (the sliding window stays full at our test rate).
expect(healthWarns.length).toBeGreaterThan(0);
for (const w of healthWarns) {
expect(w.reason).toBe('clean_restart_budget_exceeded');
expect(w.windowMs).toBe(60_000);
expect(w.count).toBeGreaterThan(2);
}
const backoffReasons = res.events
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'backoff' }> => e.kind === 'backoff')
.map((e) => e.reason);
// First 2 exits are within budget → reason='clean_exit'.
// From the 3rd exit onward → reason='budget_exceeded'.
expect(backoffReasons.slice(0, 2)).toEqual(['clean_exit', 'clean_exit']);
expect(backoffReasons.slice(2).every((r) => r === 'budget_exceeded')).toBe(true);
} finally {
h.cleanup();
}
});
it('budget config is per-instance (no module-level state leakage)', async () => {
// Run instance A with budget=2 and instance B with budget=5. Each
// tracks its own sliding window; A trips faster than B.
const hA = makeHarness('budget-a', 'exit 0');
const hB = makeHarness('budget-b', 'exit 0');
try {
const resA = await runUntilTerminal(hA, {
maxCrashes: 99,
_backoffFloorMs: 5,
cleanRestartBudget: 2,
cleanRestartBudgetBackoffMs: 5,
stopAfterEvents: 12,
});
const resB = await runUntilTerminal(hB, {
maxCrashes: 99,
_backoffFloorMs: 5,
cleanRestartBudget: 5,
cleanRestartBudgetBackoffMs: 5,
stopAfterEvents: 18,
});
const firstTripA = resA.events.findIndex(
(e) => e.kind === 'health_warn',
);
const firstTripB = resB.events.findIndex(
(e) => e.kind === 'health_warn',
);
expect(firstTripA).toBeGreaterThan(-1);
expect(firstTripB).toBeGreaterThan(-1);
// B's budget is more generous → its first health_warn appears later
// in the event stream (after more spawn/exit pairs).
expect(firstTripB).toBeGreaterThan(firstTripA);
} finally {
hA.cleanup();
hB.cleanup();
}
});
});
describe('awaitChildExit short-circuit (P2 review fix)', () => {
// Regression: pre-fix the method registered child.once('exit', ...) AFTER
// child.exitCode was already populated, so a child that drained quickly
// between killChild('SIGTERM') and awaitChildExit() would never resolve
// and the caller waited out the full timeout. Fix probes exitCode +
// signalCode first and short-circuits.
it('resolves immediately when the child has already exited', async () => {
const h = makeHarness('await-already-exited', 'exit 0');
try {
// Spin up a supervisor; drive it for ONE spawn cycle and then stop.
const events: ChildSupervisorEvent[] = [];
let stopping = false;
const sup = new ChildWorkerSupervisor({
cliPath: h.workerScript,
args: [],
maxCrashes: 1,
_backoffFloorMs: 1,
isStopping: () => stopping,
onMaxCrashesExceeded: () => { stopping = true; },
onEvent: (e) => {
events.push(e);
if (e.kind === 'worker_exited') stopping = true;
},
});
await sup.run();
// After run() returns, the child has exited; awaitChildExit on an
// already-finished cycle MUST resolve in well under the timeout.
const start = Date.now();
await sup.awaitChildExit(5_000);
const elapsed = Date.now() - start;
expect(elapsed).toBeLessThan(200);
} finally {
h.cleanup();
}
});
});
describe('event shape', () => {
it('worker_spawned + worker_exited fire on every cycle with consistent shape', async () => {
const h = makeHarness('shape', 'exit 0');
try {
const res = await runUntilTerminal(h, {
maxCrashes: 3,
_backoffFloorMs: 5,
stopAfterEvents: 9, // 3 spawn-exit-backoff triples
});
const spawned = res.events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_spawned' }> => e.kind === 'worker_spawned',
);
const exited = res.events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited',
);
expect(spawned.length).toBeGreaterThanOrEqual(2);
expect(exited.length).toBe(spawned.length);
for (const s of spawned) {
expect(typeof s.pid).toBe('number');
expect(s.pid).toBeGreaterThan(0);
expect(typeof s.tini).toBe('boolean');
}
for (const e of exited) {
expect(e.code).toBe(0);
expect(e.signal).toBeNull();
expect(typeof e.runDurationMs).toBe('number');
expect(e.likelyCause).toBe('clean_exit');
}
} finally {
h.cleanup();
}
});
});
// issue #1678: RSS-watchdog exits (code 12) are cause-keyed and must NOT
// route through the generic crash path — the >5-min stable-run reset would
// defeat max_crashes and the 400×/24h loop would never stop being silent.
describe('rss_watchdog breaker (issue #1678)', () => {
it('code=12 is labeled rss_watchdog and never increments crashCount', async () => {
const h = makeHarness('wd-nocrash', 'exit 12');
try {
const { events, maxCrashesFired } = await runUntilTerminal(h, {
maxCrashes: 3,
_backoffFloorMs: 1,
stopAfterEvents: 18, // ~6 spawn/exit/backoff triples
});
const exited = events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> =>
e.kind === 'worker_exited',
);
// Looped well past maxCrashes WITHOUT tripping it — the whole point.
expect(maxCrashesFired).toBeNull();
expect(exited.length).toBeGreaterThan(3);
for (const e of exited) {
expect(e.code).toBe(12);
expect(e.likelyCause).toBe('rss_watchdog');
expect(e.crashCount).toBe(0); // never counted as a crash
}
} finally {
h.cleanup();
}
});
it('emits rss_watchdog_loop health_warn once the window budget is exceeded', async () => {
const h = makeHarness('wd-loop', 'exit 12');
try {
const { events } = await runUntilTerminal(h, {
maxCrashes: 99,
_backoffFloorMs: 1,
watchdogLoopBudget: 2,
watchdogLoopWindowMs: 600_000,
stopAfterEvents: 24,
});
const warns = events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> =>
e.kind === 'health_warn' && e.reason === 'rss_watchdog_loop',
);
// Budget=2 → the 3rd+ watchdog exit in-window fires the loud alert.
expect(warns.length).toBeGreaterThan(0);
expect(warns[0].count).toBeGreaterThan(2);
// And every backoff after a watchdog exit is reason=rss_watchdog.
const wdBackoffs = events.filter(
(e) => e.kind === 'backoff' && e.reason === 'rss_watchdog',
);
expect(wdBackoffs.length).toBeGreaterThan(0);
} finally {
h.cleanup();
}
});
});
// issue #1994 (#2227 tail): crossing the SOFT crash budget no longer
// permanently gives up. The supervisor enters degraded mode (capped backoff
// + loud warn) so a transient outage self-heals; permanent give-up fires only
// at the much-higher hard ceiling.
describe('degraded-mode crash backoff (issue #1994)', () => {
it('crossing the soft budget does NOT give up; it warns and keeps retrying to the hard ceiling', async () => {
const h = makeHarness('degraded-softbudget', 'exit 1');
try {
const { events, maxCrashesFired } = await runUntilTerminal(h, {
maxCrashes: 3, // soft budget
hardStopMaxCrashes: 6, // hard ceiling
_backoffFloorMs: 1,
stopAfterEvents: 200,
});
// Permanent give-up fired at the HARD ceiling (6), not the soft budget (3).
expect(maxCrashesFired).not.toBeNull();
expect(maxCrashesFired!.count).toBe(6);
expect(maxCrashesFired!.max).toBe(6);
// The soft-budget crossing announced degraded mode (at least once).
const degraded = events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> =>
e.kind === 'health_warn' && e.reason === 'crash_budget_degraded',
);
expect(degraded.length).toBeGreaterThanOrEqual(1);
expect(degraded[0].max).toBe(3);
expect(degraded[0].count).toBeGreaterThanOrEqual(3);
// It kept respawning past the soft budget (more than 3 crash exits).
const crashes = events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> =>
e.kind === 'worker_exited' && e.code === 1,
);
expect(crashes.length).toBe(6);
} finally {
h.cleanup();
}
});
it('hardStopMaxCrashes=0 disables permanent give-up (retry-forever-with-backoff)', async () => {
const h = makeHarness('degraded-noforever', 'exit 1');
try {
const { events, maxCrashesFired } = await runUntilTerminal(h, {
maxCrashes: 3,
hardStopMaxCrashes: 0, // never permanently stop
_backoffFloorMs: 1,
stopAfterEvents: 40, // the safety net stops the test, not a give-up
});
// Never gave up despite many crashes past the soft budget.
expect(maxCrashesFired).toBeNull();
const crashes = events.filter(
(e) => e.kind === 'worker_exited' && (e as any).code === 1,
);
expect(crashes.length).toBeGreaterThan(3);
} finally {
h.cleanup();
}
});
});
describe('issue #1801 — restartCurrentChild + killChild liveness fix', () => {
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
// ESRCH = no such process (dead). EPERM = process exists but we can't
// signal it (alive) — under `bun test` a spawned child can land in a state
// where kill(pid,0) reports EPERM, so treat anything-but-ESRCH as alive.
const isAlive = (pid: number): boolean => {
try { process.kill(pid, 0); return true; }
catch (e) { return (e as NodeJS.ErrnoException)?.code !== 'ESRCH'; }
};
// Worker that IGNORES SIGTERM and sleeps, so only SIGKILL can stop it.
function makeSigtermIgnorer(name: string): Harness {
return makeHarness(name, "trap '' TERM\nsleep 30");
}
async function startInBackground(h: Harness): Promise<{
sup: ChildWorkerSupervisor;
events: ChildSupervisorEvent[];
firstPid: number;
stop: () => Promise<void>;
}> {
const events: ChildSupervisorEvent[] = [];
let stopping = false;
let resolveSpawn: (pid: number) => void;
const firstSpawn = new Promise<number>((r) => { resolveSpawn = r; });
const sup = new ChildWorkerSupervisor({
cliPath: h.workerScript,
args: [],
maxCrashes: 100,
_backoffFloorMs: 5,
isStopping: () => stopping,
onMaxCrashesExceeded: () => { stopping = true; },
onEvent: (e) => {
events.push(e);
if (e.kind === 'worker_spawned') resolveSpawn(e.pid);
},
});
const runPromise = sup.run();
const firstPid = await firstSpawn;
const stop = async () => {
stopping = true;
// SIGKILL-retry until run() returns (children ignore SIGTERM).
for (let i = 0; i < 60; i++) {
sup.killChild('SIGKILL');
const done = await Promise.race([
runPromise.then(() => true),
sleep(50).then(() => false),
]);
if (done) return;
}
await runPromise;
};
return { sup, events, firstPid, stop };
}
// Structural regression for Codex #1: killChild MUST gate on liveness
// (exitCode/signalCode === null), NOT on `.killed`. `.killed` flips true the
// moment a signal is *sent*, so a `!this._child.killed` guard makes a
// follow-up SIGKILL (after an ignored SIGTERM) a silent no-op — the bug that
// left the existing shutdown() drain unable to force-kill a stuck worker.
// (The SIGTERM→SIGKILL behavior is exercised end-to-end by the
// restartCurrentChild test below + standalone repros; a live-process
// assertion that a SIGTERM-ignoring child survives is unreliable under the
// `bun test` runtime, so the no-regression contract is pinned structurally.)
it('killChild gates on liveness, not .killed (Codex #1 regression)', () => {
const src = readFileSync(
join(import.meta.dir, '..', 'src', 'core', 'minions', 'child-worker-supervisor.ts'),
'utf8',
);
const killChildBody = src.slice(
src.indexOf('killChild(signal: NodeJS.Signals)'),
src.indexOf('awaitChildExit('),
);
// Strip comment lines so the doc note explaining the OLD bug (which names
// `.killed`) doesn't trip the negative assertion — we check the CODE.
const code = killChildBody
.split('\n')
.filter((l) => !l.trim().startsWith('//') && !l.trim().startsWith('*'))
.join('\n');
expect(code).toContain('exitCode === null');
expect(code).toContain('signalCode === null');
// The buggy `.killed` guard must be gone from the code.
expect(code).not.toContain('.killed');
});
it('restartCurrentChild SIGKILLs the captured child, respawns, labels wedge_restart, leaves crashCount=0', async () => {
const h = makeSigtermIgnorer('restart-current');
const ctx = await startInBackground(h);
try {
const oldPid = ctx.firstPid;
await ctx.sup.restartCurrentChild(150); // SIGTERM ignored → SIGKILL after 150ms
await sleep(400); // let the old child exit + run() respawn (ms:0 wedge backoff)
expect(isAlive(oldPid)).toBe(false); // captured child killed
const spawns = ctx.events.filter((e) => e.kind === 'worker_spawned');
expect(spawns.length).toBeGreaterThanOrEqual(2); // respawned
const wedgeExit = ctx.events.find(
(e) => e.kind === 'worker_exited' && e.likelyCause === 'wedge_restart',
);
expect(wedgeExit).toBeDefined();
if (wedgeExit && wedgeExit.kind === 'worker_exited') {
expect(wedgeExit.crashCount).toBe(0); // Codex #3 — not counted as a crash
}
const wedgeBackoff = ctx.events.find(
(e) => e.kind === 'backoff' && e.reason === 'wedge_restart',
);
expect(wedgeBackoff).toBeDefined();
if (wedgeBackoff && wedgeBackoff.kind === 'backoff') {
expect(wedgeBackoff.ms).toBe(0); // immediate respawn
}
// Codex #2 — the respawned child is alive and was NOT killed by a stale
// timer aimed at the old child.
expect(ctx.sup.childAlive).toBe(true);
} finally {
await ctx.stop();
h.cleanup();
}
});
it('repeated wedge restarts never trip max_crashes (crashCount stays 0)', async () => {
const h = makeSigtermIgnorer('restart-no-crash');
const ctx = await startInBackground(h);
try {
for (let i = 0; i < 3; i++) {
await ctx.sup.restartCurrentChild(120);
await sleep(300);
}
expect(ctx.sup.crashCount).toBe(0); // three self-heals, zero crashes
} finally {
await ctx.stop();
h.cleanup();
}
});
});
});