mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-28 14:59:47 +00:00
* test(supervisor): pin LOCK_HELD fence-exit is never counted as a crash (#2227) A duplicate supervisor loses the queue-scoped DB singleton lock (#1849) and exits LOCK_HELD before spawning a worker or emitting 'started'. summarizeCrashes counts only worker_exited, so the fence path is structurally uncountable. Pin it so a future refactor that logs worker_exited on the fence path fails here instead of silently re-introducing the crash-budget breaker-trip loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(autopilot): per-source cycle binds FS phases to source.local_path, not global repo (#2194 #2227) A per-source autopilot-cycle inherited the global sync.repo_path as brainDir while stamping DB freshness for source_id — mixed scope. FS phases (sync/lint/extract) ran against the wrong tree, so the failure-cooldown and freshness gates would attribute work to the wrong source. Resolve the source's local_path in the handler (reuse the archive-recheck SELECT) and bind brainDir to it; a pure-DB source gets null (FS phases skip) instead of falling through to the global checkout. Legacy no-source dispatch keeps the global repoPath. Prerequisite for the cooldown/split commits (codex outside-voice #8). Resolves TODOS:634. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(supervisor): detect a live supervisor via the DB lock under split $HOME (#2227) jobs supervisor status + doctor read the HOME-derived pidfile, so a supervisor started under a different $HOME (keeper=/root vs ops=/data) read as 'not running' while healthy — the false signal that drives an operator to spawn a duplicate. Both surfaces now fall back to the queue-scoped DB singleton lock (#1849), the HOME-independent authority, when the pidfile shows nothing. New isLockHolderLive keys on lock freshness (ttl + heartbeat steal-grace), never process.kill, so PID reuse can't false-positive (pid-liveness-alone-pid-reuse). Status surfaces the holder host/pid + recorded concurrency/max-rss from the latest started event. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(supervisor): degraded retry instead of permanent give-up on crash storm (#1994 #2227) max_crashes_exceeded gave up forever, so a transient DB-pooler blip that tripped the soft budget wedged the queue until a human restart (#2227's breaker-trips tail). Crossing the soft budget now enters degraded mode: keep respawning with capped exponential backoff (60s cap — a paced retry, not a hot loop) and emit a loud crash_budget_degraded health_warn. The existing stable-run reset clears the count once a respawn survives >5min, so a recovered DB self-heals. Permanent give-up fires only at a much-higher hard ceiling (maxCrashes × 10), tunable/disablable via GBRAIN_SUPERVISOR_HARD_STOP_CRASHES (0 = never). Resolves TODOS:92. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(autopilot): clamp fan-out to worker concurrency + doctor warning (#2194) Fan-out resolved to 4 (Postgres) regardless of worker --concurrency, so surplus cycles queued behind the worker and raced the stalled-sweeper. Two fixes for the same mismatch: - resolveEffectiveFanoutMax clamps to max(1, concurrency-1) (reserve a slot), gated on a LIVE DB-lock holder so a stale started-audit row can't shrink throughput (codex #9/D5); no live holder → unknown → unclamped base. Escape hatch autopilot.fanout_clamp_to_concurrency. - doctor's autopilot_fanout_concurrency check warns when fan-out exceeds effective slots — the misconfig was silent before. Advisory (started-event concurrency), wired into both doctor surfaces. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(autopilot): per-source failure cooldown — break the dead-job storm (#2194) Only SUCCESS gated dispatch, so a source whose cycle kept failing/timing-out re-fanned-out every 5-min tick forever (200+ dead jobs/24h). Now a failed source backs off with bounded exponential cooldown (10→120min). Read at DISPATCH from minion_jobs dead/failed rows (timeouts/RSS-kills dead-letter via SQL and never run handler code, so a write-only hook would miss them) AND re-checked at CLAIM time in the handler (codex #5: already-queued/retrying jobs). A success clears it (codex #7); null-source rows excluded (codex #6); engine-parity via executeRaw. Disable with autopilot.failure_cooldown_min=0. Fail-open if config/history reads error. Surfaced via fanout_cooldown_skipped + the fanout summary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(autopilot): split the cycle — per-source phases + one global-maintenance job (#2194 #2227) N per-source cycles each ran the brain-wide global phases (embed-all/orphans/ purge/…) concurrently, thrashing the same rows and taking the worker 4→10GB in <60s → RSS-kill → orphaned stalls. Split them: per-source jobs now run only source-scoped (+ mixed) phases and stamp last_source_cycle_at; a new autopilot-global-maintenance job runs the global phases ONCE per window (idempotency_key + maxWaiting:1 = structural single-flight) and stamps autopilot.last_global_at. This is the codex-endorsed design that replaced the rejected skip-and-stamp-fresh approach (codex #1/#2): no freshness poisoning, no starvation — global work always runs as its own job, never marked done when it wasn't. PHASE_SCOPE is now a runtime partition (GLOBAL ∪ NON_GLOBAL == ALL). last_full_cycle_at still written for doctor/legacy (no longer a global gate). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): guard nullable engine in supervisor DB-lock fallback (#2227) Follow-up to the supervisor-visibility commit: doctor's engine binding is BrainEngine | null, so the inspectLock fallback must guard on a non-null engine (tsc TS2345). No behavior change — a null engine simply skips the DB-lock probe and falls back to the pidfile reading, as before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doctor): categorize autopilot_fanout_concurrency check as ops (#2194) Follow-up to the fan-out/concurrency commit: the doctor-categories drift guard requires every check name in doctor.ts to belong to exactly one category set. Add the new autopilot_fanout_concurrency check to OPS_CHECK_NAMES (infrastructure liveness, alongside wedged_queue/supervisor). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update KEY_FILES for the autopilot cycle split + supervisor degraded-retry (#2194 #2227) Post-ship document-release: refresh the KEY_FILES current-state entries that drifted — cycle.ts (GLOBAL/NON_GLOBAL phase split + last_source_cycle_at / autopilot.last_global_at), jobs.ts (per-source local_path brainDir, claim-time cooldown, autopilot-global-maintenance handler), supervisor.ts + child-worker (degraded retry instead of permanent give-up; hard ceiling), db-lock.ts (isLockHolderLive), handler-timeouts (new handler). Regenerated llms bundle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(minions): handleTimeouts counts the timed-out run as a spent attempt (#1737) The per-job timeout_at dead-letter (handleTimeouts) set status='dead' without incrementing attempts_made, unlike the wall-clock and stall dead-letter siblings. It is the FIRST killer to fire for the long-lane handlers (subagent / embed-backfill / autopilot-cycle) because timeout_ms is stamped at submit, so a timed-out long job reported `attempts: 0/N (started: N)`. Mirror the siblings with attempts_made + 1 (terminal, no retry). Safe against double-count: the worker sweep runs handleStalled -> handleTimeouts -> handleWallClockTimeouts sequentially and awaited, each guarded on status='active', so the first to dead-letter excludes the row from the rest. Regression assertions added (test/minions.test.ts + e2e/minions-resilience.test.ts) so the increment can't be silently dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(agent): recognize trailing switches in `agent run`, keep prompts freeform (#1738) parseRunFlags() broke flag parsing at the first positional token, so any flag after the prompt (`gbrain agent run "do X" --detach`) was swallowed into the prompt string and silently ignored. Now the no-value switches --detach/--follow/ --no-follow are hoisted when they trail the prompt, while everything else stays verbatim: an unknown --word is treated as prompt text (no "unknown flag" throw), a --switch mid-prompt is preserved, and `--` suppresses hoisting entirely for a literal escape. Value-flags now reject a missing or flag-shaped value (and --max-turns/--timeout-ms a non-number) instead of capturing undefined/NaN. Contract change: a prompt that starts with or trails an unguarded --word no longer errors; a literal trailing --detach needs `--`. Help text updated; tests revised + extended (test/agent-cli.test.ts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): honest live-sync status + progress-aware stall-abort (#1950) Finishes the #2255 honest-freshness story for two gaps it left. (a) `gbrain sources status` printed "idle" while a sync proc held the per-source lock (the reported bug). New shared liveSyncStatus() helper in db-lock.ts reads the SAME live-lock signal `gbrain doctor` uses; runStatus now shows "running" (BACKFILL column + a sync_running field in --json) and suppresses the misleading "never synced" warning while a sync is live. One helper, so the surfaces can't drift (doctor/status retrofit tracked as a follow-up). (b) A sync wedged-but-alive kept refreshing its lock heartbeat (it fires on its own timer) and hadn't hit the wall-clock deadline, so only a manual pkill freed it. New in-band stall watchdog keys off FORWARD IMPORT PROGRESS (progress.tick), not the heartbeat: if no file completes for GBRAIN_SYNC_STALL_ABORT_SECONDS (default 900s), it aborts via a controller composed into opts.signal, so the drain returns partial() (last_commit unchanged, next run resumes from the checkpoint) and withRefreshingLock releases the lock. Limits, documented in code: a single file slower than the window trips it; a fully starved event loop won't fire the timer (the wall-clock hard deadline is that backstop). Tests: liveSyncStatus (live/expired/none/per-source) in db-lock-inspect; the resolveStallAbortSeconds env matrix in sync-hard-deadline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(status): version field + per-section --deadline-ms budget (#1984) `gbrain status` had no version in its JSON envelope and could hang on a slow connection with no way to get a partial answer. Two additions: - version: the StatusReport JSON now carries the local gbrain CLI version so a poller can pin behavior to a build. Thin-client also surfaces remote_version (the brain server's version), and the get_status_snapshot MCP op reports its version for that parity. - --deadline-ms=N / --fast: a shared wall-clock budget. Each section is raced against the REMAINING budget via Promise.race (NOT process-watchdog, which SIGKILLs and can't return partial output), so one slow/hung section can't strand the snapshot — it's marked stale and the rest still return. The envelope gains partial:true + stale_sections[]; exit code stays 0 (a snapshot was produced). Invalid --deadline-ms → exit 2. Tests: parseDeadlineFlag + withSectionDeadline (hermetic), the usage-error exit, version presence in the PGLite envelope, and the op's version key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): report stall_timeout distinctly + document in-flight limit (#1950) Pre-landing review (codex + adversarial): the stall watchdog aborted opts.signal but the per-iteration abort checks returned partial('timeout'), collapsing a wedge-reap into a user --timeout/SIGINT so JSON consumers couldn't tell them apart. Add a 'stall_timeout' reason (set via a stallAborted flag) on the three import-loop abort sites; deletes/renames-phase and checkpoint sites stay 'timeout'. Sharpen the watchdog comment: the abort is observed BETWEEN files, so a hang inside a single importFile is not interrupted until it returns (TODO: thread a cancellation signal through importFile). * fix(agent): `--` escape suppresses trailing-switch hoisting anywhere (#1738) Pre-landing review: the leading-flag loop breaks at the first positional, so the `escaped` flag only fired for a leading `--`. A `--` placed after a positional left trailing-switch hoisting active, so `agent run note -- body --detach` silently detached and dropped the `--` as junk. Suppress hoisting whenever a literal `--` appears in the prompt. Regression test added. * fix(status): deadline-ms usage-error + scoped stale_sections + cancel losing remote call (#1984) Pre-landing review (codex): (1) bare `--deadline-ms` with no value silently fell through to no-budget/--fast instead of a usage error; (2) thin-client timeout reported both sync+cycle stale even under `--section sync`, naming a section the caller excluded (local path was already correct); (3) the section race abandoned the remote promise locally but didn't cancel the in-flight MCP call — pass the budget as timeoutMs so the losing side actually cancels. Regression test added. * v0.42.52.0 fix(reliability): autopilot dead-job storm + supervisor wedge + sync/status/minion reliability (#2194 #2227 #1994 #1737 #1738 #1950 #1984) Bundles the already-reviewed autopilot/supervisor stabilization (#2194 #2227 #1994: cycle split, per-source failure cooldown, fan-out clamp, degraded supervisor retry, DB-lock live-supervisor detection) with four operational fixes: minion timeout attempt-accounting (#1737), agent-run trailing-flag parsing (#1738), honest live-sync sources status + progress-aware stall watchdog (#1950), and status version + --deadline-ms partial result (#1984). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document GBRAIN_SYNC_STALL_ABORT_SECONDS env knob (#1950) Post-ship doc sync (/document-release): add the sync stall watchdog env var to the CLAUDE.md sync-tuning table (Five → Six knobs) + regenerate the llms bundle. * test: quarantine #2249 fanout tests as *.serial (R1 env-isolation) (#2194) The cherry-picked autopilot-fanout-clamp + doctor-autopilot-fanout-concurrency tests mutate process.env.GBRAIN_AUDIT_DIR in beforeEach/afterEach, which the check:test-isolation R1 lint flags (parallel shards load multiple files per process). Rename to *.serial.test.ts (sanctioned quarantine — they run under --max-concurrency=1) instead of restructuring the reviewed test bodies. No logic change; both files stay green (9 tests). Fixes the failing verify CI check. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
679 lines
26 KiB
TypeScript
679 lines
26 KiB
TypeScript
/**
|
||
* Tests for the shared spawn-and-respawn core used by MinionSupervisor
|
||
* and src/commands/autopilot.ts. Pins the D1 lastExitCode-track behavior
|
||
* and the D2 clean-restart-budget gate so future refactors can't silently
|
||
* regress the supervisor crash-count incident this wave fixes.
|
||
*
|
||
* Strategy: each test writes a small shell script to disk that exits with
|
||
* a chosen code after an optional sleep. The class spawns the script as
|
||
* the "worker" and we assert on the event stream the class emits.
|
||
*/
|
||
|
||
import { describe, it, expect, afterEach } from 'bun:test';
|
||
import { chmodSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'fs';
|
||
import { join } from 'path';
|
||
import { tmpdir } from 'os';
|
||
import {
|
||
ChildWorkerSupervisor,
|
||
type ChildSupervisorEvent,
|
||
} from '../src/core/minions/child-worker-supervisor.ts';
|
||
|
||
interface Harness {
|
||
workerScript: string;
|
||
cleanup: () => void;
|
||
}
|
||
|
||
function makeHarness(name: string, body: string): Harness {
|
||
const root = join(tmpdir(), `gbrain-cws-test-${name}-${process.pid}-${Date.now()}`);
|
||
mkdirSync(root, { recursive: true });
|
||
const workerScript = join(root, 'worker.sh');
|
||
writeFileSync(workerScript, `#!/bin/sh\n${body}\n`, 'utf8');
|
||
chmodSync(workerScript, 0o755);
|
||
return {
|
||
workerScript,
|
||
cleanup: () => {
|
||
try {
|
||
rmSync(root, { recursive: true, force: true });
|
||
} catch {
|
||
/* noop */
|
||
}
|
||
},
|
||
};
|
||
}
|
||
|
||
interface RunResult {
|
||
events: ChildSupervisorEvent[];
|
||
maxCrashesFired: { count: number; max: number } | null;
|
||
}
|
||
|
||
async function runUntilTerminal(
|
||
h: Harness,
|
||
overrides: Partial<{
|
||
maxCrashes: number;
|
||
hardStopMaxCrashes: number;
|
||
_backoffFloorMs: number;
|
||
cleanRestartBudget: number;
|
||
cleanRestartWindowMs: number;
|
||
cleanRestartBudgetBackoffMs: number;
|
||
stableRunResetMs: number;
|
||
watchdogLoopBudget: number;
|
||
watchdogLoopWindowMs: number;
|
||
watchdogBackoffMs: number;
|
||
_now: () => number;
|
||
stopAfterEvents: number; // safety net so a buggy test can't hang
|
||
}>,
|
||
): Promise<RunResult> {
|
||
const events: ChildSupervisorEvent[] = [];
|
||
let stopping = false;
|
||
let maxCrashesFired: { count: number; max: number } | null = null;
|
||
const stopAfter = overrides.stopAfterEvents ?? 200;
|
||
|
||
const sup = new ChildWorkerSupervisor({
|
||
cliPath: h.workerScript,
|
||
args: [],
|
||
maxCrashes: overrides.maxCrashes ?? 3,
|
||
hardStopMaxCrashes: overrides.hardStopMaxCrashes,
|
||
_backoffFloorMs: overrides._backoffFloorMs ?? 5,
|
||
cleanRestartBudget: overrides.cleanRestartBudget,
|
||
cleanRestartWindowMs: overrides.cleanRestartWindowMs,
|
||
cleanRestartBudgetBackoffMs: overrides.cleanRestartBudgetBackoffMs,
|
||
stableRunResetMs: overrides.stableRunResetMs,
|
||
watchdogLoopBudget: overrides.watchdogLoopBudget,
|
||
watchdogLoopWindowMs: overrides.watchdogLoopWindowMs,
|
||
watchdogBackoffMs: overrides.watchdogBackoffMs,
|
||
_now: overrides._now,
|
||
isStopping: () => stopping,
|
||
onMaxCrashesExceeded: (count, max) => {
|
||
maxCrashesFired = { count, max };
|
||
stopping = true;
|
||
},
|
||
onEvent: (event) => {
|
||
events.push(event);
|
||
if (events.length >= stopAfter) {
|
||
stopping = true;
|
||
}
|
||
},
|
||
});
|
||
|
||
await sup.run();
|
||
return { events, maxCrashesFired };
|
||
}
|
||
|
||
afterEach(() => {
|
||
/* per-test harness.cleanup() runs in finally blocks below */
|
||
});
|
||
|
||
describe('ChildWorkerSupervisor', () => {
|
||
describe('D1 — code=0 exit classifier', () => {
|
||
it('code=0 worker exit does not count as crash; restarts immediately', async () => {
|
||
const h = makeHarness('clean-exits', 'exit 0');
|
||
try {
|
||
const res = await runUntilTerminal(h, {
|
||
maxCrashes: 3,
|
||
stopAfterEvents: 30, // ~10 spawn/exit/backoff trios
|
||
});
|
||
expect(res.maxCrashesFired).toBeNull();
|
||
|
||
const exits = res.events.filter((e) => e.kind === 'worker_exited');
|
||
expect(exits.length).toBeGreaterThanOrEqual(3);
|
||
for (const e of exits) {
|
||
if (e.kind === 'worker_exited') {
|
||
expect(e.code).toBe(0);
|
||
expect(e.likelyCause).toBe('clean_exit');
|
||
// crashCount stays at 0 across every clean exit
|
||
expect(e.crashCount).toBe(0);
|
||
}
|
||
}
|
||
|
||
const backoffs = res.events.filter((e) => e.kind === 'backoff');
|
||
expect(backoffs.length).toBeGreaterThanOrEqual(1);
|
||
// Within the default 10-restart budget, all backoffs are ms:0 / clean_exit
|
||
for (const e of backoffs) {
|
||
if (e.kind === 'backoff') {
|
||
// Once we cross the 10-restart budget the reason flips to
|
||
// budget_exceeded, but until then they're all clean_exit ms:0.
|
||
if (e.reason === 'clean_exit') {
|
||
expect(e.ms).toBe(0);
|
||
expect(e.crashCount).toBe(0);
|
||
}
|
||
}
|
||
}
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
|
||
it('interleaved code=0 and code!=0 exits still trip max_crashes', async () => {
|
||
// Worker alternates: each invocation increments a counter file and
|
||
// exits 1 on odd hits, 0 on even hits (so exit-sequence is 1,0,1,0,1).
|
||
const h = makeHarness(
|
||
'interleaved',
|
||
`
|
||
COUNTER_FILE="$(dirname "$0")/counter"
|
||
[ -f "$COUNTER_FILE" ] || echo 0 > "$COUNTER_FILE"
|
||
COUNT=$(cat "$COUNTER_FILE")
|
||
NEXT=$((COUNT + 1))
|
||
echo "$NEXT" > "$COUNTER_FILE"
|
||
# Odd-indexed runs (#1, #3, #5...) exit 1; even-indexed exit 0.
|
||
if [ $((NEXT % 2)) -eq 1 ]; then exit 1; else exit 0; fi
|
||
`,
|
||
);
|
||
try {
|
||
const res = await runUntilTerminal(h, {
|
||
maxCrashes: 3,
|
||
// issue #1994: the soft budget no longer gives up; pin the hard
|
||
// ceiling to 3 so this counting test still fires give-up at 3.
|
||
hardStopMaxCrashes: 3,
|
||
_backoffFloorMs: 5,
|
||
stopAfterEvents: 200,
|
||
});
|
||
|
||
expect(res.maxCrashesFired).not.toBeNull();
|
||
// 3 code!=0 exits → hard ceiling=3
|
||
expect(res.maxCrashesFired!.count).toBe(3);
|
||
|
||
const exits = res.events.filter((e) => e.kind === 'worker_exited');
|
||
// Should be exactly 5 exits: 1, 0, 1, 0, 1 — then max fires.
|
||
const codes = exits
|
||
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited')
|
||
.map((e) => e.code);
|
||
expect(codes).toEqual([1, 0, 1, 0, 1]);
|
||
|
||
const backoffs = res.events
|
||
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'backoff' }> => e.kind === 'backoff');
|
||
// Backoffs only fire between iterations 1-4 (not after the 5th, since
|
||
// the loop bails out via onMaxCrashesExceeded before applyBackoff).
|
||
// Even-index exits (code=0, indices 1+3) → reason='clean_exit'.
|
||
// Odd-index exits (code=1, indices 0+2) → reason='crash'.
|
||
const reasons = backoffs.map((e) => e.reason);
|
||
expect(reasons).toEqual(['crash', 'clean_exit', 'crash', 'clean_exit']);
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
|
||
it('code=0 after stable 5min+ run does not reset crashCount', async () => {
|
||
// Sequence (4 runs total): exit 1 → exit 0 (6 min, "stable") → exit 1 →
|
||
// exit 1. crashCount progression: 1, 1 (unchanged across the long
|
||
// clean exit), 2, 3 — last one trips max_crashes=3.
|
||
const h = makeHarness(
|
||
'stable-clean-no-reset',
|
||
`
|
||
COUNTER_FILE="$(dirname "$0")/counter"
|
||
[ -f "$COUNTER_FILE" ] || echo 0 > "$COUNTER_FILE"
|
||
COUNT=$(cat "$COUNTER_FILE")
|
||
NEXT=$((COUNT + 1))
|
||
echo "$NEXT" > "$COUNTER_FILE"
|
||
case $NEXT in
|
||
1) exit 1 ;;
|
||
2) exit 0 ;;
|
||
3) exit 1 ;;
|
||
4) exit 1 ;;
|
||
*) exit 0 ;;
|
||
esac
|
||
`,
|
||
);
|
||
try {
|
||
// Fake clock — each spawnOnce reads now() twice (start + exit) and
|
||
// applyBackoff may read once more. Run 2 sees a 6-minute duration
|
||
// (stable-run reset would fire IF the exit were code!=0 — we assert
|
||
// it does NOT fire when the exit is clean).
|
||
const SIX_MIN = 6 * 60_000;
|
||
const timestamps = [
|
||
0, // run 1 start
|
||
1_000, // run 1 exit (+1s) → crashCount 1
|
||
1_000, // run 2 start
|
||
1_000 + SIX_MIN, // run 2 exit (+6min) → code=0, stays at 1
|
||
1_000 + SIX_MIN, // run 3 start
|
||
1_000 + SIX_MIN + 1_000, // run 3 exit (+1s) → crashCount 2
|
||
1_000 + SIX_MIN + 1_000, // run 4 start
|
||
1_000 + SIX_MIN + 2_000, // run 4 exit (+1s) → crashCount 3, trips max
|
||
];
|
||
let idx = 0;
|
||
const last = timestamps[timestamps.length - 1];
|
||
const fakeNow = () => {
|
||
if (idx < timestamps.length) {
|
||
return timestamps[idx++];
|
||
}
|
||
return last + (idx++ - timestamps.length + 1) * 100;
|
||
};
|
||
|
||
const res = await runUntilTerminal(h, {
|
||
maxCrashes: 3,
|
||
hardStopMaxCrashes: 3, // issue #1994: pin give-up to 3 for this counting test
|
||
_backoffFloorMs: 5,
|
||
_now: fakeNow,
|
||
stopAfterEvents: 200,
|
||
});
|
||
|
||
expect(res.maxCrashesFired).not.toBeNull();
|
||
expect(res.maxCrashesFired!.count).toBe(3);
|
||
|
||
const exits = res.events
|
||
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited')
|
||
.map((e) => ({ code: e.code, crashCount: e.crashCount, runDurationMs: e.runDurationMs }));
|
||
|
||
expect(exits.length).toBeGreaterThanOrEqual(4);
|
||
expect(exits[0]).toMatchObject({ code: 1, crashCount: 1 });
|
||
expect(exits[1]).toMatchObject({ code: 0, crashCount: 1 }); // D1: unchanged
|
||
expect(exits[2]).toMatchObject({ code: 1, crashCount: 2 });
|
||
expect(exits[3]).toMatchObject({ code: 1, crashCount: 3 });
|
||
// Run 2 ran 6min, but because exit code was 0 the stable-run reset
|
||
// branch did NOT fire — crashCount stayed at 1. This is the core
|
||
// D1 invariant: clean exits never reset crashCount, even stable ones.
|
||
expect(exits[1].runDurationMs).toBe(SIX_MIN);
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('D2 — clean-restart budget', () => {
|
||
it('budget exceeded triggers health_warn + budget_exceeded backoff', async () => {
|
||
// Tight budget of 2 so we trip it on the 3rd clean exit.
|
||
const h = makeHarness('budget-trip', 'exit 0');
|
||
try {
|
||
const res = await runUntilTerminal(h, {
|
||
maxCrashes: 3, // never trips because code=0 doesn't increment
|
||
_backoffFloorMs: 5,
|
||
cleanRestartBudget: 2,
|
||
cleanRestartWindowMs: 60_000,
|
||
cleanRestartBudgetBackoffMs: 10,
|
||
stopAfterEvents: 25,
|
||
});
|
||
|
||
const healthWarns = res.events.filter(
|
||
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> => e.kind === 'health_warn',
|
||
);
|
||
// Once tripped, every subsequent clean exit re-fires health_warn
|
||
// (the sliding window stays full at our test rate).
|
||
expect(healthWarns.length).toBeGreaterThan(0);
|
||
for (const w of healthWarns) {
|
||
expect(w.reason).toBe('clean_restart_budget_exceeded');
|
||
expect(w.windowMs).toBe(60_000);
|
||
expect(w.count).toBeGreaterThan(2);
|
||
}
|
||
|
||
const backoffReasons = res.events
|
||
.filter((e): e is Extract<ChildSupervisorEvent, { kind: 'backoff' }> => e.kind === 'backoff')
|
||
.map((e) => e.reason);
|
||
// First 2 exits are within budget → reason='clean_exit'.
|
||
// From the 3rd exit onward → reason='budget_exceeded'.
|
||
expect(backoffReasons.slice(0, 2)).toEqual(['clean_exit', 'clean_exit']);
|
||
expect(backoffReasons.slice(2).every((r) => r === 'budget_exceeded')).toBe(true);
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
|
||
it('budget config is per-instance (no module-level state leakage)', async () => {
|
||
// Run instance A with budget=2 and instance B with budget=5. Each
|
||
// tracks its own sliding window; A trips faster than B.
|
||
const hA = makeHarness('budget-a', 'exit 0');
|
||
const hB = makeHarness('budget-b', 'exit 0');
|
||
try {
|
||
const resA = await runUntilTerminal(hA, {
|
||
maxCrashes: 99,
|
||
_backoffFloorMs: 5,
|
||
cleanRestartBudget: 2,
|
||
cleanRestartBudgetBackoffMs: 5,
|
||
stopAfterEvents: 12,
|
||
});
|
||
const resB = await runUntilTerminal(hB, {
|
||
maxCrashes: 99,
|
||
_backoffFloorMs: 5,
|
||
cleanRestartBudget: 5,
|
||
cleanRestartBudgetBackoffMs: 5,
|
||
stopAfterEvents: 18,
|
||
});
|
||
|
||
const firstTripA = resA.events.findIndex(
|
||
(e) => e.kind === 'health_warn',
|
||
);
|
||
const firstTripB = resB.events.findIndex(
|
||
(e) => e.kind === 'health_warn',
|
||
);
|
||
|
||
expect(firstTripA).toBeGreaterThan(-1);
|
||
expect(firstTripB).toBeGreaterThan(-1);
|
||
// B's budget is more generous → its first health_warn appears later
|
||
// in the event stream (after more spawn/exit pairs).
|
||
expect(firstTripB).toBeGreaterThan(firstTripA);
|
||
} finally {
|
||
hA.cleanup();
|
||
hB.cleanup();
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('awaitChildExit short-circuit (P2 review fix)', () => {
|
||
// Regression: pre-fix the method registered child.once('exit', ...) AFTER
|
||
// child.exitCode was already populated, so a child that drained quickly
|
||
// between killChild('SIGTERM') and awaitChildExit() would never resolve
|
||
// and the caller waited out the full timeout. Fix probes exitCode +
|
||
// signalCode first and short-circuits.
|
||
it('resolves immediately when the child has already exited', async () => {
|
||
const h = makeHarness('await-already-exited', 'exit 0');
|
||
try {
|
||
// Spin up a supervisor; drive it for ONE spawn cycle and then stop.
|
||
const events: ChildSupervisorEvent[] = [];
|
||
let stopping = false;
|
||
const sup = new ChildWorkerSupervisor({
|
||
cliPath: h.workerScript,
|
||
args: [],
|
||
maxCrashes: 1,
|
||
_backoffFloorMs: 1,
|
||
isStopping: () => stopping,
|
||
onMaxCrashesExceeded: () => { stopping = true; },
|
||
onEvent: (e) => {
|
||
events.push(e);
|
||
if (e.kind === 'worker_exited') stopping = true;
|
||
},
|
||
});
|
||
await sup.run();
|
||
// After run() returns, the child has exited; awaitChildExit on an
|
||
// already-finished cycle MUST resolve in well under the timeout.
|
||
const start = Date.now();
|
||
await sup.awaitChildExit(5_000);
|
||
const elapsed = Date.now() - start;
|
||
expect(elapsed).toBeLessThan(200);
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('event shape', () => {
|
||
it('worker_spawned + worker_exited fire on every cycle with consistent shape', async () => {
|
||
const h = makeHarness('shape', 'exit 0');
|
||
try {
|
||
const res = await runUntilTerminal(h, {
|
||
maxCrashes: 3,
|
||
_backoffFloorMs: 5,
|
||
stopAfterEvents: 9, // 3 spawn-exit-backoff triples
|
||
});
|
||
|
||
const spawned = res.events.filter(
|
||
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_spawned' }> => e.kind === 'worker_spawned',
|
||
);
|
||
const exited = res.events.filter(
|
||
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> => e.kind === 'worker_exited',
|
||
);
|
||
|
||
expect(spawned.length).toBeGreaterThanOrEqual(2);
|
||
expect(exited.length).toBe(spawned.length);
|
||
|
||
for (const s of spawned) {
|
||
expect(typeof s.pid).toBe('number');
|
||
expect(s.pid).toBeGreaterThan(0);
|
||
expect(typeof s.tini).toBe('boolean');
|
||
}
|
||
for (const e of exited) {
|
||
expect(e.code).toBe(0);
|
||
expect(e.signal).toBeNull();
|
||
expect(typeof e.runDurationMs).toBe('number');
|
||
expect(e.likelyCause).toBe('clean_exit');
|
||
}
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
});
|
||
|
||
// issue #1678: RSS-watchdog exits (code 12) are cause-keyed and must NOT
|
||
// route through the generic crash path — the >5-min stable-run reset would
|
||
// defeat max_crashes and the 400×/24h loop would never stop being silent.
|
||
describe('rss_watchdog breaker (issue #1678)', () => {
|
||
it('code=12 is labeled rss_watchdog and never increments crashCount', async () => {
|
||
const h = makeHarness('wd-nocrash', 'exit 12');
|
||
try {
|
||
const { events, maxCrashesFired } = await runUntilTerminal(h, {
|
||
maxCrashes: 3,
|
||
_backoffFloorMs: 1,
|
||
stopAfterEvents: 18, // ~6 spawn/exit/backoff triples
|
||
});
|
||
const exited = events.filter(
|
||
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> =>
|
||
e.kind === 'worker_exited',
|
||
);
|
||
// Looped well past maxCrashes WITHOUT tripping it — the whole point.
|
||
expect(maxCrashesFired).toBeNull();
|
||
expect(exited.length).toBeGreaterThan(3);
|
||
for (const e of exited) {
|
||
expect(e.code).toBe(12);
|
||
expect(e.likelyCause).toBe('rss_watchdog');
|
||
expect(e.crashCount).toBe(0); // never counted as a crash
|
||
}
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
|
||
it('emits rss_watchdog_loop health_warn once the window budget is exceeded', async () => {
|
||
const h = makeHarness('wd-loop', 'exit 12');
|
||
try {
|
||
const { events } = await runUntilTerminal(h, {
|
||
maxCrashes: 99,
|
||
_backoffFloorMs: 1,
|
||
watchdogLoopBudget: 2,
|
||
watchdogLoopWindowMs: 600_000,
|
||
stopAfterEvents: 24,
|
||
});
|
||
const warns = events.filter(
|
||
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> =>
|
||
e.kind === 'health_warn' && e.reason === 'rss_watchdog_loop',
|
||
);
|
||
// Budget=2 → the 3rd+ watchdog exit in-window fires the loud alert.
|
||
expect(warns.length).toBeGreaterThan(0);
|
||
expect(warns[0].count).toBeGreaterThan(2);
|
||
// And every backoff after a watchdog exit is reason=rss_watchdog.
|
||
const wdBackoffs = events.filter(
|
||
(e) => e.kind === 'backoff' && e.reason === 'rss_watchdog',
|
||
);
|
||
expect(wdBackoffs.length).toBeGreaterThan(0);
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
});
|
||
|
||
// issue #1994 (#2227 tail): crossing the SOFT crash budget no longer
|
||
// permanently gives up. The supervisor enters degraded mode (capped backoff
|
||
// + loud warn) so a transient outage self-heals; permanent give-up fires only
|
||
// at the much-higher hard ceiling.
|
||
describe('degraded-mode crash backoff (issue #1994)', () => {
|
||
it('crossing the soft budget does NOT give up; it warns and keeps retrying to the hard ceiling', async () => {
|
||
const h = makeHarness('degraded-softbudget', 'exit 1');
|
||
try {
|
||
const { events, maxCrashesFired } = await runUntilTerminal(h, {
|
||
maxCrashes: 3, // soft budget
|
||
hardStopMaxCrashes: 6, // hard ceiling
|
||
_backoffFloorMs: 1,
|
||
stopAfterEvents: 200,
|
||
});
|
||
// Permanent give-up fired at the HARD ceiling (6), not the soft budget (3).
|
||
expect(maxCrashesFired).not.toBeNull();
|
||
expect(maxCrashesFired!.count).toBe(6);
|
||
expect(maxCrashesFired!.max).toBe(6);
|
||
|
||
// The soft-budget crossing announced degraded mode (at least once).
|
||
const degraded = events.filter(
|
||
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> =>
|
||
e.kind === 'health_warn' && e.reason === 'crash_budget_degraded',
|
||
);
|
||
expect(degraded.length).toBeGreaterThanOrEqual(1);
|
||
expect(degraded[0].max).toBe(3);
|
||
expect(degraded[0].count).toBeGreaterThanOrEqual(3);
|
||
|
||
// It kept respawning past the soft budget (more than 3 crash exits).
|
||
const crashes = events.filter(
|
||
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> =>
|
||
e.kind === 'worker_exited' && e.code === 1,
|
||
);
|
||
expect(crashes.length).toBe(6);
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
|
||
it('hardStopMaxCrashes=0 disables permanent give-up (retry-forever-with-backoff)', async () => {
|
||
const h = makeHarness('degraded-noforever', 'exit 1');
|
||
try {
|
||
const { events, maxCrashesFired } = await runUntilTerminal(h, {
|
||
maxCrashes: 3,
|
||
hardStopMaxCrashes: 0, // never permanently stop
|
||
_backoffFloorMs: 1,
|
||
stopAfterEvents: 40, // the safety net stops the test, not a give-up
|
||
});
|
||
// Never gave up despite many crashes past the soft budget.
|
||
expect(maxCrashesFired).toBeNull();
|
||
const crashes = events.filter(
|
||
(e) => e.kind === 'worker_exited' && (e as any).code === 1,
|
||
);
|
||
expect(crashes.length).toBeGreaterThan(3);
|
||
} finally {
|
||
h.cleanup();
|
||
}
|
||
});
|
||
});
|
||
|
||
describe('issue #1801 — restartCurrentChild + killChild liveness fix', () => {
|
||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||
// ESRCH = no such process (dead). EPERM = process exists but we can't
|
||
// signal it (alive) — under `bun test` a spawned child can land in a state
|
||
// where kill(pid,0) reports EPERM, so treat anything-but-ESRCH as alive.
|
||
const isAlive = (pid: number): boolean => {
|
||
try { process.kill(pid, 0); return true; }
|
||
catch (e) { return (e as NodeJS.ErrnoException)?.code !== 'ESRCH'; }
|
||
};
|
||
|
||
// Worker that IGNORES SIGTERM and sleeps, so only SIGKILL can stop it.
|
||
function makeSigtermIgnorer(name: string): Harness {
|
||
return makeHarness(name, "trap '' TERM\nsleep 30");
|
||
}
|
||
|
||
async function startInBackground(h: Harness): Promise<{
|
||
sup: ChildWorkerSupervisor;
|
||
events: ChildSupervisorEvent[];
|
||
firstPid: number;
|
||
stop: () => Promise<void>;
|
||
}> {
|
||
const events: ChildSupervisorEvent[] = [];
|
||
let stopping = false;
|
||
let resolveSpawn: (pid: number) => void;
|
||
const firstSpawn = new Promise<number>((r) => { resolveSpawn = r; });
|
||
const sup = new ChildWorkerSupervisor({
|
||
cliPath: h.workerScript,
|
||
args: [],
|
||
maxCrashes: 100,
|
||
_backoffFloorMs: 5,
|
||
isStopping: () => stopping,
|
||
onMaxCrashesExceeded: () => { stopping = true; },
|
||
onEvent: (e) => {
|
||
events.push(e);
|
||
if (e.kind === 'worker_spawned') resolveSpawn(e.pid);
|
||
},
|
||
});
|
||
const runPromise = sup.run();
|
||
const firstPid = await firstSpawn;
|
||
const stop = async () => {
|
||
stopping = true;
|
||
// SIGKILL-retry until run() returns (children ignore SIGTERM).
|
||
for (let i = 0; i < 60; i++) {
|
||
sup.killChild('SIGKILL');
|
||
const done = await Promise.race([
|
||
runPromise.then(() => true),
|
||
sleep(50).then(() => false),
|
||
]);
|
||
if (done) return;
|
||
}
|
||
await runPromise;
|
||
};
|
||
return { sup, events, firstPid, stop };
|
||
}
|
||
|
||
// Structural regression for Codex #1: killChild MUST gate on liveness
|
||
// (exitCode/signalCode === null), NOT on `.killed`. `.killed` flips true the
|
||
// moment a signal is *sent*, so a `!this._child.killed` guard makes a
|
||
// follow-up SIGKILL (after an ignored SIGTERM) a silent no-op — the bug that
|
||
// left the existing shutdown() drain unable to force-kill a stuck worker.
|
||
// (The SIGTERM→SIGKILL behavior is exercised end-to-end by the
|
||
// restartCurrentChild test below + standalone repros; a live-process
|
||
// assertion that a SIGTERM-ignoring child survives is unreliable under the
|
||
// `bun test` runtime, so the no-regression contract is pinned structurally.)
|
||
it('killChild gates on liveness, not .killed (Codex #1 regression)', () => {
|
||
const src = readFileSync(
|
||
join(import.meta.dir, '..', 'src', 'core', 'minions', 'child-worker-supervisor.ts'),
|
||
'utf8',
|
||
);
|
||
const killChildBody = src.slice(
|
||
src.indexOf('killChild(signal: NodeJS.Signals)'),
|
||
src.indexOf('awaitChildExit('),
|
||
);
|
||
// Strip comment lines so the doc note explaining the OLD bug (which names
|
||
// `.killed`) doesn't trip the negative assertion — we check the CODE.
|
||
const code = killChildBody
|
||
.split('\n')
|
||
.filter((l) => !l.trim().startsWith('//') && !l.trim().startsWith('*'))
|
||
.join('\n');
|
||
expect(code).toContain('exitCode === null');
|
||
expect(code).toContain('signalCode === null');
|
||
// The buggy `.killed` guard must be gone from the code.
|
||
expect(code).not.toContain('.killed');
|
||
});
|
||
|
||
it('restartCurrentChild SIGKILLs the captured child, respawns, labels wedge_restart, leaves crashCount=0', async () => {
|
||
const h = makeSigtermIgnorer('restart-current');
|
||
const ctx = await startInBackground(h);
|
||
try {
|
||
const oldPid = ctx.firstPid;
|
||
await ctx.sup.restartCurrentChild(150); // SIGTERM ignored → SIGKILL after 150ms
|
||
await sleep(400); // let the old child exit + run() respawn (ms:0 wedge backoff)
|
||
|
||
expect(isAlive(oldPid)).toBe(false); // captured child killed
|
||
|
||
const spawns = ctx.events.filter((e) => e.kind === 'worker_spawned');
|
||
expect(spawns.length).toBeGreaterThanOrEqual(2); // respawned
|
||
|
||
const wedgeExit = ctx.events.find(
|
||
(e) => e.kind === 'worker_exited' && e.likelyCause === 'wedge_restart',
|
||
);
|
||
expect(wedgeExit).toBeDefined();
|
||
if (wedgeExit && wedgeExit.kind === 'worker_exited') {
|
||
expect(wedgeExit.crashCount).toBe(0); // Codex #3 — not counted as a crash
|
||
}
|
||
|
||
const wedgeBackoff = ctx.events.find(
|
||
(e) => e.kind === 'backoff' && e.reason === 'wedge_restart',
|
||
);
|
||
expect(wedgeBackoff).toBeDefined();
|
||
if (wedgeBackoff && wedgeBackoff.kind === 'backoff') {
|
||
expect(wedgeBackoff.ms).toBe(0); // immediate respawn
|
||
}
|
||
|
||
// Codex #2 — the respawned child is alive and was NOT killed by a stale
|
||
// timer aimed at the old child.
|
||
expect(ctx.sup.childAlive).toBe(true);
|
||
} finally {
|
||
await ctx.stop();
|
||
h.cleanup();
|
||
}
|
||
});
|
||
|
||
it('repeated wedge restarts never trip max_crashes (crashCount stays 0)', async () => {
|
||
const h = makeSigtermIgnorer('restart-no-crash');
|
||
const ctx = await startInBackground(h);
|
||
try {
|
||
for (let i = 0; i < 3; i++) {
|
||
await ctx.sup.restartCurrentChild(120);
|
||
await sleep(300);
|
||
}
|
||
expect(ctx.sup.crashCount).toBe(0); // three self-heals, zero crashes
|
||
} finally {
|
||
await ctx.stop();
|
||
h.cleanup();
|
||
}
|
||
});
|
||
});
|
||
});
|