mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
ed900c870e
* feat(minions): add self-health-monitoring to bare worker mode
Bare `gbrain jobs work` (without supervisor) previously had zero health
monitoring. If the Postgres connection dropped or the worker's event loop
deadlocked, the process stayed alive doing nothing — jobs piled up while
external process managers (systemd, Docker, cron) thought it was healthy.
Changes:
1. **Self-health-check timer** (worker.ts): Runs every 60s when not under
a supervisor. Two probes:
- DB liveness: `SELECT 1` — 3 consecutive failures → exit(1)
- Stall detection: waiting jobs + 0 in-flight + no completions for 5m
→ warning; 10m → exit(1)
2. **GBRAIN_SUPERVISED env var** (supervisor.ts): Supervisor sets this on
its child worker to prevent duplicate health checks. The supervisor
already has its own health monitoring.
3. **RSS watchdog default** (jobs.ts): Bare workers now default to
`--max-rss 2048` (matching supervisor default). Opt out: `--max-rss 0`.
4. **--health-interval flag** (jobs.ts): Configurable health check period.
`--health-interval 0` disables. Default: 60000ms.
5. **parseMaxRssFlag returns undefined** when flag is absent (vs 0), so
callers can distinguish 'not set' from 'explicitly disabled'.
The design ensures bare workers get supervisor-grade monitoring while
remaining compatible with any external process manager — the worker just
exits with code 1 on detected failure, letting the PM handle the restart.
Tests: 4 new tests (3 worker health, 1 supervisor env var). All 178 pass.
* fix(minions): harden bare-worker self-health-check after multi-round review
Layered fixes from 5 rounds of plan-eng-review + codex outside voice on top of
the original PR #503 (feat: bare-worker self-health-monitoring). Every change
below is in service of "fail-stop into the operator's process manager" without
introducing new ways the library can kill its caller.
worker.ts:
- MinionWorker now extends EventEmitter; emits `'unhealthy'` event with structured
reason payload (`db_dead` | `stalled`). CLI subscribes; library no longer calls
process.exit directly.
- emitUnhealthy() falls back to process.exit(1) when listenerCount('unhealthy') === 0
so direct API consumers without a listener inherit the pre-refactor fail-stop
default. Inline paths opt out via healthCheckInterval=0.
- Stall detection: count(*) query now filters by registered handler names
(`AND name = ANY($2::text[])`) so workers with handlers for {embed,sync} don't
false-positive when waiting jobs of unhandled names accumulate.
- Stall exit threshold measured from lastCompletionTime (not from warn-since), so
defaults of 5min warn / 10min exit fire at idle=10min total — matching the
documented contract.
- Recursive setTimeout pattern with running flag replaces setInterval, eliminating
callback overlap on slow DB probes.
- DB liveness probe wrapped in Promise.race against AbortController-driven
timeout (default 10s) so a hung executeRaw can't wedge the recursive chain
forever. Hung probes count as failures and feed dbFailExitAfter.
- Constructor validates stallExitAfterMs > stallWarnAfterMs and throws loudly
on misconfiguration. Internal timer-installation invariants documented inline.
- GBRAIN_SUPERVISED env-var check tightened from `!!process.env.X` to `=== '1'`.
types.ts:
- Added 5 new MinionWorkerOpts fields with documented contracts:
healthCheckInterval, stallWarnAfterMs, stallExitAfterMs, dbFailExitAfter,
dbProbeTimeoutMs.
- Exported `UnhealthyReason` discriminated union for the 'unhealthy' event payload.
supervisor.ts:
- GBRAIN_SUPERVISED=1 injected on the spawned worker child's env so the child's
self-health timer is skipped (no double-monitoring).
- setInterval(callback, healthInterval) gated behind `> 0`, so the
`--health-interval 0` documented disable contract actually disables instead
of producing a tight DB-hammer loop.
jobs.ts:
- `gbrain jobs work` subscribes to 'unhealthy' and calls process.exit(1) at the
CLI layer. Default --max-rss bumped from 0 to 2048 (matches supervisor default;
catches memory-leak stalls that previously went undetected).
- New --health-interval flag with aggressive validation (NaN/negative/sub-1000ms
rejected; parity with --max-rss) on both `jobs work` and `jobs supervisor`.
- `jobs submit --follow` and `jobs smoke` now pass healthCheckInterval=0 to
disable the self-health timer entirely. These are inline/one-shot flows with
no PM to restart them; the no-listener emitUnhealthy fallback could otherwise
trip on a DB blip and kill the user's CLI session.
- parseMaxRssFlag returns `number | undefined` (was `number`) so callers can
distinguish absent (use the default) from explicit-disable (--max-rss 0).
doctor.ts:
- New queue_health subcheck reports RSS-watchdog kills in the last 24h.
Detects via exact-match `error_text = 'aborted: watchdog'` (the worker's
failJob signature when gracefulShutdown('watchdog') aborts in-flight jobs)
scoped to status IN ('dead','failed'). Tight match avoids over-counting parent
jobs that propagate child failures via on_child_fail='fail_parent'.
* test(minions): self-health behavior + regression tests
7 new tests covering the production failure modes that drove the original PR,
plus regressions for fixes landed during multi-round review.
minions.test.ts:
- DB 3-strike → 'unhealthy' event with reason='db_dead' (the production-incident signature)
- DB recovery resets failure counter (no exit on intermittent failures)
- Stall warn-then-exit (clock-driven; idleMs > stallExitAfterMs is the new contract)
- inFlight > 0 blocks stall detection (long-running legitimate jobs don't false-trip)
- Regression for D1 fix: jobs of unregistered handler names don't trigger stall exit;
also captures the SQL via probe engine and asserts the predicate text contains
`name = ANY` so a future refactor that drops the filter is caught at test time.
- Regression for R3 constructor validation: throws when stallExitAfterMs <= stallWarnAfterMs
(covers both `<` and `=` cases); defaults still construct cleanly.
supervisor.test.ts:
- Regression for R3: supervisor with healthInterval=0 completes a normal lifecycle
within 10s. A tight setInterval(0) loop (the bug we fixed) would saturate the
event loop and slow this past the cap.
Tests use a Proxy-based engine helper (makeProbeEngine) that intercepts SELECT 1
and the count(*) WHERE status='waiting' query while passing through everything
else to the real PGLite engine. This isolates health-check semantics from claim
plumbing without mocking the entire engine surface.
* docs(v0.22.14): migration walkthrough + follow-up TODOs
skills/migrations/v0.22.14.md (new):
- Pre-flight per-PM restart-policy table (systemd Restart=always, Docker
restart: always, launchd KeepAlive, cron watchdog, supervisord autorestart).
v0.22.14 makes bare-worker behavior fail-stop — without an external restart
loop the worker exits and stays dead. Migration calls this out loudly so
OpenClaw/Hermes-style downstream agents can verify their PM before upgrade.
- Five new MinionWorkerOpts fields documented with defaults and rationale.
- Worker-side process.exit(1) fallback semantics explained: CLI subscribes to
'unhealthy', but direct API consumers without a listener inherit fail-stop.
- AskUserQuestion-driven flow for the --max-rss 2048 default (raise / opt out /
keep) with concrete edits per PM (systemd unit, cron line, Docker compose,
launchctl plist).
- Verification commands (gbrain jobs stats, gbrain doctor --json, RSS check)
and a triage paragraph for opening an issue if anything fails.
TODOS.md:
- v0.22.15 embed cooperative-abort (P0, daily pain): plumb signal through
runPhaseEmbed → embed.ts → embedBatch; check signal.aborted between OpenAI
batch calls and between slugs. Closes the daily wedge where embed > 600s
timeout dead-letters the job but keeps running, holding gbrain_cycle_locks
until the lock TTL expires. PR #503 catches the symptom (worker stalled);
this captures the cause-side fix that's the real production resolution.
- v0.23+ bare-worker engine reconnect parity: extract supervisor's
reconnect-then-fail pattern (#406) into MinionWorker so transient PgBouncer
blips don't force a full process restart.
- v0.23+ minion_workers heartbeat table for queue_health doctor check (B7
follow-up): replace lock_until proxy with ground-truth worker liveness
signal so doctor stops crying wolf on legitimately idle workers.
* chore: bump version and changelog (v0.22.14)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>