mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
ed900c870e
* feat(minions): add self-health-monitoring to bare worker mode
Bare `gbrain jobs work` (without supervisor) previously had zero health
monitoring. If the Postgres connection dropped or the worker's event loop
deadlocked, the process stayed alive doing nothing — jobs piled up while
external process managers (systemd, Docker, cron) thought it was healthy.
Changes:
1. **Self-health-check timer** (worker.ts): Runs every 60s when not under
a supervisor. Two probes:
- DB liveness: `SELECT 1` — 3 consecutive failures → exit(1)
- Stall detection: waiting jobs + 0 in-flight + no completions for 5m
→ warning; 10m → exit(1)
2. **GBRAIN_SUPERVISED env var** (supervisor.ts): Supervisor sets this on
its child worker to prevent duplicate health checks. The supervisor
already has its own health monitoring.
3. **RSS watchdog default** (jobs.ts): Bare workers now default to
`--max-rss 2048` (matching supervisor default). Opt out: `--max-rss 0`.
4. **--health-interval flag** (jobs.ts): Configurable health check period.
`--health-interval 0` disables. Default: 60000ms.
5. **parseMaxRssFlag returns undefined** when flag is absent (vs 0), so
callers can distinguish 'not set' from 'explicitly disabled'.
The design ensures bare workers get supervisor-grade monitoring while
remaining compatible with any external process manager — the worker just
exits with code 1 on detected failure, letting the PM handle the restart.
Tests: 4 new tests (3 worker health, 1 supervisor env var). All 178 pass.
* fix(minions): harden bare-worker self-health-check after multi-round review
Layered fixes from 5 rounds of plan-eng-review + codex outside voice on top of
the original PR #503 (feat: bare-worker self-health-monitoring). Every change
below is in service of "fail-stop into the operator's process manager" without
introducing new ways the library can kill its caller.
worker.ts:
- MinionWorker now extends EventEmitter; emits `'unhealthy'` event with structured
reason payload (`db_dead` | `stalled`). CLI subscribes; library no longer calls
process.exit directly.
- emitUnhealthy() falls back to process.exit(1) when listenerCount('unhealthy') === 0
so direct API consumers without a listener inherit the pre-refactor fail-stop
default. Inline paths opt out via healthCheckInterval=0.
- Stall detection: count(*) query now filters by registered handler names
(`AND name = ANY($2::text[])`) so workers with handlers for {embed,sync} don't
false-positive when waiting jobs of unhandled names accumulate.
- Stall exit threshold measured from lastCompletionTime (not from warn-since), so
defaults of 5min warn / 10min exit fire at idle=10min total — matching the
documented contract.
- Recursive setTimeout pattern with running flag replaces setInterval, eliminating
callback overlap on slow DB probes.
- DB liveness probe wrapped in Promise.race against AbortController-driven
timeout (default 10s) so a hung executeRaw can't wedge the recursive chain
forever. Hung probes count as failures and feed dbFailExitAfter.
- Constructor validates stallExitAfterMs > stallWarnAfterMs and throws loudly
on misconfiguration. Internal timer-installation invariants documented inline.
- GBRAIN_SUPERVISED env-var check tightened from `!!process.env.X` to `=== '1'`.
types.ts:
- Added 5 new MinionWorkerOpts fields with documented contracts:
healthCheckInterval, stallWarnAfterMs, stallExitAfterMs, dbFailExitAfter,
dbProbeTimeoutMs.
- Exported `UnhealthyReason` discriminated union for the 'unhealthy' event payload.
supervisor.ts:
- GBRAIN_SUPERVISED=1 injected on the spawned worker child's env so the child's
self-health timer is skipped (no double-monitoring).
- setInterval(callback, healthInterval) gated behind `> 0`, so the
`--health-interval 0` documented disable contract actually disables instead
of producing a tight DB-hammer loop.
jobs.ts:
- `gbrain jobs work` subscribes to 'unhealthy' and calls process.exit(1) at the
CLI layer. Default --max-rss bumped from 0 to 2048 (matches supervisor default;
catches memory-leak stalls that previously went undetected).
- New --health-interval flag with aggressive validation (NaN/negative/sub-1000ms
rejected; parity with --max-rss) on both `jobs work` and `jobs supervisor`.
- `jobs submit --follow` and `jobs smoke` now pass healthCheckInterval=0 to
disable the self-health timer entirely. These are inline/one-shot flows with
no PM to restart them; the no-listener emitUnhealthy fallback could otherwise
trip on a DB blip and kill the user's CLI session.
- parseMaxRssFlag returns `number | undefined` (was `number`) so callers can
distinguish absent (use the default) from explicit-disable (--max-rss 0).
doctor.ts:
- New queue_health subcheck reports RSS-watchdog kills in the last 24h.
Detects via exact-match `error_text = 'aborted: watchdog'` (the worker's
failJob signature when gracefulShutdown('watchdog') aborts in-flight jobs)
scoped to status IN ('dead','failed'). Tight match avoids over-counting parent
jobs that propagate child failures via on_child_fail='fail_parent'.
* test(minions): self-health behavior + regression tests
7 new tests covering the production failure modes that drove the original PR,
plus regressions for fixes landed during multi-round review.
minions.test.ts:
- DB 3-strike → 'unhealthy' event with reason='db_dead' (the production-incident signature)
- DB recovery resets failure counter (no exit on intermittent failures)
- Stall warn-then-exit (clock-driven; idleMs > stallExitAfterMs is the new contract)
- inFlight > 0 blocks stall detection (long-running legitimate jobs don't false-trip)
- Regression for D1 fix: jobs of unregistered handler names don't trigger stall exit;
also captures the SQL via probe engine and asserts the predicate text contains
`name = ANY` so a future refactor that drops the filter is caught at test time.
- Regression for R3 constructor validation: throws when stallExitAfterMs <= stallWarnAfterMs
(covers both `<` and `=` cases); defaults still construct cleanly.
supervisor.test.ts:
- Regression for R3: supervisor with healthInterval=0 completes a normal lifecycle
within 10s. A tight setInterval(0) loop (the bug we fixed) would saturate the
event loop and slow this past the cap.
Tests use a Proxy-based engine helper (makeProbeEngine) that intercepts SELECT 1
and the count(*) WHERE status='waiting' query while passing through everything
else to the real PGLite engine. This isolates health-check semantics from claim
plumbing without mocking the entire engine surface.
* docs(v0.22.14): migration walkthrough + follow-up TODOs
skills/migrations/v0.22.14.md (new):
- Pre-flight per-PM restart-policy table (systemd Restart=always, Docker
restart: always, launchd KeepAlive, cron watchdog, supervisord autorestart).
v0.22.14 makes bare-worker behavior fail-stop — without an external restart
loop the worker exits and stays dead. Migration calls this out loudly so
OpenClaw/Hermes-style downstream agents can verify their PM before upgrade.
- Five new MinionWorkerOpts fields documented with defaults and rationale.
- Worker-side process.exit(1) fallback semantics explained: CLI subscribes to
'unhealthy', but direct API consumers without a listener inherit fail-stop.
- AskUserQuestion-driven flow for the --max-rss 2048 default (raise / opt out /
keep) with concrete edits per PM (systemd unit, cron line, Docker compose,
launchctl plist).
- Verification commands (gbrain jobs stats, gbrain doctor --json, RSS check)
and a triage paragraph for opening an issue if anything fails.
TODOS.md:
- v0.22.15 embed cooperative-abort (P0, daily pain): plumb signal through
runPhaseEmbed → embed.ts → embedBatch; check signal.aborted between OpenAI
batch calls and between slugs. Closes the daily wedge where embed > 600s
timeout dead-letters the job but keeps running, holding gbrain_cycle_locks
until the lock TTL expires. PR #503 catches the symptom (worker stalled);
this captures the cause-side fix that's the real production resolution.
- v0.23+ bare-worker engine reconnect parity: extract supervisor's
reconnect-then-fail pattern (#406) into MinionWorker so transient PgBouncer
blips don't force a full process restart.
- v0.23+ minion_workers heartbeat table for queue_health doctor check (B7
follow-up): replace lock_until proxy with ground-truth worker liveness
signal so doctor stops crying wolf on legitimately idle workers.
* chore: bump version and changelog (v0.22.14)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Wintermute <wintermute@garrytan.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
75 lines
3.0 KiB
JSON
75 lines
3.0 KiB
JSON
{
|
|
"name": "gbrain",
|
|
"version": "0.22.14",
|
|
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
|
"type": "module",
|
|
"main": "src/core/index.ts",
|
|
"bin": {
|
|
"gbrain": "src/cli.ts"
|
|
},
|
|
"exports": {
|
|
".": "./src/core/index.ts",
|
|
"./engine": "./src/core/engine.ts",
|
|
"./types": "./src/core/types.ts",
|
|
"./operations": "./src/core/operations.ts",
|
|
"./minions": "./src/core/minions/index.ts",
|
|
"./engine-factory": "./src/core/engine-factory.ts",
|
|
"./pglite-engine": "./src/core/pglite-engine.ts",
|
|
"./link-extraction": "./src/core/link-extraction.ts",
|
|
"./import-file": "./src/core/import-file.ts",
|
|
"./transcription": "./src/core/transcription.ts",
|
|
"./embedding": "./src/core/embedding.ts",
|
|
"./config": "./src/core/config.ts",
|
|
"./markdown": "./src/core/markdown.ts",
|
|
"./backoff": "./src/core/backoff.ts",
|
|
"./search/hybrid": "./src/core/search/hybrid.ts",
|
|
"./search/expansion": "./src/core/search/expansion.ts",
|
|
"./extract": "./src/commands/extract.ts"
|
|
},
|
|
"scripts": {
|
|
"dev": "bun run src/cli.ts",
|
|
"build": "bun build --compile --outfile bin/gbrain src/cli.ts",
|
|
"build:all": "bun build --compile --target=bun-darwin-arm64 --outfile bin/gbrain-darwin-arm64 src/cli.ts && bun build --compile --target=bun-linux-x64 --outfile bin/gbrain-linux-x64 src/cli.ts",
|
|
"build:schema": "bash scripts/build-schema.sh",
|
|
"build:llms": "bun run scripts/build-llms.ts",
|
|
"test": "scripts/check-jsonb-pattern.sh && scripts/check-progress-to-stdout.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && bun run typecheck && bun test --timeout=60000",
|
|
"check:wasm": "scripts/check-wasm-embedded.sh",
|
|
"check:newlines": "scripts/check-trailing-newline.sh",
|
|
"test:e2e": "bash scripts/run-e2e.sh",
|
|
"typecheck": "tsc --noEmit",
|
|
"check:jsonb": "scripts/check-jsonb-pattern.sh",
|
|
"check:progress": "scripts/check-progress-to-stdout.sh",
|
|
"postinstall": "command -v gbrain >/dev/null 2>&1 && gbrain apply-migrations --yes --non-interactive || echo '[gbrain] postinstall skipped. If installed via bun install -g github:...: run `gbrain doctor` and `gbrain apply-migrations --yes` manually. See https://github.com/garrytan/gbrain/issues/218' 1>&2",
|
|
"prepublish:clawhub": "bun run build:all",
|
|
"publish:clawhub": "clawhub package publish . --family bundle-plugin"
|
|
},
|
|
"openclaw": {
|
|
"compat": {
|
|
"pluginApi": ">=2026.4.0"
|
|
}
|
|
},
|
|
"dependencies": {
|
|
"@anthropic-ai/sdk": "^0.30.0",
|
|
"@aws-sdk/client-s3": "^3.1028.0",
|
|
"@dqbd/tiktoken": "^1.0.22",
|
|
"@electric-sql/pglite": "0.4.3",
|
|
"@modelcontextprotocol/sdk": "^1.0.0",
|
|
"gray-matter": "^4.0.3",
|
|
"marked": "^18.0.0",
|
|
"openai": "^4.0.0",
|
|
"pgvector": "^0.2.0",
|
|
"postgres": "^3.4.0",
|
|
"tree-sitter-wasms": "0.1.13",
|
|
"web-tree-sitter": "0.22.6"
|
|
},
|
|
"devDependencies": {
|
|
"@types/bun": "latest",
|
|
"bun-types": "^1.3.13",
|
|
"typescript": "^5.6.0"
|
|
},
|
|
"trustedDependencies": [
|
|
"@electric-sql/pglite"
|
|
],
|
|
"license": "MIT"
|
|
}
|