3 Commits

Include renames

Author	SHA1	Message	Date
4 people	d838d4792b	feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency (#379) ... * feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency, shell guard Prevents stall-induced queue blockage discovered in production (OpenClaw): 1. Wall-clock timeout sweep: dead-letters active jobs exceeding 2× timeout_ms (or 2 × lockDuration × max_stalled). Catches jobs stuck while holding DB connections where FOR UPDATE SKIP LOCKED stall detection skips them. 2. Submission backpressure (maxWaiting): caps waiting jobs per name at submission time. Prevents autopilot-cycle flood when the queue is blocked. 3. --no-worker flag for autopilot: skips spawning the built-in worker child. For environments where the worker lifecycle is managed externally (systemd, Docker, OpenClaw service-manager). 4. GBRAIN_WORKER_CONCURRENCY env var: fallback for --concurrency when the worker is spawned by autopilot (which can't pass CLI flags to the child). 5. Shell job env guard with clear logging: shell handler is always registered but throws UnrecoverableError with a clear message when GBRAIN_ALLOW_SHELL_JOBS=1 is not set, instead of silently not registering. * feat: v0.19.1 Lane A — maxWaiting atomic guard, concurrency clamp, --max-waiting CLI Addresses three production-hardening findings from the CEO + Eng + Codex adversarial review of PR #379: D2/H2: maxWaiting was TOCTOU-racy — two concurrent submitters could both see waitingCount < max and both insert. Wrap the count+select+insert in pg_advisory_xact_lock keyed on (name, queue). Serializes concurrent decisions for the SAME key while leaving different keys fully parallel. Lock auto-releases on txn commit/rollback — no cleanup path to leak. Also fix the missing queue-scope bug: count and select now filter on (name, queue) not name alone, so cross-queue same-name jobs don't suppress each other. D3/H3: resolveWorkerConcurrency silently accepted NaN / 0 / negative from parseInt. `inFlight.size < NaN` is always false → worker claims nothing → silent wedge from a single-typo env var. Clamp to ≥1 with a loud stderr warning naming the bad value. D5/H5: `gbrain jobs submit` never parsed `--max-waiting N` despite the MinionJobInput field. Wire the flag with clamp [1, 100], mirror `--max-stalled`. Extract `parseMaxWaitingFlag` for unit testing. Q1: Silent coalesce was invisible by design. New src/core/minions/backpressure-audit.ts mirrors shell-audit.ts's ISO-week JSONL pattern: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Coalesce events write one JSONL line with (queue, name, waiting_count, max_waiting, returned_job_id, ts). Best-effort — disk-full never blocks submission. A2: `gbrain jobs smoke --wedge-rescue` new opt-in regression case. Forges a wedged-worker row state, invokes handleStalled + handleTimeouts + handleWallClockTimeouts in order, asserts only wall-clock evicts. Mirrors the v0.14.3 `--sigkill-rescue` shape. Tests: 23 new unit cases in test/minions.test.ts covering wall-clock timeout (3 cases + non-interference with handleTimeouts), maxWaiting (coalesce, clamp 0, floor, concurrent-submitter race via Promise.all, cross-queue isolation, unset fallthrough), concurrency clamp (7 cases incl. NaN/0/negative), parseMaxWaitingFlag (5 cases), backpressure audit file write. Part of v0.19.1 plan at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: v0.19.1 Lane B — doctor queue_health, autopilot peer probe, runbook A5 / D4: New `queue_health` check in `gbrain doctor`. Postgres-only (PGLite has no multi-process worker surface). Two subchecks, both cheap (single SELECT each, status-index-covered): - stalled-forever: any active job with started_at > 1h. Surfaces the worst offenders (top 5 by started_at ASC) with `gbrain jobs get/cancel` fix hints. The incident that motivated v0.19.1 ran 90+ min before the operator noticed. - waiting-depth: per-name waiting count exceeds threshold. Default 10, overridable via GBRAIN_QUEUE_WAITING_THRESHOLD env (D9). Signals a submitter probably needs maxWaiting set. Worker-heartbeat subcheck from the original plan dropped (D4/H4): no minion_workers table exists, and lock_until-on-active-jobs is a lossy proxy that can't distinguish idle-worker from dead-worker. Tracked as follow-up B7. A4: --no-worker peer-liveness probe in autopilot. When --no-worker is set, every cycle runs a cheap SELECT checking for any active job whose lock_until was refreshed in the last 2 minutes. After 3 consecutive idle ticks, logs a loud WARNING naming the silent-wedge vector and referencing B7 as the ground-truth follow-up. Re-arms on next live signal so the warning doesn't spam every cycle. A6: New docs/guides/queue-operations-runbook.md (one viewport, ~60 lines). "My queue looks wedged — what do I run?" in order of escalation. What each doctor subcheck means. Self-check for the --no-worker / no-worker-running footgun. CLAUDE.md: key-files updates for handleWallClockTimeouts (v0.19.0 Layer 3 kill shot), maxWaiting advisory-lock rewrite (v0.19.1 D2), queue_health doctor check (v0.19.1 D4), and backpressure-audit.ts. Tests: all 143 minions + 13 doctor unit tests pass. No new test cases required in Lane B; the doctor queue_health exercise is in the E2E verification step (needs real PG to produce meaningful stalled-forever rows). The --no-worker probe is exercised by the smoke case's wedge setup in Lane A. README: unchanged. Existing `gbrain jobs submit` examples don't show --max-stalled, so no --max-waiting precedent to extend per A6 conditional. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: v0.19.1 Lane C — CHANGELOG entry, VERSION bump, remove SPEC.md VERSION: 0.19.0 → 0.19.1 (patch; bug-fix-dominant, no schema change, no new user-facing vocabulary). CHANGELOG: new v0.19.1 entry at the top with the full release-summary template per CLAUDE.md — bold two-line headline, lead paragraph, "numbers that matter" before/after table measured against the real incident, "what this means for OpenClaw users" closer, required "To take advantage of v0.19.1" block naming the worker-restart requirement, itemized changes by area, and "For contributors" section closing the loop on the stale autopilot-idempotency narrative the CEO review was based on. Mechanism reframing per D1/H1: the 18-job pile-up was NOT caused by missing idempotency (autopilot already passes `idempotency_key: autopilot-cycle:${slot}` at autopilot.ts:241). The 18 jobs were 18 DIFFERENT slots stacking up behind the wedged one. `maxWaiting` still caps the pile; the incident just wasn't about idempotency. Adversarial review caught this before ship. SPEC.md: deleted from repo root. It was Wintermute's planning artifact for the original PR, not a shipped spec. Design docs belong under docs/designs/ per repo convention; leaving one at repo root set a precedent this repo doesn't want (A7/D11). CHANGELOG + the plan file at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md are the durable artifacts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: --wedge-rescue smoke state — both stall+timeout sweeps must skip Smoke case was setting lock_until in the past, so handleStalled's requeue path fired before handleWallClockTimeouts had a chance to evict. Production scenario is "lock_until still live (worker renewing) + timeout_at disqualified" — only wall-clock matches. Single-connection smoke can't simulate a row lock held by another txn, so we force the equivalent outcome: - lock_until = now() + 30s → handleStalled skips (not a stall) - timeout_at = NULL → handleTimeouts skips (needs NOT NULL) - started_at = now() - 10s, timeout_ms=1000 → wall-clock matches (2 × timeout_ms = 2000ms threshold exceeded) Verified: SMOKE PASS — Minions healthy + wedge rescue in 0.14s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: CI failures — shell-handler tests + llms-full.txt drift Two CI failure clusters, both pre-existing but surfaced by the v0.20.3 merge: 1) test/minions-shell.test.ts — 12 failing cases. The shell handler throws UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1' (the production RCE guard at shell.ts:210). The unit tests exercise handler mechanics, not the guard, but never set the env var — so every invocation exits through the guard path instead of the code being tested. Fix: set GBRAIN_ALLOW_SHELL_JOBS=1 in beforeAll, restore in afterAll. The env-guard IS still tested separately via the test/minions.test.ts case added in v0.20.3 Lane A which toggles the var itself. 2) llms-full.txt — stale against CLAUDE.md. Key-files entries for queue.ts, doctor.ts, and the new backpressure-audit.ts updated in v0.20.3 Lane B triggered the build-llms drift guard. Regenerated via `bun run build:llms`; no behavior change, just the inlined-docs bundle catching up to source. Full test run: 2367 pass, 0 fail across 137 files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: root <root@localhost> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-24 01:09:28 -07:00
5 people	96178d726e	fix(subagent): v0.16.3 — bind Anthropic SDK correctly + enable tsc in CI (#318) ... * fix(subagent): bind Anthropic SDK messages.create() correctly The makeSubagentHandler was casting `new Anthropic()` directly to MessagesClient, but MessagesClient.create() maps to sdk.messages.create(), not sdk.create(). Every subagent job immediately died with: client.create is not a function Fix: wrap the SDK instance so .create() delegates to .messages.create() with proper `this` binding via .bind(sdk.messages). Discovered on first production run of gbrain agent against Supabase. Co-Authored-By: Wintermute <wintermute@openclaw.ai> * chore(ci): add typescript typecheck to test pipeline + clean up baseline errors Root cause infra gap that let the v0.16.0 subagent bug ship: CI ran only `bun test`, which transpiles types without checking them. Type errors only surfaced at runtime, in production. Changes: - Add `typescript` devDep and a `typecheck` npm script (`tsc --noEmit`). - Chain `bun run typecheck` into `bun run test` so developers get the same pipeline locally that CI runs. - Flip `.github/workflows/test.yml` to invoke `bun run test` (the npm script, including typecheck) instead of `bun test` (runner only). - Clean up 100+ pre-existing type errors across 30+ files so the first run of `tsc --noEmit` is green. Root causes were: - `databaseUrl` → `database_url` rename drift in test fixtures (9 files) - `PageType` union missing `'meeting'` / `'note'` entries that are already used in both src and tests (link-extraction.ts comments acknowledged the gap) - `GBrainConfig.storage` field never declared despite being read in files.ts and operations.ts - `ErrorCode` union missing `'permission_denied'` - `OrchestratorOpts` shape changed; test callers not updated - Dead-code comparisons in migration orchestrators against narrowed status types - postgres.js `Row`-callback type drift on several `.map()` calls - Buffer-as-BodyInit assignment in supabase.ts (real but non-fatal runtime bug; Uint8Array slice works and is type-correct) - Various `as X` single-step casts that now need `as unknown as X` per TS's stricter structural-conversion rules - Bump `beforeAll` hook timeout to 30s on four PGLite-heavy tests that were flaky under parallel test execution: wait-for-completion, extract-fs, e2e/search-quality, e2e/graph-quality. All pass in isolation; timeouts only happened when dozens of PGLite instances init'd simultaneously. The new CI pipeline now fails on any type error across src/ or test/, giving us the compile-time regression guard the subagent fix depends on. * fix(subagent): bind Anthropic SDK messages.create() correctly Shipped bug: v0.16.0 cast `new Anthropic()` to `MessagesClient`, but `.create()` lives at `sdk.messages.create`, not on the top-level client. Every subagent job in production died on first LLM call with `client.create is not a function`. Discovered on the first `gbrain agent run` against Supabase. Fix: assign `sdk.messages` directly to the `MessagesClient` slot. `sdk.messages` IS the object with a callable `.create()`; the original bug was picking the wrong entry point on the SDK. No helper, no wrapper, no `.bind()` — JS method-call semantics preserve `this` at the call site because `subagent.ts:336` invokes `client.create(...)` with `client === sdk.messages`. The one-line assignment also typechecks cleanly against the existing `MessagesClient` interface (SDK's first `create` overload: `(MessageCreateParamsNonStreaming, Core.RequestOptions?) => APIPromise<Message>` is assignable structurally). This gives us compile-time regression protection: anyone reverting to `new Anthropic()` would fail tsc because `Anthropic` has no top-level `.create`. (The companion chore commit puts `tsc --noEmit` in CI so this guard is enforced.) Also adds a `makeAnthropic?: () => Anthropic` dep-injection seam so the factory default construction branch is testable without real API calls. Regression test drives one handler turn through a fake SDK, asserting `sdk.messages.create` is actually called. If someone later reverts to `new Anthropic()`, both guards fire: tsc fails AND the test fails. Co-Authored-By: Wintermute <wintermute@garrytan.com> * chore(tests): add bunfig.toml + 60s hook timeouts to stabilize PGLite-heavy suites After turning on tsc in CI (previous commit), running the full `bun run test` suite in one shot triggered flaky `beforeEach/afterEach hook timed out` failures on 8+ test files. Every failure traced to PGLite WASM init contention when many test files spin up fresh PGLite instances in parallel; each one alone passes in isolation. - `bunfig.toml` sets the global test hook timeout to 60s (default is 5s), covering every test file without per-file edits. - Individual `beforeAll(fn, 60_000)` / `beforeEach(fn, 15_000)` calls on the 8 tests that flaked most stay in place as explicit safety nets so a future bunfig config change doesn't silently re-introduce the flake. Result: 1997 pass, 0 fail on `bun run test` (117 tests added since the prior baseline by picking up typecheck-gated passes). No infrastructure flake tolerated in CI. * chore: bump version and changelog (v0.16.3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Wintermute <wintermute@garrytan.com> Co-authored-by: Wintermute <wintermute@openclaw.ai> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-22 01:34:22 -07:00
3 people	5fd9cd2644	feat: shell job type + worker abort-path fix (v0.13.0) (#217) ... * feat(minions): add protected-name constant + ctx.shutdownSignal Introduce PROTECTED_JOB_NAMES ('shell') in a side-effect-free core module so queue.ts can check it without importing from handlers/. MinionJobContext gains shutdownSignal (distinct from signal) — handlers that need to run SIGTERM-triggered cleanup subscribe to both; most handlers ignore shutdown and run through the worker's 30s cleanup race to natural completion. * fix(minions): MinionQueue.add gains trusted 4th arg + trim-normalized guard Adds allowProtectedSubmit opt-in as a separate 4th parameter (NOT folded into opts) so callers spreading user-provided opts ({...userOpts}) can't accidentally carry the trust flag. PROTECTED_JOB_NAMES check runs on the trimmed name BEFORE insert, closing the queue.add(' shell ', ...) whitespace bypass that would have evaded a has(name) check. * fix(minions): worker calls failJob on abort + wires ctx.shutdownSignal Pre-v0.13.0 worker returned silently when ctx.signal.aborted fired, leaving jobs in 'active' until stall sweep. Handlers using cooperative cancel had no deterministic status flip — timeout/cancel/lock-loss all looked the same from downstream callers (gbrain jobs get, --follow loops). Fix: derive abort reason from abort.signal.reason ('timeout' | 'cancel' | 'lock-lost' | 'shutdown') and call failJob with 'aborted: <reason>' text. failJob is idempotent via token+status match, so no-op when another path already flipped status (handleTimeouts, cancelJob, stall). Also: new shutdownAbort (instance-level AbortController) fires on process SIGTERM/SIGINT and propagates to every handler's ctx.shutdownSignal. Shell handler listens to both signals and runs SIGTERM→5s→SIGKILL on its child on either; other handlers only listen to ctx.signal so deploy restarts don't cancel them mid-flight. * feat(minions): add shell job handler + submission audit log New 'shell' job type spawns arbitrary commands under the Minions worker. Deterministic cron scripts (API fetch, token refresh, scrape+write) can move off the LLM gateway — zero Opus tokens per fire. Handler contract: - cmd or argv (exactly one required). cmd spawns via /bin/sh -c (absolute path, not 'sh', to block PATH-override shell substitution). argv spawns direct with no shell. - cwd required, must be absolute. Operator-trust boundary. - env defaults to SHELL_ENV_ALLOWLIST ({PATH, HOME, USER, LANG, TZ, NODE_ENV}) picked from process.env, with caller overrides merged on top. Prevents accidental $OPENAI_API_KEY interpolation into scripts. - stdout/stderr retained as UTF-8-safe tails (64KB/16KB) via string_decoder.StringDecoder. Prepends [truncated N bytes] marker. - Abort (either ctx.signal or ctx.shutdownSignal) fires SIGTERM → 5s grace → SIGKILL on child. Timer NOT .unref'd so worker's 30s race waits for the child to actually die. shell-audit.ts writes a JSONL line per submission to ~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl (ISO-week rotated, override via GBRAIN_AUDIT_DIR). argv logged as JSON array (not space-joined, which would flatten args with spaces). Never logs env values. Best-effort writes: failures log to stderr but don't block submission. * feat(jobs): submit_job MCP guard + CLI --timeout-ms + starvation warning submit_job operation gains timeout_ms param (was missing — couldn't plumb the existing MinionJobInput field through from either CLI or MCP). When ctx.remote=true and name is in PROTECTED_JOB_NAMES, throws OperationError('permission_denied'). Combined with the queue.add trusted guard, MCP callers can never submit shell jobs even if the env flag is on. CLI submit: new --timeout-ms N flag. Passes {allowProtectedSubmit:true} as the 4th arg to queue.add only when the submitted name is protected (not blanket-set for every job). Prints a starvation-warning block to stderr when a shell job is submitted without --follow, pointing at both --follow and 'gbrain jobs work' remediation. Fires for every shell submit regardless of the submitter's env — the submitter env is a weak proxy for the worker env. Worker handler registration: conditional on GBRAIN_ALLOW_SHELL_JOBS=1. Default: off. 'gbrain jobs submit --help' now lists handler types with a pointer to docs/guides/minions-shell-jobs.md for shell. * test(minions): 40 unit + 4 E2E cases for shell handler Unit (test/minions-shell.test.ts): - Protected names: trim-normalized, case-sensitive, whitespace bypass defense - MinionQueue.add: trusted opt-in, whitespace bypass, non-protected untouched - Handler validation: cmd|argv exclusive, cwd required/absolute, env strings - Spawn: cmd/argv happy paths, non-zero exit, ENOENT, result shape - Env allowlist: leaked-secret blocked, PATH inherited, caller override - Abort: ctx.signal, ctx.shutdownSignal, pre-aborted signal - Audit: ISO-week year boundary (2027-01-01 → W53 2026), mid-year W52/W53, GBRAIN_AUDIT_DIR override, argv as JSON array, env never logged, EACCES non-blocking - Output truncation: 100KB → last 64KB with [truncated N bytes] marker E2E (test/e2e/minions-shell.test.ts): - Full lifecycle: submit → worker claim → spawn → complete - MinionQueue.add without trusted arg throws (including whitespace bypass) - submit_job with ctx.remote=true rejects shell (MCP guard) - submit_job with ctx.remote=false allows shell (CLI path) * chore: bump version and changelog (v0.13.0) Move gateway crons to Minions. Zero LLM tokens per cron fire. Worker abort path finally marks aborted jobs dead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: reframe v0.13.0 copy for OpenClaw operators (not Wintermute-specific) gbrain is an open-source product for any OpenClaw/Hermes operator, not Garry's personal Wintermute deployment. Rewords the v0.13.0 CHANGELOG entry, the minions-shell-jobs guide, and the deferred TODOS entries to speak to "your OpenClaw" / "OpenClaw operators" instead. Replaces /data/wintermute cwd examples with the canonical /data/.openclaw/workspace path. Pre-existing Wintermute references in older CHANGELOG entries (v0.11/v0.10.3) left unchanged. * feat(migrations): add v0.13.0 adoption playbook for shell jobs Adding the migration file the CEO review originally scoped out. Without it, operators upgrade to v0.13.0 and the capability ships but adoption doesn't happen — the 60% gateway CPU reduction only lands if someone actually rewrites their crontab. skills/migrations/v0.13.0.md is the instruction manual the host agent reads on gbrain upgrade: - Enable worker: GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work (Postgres) or per-tick --follow (PGLite) - Audit cron manifest: classify LLM-requiring vs deterministic - Propose per-cron rewrites with diffs, approved one at a time - Env allowlist guidance for scripts that need API keys - Verification playbook: run one fire, compare pre/post, only then approve the next batch - Starvation sanity-check runbook item Iron rules: never auto-rewrite the operator's crontab (host-specific code per CLAUDE.md). LLM-requiring crons stay on the gateway. Ambiguous cases ask the operator. No mechanical orchestrator ships with this migration — every rewrite is operator judgment. A future gbrain crontab-to-minions helper is tracked in TODOS.md as P1. * docs: sync UPGRADING + SKILLPACK with v0.13.0 shell jobs UPGRADING_DOWNSTREAM_AGENTS.md: append v0.13.0 section per the file's convention (each release appends). No skill edits required, feature is off-by-default, optional adoption via skills/migrations/v0.13.0.md. Lists typical LLM-vs-deterministic classifications so operators know which of their crons are candidates for migration. GBRAIN_SKILLPACK.md: add shell-jobs guide row to the cron/Minions guide table so it's discoverable alongside existing Cron via Minions, Plugin Handlers, and Minions fix guides. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-20 10:54:31 +08:00