mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
73bbbde01dcdf5da4bca9fa57be44a5cd8ebd60d
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
7ea92d602c |
v0.42.48.0 feat(durability): auto-harden brain repos for git durability on PAT+URL (#2241)
* feat(git): divergence-safe pull, push-probe, default-branch detection for brain durability Add GIT_ENV_AUTH + divergenceSafePull (skip-on-dirty, conflict-abort-clean, never-mid-rebase), detectDefaultBranch, pushProbe, and an env-gated GBRAIN_GIT_ALLOW_FILE_TRANSPORT escape hatch. Export GIT_ENV. pullRepo's --ff-only contract is unchanged. * feat(durability): brain-repo hardening core (hook, helper, cron, PAT, AGENTS rules) hardenBrainRepo/unhardenBrainRepo: local untracked post-commit hook + committed brain-commit-push.sh (one shared push-retry template), repo-scoped credential with existing-helper reuse, push-probe verify, active-resolver-file rules with taxonomy from _brain-filing-rules.json, minimal DB-free pull cron. PAT redaction via redactSecretsInText. * feat(sources): harden/pull/unharden commands + auto-harden on add --url sources harden/pull/unharden subcommands; --pat-file/--no-harden on add; auto-harden managed clones on add; unharden-before-remove. cli.ts pre-connect early-exit for DB-free 'sources pull --path' (the cron entry, never opens PGLite). * test(durability): unit + integration coverage for brain-repo durability git helpers, core harden/unharden, hook+helper E2E (real background push), cron generators. 41 tests across 4 files. * chore: bump version and changelog (v0.42.48.0) Brain-repo git durability: auto-harden a brain's working tree (local auto-push hook, committed commit-push helper, always-on agent rules, DB-free pull cron, repo-scoped credential, push-probe verify) the moment gbrain gets a PAT + URL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sources): route harden exit code through setCliExitVerdict A raw process.exitCode write is zeroed by the owned-verdict flush-exit (#2084 PGLite-Emscripten pollution defense); cli-exit-verdict-pin guard caught it. Use setCliExitVerdict(3) so 'sources harden' actually reports needs-attention to cron/automation. * docs: document brain-repo durability (KEY_FILES + multi-source guide) KEY_FILES: extend git-remote.ts entry (divergenceSafePull, pushProbe, detectDefaultBranch, GIT_ENV_AUTH, GBRAIN_GIT_ALLOW_FILE_TRANSPORT) + add brain-repo-durability.ts/sources-harden.ts entry. multi-source-brains.md: add a Durability (auto-harden) how-to covering sources harden/pull/unharden, --pat-file, the guarantees, and the security posture. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a81f7e05e8 |
v0.42.43.0 feat(context): push-based context (#2095) + teardown-exit hardening (#2084) (#2175)
* fix(cli): exit deliberately after bounded teardown instead of riding the 10s backstop (#2084)
Root cause: bounded teardown (endPoolBounded, #2015) RESOLVES, but lingering
sockets — embedding-provider fetch keep-alive, PgBouncer txn-mode sockets the
bound raced past — keep Bun's event loop alive, so every `gbrain query` paid
a flat 10s tax exiting via the hard-deadline force-exit banner.
Three changes, one contract:
- flushStdoutThenExit (cli-force-exit.ts): when main() resolves and the
command is not a daemon, exit deliberately — after stdout AND stderr drain
(writableLength===0, 'drain'-event + poll loop, 2s unref'd guard for a
blocked pipe). Incident #1959 (force-exit truncating piped stdout) is the
regression class; pinned by a 256KB real-pipe subprocess test.
- drainThenDisconnect (cli.ts): ONE owner-disconnect helper at all 8 sites
(op-dispatch, CLI_ONLY fall-through, search dashboard, doctor remediation
x3, ze-switch, dream, read-only timeout path). Drains the background-work
registry, then disconnect (best-effort), bounded by the 10s unref'd
hard-deadline — which is now armed around the TEARDOWN window only, not
before the op handler (the old placement would have force-killed any op
slower than 10s). Closes the filed TODOS P3 drain-hoist: six sites
previously skipped the drain entirely and had no hang timer at all.
- Inner process.exit sweep: mid-handler exits in engine-owning/output-bearing
paths (status, friction, claw-test, smoke-test, eval cross-modal /
takes-quality replay / conversation-parser / whoknows-thin, status-thin)
become process.exitCode + return so they flow through the drains and the
flush-exit. Pre-engine usage/parse/refusal exits stay as-is.
BrainRegistry.disconnectAll deliberately unchanged: zero production callers
in src/, per-engine disconnects already bounded, and the kernel reclaims
sockets on exit (src/core/timeout.ts doctrine).
DAEMON_COMMANDS gains 'watch' ahead of the #2095 push transport.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): PgBouncer transaction-mode pooler in CI + teardown e2e (#2084)
Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs
verified only against one production deployment — CI had no transaction-mode
pooler and could never see the class. Now it can:
- docker-compose.ci.yml: `pgbouncer` service (transaction pooling) fronting
postgres-1, mirroring the production split-pool topology (direct :5432 +
pooled :6543). AUTH_TYPE=plain (pg16 SCRAM verifiers need the plaintext
password in the userlist) + IGNORE_STARTUP_PARAMETERS for the
statement_timeout/idle_in_transaction_session_timeout startup params
gbrain's client sets (the Supabase pooler whitelists the same).
- test/e2e/pgbouncer-teardown.test.ts: schema + fixture via the DIRECT url
into a dedicated `gbrain_pgbouncer` database (never races shard TRUNCATEs),
then spawns the real CLI against the POOLED url and asserts: exit 0,
stdout intact (the #1959 truncation class), and NO
"did not return within 10000ms — force-exiting" banner (pre-#2084 it
printed on 100% of query-shaped ops on this topology). Class bound, not
exact timing. Skips gracefully without GBRAIN_PGBOUNCER_URL.
- scripts/ci-local.sh: threads GBRAIN_PGBOUNCER_URL +
GBRAIN_PGBOUNCER_DIRECT_URL into all three e2e phases.
Verified live: both tests green against pgbouncer 1.25.2 in transaction mode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(schema): context_volunteer_events table (v116) — push-context feedback log (#2095)
One row per page the brain volunteers (op / reflex / watch channels).
"Used" is DERIVED, never written: pages.last_retrieved_at > volunteered_at
(the existing bumpLastRetrievedAt write-back is the open/cite signal), so
there is no second tracking path. session_id/turn are nullable
caller-supplied attribution; rationale is a deterministic template string,
never raw conversation text.
- Migration v116 (idempotent) + mirrors in src/schema.sql +
src/core/pglite-schema.ts + regenerated schema-embedded.ts (regen also
folds in pre-existing comment-only drift from the v114 links edits).
- src/core/context/volunteer-events.ts: insertVolunteerEvents (ONE
multi-row parameterized INSERT — never per-row awaited round-trips) +
purgeStaleVolunteerEvents (90-day GC, returns 0 on pre-v116 brains).
- Dream cycle purge phase prunes stale events alongside op_checkpoints /
brainstorm checkpoints / batch-retry audit files.
- RLS on Postgres comes from the v35 auto_rls_on_create_table event
trigger (the same mechanism that covered v110 page_aliases and v115
op_checkpoint_paths); the volunteer Postgres e2e pins it.
- No ::jsonb anywhere; no bootstrap probe needed (nothing references the
table pre-creation; writers guard with try/catch).
Tests: v116 shape + columns + indexes + live insert/purge round-trip on
PGLite (test/migrate.test.ts, 161 pass); schema-bootstrap-coverage green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): multi-turn window extraction + confidence-scored volunteer core (#2095)
- entity-salience.ts: extractCandidatesFromWindow(turns) — runs the existing
per-turn extractor across the last N turns (oldest→newest), merges by the
normalizeAlias form with occurrence/newest-turn/user-mention metadata, and
orders by salience (recency > frequency > user-role) so the MAX_CANDIDATES
cap drops stale assistant chatter, not the entity the user just named.
Closes the filed assistant-introduced-entities recall TODO; true pronoun
coreference (never-named antecedents) stays out of scope.
- retrieval-reflex.ts: ReflexPointer gains source_id + arm + confidence +
matchedNorm. ARM_CONFIDENCE (alias 0.9 / title 0.8 / slug-suffix 0.6)
lives next to the arm definitions so identity and score can't drift.
Arm-2 provenance is classified in JS (codex D8 — the combined OR can't
report which predicate matched). Federated sourceIds[] scope (alias arm
loops per source; arm 2 uses source_id = ANY — no engine-interface
change). Suppression gains 'slug-only' mode (codex D7, REQUIRED for
windowing): the legacy title-whole-word rule would suppress every entity
merely MENTIONED in a prior window turn, breaking the feature by
construction — slugs only enter context when a pointer/page was actually
surfaced. Default stays 'slug-and-title' for the window=1 legacy path.
- volunteer.ts (new): parseWindow (lenient user:/assistant: prefixes, CRLF,
unprefixed → one user turn), volunteerContext (zero-LLM: extract →
resolve → +0.05 multi-turn/newest-turn boost → min_confidence 0.7 gate →
cap 3/5; deterministic rationale strings, never raw conversation text),
and volunteerUsageStats (per-arm/channel precision from the
last_retrieved_at join, labeled approximate — 5-min throttle false
negatives, unrelated-read false positives; codex D9).
Tests: 35 green across volunteer-context (window parsing, pronoun follow-up
via assistant-introduced entity, confidence gating, slug-only suppression,
takes-fence privacy, multi-source scope, caps, stats join math) +
retrieval-reflex back-compat + resolve-ipc.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(ops): volunteer_context op — CLI (stdin) + MCP, drained event sink (#2095)
New read-scope op on the contract surface (CLI `gbrain volunteer-context`
with stdin → window, MCP tool for free): takes a rolling conversation
window, returns confidence-gated page pointers with rationales + synopses.
`window` is optional-unless-stats (validated in the handler, codex D9);
`stats: true` returns the volunteered-vs-used precision summary, labeled
APPROXIMATE (the 5-min last-retrieved throttle and unrelated reads both
bias the join). Source scope threads through sourceScopeOpts — federated
grants narrow the volunteer to the granted sources.
Event logging is fire-and-forget through a new `volunteer-events`
background-work sink (volunteer-events.ts, mirrors last-retrieved: tracked
dangling promise set + bounded drain + snapshot-drop on timeout so a
long-lived process never accumulates ghosts). ONE batched INSERT per call,
drained on every exit path by the commit-1 drain hoist; failure never
fails the op (pinned by an injected failing-engine test).
cli formatResult renders both shapes (pointer lines with confidence/arm/
rationale; the stats summary with per-arm precision).
Tests: op contract surface, window-required validation, sink round-trip
with session_id/turn attribution, failing-engine fail-open, federated
grant scoping, stats mode (26 green on PGLite) + a real-Postgres e2e
proving the op + sink + stats join AND that context_volunteer_events has
RLS enabled (keeps the auto-RLS event-trigger mechanism honest for v116).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(context): reflex consumes the rolling window + ambient-channel logging (#2095)
The default-on retrieval reflex now extracts entities from the last N turns
(retrieval_reflex_window_turns, default 4; env
GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; window=1 reproduces the legacy
current-turn-only behavior exactly). assemble() passes the recent
user/assistant turns (hard cap 12); the reflex slices to the configured
window. Assistant-introduced entities and "what did she invest in?"
follow-ups whose antecedent was NAMED in the window now surface pointers —
the issue's "zero agent-initiated queries" success criterion on the
ambient path.
Under windowing, suppression switches to slug-only (codex D7): the legacy
title-whole-word rule would suppress every entity merely MENTIONED in a
prior window turn, breaking the feature by construction. Slugs only enter
prior context when a pointer/page was actually surfaced, so
already-surfaced pages still suppress. The suppression mode flows through
all three resolver rungs (host opts, serve IPC request, direct Postgres).
Ambient-channel feedback (codex D11): the server-side resolver paths
(serve IPC + direct Postgres) log volunteered pointers with
channel: 'reflex' through the drained volunteer-events sink, so
`gbrain volunteer-context --stats` measures the default-on path where most
volunteering happens. Host-injected resolvers (no gbrain engine) can't
log — documented gap. Precision gates, 1.5s ceiling, fail-open, and the
pointer cap are unchanged.
Tests: prev-assistant-turn entity fires; window=1 legacy parity; slug-only
vs already-surfaced suppression; throwing resolver stays fail-open
(16 green).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): gbrain watch — push transport over stdin (#2095)
The issue's headline: the brain volunteers pages as the conversation flows,
instead of waiting to be asked. `some-transcript-feed | gbrain watch` reads
turns line-by-line ('user:'/'assistant:' prefixes set the role; unprefixed
lines are user turns), keeps a rolling window (--window-turns, default 4),
and streams confidence-gated pointers with rationales to stdout (--json for
JSONL). Session dedupe rides the core's slug-only suppression — a slug is
volunteered at most once per session. Events log on channel 'watch' with
session_id + turn through the drained sink.
Lifecycle: watch BLOCKS in the stdin iteration (like `jobs work`) — an
interactive TTY stays alive until Ctrl-C/Ctrl-D, piped input ends at EOF —
so it is deliberately NOT in DAEMON_COMMANDS (reverts the commit-1
placeholder): when main() resolves the work is over, the CLI_ONLY finally
drains volunteer events via drainThenDisconnect, and the entrypoint
flush-exit ends the process. Keeping it in the daemon set would have made
the piped EOF path hang on lingering sockets — the exact #2084 class.
SIGINT closes the stream and flows through the same drain path instead of
killing mid-write. Per-turn resolution failures are fail-open (the stream
never dies on a transient DB error).
Full wiring (eng-review D12): CLI_ONLY + CLI_ONLY_SELF_HELP (WATCH_HELP) +
THIN_CLIENT_REFUSED_COMMANDS (thin clients use the volunteer_context MCP
op) + main --help entry.
Tests: 18 green — help, per-turn volunteering + clean EOF return, rolling
window via assistant-introduced entity, session dedupe, --json shape with
turn attribution, channel-watch event rows, --min-confidence gate, CRLF/
blank tolerance, daemon-gate semantics. Live smoke: piped `gbrain watch`
on a fresh PGLite brain exits 0 at EOF with no force-exit banner.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: KEY_FILES + push-context guide + TODOS for the #2084/#2095 wave
- docs/architecture/KEY_FILES.md (current-state): context entries gain the
window extractor, arm provenance/confidence, suppression modes, volunteer
+ volunteer-events modules; background-work entry now lists FIVE sinks and
the drainThenDisconnect owner-disconnect contract; new entries for
src/core/cli-force-exit.ts (the exit contract) and src/commands/watch.ts.
- docs/guides/push-context.md (new): the three channels (reflex/op/watch),
the confidence model, CLI usage, config keys, and the approximate-stats
caveat. Linked from CLAUDE.md's reference map.
- CLAUDE.md: ops line mentions volunteer_context + the guide link;
bun run build:llms regenerated in the same commit (freshness test green).
- TODOS.md: #2095 deferrals filed (SSE push channel, policy skill + doctor
check, structured messages[] param); the #1981 entity-detection TODO
narrowed (window extraction covered assistant-introduced entities +
named-antecedent follow-ups); the drain-hoist P3 marked DONE by the
#2084 wave.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(e2e): truncate context_volunteer_events in setupDB (#2095)
The new feedback-log table wasn't in ALL_TABLES, so volunteered-event rows
persisted across e2e runs on a reused database and poisoned count/stats
assertions in volunteer-context-postgres on the second run. No FK to pages
(slug join), so position before pages is for hygiene only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): own the exit verdict — never trust ambient process.exitCode (#2084)
Caught by the full unit suite: `gbrain apply-migrations` on PGLite started
exiting 99. Root cause: PGLite's Emscripten runtime writes the WASM
backend's proc_exit status into process.exitCode (initdb at create-time,
the postmaster at close-time — `exitCode=status` in pglite's dist), and
the writes land ASYNCHRONOUSLY, outside any snapshot/restore window around
create/close (a guarded attempt verified this). The pre-#2084 success path
never read process.exitCode, so the pollution was invisible; the new
deliberate flush-exit propagated it faithfully.
Fix: gbrain records its own verdict. setCliExitCode(n)/getCliExitCode() in
cli-force-exit.ts — every gbrain-owned exit-code assignment routes through
the setter (still mirrored to process.exitCode for outside readers), and
both exit paths (entrypoint flushStdoutThenExit + the drainThenDisconnect
hard-deadline backstop) read the getter. Swept all assignment sites:
cli.ts (op error, friction, claw-test, smoke-test, eval runners, status,
import errors) + reindex/transcripts/brainstorm/frontmatter/autopilot.
Also updates the v0.42.20 structural pins to the drainThenDisconnect shape
(ordering invariant asserted INSIDE the helper + >=8 helper call sites,
superseding the two-inline-pairs assertion).
Verified: apply-migrations spawn test green; `init --migrate-only` exits 0;
an errored op still exits 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: re-pin the teardown-arming invariant at its post-#2084 home
Master's v0.42.41.0 triage wave and the #2084 wave fixed the same
pre-armed-timer bug independently; the merge keeps #2084's shape (arming
inside the shared drainThenDisconnect helper, covering all 8 exit paths).
The structural pin now asserts the same invariant — no pre-try arming;
gated, unref'd, before-drain, cleared — at the helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: coverage for ambient reflex-channel logging + watch window/cap flags
Ship coverage audit (85%, gate PASS) named five gaps; the two substantive
cheap ones close here: the codex-D11 logChannel='reflex' path now has a
behavioral pin (events land on channel 'reflex' through the drained sink;
no logChannel → no events), and gbrain watch's --window-turns / --max-pages
flags are exercised (turn-1 attribution under window=1; cap to one page).
Remaining flagged-not-blocking: the wallclock-timeout branch (untestable
without >10s real-clock flake — same rationale as the arming pin),
formatResult's volunteer case (module-private), and the cycle purge wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: close the remaining plan-audit gaps — formatResult rendering + watch SIGINT
formatResult exported for tests (same import-safety contract as cliAliases);
test/cli-format-volunteer.test.ts pins the pointer lines, empty-gate message,
and approximate stats summary. test/watch-command.test.ts gains a real
subprocess SIGINT test: piped stdin that never reaches EOF, SIGINT mid-stream,
assert exit 0 with no force-exit banner — the drain-then-exit lifecycle under
the actual signal, not just the shared exit path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: doctor's FAIL verdict was zeroed by the owned exit — sweep stragglers + class pin
The merged-state suite caught it: doctor --fast --json reported FAIL but
exited 0. Master's v0.42.41.0 brought raw `process.exitCode =` writes
(doctor.ts hasFail ternary, extract.ts) that the #2084 verdict-owning exit
silently zeroes — getCliExitCode() deliberately never reads ambient
process.exitCode (the PGLite-Emscripten pollution defense), so any setter
that bypasses setCliExitCode reports success on failure.
Swept both sites and added the structural class pin: a test greps src/ for
raw `process.exitCode =` outside cli-force-exit.ts, so the next merge that
introduces one fails loudly instead of lying about exit codes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump version and changelog (v0.42.43.0)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: quarantine the watch SIGINT subprocess test to the serial lane
The parallel unit shards flake on concurrent CLI subprocess spawns (failed
at 7ms in-suite, green solo) — same isolation rationale as
apply-migrations-pglite-spawn.serial.test.ts and #2141's R3 quarantine.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: update project documentation for v0.42.43.0
Post-ship doc verification against the release diff (#2095 push-based
context + #2084 superset hardening), with a cross-model doc review:
- push-context.md: version tag corrected to v0.42.43.0; per-call knobs
now cover prior_context/days and watch's flag surface accurately;
feedback-log writes described as best-effort; synopsis fence-strip
described as unconditional.
- CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context
release reference corrected to v0.42.43.0.
- KEY_FILES.md: ci-local entry rewritten to current topology (4-shard
parallel default, four Postgres services, transaction-mode PgBouncer
+ GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts
dropped from the selector entry.
- TESTING.md: inventory entries for the new #2084 structural pins
(cli-exit-verdict-pin, cli-pipe-truncation), the push-context test
suite (volunteer-context, watch-command, watch-sigint.serial,
cli-format-volunteer), migrate v117 coverage, and the two new E2E
files (pgbouncer-teardown env gating, volunteer-context-postgres RLS
pin); check:all row corrected (not a superset of verify).
- AGENTS.md + RELEASING.md: ci:local descriptions updated to the
sharded + pooler topology.
- CHANGELOG (wording only, entry preserved): "retrieved" instead of
"opened" for the used-signal, pooler scoped to the local CI gate,
feedback log labeled best-effort.
- llms-config.ts: index the new push-context guide; bundles
regenerated (build:llms) and freshness test green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(test): correct the v116 reference — the table shipped as migration v117
* fix: pre-landing review hardening — federated alias parallelism, trust-boundary clamps, shared protocol helpers (#2095)
Five specialist reviewers (testing/maintainability/security/performance/
data-migration) on the reconciled diff; every finding applied:
Performance: the alias arm now resolves all granted sources CONCURRENTLY
(a federated caller paid M sequential RTTs per turn — ~355ms at 5 sources
cross-region, inside the reflex's 1.5s budget); watch's session dedupe is
O(1) Set membership instead of a monotonically growing priorContext string
(O(T²) over a long-lived session); getWindowTurns iterates from the tail
(per-turn cost no longer grows with session length); the resolver's
provenance maps fold into the existing candidate pass.
Security: volunteer_context clamps caller-supplied attribution at the trust
boundary — session_id capped at 256 chars (a read-scoped token could bank
~1MiB TEXT per request, retained 90 days), turn logged only when a safe
integer (a non-integer threw inside the batched INSERT and silently dropped
the whole batch). The privacy comments now state precisely what rationale
may contain (the matched entity's surface form — which by construction
resolved to an existing alias/title/slug — never free conversation text).
Maintainability: TURN_PREFIX_RE + formatVolunteeredPage exported from
volunteer.ts and shared by watch/cli (the two surfaces can no longer
drift); volunteerEventRowsFrom is the single VolunteerEventRow assembly
site for all three channels; watch's window default now honors the same
retrieval_reflex_window_turns config knob the reflex reads; the stale
pre-v116 comments swept to pre-v117.
Testing: the two flake-class CRITICALs fixed (pipe test asserts the
backstop banner instead of a cold-CI-hostile 9s wall bound; the SIGINT test
waits on watch's new machine-readable ready line instead of a fixed 15s
sleep — 2.5s and deterministic now); new coverage for the sink's timeout
branch + ghost-reference drop, watch per-turn fail-open, untrusted knob
clamps (min_confidence/max_pages/days), window-cap ordering (newest user
mention survives), serve-IPC suppression passthrough + channel=reflex
logging, windowTurnCount edge semantics, and structural pins for the sink
registration + cycle purge wiring. The exit-verdict pin's grep is now
operator/whitespace-tolerant.
Deferred with TODOs: resolver index shapes for the per-turn query;
batched first-prune after a long dream-cycle gap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): red-team hardening — pre-cap dedupe, delivery-side reflex logging, window clamp
Four red-team findings on the #2095 push-context surface:
- RT1 starvation: watch's session-dedupe Set filtered AFTER volunteerContext's
cap, so a recurring already-pushed entity burned cap slots every turn and
starved fresh pages behind it. VolunteerOpts.excludeSlugs now skips inside
the pointer loop BEFORE the confidence gate and the cap.
- RT3 honest stats: reflex-channel event logging moved from inside the
resolver to the DELIVERY point — serve's resolve-IPC onDelivered hook fires
only after the response write succeeds, and buildReflexAddition logs only
after the per-turn timeout admits the block. A block the client's 250ms
budget abandoned was never injected and no longer counts as volunteered.
(logChannel resolver opt removed; logDeliveredReflexPointers is the seam.)
- RT5 unbounded window: --window-turns is clamped to [1, 64] so a config typo
can't reintroduce the re-scan-everything-per-turn cost class.
- RT2/RT4 documented + filed: PGLite watch connection monopoly (WATCH_HELP,
push-context guide, TODO to route watch via serve IPC); host-resolver
suppression contract at ResolveEntitiesFn (TODO for a capability gate).
Tests: starvation guard (watch + volunteerContext unit), window clamp floor +
ceiling, delivery-side logging (helper writes channel=reflex through the
drained sink; bare resolver writes nothing; empty list no-op), IPC wiring test
rewired to onDelivered. KEY_FILES.md + push-context.md updated; build:llms run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(context): env-plane window knob works config-less; harden two gateway-state-leak victims
Three CI-only check failures, two root causes:
1. windowTurnCount ignored GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS when
loadConfig() returned null (no config file AND no DATABASE_URL — a clean
CI shard with no brain). loadConfig drops its env→config mapping in that
case, so the documented escape hatch silently died and the window fell
back to 4 → windowed extraction widened when the test set window=1 →
prior-turn entity leaked. Fixed: read the env var directly in
windowTurnCount, mirroring reflexEnabled's direct process.env read. This
is a real product bug, not just a test artifact — any config-less host
using the env hatch was affected. Regression test pins it.
2. sync-cost-preview + doctor-federation-health failed only IN-SHARD: a
sibling test configured a non-legacy (ZeroEntropy 1280-d / $0.05) gateway
and never reset it. The legacy-embedding preload only restores the
OpenAI/1536 default when the gateway slot is EMPTY, so a non-empty foreign
config survives into the next file — and a file's beforeAll runs BEFORE
the preload's restoring beforeEach, so federation-health built a
vector(1280) column and its 1536-d fixture hit CheckExpectedDim. My new
test files reshuffled the deterministic file→shard assignment, exposing
this latent ordering bug. Hardened both victims to establish the gateway
state they assert (sync-cost-preview resets to the unconfigured fallback;
federation-health pins legacy 1536 before initSchema) so they're
order-independent. Verified against a simulated leaker run before them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(context): use withEnv() in the window env-hatch test (test-isolation guard)
The regression test added in
|
||
|
|
5a06af5a57 |
v0.42.32.0 fix(sync): coerce non-string frontmatter titles + bounded auto-skip failure ledger (#1939) (#1956)
* fix(import): coerce non-string frontmatter title/slug/type (#1939) YAML `title: 2024-06-01` parses to a Date and `title: 1458` to a number; the old `(frontmatter.X as string)` cast was a compile-time lie, so downstream `.toLowerCase()` threw and (via the importer failure gate) could wedge sync indefinitely. parseMarkdown now coerces via coerceFrontmatterString (Date -> UTC ISO date, deterministic), and the pure assessContentSanity self-protects against a non-string title. * feat(sync): bounded auto-skip failure ledger; poison file can't wedge indexing (#1939) New src/core/sync-failure-ledger.ts owns the failure store + a crash-safe, multi-source, concurrent bounded auto-skip valve. A file that fails N consecutive syncs (GBRAIN_SYNC_AUTOSKIP_AFTER, default 3) auto-skips so it can't freeze all indexing forever, while fresh failures still fail-closed and a `<head>` history-rewrite sentinel hard-blocks even with --skip-failed. - (source_id, path) keying — failures never merge across sources - success clears a path so attempts are truly consecutive - advance-before-ack ordering (a crash can't mark a file skipped while wedged) - shared applySyncFailureGate used by BOTH the incremental and full-sync gates - legacy-row normalization + duplicate collapse on load - cross-process lock + atomic temp-rename, age-based stale-lock break sync.ts re-exports the ledger for existing callers; import.ts records source-scoped and defers the bookmark to the gate under managedBookmark. * fix(doctor): sync_failures severity via one shared decision on both surfaces (#1939) Local buildChecks and remote doctorReportRemote now both route through decideSyncFailureSeverity, so a stuck bookmark escalates WARN -> FAIL consistently (oldest-open age > fail cadence, or large unresolved count), auto-skipped pages stay visible (WARN, not hidden), and the acknowledged/acknowledged_at field-split that caused drift is gone. The remote surface stays subprocess-free (file read + Date.parse only). * chore(test): add trailing newline to e5-lease-cap-ab baseline fixture * fix(sync): address adversarial review findings on the failure ledger (#1939) - #1: a parse-failed file that is later deleted/renamed-away no longer leaves a permanent open ledger row. Removed paths (filtered.deleted, renamed-from, and the "gone from disk" forward-delete skip branch) are treated as resolved so the ledger self-heals instead of aging doctor to a stuck FAIL. - #3: decideSyncFailureSeverity escalates to FAIL on OPEN (blocking) failures only — auto_skipped rows already advanced the bookmark, so they stay WARN-visible regardless of count, matching the state-machine contract. * chore: bump version and changelog (v0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document sync-failure ledger + auto-skip valve for v0.42.30.0 KEY_FILES.md: new src/core/sync-failure-ledger.ts entry (bounded auto-skip state machine, decideGateAction/decideSyncFailureSeverity/applySyncFailureGate, GBRAIN_SYNC_AUTOSKIP_AFTER); update sync.ts (failure store moved to ledger, re-exported), doctor.ts (sync_failures severity via shared rule on both surfaces), markdown.ts (coerceFrontmatterString), import.ts (managedBookmark). live-sync.md: poison-file auto-skip tricky-spot. Regenerated llms-full.txt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.31.0 (queue collision on 0.42.30.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: re-bump to v0.42.32.0 (queue collision) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
805814451e |
v0.42.26.0 docs(supabase): update connection-string setup to new UI + Transaction pooler (#1848) (#1875)
* docs(supabase): update connection-string setup to new UI + Transaction pooler Supabase moved the connection string under "Connect" in the top nav and now shows three options (Direct, Transaction pooler, Session pooler). Update the tutorial, gbrain init prompts, the setup skill, the verify runbook, and the live-sync guide to recommend the Transaction pooler (port 6543) — which gbrain is tuned for (prepared statements disabled, DDL/locks routed to a derived direct connection). Document the IPv4 footgun: the derived direct connection is IPv6-only, so on IPv4-only hosts reads work but sync silently skips pages. Tutorial 7c now leads with the free fix (GBRAIN_DIRECT_DATABASE_URL -> Session pooler, port 5432) and keeps the IPv4 add-on as the paid alternative. Removes stale "transaction mode breaks sync (.begin() is not a function)" warnings and the port-6543 "Session pooler" mislabels. Extends PR #1848 by @FilipHarald. * chore: bump version and changelog (v0.42.26.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f11d56cfca |
v0.42.23.0 feat(jobs): --nice scheduling-priority flag for jobs work/supervisor (#1815) (#1820)
* feat(jobs): niceness core, worker registry, shared supervisor-pid reader OS scheduling-priority primitives for issue #1815: - niceness.ts: parseNiceValue (whole-string), applyNiceness (re-reads effective in success AND failure paths), getEffectiveNiceness, formatNice. - worker-registry.ts: live workers self-register pid + requested/effective nice under gbrainPath('workers'); readWorkers prunes ESRCH (keeps EPERM) with a pid-reuse start-time guard. - supervisor-pid.ts: readSupervisorPid extracted from the copy-pasted PID-file + liveness block. * feat(jobs): --nice flag for jobs work/supervisor + doctor niceness check Wires the --nice <n> flag (and GBRAIN_NICE env) through the CLI (issue #1815): - jobs work: applies niceness + registers the worker; cleanup on finally and process.on('exit'). - jobs supervisor: applies in the foreground-start path only (after the --detach fork), passes the apply result into MinionSupervisor. - supervisor.ts: nice opts, extracted testable buildWorkerArgs (appends --nice), emits niceness on started/worker_spawned audit events. - jobs stats / supervisor status: surface effective worker + supervisor nice. - doctor: separate supervisor_niceness check (warns on requested != effective) so it can't clobber the supervisor crash-check precedence; registered in doctor-categories. * test(jobs): cover niceness, worker registry, supervisor-pid, build args Unit tests for issue #1815: parseNiceValue rejects 3.5/10abc that parseInt would accept; applyNiceness re-reads effective on EPERM; registry ESRCH/EPERM + pid-reuse guard + brain-isolated path; readSupervisorPid states; parseNiceFlag flag>env precedence; buildWorkerArgs --nice propagation. * chore: bump version and changelog (v0.42.23.0) --nice flag for jobs work/supervisor (issue #1815). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document --nice flag for jobs work/supervisor (v0.42.23.0) - minions-deployment.md: niceness tuning section (full concurrency, low priority). - KEY_FILES.md: entries for niceness.ts, worker-registry.ts, supervisor-pid.ts; supervisor.ts entry notes buildWorkerArgs + nice opts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): add enrich_thin to dream cycle EXPECTED_PHASES The enrich_thin cycle phase (src/core/cycle.ts ALL_PHASES, between conversation_facts_backfill and skillopt) shipped without updating the e2e phase-order expectation, so dream-cycle-phase-order-pglite failed on master. Sync the expected list to the real ALL_PHASES order. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): align sync-lock-recovery with the shipped --break-lock --all contract v0.41.13.0 intentionally dropped the "--break-lock + --all is refused" guard so cron can self-heal every source in one call (sync.ts runBreakLock iterates sources under --all). The e2e test still asserted the old exit-1 refusal and failed on master. Assert the current contract: the combination is accepted and takes the iterate / no-active-sources path (exit 0, no refusal message). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): de-flake ingestion-roundtrip chokidar first-drop race The native fsevents watcher occasionally missed a freshly written file, timing out the 15s waitFor (~1/3 on master under load). Three fixes: - inject a polling chokidar watcher via the source's _watchFactory seam (usePolling, 20ms interval) so detection never depends on fsevents timing; - drop deterministic fixtures BEFORE start so the initial scan (ignoreInitial:false) emits them, keeping live-watch coverage only where it's robust; - poll for the dedup hit instead of a fixed 600ms sleep. 15/15 green under stress. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): make hermetic-PGLite serve tests actually hermetic connect-bearer and serve-stdio-roundtrip init a PGLite brain and spawn serve, but passed {...process.env} through — leaking an ambient DATABASE_URL / GBRAIN_DATABASE_URL into the subprocess, which then came up on Postgres and failed the `engine: pglite` assertion. Strip both DB vars from the spawned env so the tests are deterministic whether or not the shell/CI has a DB URL set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): type the hermetic-PGLite env so tsc passes The DATABASE_URL/GBRAIN_DATABASE_URL strip used `delete` on a narrowly-typed env literal (tsc-only failure; bun test doesn't typecheck). Annotate connect-bearer's env as Record<string,string|undefined> and build serve-stdio's as a concrete Record<string,string> (StdioClientTransport.env rejects undefined). Runtime behavior unchanged (7/7 + 3/3 green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: quarantine worker-registry to *.serial (R1 env-mutation isolation) worker-registry.test.ts sets process.env.GBRAIN_HOME per-test so gbrainPath resolves to a temp dir, then lazy-imports the module — a process-global mutation the parallel isolation lint (rule R1) forbids. Rename to worker-registry.serial.test.ts: it runs in the serial pass (own bun process, max-concurrency=1) where env mutation is safe, and the lint skips *.serial files. No logic change (6/6 green); fixes the failing `verify` CI job. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f4959348c2 |
v0.42.22.0 fix(minions): supervisor progress watchdog + worker DB self-defense — alive-but-wedged worker self-heals (#1801) (#1824)
* fix(minions): supervisor progress watchdog + worker DB self-defense under supervision (#1801) Alive-but-wedged worker (dead DB pool, process still up) now self-heals in minutes instead of a silent 15h halt. - supervisor: progress watchdog restarts a child that makes no forward progress on claimable work (name+queue-scoped, active_healthy/due-delayed aware, startup-grace + loop-budget bounded); runtime handler-name derivation. - child-worker-supervisor: killChild gates on liveness not .killed (also fixes the existing shutdown SIGKILL no-op); restartCurrentChild kills the captured child ref; intentional restart doesn't count toward max_crashes. - worker: DB-liveness probe runs under supervision (db_dead self-exit), stall detection stays supervised-off. - doctor: standalone per-queue wedged_queue check + state->status fix in the remote queue_health check. - jobs/queue: queue-scoped getStats wedge fields + jobs stats WEDGED line. * fix(minions): wedge_restart_loop one-shot + supervised-probe comment + jobs-stats threshold (review) Pre-landing adversarial review findings: - wedge_restart_loop warn now fires once per exhausted window via a re-arming flag, not every health tick (was flooding the audit log for the full window). - Correct the stale GBRAIN_SUPERVISED comment: the DB probe runs under supervision now; only stall detection is skipped. - jobs stats WEDGED line reads GBRAIN_WEDGED_QUEUE_WARN_MINUTES so it agrees with the doctor wedged_queue threshold. * chore: bump version and changelog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: queue-ops runbook + KEY_FILES for the #1801 wedge watchdog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a57d98b813 |
v0.42.12.0 feat: self-upgrading gbrain — invocation-riding update check + opt-in auto-upgrade (#1798)
* feat(self-upgrade): decision/cache/snooze foundation + atomic binary self-update Pure decideSelfUpgrade (invocation + autopilot channels), atomic untrusted cache + escalating snooze + shared marker grammar (forged-marker rejection), semver helpers, and real darwin-arm64/linux-x64 binary self-update (download -> fsync -> smoke -> atomic rename; failure leaves old binary intact). Tests incl. real-HTTP-server swap E2E. * feat(self-upgrade): check-update cache/markers, self-upgrade command, CLI heartbeat hook check-update gains gstack-style cache/snooze/markers + refreshUpdateCache + exported fetchLatestRelease. New 'gbrain self-upgrade' command. cli.ts emits the update marker on every invocation (cache-read-only hot path, detached single-flight refresh, skip-set + recursion guard + NODE_ENV=test gate). * feat(self-upgrade): autopilot silent channel, doctor check, runPostUpgrade setup, config + identity marker autopilot opt-in silent channel (auto+quiet+idle, swap-only+breadcrumb+exit-relaunch) + installSystemd Restart=always + migrateSystemdUnitToRestartAlways. doctor self_upgrade_health. runPostUpgrade applySelfUpgradeSetup (one-time consent + systemd rewrite). init defaults mode=notify. config self_upgrade plane + KNOWN_CONFIG_KEYS. get_brain_identity carries update marker. * docs(self-upgrade): gbrain-upgrade agent skill, RESOLVER/manifest, auto-update doc reversal, HEARTBEAT New skills/gbrain-upgrade agent flow (mirror gstack-upgrade) wired into RESOLVER + manifest. upgrades-auto-update.md reversed to document opt-in auto + conservative gates. HEARTBEAT self-upgrade --check-only line. llms-full regenerated. * chore: bump version and changelog (v0.42.12.0) Self-upgrading gbrain: invocation-riding update marker + opt-in autopilot silent channel + real atomic binary self-update. Mirrors gstack's mechanism. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(self-upgrade): write just-upgraded-from breadcrumb + clear stale cache after upgrade Codex ship-review P3: the CLI startup hook reads just-upgraded-from to print the one-time JUST_UPGRADED confirmation, but nothing wrote it — dead path. runUpgrade now writes the breadcrumb (covers full + --swap-only) and clears the update-check cache + snooze so a now-applied 'upgrade available' marker stops nudging. * feat(self-upgrade): surface what's-new in notify + wire agent integration (AGENTS.md, HEARTBEAT) + e2e - self-upgrade --check-only --json now includes changelog_diff + release_url (export fetchChangelog); the gbrain-upgrade skill shows 3-5 what's-new bullets before the 4-option prompt instead of just version numbers. - setup injects a self-upgrade marker protocol into AGENTS.md so interactive agents (Claude Code, Codex) act on the UPGRADE_AVAILABLE stderr marker — the piece that makes notify actually fire for them. - HEARTBEAT daily beat routes through the gbrain-upgrade skill (OpenClaw/Hermes cron cadence); auto-mode daemons ride the autopilot tick. - e2e: real subprocess invocation proves the marker fires (notify emits; off/snooze/up-to-date silent; JUST_UPGRADED fires+clears; --quiet suppresses). Serial test: --check-only surfaces the changelog. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
d4211f4176 |
v0.42.11.0 feat(skillopt): held-out eval gate, honest receipts, ENFORCE + ablation opts (#1759)
* feat(skillopt): wire held-out gate, honest receipts, ENFORCE + ablation opts Wire the F11 held-out gate into the orchestrator at checkpoint acceptance (runHeldOutGate was dead code); parse + thread --held-out through CLI, batch, fleet, background job, and the run_skillopt MCP op. Populate the real receipt.baseline_sel_score (was hardcoded 0) and add a final-test eval (test_score + baseline_test_score) via a shared scoreSkillOnTasks primitive. Fix the --no-mutate proposed.md write (was a stub) and enforce maxRuntimeMin. D16 ENFORCE in core mutation policy (assertBundledMutationHeldOut): mutating a bundled skill in place requires a non-empty (>=5), benchmark-disjoint held-out set or hard-refuses. Add three eval-internal ablation opts (reflectMode, disableValidationGate, optimizerMode='one-shot-rewrite') recorded in the receipt + audit; ROLLOUT_SUCCESS_THRESHOLD named constant. Security: run_skillopt MCP op validates skill_name (kebab-only) and confines caller-supplied benchmark/held-out paths to the skills dir for remote callers. * test(skillopt): held-out gate, ENFORCE, one-shot rewrite, runtime + receipt honesty New test/skillopt/rollout.test.ts (rollout had zero coverage). Held-out ENFORCE unit cases + one-shot-rewrite fence handling (whole-response unwrap, embedded-fence preserved, error path). E2E: F11 held-out BLOCKS/ALLOWS, bundled no-mutate write, reflectMode/disableValidationGate/optimizerMode, maxRuntimeMin abort, receipt baseline/test-score honesty, held-out/benchmark disjointness, D2 no-DB-pollution. * chore: bump version and changelog (v0.42.9.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document skillopt held-out gate + bundled mutation requirement for v0.42.9.0 Wire --held-out into the skill-optimizer SKILL.md, guide flags/safety tables, and the tutorial's bundled-skill step: mutating a bundled skill in place now requires --allow-mutate-bundled AND --held-out (>=5 benchmark-disjoint tasks) or it hard-refuses. Add the --held-out flag row + F11 held-out gate to the guide; update the receipt contract to the honest baseline/test-score fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): AI SDK v6 toolLoop compat — multi-turn tool calls work again The ai@6.x bump tightened ModelMessage + tool-schema validation, which silently broke every multi-turn tool loop. Both `gbrain skillopt` rollouts and production background `subagent` jobs route through `chat()`/`toolLoop` and crashed the moment the model called a tool ("messages do not match the ModelMessage[] schema" / "schema is not a function"). Surfaced end-to-end by the SkillOpt real-LLM eval. Three fixes: - chat(): wrap tool defs with the SDK's `jsonSchema()` helper instead of a bare `{jsonSchema}` object (v6 asSchema() treated the bare object as a thunk and threw). - chat(): new exported pure `toModelMessages()` converts gbrain's provider-neutral ChatMessage[] into v6 ModelMessage[] — tool results ride a dedicated `role:'tool'` message with structured `{type,value}` output; null output preserved as json null. Load-bearing for the production subagent path, not just skillopt. - rollout.ts: replace the inline params→schema mapper (dropped `items` on array params) with the shared `paramDefToSchema` single source of truth. Pinned by test/gateway-model-messages.test.ts (8 cases). Folds into the open v0.42.9.0 PR (#1759) — these complete the eval-readiness wave by making skillopt actually run against a live model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skillopt): budget no-pricing for Haiku silently scored every rollout 0 Surfaced by the SkillOpt real-LLM eval (Track B). Two coupled bugs that made a budget-capped Haiku run report a vacuous "0/N" measurement in ~2ms with zero LLM calls — indistinguishable from a real deficient-skill score: 1. Claude Haiku 4.5's canonical dateless id (`claude-haiku-4-5`) was missing from anthropic-pricing.ts (only the dated `-20251001` was present). With `--max-cost` set, BudgetTracker.reserve() threw no_pricing on the FIRST chat() of every rollout. Added the dateless entry (sonnet already had its dateless form). 2. runValidationGate swallowed that BUDGET_EXHAUSTED error — runWithLimit settled it as {ok:false}, which the gate turned into median:0. A pricing/cap crash became a fake score. The gate now scans settled results for isMustAbortError() and re-throws so the caller aborts loudly; ordinary (non-abort) rollout errors still fail-open to 0 (judge-hiccup posture kept). Pinned by test/skillopt/validate-gate-abort.test.ts (3 cases). Folds into the open v0.42.9.0 PR (#1759). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): llms-full.txt over size budget — drop what-schemas-unlock from full bundle The toolLoop + budget bug-fix annotations grew CLAUDE.md, pushing llms-full.txt to 756KB over the 750KB FULL_SIZE_BUDGET (the `build-llms > size budget` test failed, failing the `test` CI job). CLAUDE.md stays inlined by design (it's the point of the one-fetch bundle), so per the budget comment's own guidance ("ship with includeInFull=false exclusions") this excludes docs/what-schemas-unlock.md (15.4KB value-explainer, not load-bearing operational reference) from llms-full.txt; it stays linked in llms.txt. Bundle now 740KB with ~9KB headroom. No budget bump — 750KB is near the ~190k-token-context fit ceiling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ci): re-admit policy docs into ci-cache-hash before doc relocation docs/**/*.md is deny-listed from the CI cache hash (test-irrelevant). The CLAUDE.md restructure moves test/release POLICY into docs/TESTING.md + docs/RELEASING.md, which DO carry contracts the test suite reads. Without re-admitting them, a policy-only edit would produce the same cache hash and skip the test shard that runs the build-llms + doc-history guards (false-pass). Adds an ALLOW_PATTERNS re-admit step after the deny, scoped to the named policy docs (not a blanket docs un-deny). Lands FIRST, before any doc moves. Pinned by 3 new cases in test/scripts/ci-cache-hash.test.ts: TESTING.md + RELEASING.md edits MUST change the hash; docs/guide.md still must not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): relocate Key files / thin-client / Testing out of CLAUDE.md (verbatim) CLAUDE.md had grown to 592KB / ~147k tokens auto-loaded every session (~77% of the llms-full.txt single-fetch bundle). The per-file index was append-only by mandate. This is the exact thin-dispatcher-vs-fat-blob anti-pattern gbrain exists to fix, so CLAUDE.md becomes a thin orientation + resolver that points at on-demand docs. This commit is the VERBATIM move (content-preserving — the next commit compresses): - docs/architecture/KEY_FILES.md <- ## Key files + the calibration key-files cluster + Schema Cathedral v3 impl detail - docs/architecture/thin-client.md <- ## Thin-client routing - docs/TESTING.md <- ## Testing - ## Commands DROPPED (18 'added in vX.Y' history blocks; current surface is gbrain 0.41.38.0 -- personal knowledge brain USAGE gbrain <command> [options] SETUP init [--pglite|--supabase|--url] Create brain (PGLite default, no server) migrate --to <supabase|pglite> Transfer brain between engines upgrade Self-update check-update [--json] Check for new versions doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings) integrations [subcommand] Manage integration recipes (senses + reflexes) PAGES get <slug> Read a page put <slug> [< file.md] Write/update a page delete <slug> Delete a page list [--type T] [--tag T] [-n N] List pages SEARCH search <query> Keyword search (tsvector) query <question> [--no-expand] Hybrid search (RRF + expansion) ask <question> [--no-expand] Alias for query IMPORT/EXPORT import <dir> [--no-embed] Import markdown directory sync [--repo <path>] [flags] Git-to-brain incremental sync sync --watch [--interval N] Continuous sync (loops until stopped) sync --install-cron Install persistent sync daemon export [--dir ./out/] Export to markdown export --restore-only [--repo <p>] Restore missing supabase-only files [--type T] [--slug-prefix S] With optional filters FILES files list [slug] List stored files files upload <file> --page <slug> Upload file to storage files upload-raw <file> --page <s> Smart upload (size routing + .redirect.yaml) files signed-url <path> Generate signed URL (1-hour) files sync <dir> Bulk upload directory files verify Verify all uploads EMBEDDINGS embed [<slug>|--all|--stale] Generate/refresh embeddings LINKS link <from> <to> [--type T] Create typed link unlink <from> <to> Remove link backlinks <slug> Incoming links graph <slug> [--depth N] Traverse link graph (returns nodes) graph-query <slug> [--type T] Edge-based traversal with type/direction filters [--depth N] [--direction in|out|both] TAGS tags <slug> List tags tag <slug> <tag> Add tag untag <slug> <tag> Remove tag TIMELINE timeline [<slug>] View timeline timeline-add <slug> <date> <text> Add timeline entry TOOLS extract <links|timeline|all> Extract links/timeline (idempotent) [--source fs|db] fs (default) walks .md files; db iterates engine pages [--dir <brain>] brain dir for fs source [--type T] [--since DATE] filters (db source) [--dry-run] [--json] publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256) check-backlinks <check|fix> [dir] Find/fix missing back-links across brain lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter orphans [--json] [--count] Find pages with no inbound wikilinks salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type) transcripts recent [--days N] v0.29: recent raw .txt transcripts (local-only) dream [--dry-run] [--json] Run the overnight maintenance cycle once (cron-friendly). See also: autopilot --install (continuous daemon). check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY) report --type <name> --content ... Save timestamped report to brain/reports/ BRAIN (capture / ideate / explore — v0.37/v0.38) capture [content] [--file PATH] Single entrypoint for getting content into the brain [--stdin] [--slug s] [--type t] Inline content / file / stdin; writes to inbox/ by default [--source ID] [--quiet|--json] Multi-source brains: route to a non-default source brainstorm <question> [--json] Bisociation idea generator (hybrid search + far-set + judge) [--save|--no-save] [--limit N] lsd <question> [--json] Lateral Synaptic Drift: inverted-judge brainstorm [--save|--no-save] [--limit N] rewarding far-from-obvious + axiomatic inversions SOURCES (multi-repo / multi-brain) sources list Show registered sources sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki') sources remove <id> Remove a source + its pages sync --all Sync all sources with a local_path sync --source <id> Sync one specific source repos ... DEPRECATED alias for 'sources' (v0.19.0) CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II) code-def <symbol> [--lang l] Find the definition of a symbol across code pages code-refs <symbol> [--lang l] Find all references to a symbol (JSON-first) code-callers <symbol> Who calls this symbol? (v0.20.0 A1) code-callees <symbol> What does this symbol call? (v0.20.0 A1) query <q> --lang <l> Filter hybrid search to one language (v0.20.0) query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0) reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0) reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0) sync --strategy code Sync code files into the brain JOBS (Minions) jobs submit <name> [--params JSON] Submit background job [--follow] [--dry-run] jobs list [--status S] [--limit N] List jobs jobs get <id> Job details + history jobs cancel <id> Cancel job jobs retry <id> Re-queue failed/dead job jobs prune [--older-than 30d] Clean old jobs jobs stats Job health dashboard jobs work [--queue Q] Start worker daemon (Postgres only) ADMIN stats Brain statistics health Brain health dashboard history <slug> Page version history revert <slug> <version-id> Revert to version features [--json] [--auto-fix] Scan usage + recommend unused features autopilot [--repo] [--interval N] Self-maintaining brain daemon config [show|get|set] <key> [val] Brain config storage status [--repo <path>] Storage tier status and health [--json] (git-tracked vs supabase-only) serve MCP server (stdio) serve --http [--port N] HTTP MCP server with OAuth 2.1 --token-ttl N Access token TTL in seconds (default: 3600) --enable-dcr Enable Dynamic Client Registration --public-url URL Public issuer URL (required behind proxy/tunnel) call <tool> '<json>' Raw tool invocation version Version info --tools-json Tool discovery (JSON) Run gbrain <command> --help for command-specific help. + the per-command KEY_FILES entries; content stays in git) CLAUDE.md gains: a Reference map (resolver), a Maintaining section (the anti-disease rule), and a Cross-cutting invariants subsection under Architecture so the must-never-violate rules (trust fail-closed, sourceScopeOpts isolation, JSONB trap, engine parity, contract-first, migrations, multi-source) still auto-load after the index moved out. Result: CLAUDE.md 592KB -> 61KB; llms-full.txt 740KB -> 210KB (new docs link-only until compressed). build-llms drift + budget test green; verify 29/29 green. The pre-move content is recoverable at git show <this^>:CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): compress relocated docs to current-state + add recurrence guard Compresses the verbatim-relocated reference docs from append-only release-history to current-state-only (the disease cure), then makes recurrence structurally impossible via a CI guard. Compression (fan-out subagents + adversarial verify, audited mechanically): - KEY_FILES.md 453KB -> 356KB; TESTING.md 42KB -> 38KB; thin-client.md already clean. - 393/393 entries preserved; every src/test/scripts path from the verbatim original survives (mechanical comm-check); zero bolded **v0. markers remain. - Conservative ratio (~22%) because the content is invariant-dense — correctness over brevity. Dropped: **vX.Y.Z (#NNN):** clauses, codex/review tags, contributor credits, PR-numbers-as-ids, pre-fix/then/was-now history deltas. Kept: every exported symbol, invariant, and Pinned-by reference. Verbatim original recoverable at git show <relocation-commit>:docs/architecture/KEY_FILES.md. Recurrence guard (scripts/check-key-files-current-state.sh, wired into verify + check:all): - HARD: bans the bolded **v0.<digit> marker in the reference docs (scoped — plain 'as of pgvector 0.7' prose is fine, no false positives). - HARD: CLAUDE.md size cap (90KB; currently 61KB) — the structural backstop. - Pinned by test/scripts/check-key-files-current-state.test.ts (7 cases). Content contracts (test/build-llms.test.ts, +5 cases per codex outside-voice): CLAUDE.md keeps inline ship IRON RULES (version format, document-release, never-hand-roll); AGENTS.md keeps its boot order; llms indexes the new docs; KEY_FILES stays link-only (not inlined). Privacy: scrubbed the relocated 'wintermute/chat/' source-boost examples + the literal harvest-lint regex to generic placeholders (legitimate in allowlisted CLAUDE.md; genericized for the new public docs per the privacy rule). Reverts the |
||
|
|
eefe8b5741 |
v0.42.1.0 feat: gbrain skillopt — self-evolving skills (closes #1481) (#1563)
* feat(skillopt): foundation modules — types, lr-schedule, benchmark, score, audit, lock
* feat(skillopt): edit primitives — apply-edits (D5+D9), rejected-buffer LRU, version-store (D8 history-intent-first)
* feat(skillopt): rollout (D2 gateway.toolLoop + D13 read-only allowlist), reflect (D7 two calls), validate-gate (D12 median+epsilon, D4 parallel), preflight (D3), bundled-skill-gate (D16)
* feat(skillopt): orchestrator (D6 slow-update, D10 ASCII diagrams, D11 caching), checkpoint, bootstrap (D15 sentinel), CLI dispatch + help
* feat(skillopt): cycle phase (F1 dream-loop wiring), PROTECTED_JOB_NAMES + MCP op (F6 admin scope + allowlist) + Minion handler (F7 --background)
* feat(skillopt): full cathedral — --all batch (F4), --target-models fleet (F5), write-capture (F10), held-out scaffold (F11), adversarial suite 41 cases (F2), E2E PGLite (F3), meta-skill bundle (T7), reflect+judge evals (F8+F9), docs (T10)
* chore: bump version to v0.42.0.0 (MINOR — significant new feature)
* fix(skillopt): wire trajectories from forward gate to reflect + fix parseEditsResponse parser misuse
Two related v0.42.0.0 bugs that conspired to make `runSkillOpt` structurally
unable to accept any candidate edit. Either alone would have killed self-evolution;
together they made the loop a no-op for every input.
**Bug 1 (orchestrator gap):** `runOptimizationLoop` in orchestrator.ts called
`runReflect({successes: [], failures: []})` with hardcoded empty arrays. The
forward gate's `scoredRollouts` were computed then voided. `runReflect`
short-circuits both modes when their batches are empty, so the optimizer was
never asked to propose an edit. Every step hit the no_edits_applied branch.
Fix: add `scoredRollouts: ScoredRollout[]` to `GateResult` and
`runsPerTask?: number` to `ValidateGateOpts`. Forward pass uses
`runsPerTask: 1`; orchestrator partitions returned rollouts by `score >= 0.5`
and threads real successes + failures into `runReflect`.
**Bug 2 (parser misuse):** `parseEditsResponse` in reflect.ts routed every
optimizer response through `parseJudgeJson` first. `parseJudgeJson` looks for
a `score` key (it's a judge-output parser, not an edits parser) and returns
null for any JSON without one — including the well-formed `{"edits": [...]}`
the optimizer is contractually required to emit. The function then early-
returned `[]` and the actual `tryExtractEdits` path on the next line was
unreachable dead code.
Fix: drop the wrong-typed guard. `parseEditsResponse` now calls
`tryExtractEdits` directly. Export it so `reflect.test.ts` can pin the
contract independently of the chat transport.
**Why this slipped through 152 prior skillopt tests:** zero unit coverage
of `parseEditsResponse` or `runReflect`. The existing E2E `all-reject` case
asserted no_improvement (which was true for the wrong reason — empty edits,
not gate rejection). Both bugs were structurally invisible to the existing
test surface.
**New coverage:**
- `test/skillopt/reflect.test.ts` (15 cases):
- 8 `parseEditsResponse` cases including the IRON-RULE regression pin
for the v0.42.0.1 fix (`{"edits": [...]}` JSON must survive the parser).
- 7 `runReflect` D7 contract cases: both modes fire, empty-batch skips,
additive token usage, one-mode-throws-other-still-works, rejected-buffer
flows into anti-bias prompt.
- Documents the trailing-comma limitation as an explicit out-of-scope pin
(so a future tightening of `tryExtractEdits` lights this test up
intentionally).
- `test/e2e/skillopt-loop.serial.test.ts` (7 cases):
- HAPPY PATH: stubbed `gateway.chat` acts as both target agent (emits
sections based on skill content) and optimizer (proposes a real
add-Citations edit). Drives `runSkillOpt` end-to-end against PGLite.
Asserts outcome=accepted, SKILL.md mutated with new section,
frontmatter preserved (D5), history has one committed row,
best.md mirrors disk, delta > epsilon, receipt fields populated.
- 5 broken cases (each isolates a distinct orchestrator-visible failure):
1. Below-baseline regression: optimizer proposes a destructive edit;
gate rejects with reason=below_baseline; SKILL.md unchanged;
rejected-buffer captures the bad edit for anti-bias context.
2. Malformed reflect JSON: orchestrator degrades gracefully to
no_improvement without crashing.
3. Anchor-not-found: applyEditBatch rejects all; sel gate skipped;
rejected-buffer captures with reason=apply_failed.
4. Budget exhausted mid-step: outcome=aborted, no pending rows survive.
5. Converged-skill re-run: starting from already-perfect skill →
no_improvement (no thrash on a well-tuned starting point).
- IDEMPOTENT RE-RUN: drive runSkillOpt twice in sequence. Run 1 accepts.
Run 2 sees improved baseline, no failures, returns no_improvement.
SKILL.md byte-identical to post-run-1; history still has exactly 1
committed row. Proves stability at the fixed point.
All hermetic (no DATABASE_URL, no API keys). PGLite in-memory engine,
tempdir SKILL.md + benchmark, stubbed gateway.chat via
`__setChatTransportForTests`. `.serial.test.ts` because the stub installs
module state and the loop walks shared disk state across epochs.
Test counts after fix: 174 skillopt-surface tests pass (149 pre-existing
unit + 15 new reflect unit + 3 existing E2E + 7 new E2E). Typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cycle): align ALL_PHASES skillopt position with actual dispatch order
v0.42.0.0 added skillopt to ALL_PHASES right after `patterns` (line 127), but
the dispatch block in runCycle (line ~1912) actually runs skillopt between
`conversation_facts_backfill` and `embed`. The two were inconsistent, and the
serial test `report.phases.map(p => p.phase)).toEqual(ALL_PHASES)` was failing
on master because of it.
A second pre-existing failure: the two phase-count assertions in
`test/core/cycle.serial.test.ts` still said `toBe(20)` even though
ALL_PHASES grew to 21 when skillopt was added. The author bumped the array
but forgot the test.
Two fixes, one commit:
1. Move `'skillopt'` in ALL_PHASES from after `patterns` to between
`conversation_facts_backfill` and `embed`, matching where runCycle
actually dispatches it. Runtime behavior is unchanged — only the
declaration order moves. Updated the surrounding comment to call out
the position invariant and reference the test that pins it.
2. Update both `toBe(20)` assertions in cycle.serial.test.ts to `toBe(21)`
with a v0.42.0.0 history line in the running comments.
Why declaration follows runtime (not the other way around): the comment
intent ("Runs AFTER patterns — graph-fresh") is still satisfied because
"after the entire main graph-mutating cluster" is strictly fresher than
"right after patterns". No design intent is lost.
Test result: cycle.serial.test.ts is now 28/28 (was 27/28 on master + my
prior commit). Skillopt suite still 174/174.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): bump PHASE_SCOPE assertion to 21 + fix skill-optimizer Anti-Patterns case
Two CI failures pre-existing on this branch since the v0.42.0.0 skillopt
cathedral landed; master is green because skillopt didn't exist there yet.
1. test/phase-scope-coverage.test.ts asserted ALL_PHASES.length === 20.
skillopt is the 21st phase. Bumped to 21 with v0.42.0.0 history line
in the comment chain. Sibling fix to the cycle.serial.test.ts bump
in commit
|
||
|
|
374deff579 |
v0.41.7.0 feat: compact list-format resolver + 300-skill scaling tutorial (#1407)
* feat(check-resolvable): parseResolverEntries accepts compact list format
Add the second parser branch alongside the existing markdown-table branch
so RESOLVER.md and AGENTS.md can use the OpenClaw-native list shape:
- **skill-name**: trigger1 | trigger2 | trigger3
- skill-name: trigger1 | trigger2
Constraints:
- Skill names must be kebab-lowercase ([a-z][a-z0-9-]+). Bold names
starting with an uppercase letter (e.g. **Note**, **Convention**)
are deliberately skipped so prose bullets in real-world AGENTS.md
files don't get mis-parsed as fake skill rows.
- skillPath is always derived as skills/<name>/SKILL.md. An optional
arrow suffix (Unicode -> or ASCII ->) is stripped from the trigger
string but NOT honored as a path. Downstream consumers
(routing-eval.ts skillSlugFromPath, the manifest check at line 367)
assume the convention. For non-conventional paths, use the table
format.
- Multiple triggers fan out to one entry per trigger. checkResolvable
dedupes by skillPath downstream, so the reachability count counts
each skill once regardless of trigger fan-out.
The parser body is restructured to an if/else-if shape so the existing
'continue' on non-table rows no longer short-circuits the list branch.
Unit tests cover 11 new cases: bold + plain name shapes, multi-trigger
fan-out, Unicode and ASCII path-suffix strip, ellipsis filter, empty
pipe segments, mixed-shape files, section tracking, and two D4
regression cases (prose-bullet rejection + convention-violation
silent-skip).
Closes #1370 — credit @garrytan-agents for the original PR that flagged
the parser gap.
* test(check-resolvable): integration fixtures + regression suite for compact format
Two fixtures pin the v0.41.7.0 parser fix at the integration layer:
test/fixtures/openclaw-compact-resolver/
List-format only RESOLVER.md with 10 fictional skills (gift-advisor,
flight-tracker, email-triage, etc.), each with valid frontmatter
triggers. A trailing 'Notes' section embeds 4 prose bullets
(- **Note**:, - **Convention**:, - **TODO**:, - **Important**:)
that pin the D4 kebab-lowercase regex tighten: if the regex ever
regresses to permissive [\w-]+, those prose bullets would surface
as orphan_trigger warnings and the test fails loudly.
test/fixtures/openclaw-mixed-merge/
Tests the v0.31.7 D-CX-14 multi-resolver merge: workspace-root
AGENTS.md (compact list, 3 skills) + skills/RESOLVER.md (table
format, 5 skills). The merge dedups by skillPath and counts each
skill once.
The regression test (test/check-resolvable-openclaw-compact.test.ts)
runs 8 assertions across both fixtures:
1. unreachable === 0 on the compact fixture (the 'pre-v0.41.7.0
reported 238 FAILs on a 306-skill OpenClaw, post-fix 0' headline).
2. zero error-severity issues; report.ok === true.
3. zero mece_gap warnings (every stub ships valid triggers).
4. zero orphan_trigger warnings for the 4 prose-bullet names — D4
regex regression guard at integration level.
5. zero missing_file warnings.
6. mixed-merge: total_skills === 8 (5 table + 3 list), all reachable.
7. mixed-merge: errors.length === 0; report.ok === true.
8. mixed-merge: each expected skill from BOTH shapes is non-unreachable
(catches the bug where one shape silently swallows the other via
dedup-by-skillPath).
* docs(guides): scaling-skills.md walkthrough for 300-skill agents
Three-tier architecture for agents that have outgrown the always-loaded
skill manifest:
Tier A — always loaded (~35 skills, in the system prompt every turn)
Tier B — resolver-routed (~85 skills, looked up via RESOLVER.md/AGENTS.md
only when no Tier A match)
Tier C — dormant (~180 skills, on disk but not injected into the prompt)
Real numbers from Garry's 306-skill OpenClaw: 25K tokens of skill
descriptions per turn collapsed to 4K tokens (~21K tokens freed per
turn) with zero capability loss. The compact list-format resolver
(v0.41.7.0) is the parser-level enabler for this pattern.
The guide covers:
- The scaling wall (when the always-loaded manifest stops working)
- The three tiers + per-turn token math
- What the resolver actually does (routing-table-but-cheaper pattern)
- The compact list format (kebab-lowercase contract, optional path
suffix, mixed-shape support)
- The 'gbrain doctor' / 'gbrain check-resolvable --strict' safety net
- Implementation walkthrough (audit → tier → disable → resolver →
doctor)
- The scaling curve (50 → 100 → 200 → 300 → 1000, no ceiling)
Voice + privacy cleanup applied per CLAUDE.md rules:
- Wintermute → 'Garry's OpenClaw' / 'your OpenClaw'
- Unicode em dashes stripped; ASCII '--' preserved in command flags
- Made-up 'check_resolvable' invocation replaced with real
'gbrain doctor' and 'gbrain check-resolvable --json'/'--strict'
- Blog-style 'Previous in this series' footer dropped
Wiring:
- scripts/llms-config.ts registers the new guide in the curated
array so 'bun run build:llms' picks it up. docs/UPGRADING_
DOWNSTREAM_AGENTS.md excluded from the inlined bundle to stay
under the 600KB FULL_SIZE_BUDGET after adding the new content.
- docs/tutorials/README.md gains a one-line entry pointing at the
guide under Related documentation.
- llms.txt + llms-full.txt regenerated.
* chore: bump version and changelog (v0.41.7.0)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs: update CLAUDE.md for v0.41.7.0 compact-format resolver
Annotate the src/core/check-resolvable.ts entry with the v0.41.7.0
parseResolverEntries compact list-format support: kebab-lowercase name
gate (closes the prose-bullet false-positive class), path-suffix strip
contract (skillPath always derived as skills/<name>/SKILL.md so
routing-eval and the manifest check don't drift), multi-trigger fan-out
plus checkResolvable downstream dedupe, the 238 FAILs to 0 OpenClaw
headline, the two integration fixtures pinning the regression, and the
docs/guides/scaling-skills.md pointer for the tutorial context.
Regenerate llms-full.txt to match (CLAUDE.md edit chaser, per the
CLAUDE.md own rule about test/build-llms.test.ts catching drift).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
|
e227965024 |
v0.36.5.0 feat: secure DATABASE_URL access for shell jobs (inherit: ["database_url"]) (#1192)
* v0.36.5.0 feat: secure DATABASE_URL access for shell jobs (inherit: ["database_url"]) Replaces PR #1137's plaintext-config / plaintext-env workarounds with code. Shell-job params gain `inherit: ["database_url"]`, validated pre-enqueue in both the CLI (`gbrain jobs submit`) and `submit_job` MCP op handler. Worker resolves the value from its own loadConfig() at child-spawn time; the persisted `minion_jobs.data` row stores only the name. Plain `env: { GBRAIN_DATABASE_URL: ... }` / `env: { DATABASE_URL: ... }` / `env: { GBRAIN_DIRECT_DATABASE_URL: ... }` are rejected pre-enqueue with a paste-ready hint pointing at `inherit:`. Codex pre-landing review caught two bypasses + one missing shadow name: - H1: cmd/argv inline-secret regex scan (cmd:"GBRAIN_DATABASE_URL=... gbrain sync" was a clean bypass — fixed) - H3: GBRAIN_DIRECT_DATABASE_URL added to shadowKeys - H2: honest docs about output-side leakage (stdout_tail/stderr_tail can still carry the value if the script prints it; that's the script author's responsibility, not gbrain's) Also: gbrain doctor learns home_dir_in_worktree (warns when ~/.gbrain lives inside a git worktree); ~/.gbrain/.gitignore retroactive via saveConfig + post-upgrade. New canonical guide: docs/guides/agent-to-gbrain.md (two-domain framing for downstream agent authors: MCP ops via OAuth vs localOnly admin ops via shell-job inherit:). Closes #1137. Tests: +53 new (21 validator + 12 inherit-record + 6 ensureGitignore + 5 doctor + 2 PGLite E2E + 7 codex-driven H1/H3 cases). Credit: @wintermute filed PR #1137 which made the env-stripping gap visible enough to fix in code. Thank you. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * v0.36.5.0 redesign: free-form inherit:, drop closed enum User feedback: "agent spawning minions should have agency to do what it wants with secrets and pass only the ones that it needs. don't be a security nazi please." Replaces the closed INHERITABLE enum (database_url only) with three small helpers in shell-inherit.ts: - INHERIT_NAME_RE: snake_case shape guard. Rejects __proto__, leading underscore, uppercase, path-traversal. Prototype-pollution defense. - deriveEnvKey(name): config-key → child-env-key. Uppercase by default with one override: database_url → GBRAIN_DATABASE_URL. - resolveInheritValue(cfg, name): value lookup with Object.hasOwn. inherit: now accepts any snake_case config-key the worker has. Agent picks what it needs per-job (database_url, anthropic_api_key, voyage_api_key, or any custom field). Validator does NOT police WHICH keys — single-uid trust model treats agent as peer of worker. Drops the v0.36.5.0-RC rules that were paternalistic for the actual threat model: - closed-enum check - env-shadow rejection - cmd/argv inline-secret scan Keeps the parts that defend real problems: - pre-enqueue validation (closes the persistence-before-throw window) - snake_case regex (prototype-pollution + audit-log readability) - fail-fast on missing config value (UX guardrail, not security) Tests: shell-validate (existing rules + new free-form + prototype-pollution defense + T1 regression guard) and shell-inherit (regex matrix, deriveEnvKey per-name, resolveInheritValue with hasOwn defense). E2E case now exercises inherit:["anthropic_api_key"] to prove genuinely free-form. Docs and CHANGELOG rewritten to reflect the open design + the design-arc story (closed → cut → free-form). Migration file too. 7653 unit tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * v0.36.5.0 add: redact_secrets opt-in for stdout/stderr scrubbing Honest defense for the documented output-side leakage. When a script prints an inherited secret, the value lands plaintext in result.stdout_tail / result.stderr_tail / error_text. v0.36.5.0 adds: - `redact_secrets: true` ShellJobParams field - `--redact-secrets` CLI convenience flag on `gbrain jobs submit shell` - shell-redact.ts: pure `redactSecretsInText(text, secrets)` helper (string-mode replaceAll; regex metachars in values stay literal) - Handler post-processes both tails before throw/return, so the persisted row carries `<REDACTED:name>` tokens instead of values Only inherit-resolved values are scrubbed. env: values are not (those are the agent's "fine in the row" channel by design). Heuristic — defeats accidental `echo "$GBRAIN_DATABASE_URL"`, not adversarial encode-then-print. Default false for back-compat. Tests: - test/minions-shell-redact.test.ts (9 cases): pure-function behavior, regex-metachar safety, multi-secret independent redaction, substring overlap, empty-input/map edge cases - test/minions-shell-validate.test.ts: +4 cases for redact_secrets shape - test/e2e/minions-shell-pglite.test.ts: +2 cases proving redact_secrets: true scrubs persisted row AND redact_secrets:false preserves plaintext (back-compat regression guard) Docs + CHANGELOG + migration file + CLAUDE.md updated. 7667 unit tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
|
03947665e4 |
v0.36.0.0 feat(skillpack): scaffold + reference + harvest (retire managed-block install) (#1130)
* feat(skillpack): extract copyArtifacts shared helper (T1)
Pure file-copy primitive for scaffold (gbrain→host) and harvest (host→gbrain).
Atomic-refusal contract: symlink-reject + canonical-path containment validate
every item before any write. Used by both directions of the v0.33 loop.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(skillpack): scaffold subcommand + SKILL.md frontmatter sources (T2)
New scaffold.ts replaces the managed-block installer. One-time additive copy
into the user's repo via copyArtifacts; refuses to overwrite existing files
(user owns them). Partial-state policy: copies missing paired sources even
when the skill dir already exists.
bundle.ts extended with loadSkillSources + enumerateScaffoldEntries — paired
source files declared in each SKILL.md's frontmatter sources: array, not in
openclaw.plugin.json. Single source of truth, co-located with the skill.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(skillpack): reference command + apply-clean-hunks (T4 + T15)
reference is the read-only diff lens with an agent-readable framing line. Pure-JS
unified-diff producer + parser + applier (no patch(1) dependency). Two-way merge
with documented limitation: without scaffold-time base tracking, applied hunks
align everything to gbrain. The agent dry-runs reference first, then decides.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(skillpack): migrate-fence + scrub-legacy-fence-rows (T5 + T16)
migrate-fence is the one-shot transition from the pre-v0.36 managed-block model.
Strips begin/end markers and the cumulative-slugs receipt comment; preserves
fence rows verbatim as user-owned routing during the transition to frontmatter
discovery. Receipt-then-row fallback (F-CDX-8) covers stale/missing receipts.
scrub-legacy-fence-rows is the opt-in cleanup after migrate-fence. Two-condition
gate: removes a row only when skills/<slug>/ exists AND that skill's frontmatter
declares non-empty triggers (proof frontmatter discovery covers it).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(skillpack): harvest + privacy linter (T6 + T7)
The inverse loop: lift a proven skill from a host repo (~/git/wintermute, etc.)
back into gbrain so other clients can scaffold it. --from <host-repo-root> is
symmetric with scaffold's --workspace.
Security: symlink rejection + canonical-path containment (mirrors validateUploadPath).
Privacy: default-on linter scans harvested files against ~/.gbrain/harvest-private-patterns.txt
plus built-in defaults (Wintermute, email, Slack channel patterns). Any match
rolls back the copy and exits non-zero. --no-lint bypasses for the editorial
workflow after a manual scrub.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(repo-root): cwd_walk_up tier for non-OpenClaw hosts (T9 + D3)
autoDetectSkillsDir now walks up from cwd looking for any skills/ directory,
ahead of the implicit ~/.openclaw/workspace fallback. cd ~/git/wintermute &&
gbrain skillpack scaffold ... finds wintermute automatically without requiring
a RESOLVER.md/AGENTS.md to exist yet.
R5 regression preserved: $OPENCLAW_WORKSPACE still wins when explicitly set.
+5 test cases in test/repo-root.test.ts pin the new tier order and the R5 guard.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(skillpack): rewrite CLI dispatch, drop install + uninstall (T3 + T10)
skillpack.ts dispatcher rewritten for the v0.36 contract: scaffold, reference
(+ --apply-clean-hunks), migrate-fence, scrub-legacy-fence-rows, harvest, plus
the existing list / diff / check.
install and uninstall are gone — both exit non-zero with a hint pointing at
scaffold / migrate-fence. Clean break, no deprecated alias.
skillpack-check gains --strict for CI gating. When invoked as the subcommand
`gbrain skillpack check`, default is informational (exit 0 even with drift);
--strict opts back into the cron-friendly exit-1-on-issues behavior. Top-level
gbrain skillpack-check preserves its existing exit semantics for backwards compat.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(skills): skillpack-harvest editorial workflow + resolver wiring (T8)
The companion editorial skill for the gbrain skillpack harvest CLI. Walks the
genericization checklist (scrub fork names, generalize triggers, lift fork-
specific conventions to references) before the CLI runs. Routing-eval fixtures
use paraphrased intents to avoid the intent_copies_trigger lint.
Wires the new slug into openclaw.plugin.json#skills, skills/manifest.json, and
skills/RESOLVER.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* test(skillpack): 9-case real-subprocess E2E flow (T11)
Spawns gbrain as a subprocess against tempdir workspaces. Covers: scaffold
first-run + re-run no-op, reference diff + --apply-clean-hunks, migrate-fence,
scrub-legacy-fence-rows, harvest privacy-lint catch + --no-lint bypass, and
the install removed-error path. No DATABASE_URL needed — skillpack is
filesystem-only.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* chore: docs + VERSION + CHANGELOG for v0.36.0.0 (T13 + T14)
Skillpacks as scaffolding, not amber.
v0.36 retires the managed-block install model. Six new subcommands replace
install + uninstall: scaffold, reference (with --apply-clean-hunks), migrate-fence,
scrub-legacy-fence-rows, harvest, plus the existing list / diff / check
(check gains --strict for CI gating). Routing comes from each skill's
frontmatter triggers — gbrain does not touch your RESOLVER.md or AGENTS.md.
Companion editorial skill skillpack-harvest drives the genericization
checklist; default-on privacy linter catches Wintermute / email / Slack
references before they leak into gbrain core.
New docs guide at docs/guides/skillpacks-as-scaffolding.md walks the model
and the migration path for pre-v0.36 installs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(ci): privacy checks — allow-list harvest-lint tests, scrub user-facing fork-name references
CI's check-privacy.sh and check-test-real-names.sh both flagged the literal
fork name across the v0.36 skillpack diff. Two failure modes, two fixes:
1. **Meta-rule-enforcement files** added to both allow-lists. The harvest
privacy linter's whole job is to catch the banned literal leaking into
gbrain; its source has the regex pattern, its tests verify the linter
fires by feeding it the banned string, and the skill markdown documents
the substitution policy. Same exception status as check-privacy.sh and
check-proposal-pii.sh themselves. Files allow-listed:
- src/core/skillpack/harvest-lint.ts
- test/skillpack-harvest-lint.test.ts
- test/skillpack-harvest.test.ts
- test/e2e/skillpack-flow.test.ts
- skills/skillpack-harvest/SKILL.md
2. **User-facing references** swapped for canonical phrasing per CLAUDE.md's
responsible-disclosure rule. README + new docs guide + 4 src docstrings
+ 1 test now say 'your OpenClaw' / 'host agent repo' / 'agentRepo' var
name. Behavior unchanged — only documentation strings touched.
Verify gate (the script CI runs) passes locally: EXIT=0.
Tests still pass: 60/60 across the affected files.
llms-full.txt regenerated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(test): update check-resolvable-cli expectation for cwd_walk_up tier
Sister fix to the test/repo-root.test.ts update in commit
|
||
|
|
9c2dc4cd54 |
v0.26.8 feat(migration): v35 auto-RLS event trigger — new tables always secure (#612)
* feat(migration): v35 auto-RLS event trigger — new tables always secure Postgres event trigger that fires on every CREATE TABLE and auto-enables Row Level Security. Prevents the face_detections bug: tables created outside gbrain migrations (Baku, manual SQL, other apps sharing the same Supabase project) were silently unprotected until gbrain doctor caught it. This is the Supabase-recommended approach — no dashboard toggle exists. Migration v35 (auto_rls_event_trigger): - CREATE FUNCTION auto_enable_rls() — event trigger handler - CREATE EVENT TRIGGER auto_rls_on_create_table — fires on ddl_command_end - PGLite: no-op (no RLS engine, no event triggers) Tests (3 cases): - Event trigger exists after migration - New table automatically gets RLS enabled - auto_enable_rls function exists Closes the gap identified in production on 2026-05-04 when face_detections was found without RLS. * feat(migration): v35 — drop FORCE, public-only, bundle backfill, cover CTAS+SELECT INTO Apply the corrections surfaced by /plan-eng-review + /codex consult against the original PR #612. The trigger now matches v24/v29/schema.sql posture (ENABLE only, no FORCE), scopes to the public schema, and covers all three table-creation syntaxes Postgres reports. Bundles a one-time backfill of every existing public.* table without RLS, honoring doctor.ts's GBRAIN:RLS_EXEMPT regex and quoting identifiers via format('%I.%I'). Drops the EXCEPTION wrap inside the trigger so per-table failures abort the offending CREATE TABLE (loud rollback) rather than producing a silent permissive default. Drops the hand-rolled privilege pre-check — the runner already fails loud on permission errors and gates the version bump. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment before upgrading or the backfill will flip them on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Wintermute <wintermute@garrytan.com> Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
d838d4792b |
feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency (#379)
* feat: queue resilience — wall-clock timeouts, backpressure, --no-worker, env concurrency, shell guard Prevents stall-induced queue blockage discovered in production (OpenClaw): 1. Wall-clock timeout sweep: dead-letters active jobs exceeding 2× timeout_ms (or 2 × lockDuration × max_stalled). Catches jobs stuck while holding DB connections where FOR UPDATE SKIP LOCKED stall detection skips them. 2. Submission backpressure (maxWaiting): caps waiting jobs per name at submission time. Prevents autopilot-cycle flood when the queue is blocked. 3. --no-worker flag for autopilot: skips spawning the built-in worker child. For environments where the worker lifecycle is managed externally (systemd, Docker, OpenClaw service-manager). 4. GBRAIN_WORKER_CONCURRENCY env var: fallback for --concurrency when the worker is spawned by autopilot (which can't pass CLI flags to the child). 5. Shell job env guard with clear logging: shell handler is always registered but throws UnrecoverableError with a clear message when GBRAIN_ALLOW_SHELL_JOBS=1 is not set, instead of silently not registering. * feat: v0.19.1 Lane A — maxWaiting atomic guard, concurrency clamp, --max-waiting CLI Addresses three production-hardening findings from the CEO + Eng + Codex adversarial review of PR #379: D2/H2: maxWaiting was TOCTOU-racy — two concurrent submitters could both see waitingCount < max and both insert. Wrap the count+select+insert in pg_advisory_xact_lock keyed on (name, queue). Serializes concurrent decisions for the SAME key while leaving different keys fully parallel. Lock auto-releases on txn commit/rollback — no cleanup path to leak. Also fix the missing queue-scope bug: count and select now filter on (name, queue) not name alone, so cross-queue same-name jobs don't suppress each other. D3/H3: resolveWorkerConcurrency silently accepted NaN / 0 / negative from parseInt. `inFlight.size < NaN` is always false → worker claims nothing → silent wedge from a single-typo env var. Clamp to ≥1 with a loud stderr warning naming the bad value. D5/H5: `gbrain jobs submit` never parsed `--max-waiting N` despite the MinionJobInput field. Wire the flag with clamp [1, 100], mirror `--max-stalled`. Extract `parseMaxWaitingFlag` for unit testing. Q1: Silent coalesce was invisible by design. New src/core/minions/backpressure-audit.ts mirrors shell-audit.ts's ISO-week JSONL pattern: `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Coalesce events write one JSONL line with (queue, name, waiting_count, max_waiting, returned_job_id, ts). Best-effort — disk-full never blocks submission. A2: `gbrain jobs smoke --wedge-rescue` new opt-in regression case. Forges a wedged-worker row state, invokes handleStalled + handleTimeouts + handleWallClockTimeouts in order, asserts only wall-clock evicts. Mirrors the v0.14.3 `--sigkill-rescue` shape. Tests: 23 new unit cases in test/minions.test.ts covering wall-clock timeout (3 cases + non-interference with handleTimeouts), maxWaiting (coalesce, clamp 0, floor, concurrent-submitter race via Promise.all, cross-queue isolation, unset fallthrough), concurrency clamp (7 cases incl. NaN/0/negative), parseMaxWaitingFlag (5 cases), backpressure audit file write. Part of v0.19.1 plan at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: v0.19.1 Lane B — doctor queue_health, autopilot peer probe, runbook A5 / D4: New `queue_health` check in `gbrain doctor`. Postgres-only (PGLite has no multi-process worker surface). Two subchecks, both cheap (single SELECT each, status-index-covered): - stalled-forever: any active job with started_at > 1h. Surfaces the worst offenders (top 5 by started_at ASC) with `gbrain jobs get/cancel` fix hints. The incident that motivated v0.19.1 ran 90+ min before the operator noticed. - waiting-depth: per-name waiting count exceeds threshold. Default 10, overridable via GBRAIN_QUEUE_WAITING_THRESHOLD env (D9). Signals a submitter probably needs maxWaiting set. Worker-heartbeat subcheck from the original plan dropped (D4/H4): no minion_workers table exists, and lock_until-on-active-jobs is a lossy proxy that can't distinguish idle-worker from dead-worker. Tracked as follow-up B7. A4: --no-worker peer-liveness probe in autopilot. When --no-worker is set, every cycle runs a cheap SELECT checking for any active job whose lock_until was refreshed in the last 2 minutes. After 3 consecutive idle ticks, logs a loud WARNING naming the silent-wedge vector and referencing B7 as the ground-truth follow-up. Re-arms on next live signal so the warning doesn't spam every cycle. A6: New docs/guides/queue-operations-runbook.md (one viewport, ~60 lines). "My queue looks wedged — what do I run?" in order of escalation. What each doctor subcheck means. Self-check for the --no-worker / no-worker-running footgun. CLAUDE.md: key-files updates for handleWallClockTimeouts (v0.19.0 Layer 3 kill shot), maxWaiting advisory-lock rewrite (v0.19.1 D2), queue_health doctor check (v0.19.1 D4), and backpressure-audit.ts. Tests: all 143 minions + 13 doctor unit tests pass. No new test cases required in Lane B; the doctor queue_health exercise is in the E2E verification step (needs real PG to produce meaningful stalled-forever rows). The --no-worker probe is exercised by the smoke case's wedge setup in Lane A. README: unchanged. Existing `gbrain jobs submit` examples don't show --max-stalled, so no --max-waiting precedent to extend per A6 conditional. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: v0.19.1 Lane C — CHANGELOG entry, VERSION bump, remove SPEC.md VERSION: 0.19.0 → 0.19.1 (patch; bug-fix-dominant, no schema change, no new user-facing vocabulary). CHANGELOG: new v0.19.1 entry at the top with the full release-summary template per CLAUDE.md — bold two-line headline, lead paragraph, "numbers that matter" before/after table measured against the real incident, "what this means for OpenClaw users" closer, required "To take advantage of v0.19.1" block naming the worker-restart requirement, itemized changes by area, and "For contributors" section closing the loop on the stale autopilot-idempotency narrative the CEO review was based on. Mechanism reframing per D1/H1: the 18-job pile-up was NOT caused by missing idempotency (autopilot already passes `idempotency_key: autopilot-cycle:${slot}` at autopilot.ts:241). The 18 jobs were 18 DIFFERENT slots stacking up behind the wedged one. `maxWaiting` still caps the pile; the incident just wasn't about idempotency. Adversarial review caught this before ship. SPEC.md: deleted from repo root. It was Wintermute's planning artifact for the original PR, not a shipped spec. Design docs belong under docs/designs/ per repo convention; leaving one at repo root set a precedent this repo doesn't want (A7/D11). CHANGELOG + the plan file at ~/.claude/plans/ok-wintermute-wrote-this-polished-matsumoto.md are the durable artifacts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: --wedge-rescue smoke state — both stall+timeout sweeps must skip Smoke case was setting lock_until in the past, so handleStalled's requeue path fired before handleWallClockTimeouts had a chance to evict. Production scenario is "lock_until still live (worker renewing) + timeout_at disqualified" — only wall-clock matches. Single-connection smoke can't simulate a row lock held by another txn, so we force the equivalent outcome: - lock_until = now() + 30s → handleStalled skips (not a stall) - timeout_at = NULL → handleTimeouts skips (needs NOT NULL) - started_at = now() - 10s, timeout_ms=1000 → wall-clock matches (2 × timeout_ms = 2000ms threshold exceeded) Verified: SMOKE PASS — Minions healthy + wedge rescue in 0.14s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: CI failures — shell-handler tests + llms-full.txt drift Two CI failure clusters, both pre-existing but surfaced by the v0.20.3 merge: 1) test/minions-shell.test.ts — 12 failing cases. The shell handler throws UnrecoverableError when GBRAIN_ALLOW_SHELL_JOBS !== '1' (the production RCE guard at shell.ts:210). The unit tests exercise handler mechanics, not the guard, but never set the env var — so every invocation exits through the guard path instead of the code being tested. Fix: set GBRAIN_ALLOW_SHELL_JOBS=1 in beforeAll, restore in afterAll. The env-guard IS still tested separately via the test/minions.test.ts case added in v0.20.3 Lane A which toggles the var itself. 2) llms-full.txt — stale against CLAUDE.md. Key-files entries for queue.ts, doctor.ts, and the new backpressure-audit.ts updated in v0.20.3 Lane B triggered the build-llms drift guard. Regenerated via `bun run build:llms`; no behavior change, just the inlined-docs bundle catching up to source. Full test run: 2367 pass, 0 fail across 137 files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: root <root@localhost> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
e3f704229b |
v0.20.2 feat: gbrain jobs supervisor — self-healing worker process manager (#364)
* feat: add `gbrain jobs supervisor` — self-healing worker process manager Adds a first-class supervisor command that: - Spawns `gbrain jobs work` as a child process - Restarts on crash with exponential backoff (1s→60s cap) - Resets crash counter after 5min of stable operation - PID file locking prevents duplicate supervisors - Periodic health checks (stalled jobs, completion gaps) - Graceful shutdown (SIGTERM→35s→SIGKILL) Usage: gbrain jobs supervisor --concurrency 4 Replaces ad-hoc nohup patterns in bootstrap scripts. The autopilot command's internal supervisor can be migrated to use this in a follow-up. Tests: 7 pass (backoff calc, PID management, crash tracking) * supervisor: atomic PID lock, queue-scoped health, env safety, unified exit Lane A of PR #364 review fixes (20-item multi-lane plan). Addresses the codex-tier + CEO + Eng findings on src/core/minions/supervisor.ts: Safety + correctness: - Atomic O_CREAT|O_EXCL PID lock via openSync('wx') with stale-file liveness check. Prevents two supervisors racing on the same PID file. (codex #1) - Health check now queries status='active' AND lock_until < now() matching queue.ts:848's authoritative stalled definition. The prior `status = 'stalled'` predicate returned zero rows forever because 'stalled' is not a persisted value in the schema. (codex #2) - All health queries scoped to WHERE queue = $1 via opts.queue binding. Multi-queue installs no longer see cross-queue false positives. (codex #3) - Class default allowShellJobs flipped true→false AND explicit `delete env.GBRAIN_ALLOW_SHELL_JOBS` when false, so child workers don't silently inherit the var from the parent shell. (eng #8, codex #9) - Unified shutdown(reason, exitCode) — max-crashes now routes through the same drain path as SIGTERM. Single source of truth for lifecycle cleanup; prerequisite for trustworthy audit events (Lane C). (eng #1) - Default PID path moves from /tmp to ~/.gbrain/supervisor.pid with mkdirSync recursive + GBRAIN_SUPERVISOR_PID_FILE env override. Matches the rest of the product's ~/.gbrain/ convention; fresh installs no longer hit ENOENT. (CEO #2 + codex #6) Refinements: - crashCount = 1 after 5-min stable-run reset (was 0, produced calculateBackoffMs(-1) = 500ms by accident). Now reads as 'first crash of a new cycle' with a clean 1s backoff. (Nit 1) - Top-of-file POSTGRES-ONLY docstring documenting why the supervisor can't run against PGLite. (Nit 2) - inBackoff flag suppresses 'worker not alive' warn during the expected null-child window (crash → sleep → next spawn). (eng #2) - Tracked listener refs for SIGTERM/SIGINT removed in shutdown() so integration tests spinning up/tearing down multiple supervisors on one process don't leak handlers. (eng #3) - Single FILTER query replaces two SELECT counts — one round-trip instead of two, three metrics in one pass. (eng #10) - child.on('error') listener emits worker_spawn_failed event for ENOENT/EACCES; exit handler still increments crashCount as usual so max-crashes bounds permanent misconfigurations. (codex #7) - healthInFlight boolean guard with try/finally prevents overlapping health checks from stacking on a hung DB. (codex #8) Documented exit codes (ExitCodes const): 0 CLEAN, 1 MAX_CRASHES, 2 LOCK_HELD, 3 PID_UNWRITABLE Agent can branch on exit=2 ('another supervisor, I'm fine') vs exit=1 ('escalate to human'). Event emitter surface: - started / worker_spawned / worker_exited / worker_spawn_failed - backoff / health_warn / health_error / max_crashes_exceeded - shutting_down / stopped Plumbed through emit() with an onEvent callback hook for Lane C's audit writer. json:false is the default; Lane C's --json mode flips it and writes JSONL to stderr. CLI changes (src/commands/jobs.ts): - `gbrain jobs supervisor` gains --allow-shell-jobs (explicit opt-in mirroring the env-var gate), --cli-path (override auto-resolution for exotic setups), and --json (JSONL lifecycle events on stderr). - Expanded --help body with description, 3 examples, and exit-code table. (DX Fix A per review) - Three-tier PID path resolution: --pid-file > GBRAIN_SUPERVISOR_PID_FILE > ~/.gbrain/supervisor.pid (via exported DEFAULT_PID_FILE). - Removed the catch-fallback to process.argv[1] — resolveGbrainCliPath() throws its own actionable install-hint error, which is what dev users need instead of a cryptic spawn failure on a .ts path. (codex #5) Tests: existing 7 supervisor.test.ts cases continue to pass. Integration tests (crash-restart, max-crashes, SIGTERM-during-backoff, env-inheritance regression) land in Lane E. Out of scope for this lane (tracked in follow-up lanes): - Audit file writer at ~/.gbrain/audit/supervisor-YYYY-Www.jsonl (Lane C) - Documentation pass (Lane B) - supervisor start/status/stop subcommands (Lane C) - gbrain doctor supervisor check (Lane D) - /ship release hygiene (Lane F) - autopilot.ts migration to MinionSupervisor (deferred to follow-up PR per codex — requires non-blocking start() API redesign, not ~30 lines) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: supervisor as canonical worker deployment pattern Lane B of PR #364 review fixes. Reframes docs/guides/minions-deployment.md around `gbrain jobs supervisor` as the default answer (blocker 7), deletes the 68-line legacy bash watchdog (F10), and updates README + deployment snippets to match. docs/guides/minions-deployment.md: - New 'Worker supervision' section at the top with the canonical 3-command agent pattern (start --detach / status --json / stop) and a documented exit-code table (0 clean, 1 max-crashes, 2 lock-held, 3 PID-unwritable). - 'Which supervisor when?' decision table: container = supervisor as PID 1, Linux VM = systemd-over-supervisor, dev laptop = bare terminal. - New 'Agent usage' section for OpenClaw / Hermes / Cursor / Codex — the 3-turn discover-start-maintain workflow that replaces shell archaeology with machine-parseable JSON events + an audit file at ~/.gbrain/audit/supervisor-YYYY-Www.jsonl. - Demoted the 'Option 1: watchdog cron' path entirely; replaced with a straightforward upgrade migration block (stop script, remove cron line, start supervisor, verify via doctor). - Preconditions now check Postgres connectivity directly (supervisor is Postgres-only; the CLI rejects PGLite with a clear error). Snippets: - systemd.service: ExecStart now invokes `gbrain jobs supervisor` instead of raw `gbrain jobs work`. Two-layer supervision (systemd → supervisor → worker) buys automatic restart on reboot plus fast crash recovery. ReadWritePaths expanded to cover $HOME/.gbrain (supervisor PID + audit). - Procfile + fly.toml.partial: same change — platform restarts the container on host events, supervisor restarts the worker on crashes. - minion-watchdog.sh: deleted (git history retains it for anyone in an exotic deployment). Supervisor subsumes every capability it had plus atomic PID locking, structured audit events, queue-scoped health checks, and graceful drain on SIGTERM. README.md: - Added a paragraph under the Minions section pointing `gbrain jobs supervisor` as canonical, noting the --detach / status / stop surface and the audit file path, with a link to the full deployment guide. Kept `gbrain jobs work` documented for direct raw invocation but flagged 'prefer supervisor' for any long-running use. The supervisor `--help` body itself (3 examples + exit-code table in src/commands/jobs.ts) landed with Lane A — this lane finishes the discoverability story by making the supervisor findable via doc grep, README landing, and deployment-guide landing paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * supervisor: daemon-manager subcommands + JSONL audit writer Lane C of PR #364 review fixes. Adds the daemon-manager CLI surface so agents can drive `gbrain jobs supervisor` in 3 turns instead of 10, and the audit writer that makes lifecycle events inspectable across process restarts. (Blocker 8, closes DX Fix A/B/C.) New: src/core/minions/handlers/supervisor-audit.ts - writeSupervisorEvent(emission, supervisorPid) appends JSONL to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`. ISO-week rotation via a `computeSupervisorAuditFilename()` helper that mirrors `shell-audit.ts` exactly (year-boundary ISO week math, Thursday anchor, etc). - readSupervisorEvents({sinceMs}) returns parsed events from the current week's file, oldest-first, for Lane D's doctor check. Malformed lines are skipped silently (disk-full truncation is already best-effort at write time). - Reuses `resolveAuditDir()` from shell-audit.ts so the `GBRAIN_AUDIT_DIR` env var override works identically across all gbrain audit trails. src/commands/jobs.ts: supervisor subcommand dispatcher - `gbrain jobs supervisor [start] [--detach] [--json] ...` — default subcommand. Without --detach, runs foreground as before. With --detach, forks a background child (inheriting stderr so the caller can still tail JSONL events), writes a stdout payload: {"event":"started","supervisor_pid":N,"pid_file":"...","detached":true} and exits 0. Stdin/stdout on the detached child are /dev/null so the parent shell isn't held open. - `gbrain jobs supervisor status [--json]` — reads the PID file, checks liveness via `kill -0`, then reads the last 24h from the supervisor audit file to compute crashes_24h / last_start / max_crashes_exceeded. Exits 0 if running, 1 if not. JSON output is machine-parseable; human output is a 5-line ASCII report. - `gbrain jobs supervisor stop [--json]` — reads PID, sends SIGTERM, polls `kill -0` every 250ms for up to 40s (supervisor's own 35s worker-drain + 5s slack). Reports outcome: drained / timeout_40s / pid_file_missing / pid_file_corrupt / process_gone. Exit 0 on clean stop. - `--json` flag is already plumbed through to the supervisor opts from Lane A — this lane adds the onEvent audit-writer callback so every supervisor emission (started, worker_spawned, worker_exited, worker_spawn_failed, backoff, health_warn, health_error, max_crashes_exceeded, shutting_down, stopped) lands in the JSONL file with the supervisor's PID attached. --help body updated: - Three separate usage lines (start / status / stop). - SUBCOMMANDS block with one-line summaries each. - EXIT CODES block (unchanged from Lane A, moved under SUBCOMMANDS). - EXAMPLES block updated with status --json + stop + --detach forms. Tests: existing 127 supervisor + minions tests continue to pass. Integration tests for the new subcommands + audit writer land with Lane E. Follow-up (Lane D): `gbrain doctor` will read readSupervisorEvents() from this module to surface a `supervisor` health check alongside its existing checks (DB connectivity, schema version, queue health). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * doctor: add supervisor health check Lane D of PR #364 review fixes. Closes the observability loop: now that Lane C writes supervisor lifecycle events to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl`, `gbrain doctor` surfaces a `supervisor` check alongside its existing health indicators. Implementation (src/commands/doctor.ts, filesystem-only block 3b-bis): - Resolves DEFAULT_PID_FILE via the same three-tier logic as the start path (--pid-file > GBRAIN_SUPERVISOR_PID_FILE > ~/.gbrain/supervisor.pid). - Reads the PID file + `kill -0 <pid>` for liveness. - Calls readSupervisorEvents({sinceMs: 24h}) from the audit module to derive last_start / crashes_24h / max_crashes_exceeded. - Suppresses the check entirely when the user has never invoked the supervisor (no PID file AND no audit events) — avoids noise on installs that don't use the feature. Status thresholds: fail max_crashes_exceeded event seen in last 24h (supervisor gave up; operator needs to restart or triage) warn supervisor not running but audit shows prior use (unexpected stop — likely crash or manual kill) warn running but > 3 crashes in last 24h (supervisor recovering but worker is unstable) ok running + ≤ 3 crashes + no max_crashes event All failure paths emit a paste-ready recovery command. Read/import errors are swallowed (best-effort like the other doctor checks). Tests: all 127 supervisor + minions tests still green; 13 existing doctor tests unaffected. F3 done. All four lanes A/B/C/D are now committed; Lane E (integration tests) and Lane F (/ship v0.20.2) remain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: 4 critical integration tests for supervisor lifecycle Lane E of PR #364 review fixes (blocker 10). Fills the ~15% coverage gap flagged in the eng review by actually exercising the code paths that will break in production — crash-restart loop, max-crashes exit, SIGTERM-during-backoff, env-var inheritance — via real spawn() calls against fake shell-script workers. No mocks: real fork, real signals, real env propagation, real audit file writes. test/fixtures/supervisor-runner.ts (new, 55 lines): A standalone bun script that constructs a MinionSupervisor from env vars (SUP_PID_FILE / SUP_CLI_PATH / SUP_MAX_CRASHES / SUP_BACKOFF_FLOOR_MS / SUP_HEALTH_INTERVAL_MS / SUP_ALLOW_SHELL_JOBS / SUP_AUDIT_DIR) and calls start(). Mock engine returns empty rows for executeRaw (health check path still exercised without Postgres). Tests spawn this as a subprocess because MinionSupervisor.start() calls process.exit() on shutdown — can't run it in the test runner's own process. test/supervisor.test.ts (existing; 91 → 300 lines): - Added IntegrationHarness helper: creates a unique tmpdir per test, a fake worker shell script, a PID-file path, and an audit-dir path; cleanup runs in finally. - spawnSupervisor() forks bun on the runner with env vars set. - readAudit() reads the supervisor-YYYY-Www.jsonl file via the existing readSupervisorEvents() helper (Lane C), threading GBRAIN_AUDIT_DIR through so tests don't collide on ~/.gbrain. - waitFor(pred, timeoutMs) polls helper for event-driven tests. Four integration tests (with _backoffFloorMs=5 for <1s suite runs): 1. "respawns the worker after a crash and eventually exits with max-crashes code=1" Worker always `exit 1`. maxCrashes=3. Asserts: exit code 1, PID file cleaned up, audit contains started + 3x worker_spawned + 3x worker_exited + max_crashes_exceeded + shutting_down + stopped, and the stopped event carries {reason:'max_crashes', exit_code:1}. Locks in blockers 1 (PID lock), 2+3+6 (health SQL doesn't 500), 5 (unified shutdown emits right events), F8 (spawn errors counted). 2. "receives SIGTERM while sleeping between crashes and exits 0 cleanly" Worker always `exit 1`, backoff floor 800ms to catch the sleep. Asserts: SIGTERM during backoff → exit code 0 (not 1) in <5s, no signal kill (process.exit via shutdown), audit contains shutting_down {reason:'SIGTERM'} + stopped, PID file cleaned up. Locks in eng Issue 1 (unified exit path), eng Issue 3 (signal handlers don't accumulate across shutdowns). 3. "strips inherited GBRAIN_ALLOW_SHELL_JOBS when allowShellJobs=false, even if parent has it set" ⚠ CRITICAL regression test Parent env has GBRAIN_ALLOW_SHELL_JOBS=1. SUP_ALLOW_SHELL_JOBS=0. Worker writes $GBRAIN_ALLOW_SHELL_JOBS (or 'UNSET' if absent) to an OUT_FILE. Asserts child sees 'UNSET'. Locks in codex #9 + eng #8: the `else delete env.GBRAIN_ALLOW_SHELL_JOBS` branch from Lane A is load-bearing for the supervisor's security posture; this test prevents a future refactor silently re-opening the inheritance hole. 4. "DOES pass GBRAIN_ALLOW_SHELL_JOBS to child when allowShellJobs=true" Positive-path companion to #3. SUP_ALLOW_SHELL_JOBS=1 → worker sees '1'. Confirms the else-branch doesn't over-strip and that operators who explicitly opt in still get shell-exec enabled. Plus two audit-format unit tests: - computeSupervisorAuditFilename format (regex match) - Year-boundary ISO week: 2027-01-01 → supervisor-2026-W53.jsonl (matches the shell-audit.ts pattern exactly) Before: 7 tests covering backoff math + PID helpers (~15% behavioral coverage per eng review). After: 13 tests across all critical lifecycle paths (crash-restart, max-crashes, SIGTERM, env-inheritance, audit rotation). All 146 tests in supervisor + minions + doctor suites green in ~8s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.20.2) Lane F of PR #364 review fixes. Closes the multi-lane plan with release hygiene: VERSION bump 0.19.0 → 0.20.2, package.json sync, CHANGELOG entry in GStack voice with release summary + "numbers that matter" table + "To take advantage of v0.20.2" migration block + itemized changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: escape template-literal interpolation in supervisor --help The --help body in src/commands/jobs.ts is one big backtick template literal. The supervisor subcommand description I added in Lane B used both `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}` (parsed as a template interpolation into an undefined variable) and inline `code` backticks (parsed as nested template literals). CI caught it with ~200 tsc parse errors across the file. Fix: - Escape `${...}` → `\${...}` so the audit-file path renders literally. - Replace prose inline-code backticks with plain single-quote fences (`gbrain jobs work` → 'gbrain jobs work', `~/.gbrain/supervisor.pid` → ~/.gbrain/supervisor.pid). `--help` output is human prose; the single-quote form reads cleanly in a terminal without needing to smuggle nested backticks through a template literal. `bunx tsc --noEmit` is clean. 146 tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate llms-full.txt after Lane B doc rewrite CI drift guard caught that `llms-full.txt` didn't match the current generator output. Root cause: the Lane B rewrite of `docs/guides/minions-deployment.md` (supervisor as canonical, watchdog deleted) changed content that gets inlined into `llms-full.txt`, but I didn't run `bun run build:llms` to regenerate. `bun test test/build-llms.test.ts` now clean (7/7 pass). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: root <root@localhost> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
275158137a |
fix: v0.18.1 — RLS hardening + schema backfill (supersedes #336) (#343)
* fix(doctor): check ALL public tables for RLS, not just gbrain's own The RLS check was hardcoded to only verify 10 gbrain-managed tables: pages, content_chunks, links, tags, raw_data, page_versions, timeline_entries, ingest_log, config, files. Any other table in the public schema (created by the application, extensions, or manually) was invisible to the check. This allowed 12 tables to exist without RLS for months — publicly readable by anyone with the Supabase anon key. Changes: - Query ALL tables in public schema, not a hardcoded list - Upgrade severity from 'warn' to 'fail' — missing RLS is a security issue, not a suggestion - Include table count in success message for visibility - Include remediation SQL in failure message Supabase exposes the public schema via PostgREST. Any table without RLS is readable/writable by the anon key by default. * fix(schema): enable RLS on 10 gbrain-managed public tables The base schema and prior migrations shipped 10 public tables without Row Level Security enabled: access_tokens, mcp_request_log, minion_inbox, minion_attachments, subagent_messages, subagent_tool_executions, subagent_rate_leases, gbrain_cycle_locks, budget_ledger, budget_reservations. Supabase exposes the public schema via PostgREST, so tables without RLS are readable and writable by anyone holding the anon key. access_tokens and the subagent conversation history tables carry the most sensitive data in the set. Fix: add the missing ENABLE RLS statements to src/schema.sql (inside the existing BYPASSRLS-gated DO block, so dev sessions without bypass don't get locked out). Add a new schema migration v17 rls_backfill_missing_tables that does the same on existing brains. budget_ledger and budget_reservations were previously migration-only (v12); promoted to the base schema so fresh installs pick up RLS from the standard gate. Regenerated src/core/schema-embedded.ts. * fix(doctor): widen RLS check to all public tables, add GBRAIN:RLS_EXEMPT escape hatch The RLS check was hardcoded to 10 gbrain-managed tables; any other table in the public schema (plugin-created, user-created, extension- created) was invisible to the check. Widen the scan to every pg_tables row in the public schema. Upgrade severity warn to fail. Missing RLS is a security issue, not a suggestion. gbrain doctor now exits 1 when any public table lacks RLS. Cron and CI wrappers that call gbrain doctor should be aware of the exit-code flip. Add an explicit escape hatch for tables that should stay readable by the anon key on purpose (analytics, public materialized views, plugin tables). The doctor reads pg_description for each non-RLS table and treats a comment matching GBRAIN:RLS_EXEMPT reason=<why> as an intentional exemption. Doctor enumerates exempt tables by name on every successful run so they never go invisible. There is no gbrain rls-exempt CLI subcommand by design. The escape hatch is deliberately painful: operators drop to psql and type the justification as raw SQL. Comment lives in pg_description, survives pg_dump, shows up in schema diffs, and appears in shell history. PGLite is now explicitly skipped with an ok status (embedded and single-user, no PostgREST exposure). Previously hit the db.getConnection() throw-catch path and surfaced a misleading warn. Remediation SQL now quotes identifiers (ALTER TABLE "public"."<name>" ...) so it works on tables with hyphens, reserved words, or mixed case. See docs/guides/rls-and-you.md for the full user-facing guide. * test: coverage for RLS hardening (doctor + migration + e2e) Four layers of guard for the v0.18 RLS changes: test/doctor.test.ts: source-grep structural regression guards on the doctor RLS block — absence of the old tablename IN filter, presence of status=fail on the gap branch, quoted-identifier remediation SQL, PGLite skip wrapper, GBRAIN:RLS_EXEMPT parsing with required reason=. Fast, no DB needed. Mirrors the statement_timeout regression pattern in test/postgres-engine.test.ts. test/migrate.test.ts: structural guard for migration v17. Asserts the migration exists with the expected name, all 10 ALTER TABLE statements are present, BYPASSRLS gating is in place, and LATEST_VERSION has caught up. test/e2e/mechanical.test.ts: rewrote the E2E RLS Verification block. The old hardcoded-allowlist query is replaced with an every-public-table-has-RLS assertion. Four new CLI-spawn cases verify real end-to-end behavior: (a) no-RLS public table makes gbrain doctor --json return status=fail with ALTER TABLE in the message and exit code 1, (b) a GBRAIN:RLS_EXEMPT comment with a valid reason makes doctor report the table as explicitly exempt and keep status=ok, (c) a GBRAIN:RLS_EXEMPT prefix without a reason= segment still fails doctor, (d) an unrelated comment on a no-RLS table still fails doctor. All helpers use try/finally with unique-per-run suffixes (gbrain_rls_..._<pid>_<timestamp>) so assertion failures don't pollute subsequent tests. * docs: one-page guide for RLS and GBRAIN:RLS_EXEMPT escape hatch Covers why RLS matters on Supabase (PostgREST exposes the public schema to the anon key), what to do when gbrain doctor fails, the exact SQL template for an intentional exemption, how to audit exemptions later, and how the check behaves on PGLite vs self-hosted Postgres. Emphasizes that the escape hatch is deliberately painful on purpose: there is no gbrain rls-exempt CLI subcommand and no config-file allowlist. The operator drops to psql and writes the justification in SQL, which makes the action visible in shell history, pg_dump, schema diffs, and doctor output on every run. Referenced from gbrain doctor's failure message when any public table lacks RLS. * chore: bump version and changelog (v0.18.0) Reconciles VERSION and package.json (were drifting: 0.17.0 vs 0.16.4). Runtime gbrain --version reads from package.json via src/version.ts, so prior ships were reporting 0.16.4. Both now land on 0.18.0. Minor bump (not patch) because gbrain doctor's exit code semantics change: missing RLS on a public table was warn+exit-0, is now fail+exit-1. Any external cron, CI, or skillpack-check wrapper around gbrain doctor needs to be aware. skillpack-check.ts itself is unaffected (uses --fast, skips DB checks). CHANGELOG entry follows the release-summary format from CLAUDE.md: headline, lead paragraph, numbers-that-matter table, what-this- means-for-your-workflow, To take advantage of v0.18.0 block with remediation SQL + exemption format, itemized changes. Also sweeps a stale @Wintermute reference in the 0.17.0 entry to "Garry's OpenClaw" per the CLAUDE.md privacy rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(v0.18.1): address codex review (orchestrator wiring + fail-closed + identifier escape) Four fixes from `/codex` review of the merged diff: 1. HIGH — wire migration v24 into the `gbrain apply-migrations` upgrade path. Without an orchestrator entry, `gbrain upgrade`'s post-upgrade step runs `apply-migrations --yes`, which walks the registry in `src/commands/migrations/index.ts`. The registry stopped at v0_18_0, so v24 never fired on upgrade (connectEngine and doctor do not call initSchema). New `v0_18_1.ts` orchestrator mirrors v0.18.0's Phase A: shells out to `gbrain init --migrate-only`, which triggers initSchema → runMigrations → v24 applies. Registered in the migrations array. 2. HIGH — fail loudly when v24 runs under a non-BYPASSRLS role instead of RAISE WARNING-then-silently-bumping-version. The runner at migrate.ts:773 unconditionally calls `setConfig('version', String(m.version))` when a migration completes without throwing, so a WARNING-and-continue path would permanently lock the backfill out: schema_version=24 on the next run means `m.version > current` is false and v24 is skipped forever, even after the role gets BYPASSRLS. Changed `RAISE WARNING` → `RAISE EXCEPTION` so the transaction aborts, schema_version stays at 23, and a subsequent initSchema retries cleanly after the role is fixed. Test asserts the SQL uses EXCEPTION and does not use WARNING. 3. MEDIUM — escape double-quote characters in the remediation SQL output. doctor.ts was building `ALTER TABLE "public"."${n}"` with `n` un-escaped, so a pathological table name containing a literal `"` would break out of the quoted identifier and produce invalid copy-paste SQL. Double the `"` before interpolating, matching Postgres quoted-identifier escaping rules. Extremely rare in practice, cheap to get right. 4. LOW — CHANGELOG cleanup: corrected the upgrade-behavior claim (v24 runs via `apply-migrations --yes` through the new orchestrator, not during `gbrain doctor`) and split the "tables with RLS" row into two metrics (21 base-schema tables + 2 migration-only budget_* tables = 23 managed total, all covered). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: add v0.18.1 to apply-migrations skippedFuture expectations CI-only failure: test/apply-migrations.test.ts hardcodes the orchestrator-migration version list in two `skippedFuture` expectations. The v0.18.1 orchestrator I added in the prior commit pushed the list to 8 entries. Both assertions now include 0.18.1 at the tail. Caught by the gbrain CI run on the merged branch — locally the rest of the unit suite (dream/orphans) is flaky due to unrelated PGLite parallelism, but `bun test test/apply-migrations.test.ts` now passes 18/18. CI should follow. * docs: scrub v0.18.1 CHANGELOG — remove specific-table attack surface Responsible-disclosure pass on the public-facing release notes. The prior CHANGELOG entry enumerated which gbrain-managed public tables had shipped without RLS and highlighted the most sensitive ones by name. That gives anyone reading the CHANGELOG a directed probe list for unpatched Supabase installs before operators have had a chance to run `gbrain upgrade`. Rewritten to describe the change at a functional level (what doctor does now, what the upgrade path does, what the escape hatch is) without naming the specific tables or quantifying the gap. The actual SQL remains in the binary — anyone reverse-engineering can find it there — but we shouldn't put it on the release page with a banner. User-facing content kept intact: the "To take advantage of" block, the upgrade commands, the exemption SQL template, the breaking exit-code note. * docs(CLAUDE.md): add responsible-disclosure rule for release notes Prior incident on this branch: the original v0.18.1 CHANGELOG entry enumerated the specific public tables that had shipped without RLS, quantified the exposure duration, and highlighted the most sensitive ones by name. Garry caught it. Scrubbed in |
||
|
|
90c5d93fce |
feat: v0.18.0 — multi-source brains (one DB, many repos, federation + dotfile resolution) (#337)
* feat(v0.17.0 step 1/9): sources primitive — additive-only multi-source foundation
Lane A of the multi-repo plan. Installs the sources table and seeds a
'default' row that inherits sync.repo_path/last_commit from existing
config. This is the bisectable foundation every later step builds on;
the breaking schema changes (composite UNIQUE, files FK rewrite,
resolution_type, ingest_log.source_id) land with their paired code
rewrites in Steps 2/4/5/7 so no single commit breaks the engine.
- migration v16 (sources_table_additive) + v0_17_0 orchestrator skeleton
- sort-by-version guard in runMigrations (array insertion order can
never cause a later migration to skip a lower one again)
- default source seeded with config '{"federated": true}' so pre-v0.17
brains keep single-namespace search semantics after upgrade
- orchestrator phase B detects absence of file_migration_ledger and
no-ops until Step 7 lands it
- 8 new structural tests in test/migrate.test.ts (shape, idempotency,
scope-guard that nothing else was smuggled into v16)
- apply-migrations tests include v0.17.0 in the registered list
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.17.0 step 2/9): pages.source_id + composite UNIQUE (Lane B)
Migration v17 adds pages.source_id with DEFAULT 'default' and swaps the
global UNIQUE(slug) for composite UNIQUE(source_id, slug). Ships atomically
with the engine's ON CONFLICT rewrite so the constraint swap and the code
that writes under it land in the same commit — no window where the engine
sees one shape and the schema has another.
Minimum-surface engine change: only putPage's ON CONFLICT target needs
re-targeting. Other slug-based queries work unchanged because single-
source brains (the only brain shape pre-Step-5) have exactly one source
'default', so slug remains effectively unique within it. Step 5+ will
surface an explicit sourceId param on putPage for cross-source sync.
- migration v17 (pages_source_id_composite_unique) in src/core/migrate.ts
- pages.source_id + composite UNIQUE added to schema.sql + pglite-schema.ts
for fresh installs
- ON CONFLICT (slug) → ON CONFLICT (source_id, slug) in both pglite-engine
and postgres-engine putPage
- DEFAULT 'default' closes the Codex-flagged race where an INSERT between
ADD COLUMN and SET NOT NULL could leave source_id NULL
- 5 new v17 structural tests (29 pass / 0 fail in migrate.test.ts)
- Full suite: 1979 pass / 3 fail (same as baseline — no regressions)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.17.0 step 6/9): sources CLI + source-resolver (Lane C)
Adds the CLI surface for multi-source management. Users can now register,
list, rename, federate/unfederate, and attach-to-directory a source. The
source-resolver is the shared 6-priority helper that Steps 4/5 will use
when they start surfacing an explicit --source flag on sync/extract/query.
Commands:
gbrain sources add <id> --path <p> [--name <n>] [--federated|--no-federated]
gbrain sources list [--json]
gbrain sources remove <id> [--yes] [--dry-run] [--keep-storage]
gbrain sources rename <id> <new-name>
gbrain sources default <id>
gbrain sources attach <id> — writes .gbrain-source in CWD
gbrain sources detach
gbrain sources federate <id> / unfederate <id>
Resolution priority (source-resolver.ts) — highest first:
1. --source flag 2. GBRAIN_SOURCE env 3. .gbrain-source dotfile walk-up
4. longest-prefix match on registered local_path (Codex #2 fix)
5. sources.default config 6. fallback 'default'
- add: validates id format (kebab-case alnum, 1-32), rejects overlapping
paths (eng review §4 finding 4.1), supports federated default opt-in
- remove: guards against --yes omission + refuses to remove 'default',
supports --dry-run, reports cascade page count
- attach/detach: matches kubectl/terraform context-pinning semantics
- Throws on overlap rather than process.exit() so the CLI error wrapper
reports it consistently (also makes unit testing clean)
28 new tests across sources.test.ts (dispatcher + validation + overlap
guard) and source-resolver.test.ts (full 6-priority coverage including
longest-prefix). Full suite: 2012 pass / 3 fail (pre-existing PGLite
infra timeouts).
NOT in scope for Step 6 (deferred):
- import-from-github (SSRF + clone integration)
- prune (retention/TTL, lands v0.18)
- MCP tool-defs regen for source-scoping on read ops (Step 5)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(v0.17.0 step 8/9): getting-started guide + migration skill + citation rule
Step 8 (Lane F) documents what Steps 1+2+6 have shipped and sets up
the agent-facing rules for multi-source.
New files:
- skills/migrations/v0.17.0.md — migration skill read by host agents
after `gbrain apply-migrations`. Covers the v16+v17 chain, what's
in v0.17.0 vs what lands later (v0.17.1 ACL, v0.18 sessions), and
the new sources CLI surface. Cites docs/guides/multi-source-brains.md
as the recipe.
- docs/guides/multi-source-brains.md — getting-started for end users.
Three canonical scenarios (unified wiki+gstack / purpose-separated
yc-media+garrys-list / mixed), full resolution priority, federation
flag semantics, command reference, and citation format.
skills/brain-ops/SKILL.md — new "Cross-source citation format"
section mandating `[source-id:slug]` when the brain has multiple
sources. Matches the contract the /plan-devex-review DX review
pinned down (DX Finding 5: surface source_id in every page payload
+ citation contract). Key must be sources.id (immutable), never
sources.name.
No behavior change — this is pure documentation for what already
exists in the binary. 144 skills conformance tests still pass.
NOT in this commit (deferred to later steps):
- docs/guides/repo-architecture.md rewrite (lands with the full
v0.17.0 PR description + release notes)
- skills/_brain-filing-rules.md "which source to file into"
guidance (lands with Step 5 when sync surfaces --source)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.17.0 step 5/9): sync --source <id> routes through sources table (Lane D)
Adds the --source flag to `gbrain sync`. When set, sync reads local_path
+ last_commit from the matching sources(id) row instead of the global
sync.repo_path / sync.last_commit config keys, and writes last_commit +
last_sync_at back to the same row. Backward compat: --source omitted =
pre-v0.17 behavior exactly, global config path unchanged.
- SyncOpts.sourceId threaded through performSync + performFullSync
- readSyncAnchor/writeSyncAnchor helpers centralize the sources-vs-config
branch so every read/write goes through one decision point. Makes
Step 5's later per-source sync-failures tracking a one-file change.
- --source resolved via src/core/source-resolver.ts (Step 6), so any
command that shell-exposes resolveSourceId gets env var + dotfile
walk-up + longest-prefix for free.
- Error message for missing source local_path is actionable:
Source "gstack" has no local_path. Run: gbrain sources add gstack --path <path>
- last_sync_at auto-updates on every last_commit advance so `gbrain
sources list` shows real recency.
No regression: 2012 pass / 3 fail (same as baseline).
NOT in this commit (deferred per plan):
- Per-source failure tracking (~/.gbrain/sources/<id>/sync-failures.jsonl)
- runImport source-awareness (import.ts path — Step 5 continuation)
- Partial-success semantics when walking N sources — single-source flow
today, multi-walk lands when the top-level `gbrain sync` without
--source starts iterating all sources.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.17.0 step 4/9): qualified [[source:slug]] + links.resolution_type (Lane B)
Adds source-pinned wikilink syntax and records the resolution kind on
each edge so `gbrain extract --refresh-unqualified` (future) can
re-resolve bare references when the source topology changes.
Wikilink syntax extension:
[[concepts/ai]] — unqualified; resolves via local-first fallback
[[wiki:concepts/ai]] — qualified; target pinned to sources.id='wiki'
[[gstack:projects/foo|Display]] — qualified + display name
The qualified regex runs first and masks matched spans so the
unqualified pass can't double-emit. Source id format enforced to match
the sources CLI validation: [a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?
Schema:
- migration v18 adds links.resolution_type TEXT with CHECK constraint
('qualified'|'unqualified' or NULL for legacy/manual/frontmatter edges)
- schema.sql + pglite-schema.ts updated for fresh installs
EntityRef type:
- sourceId is OPTIONAL (only set on qualified wikilinks). Markdown
[Name](path) and unqualified wikilinks omit it so strict toEqual
tests pre-v0.17 keep working (69 existing tests still pass).
Tests:
- 5 new qualified-wikilink extraction tests + 1 migration v18 structural
assertion. 75 tests in test/link-extraction.test.ts (up from 69).
- Full suite: 2018 pass / 3 fail (pre-existing PGLite infra timeouts).
NOT in this commit (deferred to Step 3 / Step 5 continuation):
- Writing resolution_type to the DB (addLink / addLinksBatch don't
carry the field yet — that's the plumb-through that lands with
Step 3 when search/dedup also needs source-aware result keys).
- `gbrain extract --refresh-unqualified` re-resolver.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.17.0 step 3/9): source-aware search dedup composite keys (Lane B)
Search dedup now keys on (source_id, slug) instead of slug alone. Pre-
v0.17 would collapse two same-slug pages in different sources into
one, destroying cross-source recall. Codex outside-voice review flagged
this as regression-critical — this commit ships the fix plus tests
that lock the invariant in.
Dedup pipeline (src/core/search/dedup.ts):
- pageKey(r) helper — one canonical composite-key derivation. Falls
back to source_id='default' for pre-v0.17 rows so single-source
brains behave identically to before.
- Layer 1 (dedupBySource): group-by composite key.
- Layer 4 (capPerPage): count-by composite key.
- guaranteeCompiledTruth: swap scoped to matching (source_id, slug),
so wiki:topics/ai can't accidentally pull gstack:topics/ai's
compiled_truth chunk.
SearchResult type gains optional source_id — populated by SQL JOINs
in both engines, falls through as 'default' for legacy callers.
Engine SQL:
- pglite-engine.ts + postgres-engine.ts: search SELECTs add p.source_id
- rowToSearchResult (utils.ts): maps row.source_id → result.source_id
when present. Shape stays backward compatible (field optional).
Tests — 4 new in test/dedup.test.ts:
- same-slug-different-source does NOT collapse (the critical regression
guard Codex called out)
- same-slug-same-source DOES still collapse (no over-correction)
- missing source_id falls back to 'default' for pre-v0.17 compat
- compiled_truth guarantee scopes to composite key (Codex second pass
caught this specific path would leak otherwise)
Full suite: 2022 pass / 3 fail (3 pre-existing PGLite infra timeouts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.17.0 step 7/9): file_migration_ledger + phase-B storage backfill (Lane E)
Adds files.source_id + files.page_id + the file_migration_ledger
state machine that drives storage object rewrites. Each per-file
transition is its own transaction so crash-point recovery is a
ledger read, not a filesystem inspection. Codex second-pass review
flagged that "skip if already has source prefix" was an unsafe
heuristic — the ledger replaces it with explicit state tracking.
Schema:
- migration v19 (files_source_id_page_id_ledger): handler-only
(PGLite has no files table; Postgres-only gate). ADDs
source_id + page_id to files, backfills page_id from page_slug
scoped to source_id='default', creates file_migration_ledger
with PK on file_id (Codex: not storage_path_old — two sources
can share an old path during migration).
- schema.sql updated for fresh Postgres installs; file_migration_ledger
gets RLS alongside other tables.
Runtime:
- src/commands/migrations/v0_17_0-storage-backfill.ts: drives the
ledger state machine pending → copy_done → db_updated → complete.
Idempotent per row: re-running resumes from whichever state
crashed. Old objects preserved (no delete) so operators can
verify the soak window before a future cleanup release.
- phase B in v0_17_0.ts orchestrator: wires the storage backend
(Supabase/S3/local) through createStorage, runs runStorageBackfill,
reports per-state counts + first-three error details.
Tests — 13 new in test/storage-backfill.test.ts:
- pending → copy_done → db_updated → complete happy path
- 3 crash-point recovery tests (resume from copy_done, resume from
db_updated, failed rows don't auto-retry)
- already-complete rows are skipped with zero side effects
- idempotent re-upload (exists-check skips redundant upload)
- dry-run mode (no storage, reports counts without mutating)
Plus 5 new migrate.test.ts assertions for v19 structure (handler-
only, PGLite gate, source_id + page_id + ledger DDL, default-source
backfill scope, state machine values).
Full suite: 2035 pass / 3 fail (3 pre-existing PGLite infra
timeouts).
NOT in this commit (explicitly deferred):
- DROP old page_slug column — kept for backward compat until
operators have time to verify page_id everywhere.
- DROP old UNIQUE(storage_path) in favor of UNIQUE(source_id,
storage_path) — same reason, deferred to later cleanup.
- Actual cleanup phase that deletes old objects post-soak.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(v0.17.0 step 9/9): full multi-source PGLite integration suite (Lane G)
End-to-end exercise of every v0.17.0 surface against real PGLite
(in-memory, fast — no DATABASE_URL needed). The migration chain
v2→v19 runs start-to-finish and the test asserts each Step's
invariants hold together.
16 new integration tests across 7 describes:
1. Migration-installed state:
- sources('default') exists with federated=true config
- pages.source_id column has DEFAULT 'default'
- composite UNIQUE (source_id, slug) is installed
2. Default-source write path:
- putPage without explicit source → source_id='default' via schema
default clause (no engine API change needed for single-source brains)
3. Composite UNIQUE regression guards (Codex-flagged):
- Same slug in two different sources coexists
- Third insert with same (source_id, slug) hits the UNIQUE constraint
4. sources CLI round-trip:
- federate / unfederate flips config.federated
- rename changes display, id stays immutable
5. Source resolution priority (integration):
- Explicit flag > env var > fallback to default
- Unregistered explicit source errors with actionable message
6. Cascade semantics:
- sources remove cascades to pages; default source untouched
7. links.resolution_type (Step 4):
- Qualified/unqualified values accepted
- CHECK constraint rejects invalid values
All 16 tests pass. Full suite: 2042 pass / 4 fail (4 pre-existing
PGLite beforeEach timeouts in test/wait-for-completion,
test/extract-fs, test/e2e/search-quality, test/e2e/graph-quality
— count fluctuated 3-5 on baseline from variance alone).
Total new tests across Steps 1-9: ~85 unit + integration tests
(sources, source-resolver, migrate v16/v17/v18/v19 structural,
link-extraction qualified wikilinks, dedup regression-critical,
storage-backfill state machine + crash recovery, full
multi-source PGLite integration).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump to v0.18.0 + CHANGELOG entry (multi-source brains)
One-viewport release summary + itemized changes covering all 9 steps
of the multi-source primitive. Notes the v0.17 → v0.18 version bump
rationale (master shipped gbrain dream as v0.17 while this branch was
in flight).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): v0_18_0 orchestrator TS narrow + mechanical test ON CONFLICT
Two CI failures on PR #337:
1. tsc TS2367 at src/commands/migrations/v0_18_0.ts:190 —
after the early-return on `a.status === 'failed'` (line 179),
TypeScript narrows `a.status` to `'skipped' | 'complete'`, so the
subsequent `a.status === 'failed' ? 'failed' :` branch was dead
code and refused to compile. Dropped the redundant check.
2. E2E `file_list LIMIT enforcement` at test/e2e/mechanical.test.ts:636 —
the test pre-seeded a pages row with `ON CONFLICT (slug) DO NOTHING`
but v21 swapped the global UNIQUE for `UNIQUE (source_id, slug)`, so
Postgres rejects with "no unique or exclusion constraint matching".
Updated the conflict target to the composite key.
Tier-1 E2E had only this one failing test; everything else passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(e2e): v0.18.0 multi-source against real Postgres (v20-v23 schema + cascade + sync)
Closes the three biggest confidence gaps the author flagged in the
self-audit of PR #337:
1. No real Postgres E2E — PGLite has no files table, so v23's
files.source_id + files.page_id rewrite + file_migration_ledger
seed was NEVER executed against the real DB. This file covers it.
2. `gbrain sync --source <id>` had zero direct tests. Now has two:
one that asserts performSync({sourceId}) reads local_path from the
sources row (not the global config), one that asserts no-sourceId
falls back to the global sync.repo_path.
3. Cascade delete coverage — previously verified only pages count
after source removal. Now verifies pages + content_chunks +
timeline_entries + links + files ALL cascade-delete when a source
is removed.
6 describes, 16 tests total:
- Schema shape (fresh install): 6 tests confirming sources('default'),
pages.source_id NOT NULL with DEFAULT, composite UNIQUE pages
(source_id, slug) replaces global UNIQUE(slug), links.resolution_type
column + CHECK, files.source_id + page_id columns, file_migration_ledger
table + status CHECK.
- Composite UNIQUE semantics: 3 tests confirming same-slug in two
sources coexists (Codex-critical regression guard), duplicate
(source_id, slug) hits the UNIQUE, putPage targets default source
by schema DEFAULT.
- Cascade delete: 1 test building a fully populated source (2 pages,
chunks, timeline, links, files) then removing it + asserting every
dependent row is gone.
- Sync routing: 2 tests confirming performSync({sourceId}) reads
per-source local_path vs global config.
- Sources surface: 3 tests for federate/unfederate flipping + rename
preserving id.
- Storage backfill: 1 end-to-end test seeding ledger + running
runStorageBackfill against a stub StorageBackend, asserting
pending → complete transition and files.storage_path rewrite.
Gated by DATABASE_URL per CLAUDE.md E2E lifecycle. Each describe's
beforeAll defensively DELETEs non-default sources + file_migration_ledger
rows so reruns are hermetic (sources isn't in helpers.ALL_TABLES).
Verified: 16/16 pass on first run AND second run (residual-state fix
holds). Full E2E suite still green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): TS2352 in multi-source E2E — cast postgres.js RowList via unknown
tsc rejects the direct
`(rows as { column_name: string }[]).map(...)`
cast because postgres.js RowList rows have an iterable-row shape that
doesn't overlap with the plain-object target. Standard fix: cast via
`unknown` first so the narrowing is explicit.
Verified: `bunx tsc --noEmit` clean (ignoring the pre-existing baseUrl
deprecation warning).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v0.18.0): addLinksBatch + addTimelineEntriesBatch source-aware JOINs
Batch APIs JOINed on pages.slug globally, so two pages sharing the same
slug across sources would silently fan out — addLinksBatch(['a->b']) in
a brain with 'a' in both 'default' and 'alt' wrote 2 edges instead of 1.
Same bug on addTimelineEntriesBatch.
Fix:
- LinkBatchInput + TimelineBatchInput gain optional source_id fields
(from_source_id, to_source_id, origin_source_id for links; source_id
for timeline). All default to 'default' so existing callers are
backward-compatible on single-source brains.
- pglite-engine + postgres-engine batch JOINs now composite-key on
(slug, source_id). Postgres adds 3 more unnest arrays for links + 1
for timeline — still one bind per column, no 65535-param cap risk.
- LEFT JOIN for origin pages also source-qualified so frontmatter-
provenance edges don't cross-pollinate across sources.
Regression coverage:
- test/pglite-engine.test.ts: 5 new tests covering default-path isolation,
explicit alt-source writes, and cross-source edges.
- test/e2e/multi-source.test.ts: 4 new tests against real Postgres so
postgres-js's unnest() bind path is exercised (structurally different
from PGLite's).
Gap #4 from the PR self-audit — latent bug, not previously reachable
because every existing caller wrote to the default source only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
418d955fd3 |
docs: v0.16.1 — minions worker deployment guide (from #287) (#317)
* docs: v0.16.1 — minions worker deployment guide (from #287) New docs/guides/minions-deployment.md covering persistent worker deploy patterns (watchdog cron, inline --follow for cron-only workloads) plus the sharp edges of running gbrain jobs work against Supabase in production. Addresses a real gap: existing minions docs (minions-fix.md, minions-shell-jobs.md) cover schema repair and shell-job security, not deploy patterns. With v0.16.0's durable agent runtime, the persistent worker is now load-bearing for subagent + subagent_aggregator handlers too, so a supervised deploy story matters. Pre-landing accuracy pass corrected five factual bugs against current source: - max_stalled column default (5, not 1 or 3) - stalled-jobs smoke-test query (active, not waiting) - watchdog SIGTERM-to-SIGKILL grace (10s minimum, not 2s) - cron env pattern (crontab env lines, not source ~/.bashrc) - --follow exit semantics (blocks until submitted job is terminal, not until queue is empty) Docs-only. No code changed. Zero migration required. Contributed by a downstream agent fork via #287. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: credit Wintermute correctly in v0.16.1 CHANGELOG Wintermute is gbrain's own OpenClaw instance running in production, not a community contributor. The original CHANGELOG framing ("community contributor @wintermute") understated the funnier truth: the agent built on top of the project wrote the deploy guide for the project after hitting its sharp edges in production. Dogfooding with extra steps. Co-Authored-By: Wintermute (OpenClaw) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: rewrite minions deployment guide for agent line-by-line execution Fixes 12 findings from reading v0.16.1 guide as-an-agent would: Real bugs: - Crontab syntax wrong for user crontabs (6-field format dumped into `crontab -e` got "bad minute" or parsed `user` as the command). Now two labeled blocks: 5-field for `crontab -e`, 6-field for `/etc/crontab`. - Watchdog restart loop (old shutdown lines in unrotated log re-matched every 5 min forever). New `minion-watchdog.sh` writes 2-line PID file (PID + restart epoch) and only considers log lines newer than the epoch. Regex rewritten explicit (mawk rejects `{n}` intervals). - Credentials in world-readable /etc/crontab. Secrets move to /etc/gbrain.env (mode 600), referenced via BASH_ENV in crontab. Structural: - Preconditions block (5 fail-fast checks). - "Which option?" decision tree. - Template variable table (6 vars documented). - Upgrade section (v0.13.x -> v0.16.2 checklist). - Option 3: systemd.service + Procfile + fly.toml.partial snippets. - Uninstall section. - `--follow` example uses `gbrain embed --stale` (a real command) instead of the fictional `gbrain enrich`. - Dead-end "Proposed CLI flags (not yet implemented)" replaced with a "Tune per-job today" callout pointing at flags that exist. - Known Issues rewritten as imperatives. Also wires `docs/guides/minions-deployment.md` into `scripts/llms-config.ts` under the Configuration section so remote agents fetching llms.txt / llms-full.txt see the guide by name. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.16.2) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: sync v0.16.2 CHANGELOG with the actual --follow example in the guide The shipped docs/guides/minions-deployment.md uses `gbrain embed --stale` (a real command) but the v0.16.2 CHANGELOG entry still referenced `gbrain enrich --brain $GBRAIN_WORKSPACE` (the older draft). Bring the CHANGELOG in line with what actually shipped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
0e9f8814a5 |
feat: v0.16.0 — durable agent runtime (gbrain agent + subagent handler + plugin loader) (#258)
* refactor(mcp): extract buildToolDefs helper for subagent tool registry reuse
The inline operations.map(...) block in src/mcp/server.ts became the only
source of truth for agent-facing tool definitions. Extract into a reusable
exported helper so the v0.15 subagent tool registry can call it with a
filtered OPERATIONS subset instead of duplicating the shape.
Byte-for-byte equivalence regression pinned in test/mcp-tool-defs.test.ts —
legacy inline mapping kept verbatim inside the test so any future drift
between the new helper and the pre-extraction MCP schema fails loudly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(operations): subagent-aware OperationContext + put_page namespace
Adds three optional fields to OperationContext:
- jobId?: number — the currently running Minion job id
- subagentId?: number — the owning subagent job id for tool-dispatched calls
- viaSubagent?: boolean — FAIL-CLOSED flag for agent-path gating
put_page now enforces a namespace rule when invoked on the subagent tool
dispatch path (viaSubagent=true): writes MUST target
`wiki/agents/<subagentId>/...`. Anchored, slash-boundary enforced so a
collision like `wiki/agents/12evil/...` can't impersonate subagent 12.
The check runs BEFORE the dry-run short-circuit so preview calls surface
the same rejection. Fail-closed: a missing subagentId with viaSubagent=true
rejects every slug rather than letting a dispatcher bug open a hole.
Existing callers unaffected — all three fields are optional and the legacy
put_page behavior is unchanged when viaSubagent is undefined/false.
12 regression + namespace tests pin:
- local CLI writes (viaSubagent unset) accept arbitrary slugs
- MCP writes (remote=true, viaSubagent unset) accept arbitrary slugs
- subagent-path: anchored prefix accepted, wrong id rejected, prefix-
collision defeated, leading-slash rejected, bare-prefix rejected,
fail-closed on missing/NaN subagentId, permission_denied code emitted
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(schema): v0.15.0 subagent runtime tables + migration orchestrator
Adds three new tables for the durable LLM agent runtime:
subagent_messages — Anthropic message-block persistence.
Parallel tool_use blocks in one assistant
message live in content_blocks JSONB, not
across rows (fixes the (job_id, turn_idx, role)
misdesign codex caught in v0.13 drafting).
subagent_tool_executions — Two-phase tool ledger. INSERT pending before
execute, UPDATE complete/failed after. Replay
re-runs pending rows only if the tool is
idempotent (v1 ships only idempotent tools so
this is preventive).
subagent_rate_leases — Lease-based concurrency cap for outbound
providers (e.g. anthropic:messages). Stale
leases auto-prune on next acquire so crashed
workers can't strand capacity.
All DDL uses CREATE TABLE/INDEX IF NOT EXISTS — order-independent vs
PR #244's initSchema() reorder, and idempotent across fresh-install +
upgrade paths. Shipped in both src/schema.sql (Postgres) and
src/core/pglite-schema.ts (PGLite); schema-embedded.ts regenerated.
Migration orchestrator v0_15_0.ts (phases: schema → verify → record).
v0_14_0.ts is a no-op stub so the registry's version sequence stays
gapless (v0.14.0 shipped shell-jobs — code change, no DB migration).
10 unit tests for registry wiring, ordering, dry-run phase behavior, and
schema-embedded table presence. test/apply-migrations.test.ts updated for
the two new registry entries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): emit child_done on every terminal + max_stalled per-job + terminal set fix
Three correctness fixes the v0.15 subagent aggregator spine depends on:
1. child_done emission on ALL terminal transitions, not just success.
- completeJob already emitted on success — now also tags outcome='complete'.
- failJob newly emits on terminal 'failed' or 'dead' (outcome='failed'|'dead',
error=<text>), BEFORE the parent-terminal UPDATE so the EXISTS guard on
the inbox INSERT doesn't skip it on fail_parent paths (codex catch).
- cancelJob now emits outcome='cancelled' per descendant with a parent.
- handleTimeouts now emits outcome='timeout' per timed-out child.
ChildDoneMessage gains optional { outcome, error } — backwards compatible
(legacy writers omitted them; consumers treat absent outcome as 'complete').
2. Parent-resolution terminal set now includes 'failed'.
Pre-v0.15 the `NOT EXISTS (... status NOT IN ('completed','dead','cancelled'))`
guard treated a failed child as still-pending, stranding aggregator parents
that chose on_child_fail='continue' or 'ignore' in waiting-children forever.
Expanded to {completed, failed, dead, cancelled} everywhere parent resolution
reads child status (completeJob inline, failJob remove_dep + continue,
cancelJob sweep, handleTimeouts sweep, and the resolveParent method itself).
3. MinionJobInput.max_stalled threads through MinionQueue.add() on INSERT.
Column exists with default 1 — that is "first stall → dead", which defeats
crash recovery for long-running handlers. Subagent children will set
max_stalled: 3 to survive mid-run worker kills. Second-submitter under an
idempotency-key hit does NOT mutate the existing row (codex-flagged
footgun — first-submit options are load-bearing state).
13 unit tests pin: emission on each of completeJob/failJob/cancelJob/
handleTimeouts, insertion order on fail_parent, terminal-set expansion with
continue policy, max_stalled default + override + idempotency behavior.
E2E tier 1 (Postgres) passes 141 tests unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): rate-leases + waitForCompletion infra for v0.15 subagent
Two infrastructure modules the subagent handler spine depends on:
rate-leases.ts — lease-based concurrency cap for outbound providers
(anthropic:messages, openai:*, etc.). Counter-based limiters leak capacity
on worker crash; leases are owner-tagged rows with expires_at that
auto-prune on the next acquire. Two-phase: txn-scoped pg_advisory_xact_lock
guards the check-then-insert so concurrent acquires can't both win the
"last slot". renewLeaseWithBackoff retries 3x (250/500/1000ms) for mid-
call DB blips — on persistent failure the LLM-loop caller aborts with a
renewable error so the worker re-claims and the rate invariant is
preserved. Owner FK cascades clean up leases on job deletion.
wait-for-completion.ts — poll-until-terminal helper for CLI callers.
Minions' NOTIFY is worker-side only; `gbrain agent run --follow` polls
getJob() until status is {completed, failed, dead, cancelled}. TimeoutError
carries jobId + elapsedMs and does NOT cancel the job — the user can
inspect via `gbrain jobs get <id>` later. Supports AbortSignal for Ctrl-C
without throwing. Default pollMs is 1000 on Postgres, 250 on PGLite (inline
CLI has no network RTT).
21 unit tests cover: single/multi acquire under cap, rejection past cap,
release frees slot, different keys are independent, stale prune, cascade
on owner delete, renew bumps expires_at, renew on missing is false,
backoff path success + pruned short-circuit. waitForCompletion: fast-path
terminal, transitions mid-wait (completed/failed/cancelled), TimeoutError
shape, abort-signal early exit, non-existent job error.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): subagent ToolDef types + brain-tool registry (v0.15)
Types first so the handler has a stable contract:
- SubagentHandlerData / AggregatorHandlerData — the two job.data shapes
- ToolCtx (engine, jobId, remote, signal) + ToolDef (name, description,
input_schema, idempotent, execute) — Anthropic-envelope, distinct from
the MCP McpToolDef extraction landed earlier
- ContentBlock discriminated union for subagent_messages.content_blocks
- SubagentStopReason + SubagentResult emitted on terminal completion
brain-allowlist.ts derives one ToolDef per allow-listed OPERATION. Reuses
the ParamDef → JSONSchema shape from the MCP extraction in a local helper
(Anthropic's input_schema field diverges from MCP's inputSchema by a
character). The 11-name allow-list is read-safe + put_page — every
destructive / filesystem / identity-mutating op stays off by default.
put_page gets a namespace-wrapped tool schema: `slug` pattern = anchored
`^wiki/agents/<subagentId>/.+`. The server-side check in put_page op
(shipped in prior commit) is still the authoritative gate — the schema
just helps the model write correct slugs first-try. `subagentId` is
plumbed into the ToolCtx so the viaSubagent=true fail-closed path lights
up on every tool-dispatched put_page.
filterAllowedTools narrows a registry by subagent_def's allowed_tools
frontmatter field. Rejects unknown names at load time (no silent drop —
typos in a skills/subagents/*.md would otherwise ship to prod with a
tool silently missing).
18 tests pin: every allowlist name exists in OPERATIONS (catches upstream
rename), Anthropic name regex, put_page namespace pattern per-subagent,
execute() routes through the op handler with viaSubagent=true, out-of-
namespace put_page throws permission_denied, filter passes prefixed +
unprefixed names, rejects unknowns, deduplicates.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): subagent-audit JSONL + transcript renderer
Two small plumbing pieces the v0.15 subagent handler + `gbrain agent logs`
depend on:
subagent-audit.ts — JSONL-rotated audit log mirroring the shell-audit
pattern. Two event flavors: submission (one line per job submit) and
heartbeat (one line per turn boundary — llm_call_started / completed /
tool_called / tool_result / tool_failed). Heartbeats fix the "--follow on
a long Anthropic call shows nothing for 30 seconds" problem codex flagged.
Never logs prompts or tool inputs (PII risk — subagent input_vars may
carry user-supplied free text); DOES log tokens, ms_elapsed, tool_name,
first 200 chars of error text. Rotates weekly via ISO week. `readSubagent
AuditForJob` is the readback path for `gbrain agent logs` — scans the
current + prior week file so job boundaries across weeks still resolve.
`GBRAIN_AUDIT_DIR` overrides the default ~/.gbrain/audit/ for container
deploys.
transcript.ts — renders subagent_messages + subagent_tool_executions to
markdown. Message order is authoritative; tool rows splice under their
owning assistant tool_use by tool_use_id. Handles text, tool_use (with
pending / complete / failed execution rows), tool_result (skipped if
we already rendered the owning tool_use — avoids double-printing), and
unknown block types (fenced JSON dump for diagnostics). Output is
UTF-8-safe truncated at maxOutputBytes.
21 unit tests: ISO week filename rotation (incl. 2027-01-01 → W53-2026
boundary), submission + heartbeat write shapes, 200-char error cap, best-
effort write failure doesn't throw, readback filters by job_id and
sinceIso. Transcript: empty input, ordering, token line, tool_use +
complete/failed/pending execution rendering, truncation, unknown-block
diagnostic dump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): subagent LLM-loop handler with crash-resumable replay
The main event: runs one Anthropic Messages API conversation with tool
use, persists every turn + tool execution, and resumes cleanly after a
worker kill anywhere in the loop.
Design points that carry the v0.15 guarantees:
1. Two-phase tool persistence. INSERT status='pending' before dispatch,
UPDATE to 'complete' or 'failed' after. subagent_messages rows are
the canonical conversation; subagent_tool_executions rows are the
canonical "did this tool run + what did it return". Either DB commit
is atomic, so replay has a single source of truth.
2. Replay reconciliation. If the last persisted message is an assistant
with tool_use blocks AND no following synthesized user message, we
crashed mid-dispatch. On resume, finish those tools first (respecting
idempotent flag for 'pending' rows), synthesize the user turn, and
THEN call the LLM again. Non-idempotent pending rows abort the job
with a clear error — v0.15 ships only idempotent tools so this is
preventive.
3. Rate lease around every LLM call. acquireLease before, releaseLease
after (both success and error paths). acquired=false throws
RateLeaseUnavailableError — the worker treats it as a renewable
error and re-claims later, so a temporary capacity cap doesn't fail
the job terminally.
4. Anthropic prompt caching. system block gets cache_control=ephemeral;
the LAST tool def gets it too (Anthropic caches everything up to and
including the marked block). ~10x cost reduction on multi-turn
agents per the plan.
5. Dual-signal abort. AbortSignal.any merges ctx.signal (timeout / lock
loss / cancel) with ctx.shutdownSignal (worker SIGTERM). Both feed
the Anthropic call's AbortSignal; mid-turn abort bails before the
next LLM call with whatever turns are already persisted. Node ≥ 20
has AbortSignal.any; older runtimes get a manual-merge polyfill.
6. Injectable Anthropic client. The real SDK implements MessagesClient
structurally; tests inject a FakeMessagesClient that scripts
responses.
12 unit tests pin: no-tool happy path, single tool_use complete, tool
throws → failed row + loop continues, unknown tool name rejection,
max_turns cap, crash-then-resume with partial state, replay skips already-
complete tool execs without re-invoking execute, non-idempotent pending
rejects on resume, lease acquire + release roundtrip, RateLeaseUnavailable
under cap-full, missing prompt validation, allowed_tools unknown-name.
NOT in v0.15: refusal detection (stop_reason + content shape), stop_reason
=max_tokens partial recovery, mid-call lease renewal with backoff loop.
All three are documented as P2 items in the plan file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): subagent_aggregator handler with mixed-outcome rendering
Claims AFTER all subagent children resolve — by then Lane 1B's queue
changes have posted one child_done message per terminal transition into
this job's inbox (complete / failed / dead / cancelled / timeout). The
aggregator reads those, builds a deterministic markdown summary, and
returns it as the handler result.
Not an LLM call in v0.15 — output is reproducible concatenation so
fan-out runs stay comparable. v0.16+ can add an LLM synthesis pass
behind an opt-in flag.
Contract:
- empty children_ids → `(no children)` marker
- missing child_done (shouldn't happen under v0.15 invariants but
possible if a terminal-state path slipped past Lane 1B) → counted as
failed with "no child_done message observed" error
- non-complete outcomes: result is null in the output so no payload
leaks alongside a failure label
- children appear in the order children_ids was supplied
- custom aggregate_prompt_template replaces the markdown header
13 unit tests cover: empty input, all-success, mixed outcomes, result
suppression on failure, missing child_done handling, order preservation,
custom template, progress + log emission, stringified JSONB payload
parsing, non-child_done inbox filtering, legacy-writer outcome fallback,
and internal helper edges.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): GBRAIN_PLUGIN_PATH loader + plugin-authors guide (v0.15)
Plumbing that makes Wintermute (and future downstream agents) day-1
usable on v0.15. Host repos drop a `gbrain.plugin.json` + `subagents/`
directory somewhere, set GBRAIN_PLUGIN_PATH (colon-separated like \$PATH),
and their custom subagent defs load at worker startup.
Path policy is strict: absolute paths only. Relative, ~-prefixed, and
URL-style (https://, file://) all rejected with warnings — the user
controls where plugins live. Non-existent paths and files (not dirs) are
warned and skipped so a typo doesn't crash worker startup.
Collision policy: left-wins. If two plugins ship a subagent with the same
name, the first one in GBRAIN_PLUGIN_PATH keeps it and the other gets a
warning naming both sources. Deterministic + debuggable.
Trust policy: plugins ship subagent defs ONLY. Cannot declare new tools,
cannot extend the brain allow-list, cannot override safety flags. The
subagent def's `allowed_tools:` frontmatter MUST subset the derived
registry — validation happens at load time (worker startup), not at
dispatch time, so a typo in a skill gives a loud startup error instead
of silently "tool never fires at 3am."
Manifest `plugin_version: "gbrain-plugin-v1"` locks the contract. Unknown
versions rejected. `subagents` field escape attempts (`../../../etc` etc)
rejected. gray-matter handles the markdown frontmatter parse — subagent
defs don't conform to the page schema, so we don't use parseMarkdown.
docs/guides/plugin-authors.md is the Wintermute-facing walkthrough.
Covers the minimum viable plugin shape, the three policies, the
frontmatter fields, known caveats (audit JSONL is local-only, tool calls
always run remote=true, put_page is namespace-scoped).
22 unit tests pin path rejection, missing/invalid manifest, unsupported
version, escape-attempt, basename fallback for missing frontmatter.name,
allowed_tools round-trip, unknown-tool rejection with validAgentToolNames,
empty env, multi-path, collision warning with left-wins, trimmed paths,
manifest-rejection as warning.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): gbrain agent run + logs + worker registration (v0.15 Lane 4H)
Three integration seams wired:
src/commands/agent.ts — \`gbrain agent run\`. Submits subagent jobs (or a
fan-out of N + aggregator) under the trusted-submit flag so the
PROTECTED_JOB_NAMES guard doesn't reject. Fan-out path creates the
aggregator first (so children can reference its id as parent), submits
each child with on_child_fail='continue' (required by Lane 1B's terminal-
set + child_done machinery), then jsonb_set's the aggregator's
children_ids. Short-circuits a 1-entry manifest to a single subagent
with no aggregator. Follow mode runs agent-logs streaming + waitFor
Completion in parallel and exits on terminal status; detach prints the
job id and exits. Ctrl-C is handled as detach, not cancel — the job
keeps running, consistent with durability invariants.
src/commands/agent-logs.ts — \`gbrain agent logs\`. Merges ~/.gbrain/audit/
subagent-jobs-*.jsonl (heartbeats + submissions) with subagent_messages
(persisted conversation) in one chronological stream. --follow polls at
1s and exits when the job hits terminal. --since accepts ISO-8601 OR
relative shorthand (5m / 1h / 2d). Writes transcript tail (full message
+ tool tree) only for terminal jobs, so mid-run --follow doesn't spam a
half-rendered transcript.
src/commands/jobs.ts registerBuiltinHandlers — matches the shell-handler
opt-in shape. GBRAIN_ALLOW_LLM_JOBS=1 registers the subagent +
subagent_aggregator handlers, then loads plugins from GBRAIN_PLUGIN_PATH
with validAgentToolNames pulled from BRAIN_TOOL_ALLOWLIST. Every plugin
warning + loaded-plugin line prints to stderr, mirroring the openclaw-
seam startup convention.
src/core/minions/protected-names.ts — subagent + subagent_aggregator
join the protected set. MCP submit_job returns permission_denied; only
trusted-CLI callers (with allowProtectedSubmit) can insert these rows.
src/cli.ts — adds 'agent' to CLI_ONLY + dispatches it like 'jobs'.
Test fallout: subagent-handler.test.ts + subagent-transcript.test.ts
helpers now submit under allowProtectedSubmit (they insert rows named
'subagent' directly against the queue). 23 new tests in agent-cli.test.ts
cover: flag parsing (including --detach implies !follow, --tools comma
split, -- terminator, unknown flag throw), --since parse (ISO, relative
5m/2h/1d, unparseable error), protected-name guard for all three names,
trusted-submit gate, and a fan-out integration check that verifies the
aggregator + children shape after --fanout-manifest.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(e2e): rename max_children test's spawned jobs off the protected 'subagent' name
The spawn-storm test submitted 50 literal-string 'subagent' children to
exercise the max_children row-lock serialization. In v0.15 'subagent' is
a PROTECTED_JOB_NAME (CLI-only; trusted submit required), so the old
literal submission now throws before reaching the row-lock check.
The test is about max_children semantics, not the v0.15 subagent runtime
specifically — rename the child name to 'child_worker' so the test
exercises the exact same queue.add path without tripping the new guard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ship): v0.15.0 — VERSION, CHANGELOG, README, upgrading-agents, CLAUDE.md
Bumps VERSION → 0.15.0 and package.json → 0.15.0 (resolves the pre-existing
drift — on master, VERSION=0.14.0 but package.json=0.13.1; src/version.ts
reads package.json, so this is what the binary prints now).
CHANGELOG lands the release-summary entry in the GStack voice + the full
itemized change list (11 new modules, 3 new tables, queue correctness
fixes, trust-model additions, 159 new unit tests). Voice rules respected
— no em dashes, no AI vocabulary, real file names + real numbers.
README gets a "Durable agents: `gbrain agent` (v0.15)" section next to
the Minions block, with the three canonical CLI shapes (single run,
fanout-manifest, logs --follow) and a pointer to plugin-authors.md.
docs/UPGRADING_DOWNSTREAM_AGENTS.md gets a full v0.15.0 section covering
the four adoption steps downstream agents (Wintermute and similar) need:
(1) worker opt-in via GBRAIN_ALLOW_LLM_JOBS, (2) moving custom subagent
defs to a plugin repo, (3) replacing ephemeral subagent runs with durable
`gbrain agent run`, (4) the put_page namespace rule for agent-driven writes.
CLAUDE.md updated with concise per-file descriptions for every new module:
the handler, aggregator, audit, rate-leases, wait-for-completion,
transcript, plugin-loader, brain-allowlist, tool-defs extraction, agent
CLI + logs CLI, and the registerBuiltinHandlers wiring for subagent
handlers + plugin-loader.
Verified: binary builds (940 modules, 89ms compile), prints `gbrain 0.15.0`,
`gbrain agent --help` shows the new subcommand shape. 170 new tests pass
(full v0.15 surface). Full unit suite passes bar one parallel-load
flake on a pre-existing E2E (graph-quality, passes in isolation).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(minions): drop GBRAIN_ALLOW_LLM_JOBS flag — subagent handlers always-on
The env flag was ceremony. Shell jobs need the flag because they execute
arbitrary CLI commands (RCE surface). Subagent jobs don't — they call the
Anthropic API with whatever ANTHROPIC_API_KEY is in env, so the key is
already the cost gate (no key → SDK fails on the first turn). And
who-can-submit is already protected by PROTECTED_JOB_NAMES +
TrustedSubmitOpts: MCP callers get permission_denied; only `gbrain agent
run` with allowProtectedSubmit can insert subagent / subagent_aggregator
rows. The flag added nothing the existing guards didn't already give us.
registerBuiltinHandlers now always registers subagent + subagent_aggregator
and loads GBRAIN_PLUGIN_PATH plugins. Worker startup prints:
[minion worker] subagent handlers enabled
instead of the conditional enabled/disabled pair. Plugin discovery runs
unconditionally — empty PATH is a no-op.
README, CHANGELOG, docs/UPGRADING_DOWNSTREAM_AGENTS, CLAUDE.md, agent CLI
help text, and subagent handler docstring all updated to drop the flag
reference. Shell handler's GBRAIN_ALLOW_SHELL_JOBS gate is untouched —
separate concern (RCE, not billing).
Full suite: 1859 pass, 0 fail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: scrub private agent-fork name from all public artifacts
Enforces the rule added to CLAUDE.md (privacy section): never say
`Wintermute` in any CHANGELOG, README, doc, PR, or commit message.
Reader-facing copy says `your OpenClaw` (the term covers every
downstream OpenClaw deployment — Wintermute, Hermes, AlphaClaw — in
one umbrella the reader already recognizes). First-person /
origin-story copy says `Garry's OpenClaw` (honest that this is the
production deployment driving the feature, without exposing the
private agent's name).
Swept across:
CHANGELOG.md (v0.15 entry + 4 historical mentions)
README.md
TODOS.md
docs/UPGRADING_DOWNSTREAM_AGENTS.md
docs/guides/plugin-authors.md (including example plugin names)
docs/guides/plugin-handlers.md
docs/guides/minions-fix.md
docs/designs/KNOWLEDGE_RUNTIME.md (27 refs, mostly analytical)
docs/benchmarks/2026-04-18-minions-vs-openclaw-production.md
skills/migrations/v0.11.0.md
skills/skillpack-check/SKILL.md
scripts/skillify-check.ts
src/commands/doctor.ts
src/commands/migrations/v0_15_0.ts
src/commands/skillpack-check.ts
src/core/enrichment/completeness.ts
src/core/minions/plugin-loader.ts
src/core/operations.ts
src/core/output/scaffold.ts
Intentionally kept (these mentions define/test the rule itself):
CLAUDE.md — the privacy rule section necessarily uses the literal
name to define the restriction and examples
test/plugin-loader.test.ts — fixture name in a plugin-loading test;
renaming risks breaking assertion logic
test/integrations.test.ts — the word appears in a privacy-regex
test that explicitly enforces name redaction
test/doctor-minions-check.test.ts — a comment referencing the rule
CEO plan artifact at ~/.gstack/projects/… — private, not distributed
Binary builds (941 modules), 198/198 relevant tests pass, `gbrain --version`
prints `0.15.0`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: gitignore bun --compile artifacts with a glob, not specific hashes
Each `bun build --compile` emits a fresh hash-named `.*-*.bun-build` file
in cwd. The prior entries listed two specific hashes that were already
stale, so every build after those created a new untracked file requiring
manual cleanup.
Replace the two stale entries with `*.bun-build` so any current or future
compile artifact is ignored automatically.
Verified: ran `bun build --compile`, got two new `.*-*.bun-build` files,
`git status` stays clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ship): rename v0.15.0 → v0.16.0
gbrain master is at 0.14.2. Other 0.15.x PRs may land before/after
this one — we bump the minor (new capability) and lock to 0.16.0 so
ordering with concurrent work doesn't matter.
Touches:
- VERSION: 0.15.0 → 0.16.0
- package.json: 0.15.0 → 0.16.0
- Rename src/commands/migrations/v0_15_0.ts → v0_16_0.ts (+ all
version strings inside + import in index.ts registry)
- Rename test/migrations-v0_15_0.test.ts → migrations-v0_16_0.test.ts
- test/apply-migrations.test.ts: skippedFuture lists now reference
'0.16.0'
- test/put-page-namespace.test.ts + test/mcp-tool-defs.test.ts: Lane
comment refs updated
- src/schema.sql + src/core/pglite-schema.ts: "v0.15.0" section
comment updated; src/core/schema-embedded.ts regenerated
- CHANGELOG.md: top entry renamed to [0.16.0]; inline v0_15_0 /
v0.15.0 refs swept
- docs/UPGRADING_DOWNSTREAM_AGENTS.md: section heading v0.15.0 → v0.16.0
Verified: `gbrain --version` prints 0.16.0, migration registry /
buildPlan / put_page / mcp-tool-defs / handlers tests all green
(49/49).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: reframe v0.16 durability headline around OpenClaw crashes
"Laptop closed mid-run" framing implied a consumer workflow. Real pain is
OpenClaw subagents dying daily on worker kill, memory blip, or timeout.
Headline + README copy match the body now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: regenerate llms-full.txt after README copy change
Regen drift guard caught the README edit from
|
||
|
|
5fd9cd2644 |
feat: shell job type + worker abort-path fix (v0.13.0) (#217)
* feat(minions): add protected-name constant + ctx.shutdownSignal
Introduce PROTECTED_JOB_NAMES ('shell') in a side-effect-free core module
so queue.ts can check it without importing from handlers/. MinionJobContext
gains shutdownSignal (distinct from signal) — handlers that need to run
SIGTERM-triggered cleanup subscribe to both; most handlers ignore shutdown
and run through the worker's 30s cleanup race to natural completion.
* fix(minions): MinionQueue.add gains trusted 4th arg + trim-normalized guard
Adds allowProtectedSubmit opt-in as a separate 4th parameter (NOT folded into
opts) so callers spreading user-provided opts ({...userOpts}) can't accidentally
carry the trust flag. PROTECTED_JOB_NAMES check runs on the trimmed name BEFORE
insert, closing the queue.add(' shell ', ...) whitespace bypass that would have
evaded a has(name) check.
* fix(minions): worker calls failJob on abort + wires ctx.shutdownSignal
Pre-v0.13.0 worker returned silently when ctx.signal.aborted fired, leaving
jobs in 'active' until stall sweep. Handlers using cooperative cancel had
no deterministic status flip — timeout/cancel/lock-loss all looked the same
from downstream callers (gbrain jobs get, --follow loops).
Fix: derive abort reason from abort.signal.reason ('timeout' | 'cancel' |
'lock-lost' | 'shutdown') and call failJob with 'aborted: <reason>' text.
failJob is idempotent via token+status match, so no-op when another path
already flipped status (handleTimeouts, cancelJob, stall).
Also: new shutdownAbort (instance-level AbortController) fires on process
SIGTERM/SIGINT and propagates to every handler's ctx.shutdownSignal.
Shell handler listens to both signals and runs SIGTERM→5s→SIGKILL on its
child on either; other handlers only listen to ctx.signal so deploy
restarts don't cancel them mid-flight.
* feat(minions): add shell job handler + submission audit log
New 'shell' job type spawns arbitrary commands under the Minions worker.
Deterministic cron scripts (API fetch, token refresh, scrape+write) can
move off the LLM gateway — zero Opus tokens per fire.
Handler contract:
- cmd or argv (exactly one required). cmd spawns via /bin/sh -c (absolute
path, not 'sh', to block PATH-override shell substitution). argv spawns
direct with no shell.
- cwd required, must be absolute. Operator-trust boundary.
- env defaults to SHELL_ENV_ALLOWLIST ({PATH, HOME, USER, LANG, TZ,
NODE_ENV}) picked from process.env, with caller overrides merged on top.
Prevents accidental $OPENAI_API_KEY interpolation into scripts.
- stdout/stderr retained as UTF-8-safe tails (64KB/16KB) via
string_decoder.StringDecoder. Prepends [truncated N bytes] marker.
- Abort (either ctx.signal or ctx.shutdownSignal) fires SIGTERM → 5s grace
→ SIGKILL on child. Timer NOT .unref'd so worker's 30s race waits for
the child to actually die.
shell-audit.ts writes a JSONL line per submission to
~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl (ISO-week rotated, override via
GBRAIN_AUDIT_DIR). argv logged as JSON array (not space-joined, which would
flatten args with spaces). Never logs env values. Best-effort writes:
failures log to stderr but don't block submission.
* feat(jobs): submit_job MCP guard + CLI --timeout-ms + starvation warning
submit_job operation gains timeout_ms param (was missing — couldn't plumb
the existing MinionJobInput field through from either CLI or MCP). When
ctx.remote=true and name is in PROTECTED_JOB_NAMES, throws
OperationError('permission_denied'). Combined with the queue.add trusted
guard, MCP callers can never submit shell jobs even if the env flag is on.
CLI submit: new --timeout-ms N flag. Passes {allowProtectedSubmit:true}
as the 4th arg to queue.add only when the submitted name is protected
(not blanket-set for every job). Prints a starvation-warning block to
stderr when a shell job is submitted without --follow, pointing at both
--follow and 'gbrain jobs work' remediation. Fires for every shell submit
regardless of the submitter's env — the submitter env is a weak proxy for
the worker env.
Worker handler registration: conditional on GBRAIN_ALLOW_SHELL_JOBS=1.
Default: off. 'gbrain jobs submit --help' now lists handler types with a
pointer to docs/guides/minions-shell-jobs.md for shell.
* test(minions): 40 unit + 4 E2E cases for shell handler
Unit (test/minions-shell.test.ts):
- Protected names: trim-normalized, case-sensitive, whitespace bypass defense
- MinionQueue.add: trusted opt-in, whitespace bypass, non-protected untouched
- Handler validation: cmd|argv exclusive, cwd required/absolute, env strings
- Spawn: cmd/argv happy paths, non-zero exit, ENOENT, result shape
- Env allowlist: leaked-secret blocked, PATH inherited, caller override
- Abort: ctx.signal, ctx.shutdownSignal, pre-aborted signal
- Audit: ISO-week year boundary (2027-01-01 → W53 2026), mid-year W52/W53,
GBRAIN_AUDIT_DIR override, argv as JSON array, env never logged, EACCES
non-blocking
- Output truncation: 100KB → last 64KB with [truncated N bytes] marker
E2E (test/e2e/minions-shell.test.ts):
- Full lifecycle: submit → worker claim → spawn → complete
- MinionQueue.add without trusted arg throws (including whitespace bypass)
- submit_job with ctx.remote=true rejects shell (MCP guard)
- submit_job with ctx.remote=false allows shell (CLI path)
* chore: bump version and changelog (v0.13.0)
Move gateway crons to Minions. Zero LLM tokens per cron fire.
Worker abort path finally marks aborted jobs dead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: reframe v0.13.0 copy for OpenClaw operators (not Wintermute-specific)
gbrain is an open-source product for any OpenClaw/Hermes operator, not
Garry's personal Wintermute deployment. Rewords the v0.13.0 CHANGELOG
entry, the minions-shell-jobs guide, and the deferred TODOS entries to
speak to "your OpenClaw" / "OpenClaw operators" instead.
Replaces /data/wintermute cwd examples with the canonical
/data/.openclaw/workspace path. Pre-existing Wintermute references in
older CHANGELOG entries (v0.11/v0.10.3) left unchanged.
* feat(migrations): add v0.13.0 adoption playbook for shell jobs
Adding the migration file the CEO review originally scoped out. Without
it, operators upgrade to v0.13.0 and the capability ships but adoption
doesn't happen — the 60% gateway CPU reduction only lands if someone
actually rewrites their crontab.
skills/migrations/v0.13.0.md is the instruction manual the host agent
reads on gbrain upgrade:
- Enable worker: GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs work (Postgres)
or per-tick --follow (PGLite)
- Audit cron manifest: classify LLM-requiring vs deterministic
- Propose per-cron rewrites with diffs, approved one at a time
- Env allowlist guidance for scripts that need API keys
- Verification playbook: run one fire, compare pre/post, only then
approve the next batch
- Starvation sanity-check runbook item
Iron rules: never auto-rewrite the operator's crontab (host-specific
code per CLAUDE.md). LLM-requiring crons stay on the gateway. Ambiguous
cases ask the operator.
No mechanical orchestrator ships with this migration — every rewrite
is operator judgment. A future gbrain crontab-to-minions helper is
tracked in TODOS.md as P1.
* docs: sync UPGRADING + SKILLPACK with v0.13.0 shell jobs
UPGRADING_DOWNSTREAM_AGENTS.md: append v0.13.0 section per the file's
convention (each release appends). No skill edits required, feature is
off-by-default, optional adoption via skills/migrations/v0.13.0.md.
Lists typical LLM-vs-deterministic classifications so operators know
which of their crons are candidates for migration.
GBRAIN_SKILLPACK.md: add shell-jobs guide row to the cron/Minions guide
table so it's discoverable alongside existing Cron via Minions, Plugin
Handlers, and Minions fix guides.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
d8613366a5 |
Minions v7 + v0.11.1 canonical migration + skillify (#130)
* feat: add minion_jobs schema, migration v5, and executeRaw to BrainEngine Foundation for the Minions job queue system. Adds: - minion_jobs table (20 columns) with CHECK constraints, partial indexes, and RLS. Inspired by BullMQ's job model, adapted for Postgres. - Migration v5 creates the table for existing databases. - executeRaw<T>() method on BrainEngine interface for raw SQL access, needed by the Minions module for claim queries (FOR UPDATE SKIP LOCKED), token-fenced writes, and atomic stall detection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions job queue — queue, worker, backoff, types BullMQ-inspired Postgres-native job queue built into GBrain. No Redis. No external dependencies. Postgres transactions replace Lua scripts. - MinionQueue: submit, claim (FOR UPDATE SKIP LOCKED), complete/fail (token-fenced), atomic stall detection (CTE), delayed promotion, parent-child resolution, prune, stats - MinionWorker: handler registry, lock renewal, graceful SIGTERM, exponential backoff with jitter, UnrecoverableError bypass - MinionJobContext: updateProgress(), log(), isActive() for handlers - 8-state machine: waiting/active/completed/failed/delayed/dead/ cancelled/waiting-children Patterns stolen from: BullMQ (lock tokens, stall detection, flows), Sidekiq (dead set, backoff formula), Inngest (checkpoint/resume). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: 43 tests for Minions job queue Full coverage of the Minions module against PGLite in-memory: - Queue CRUD (9): submit, get, list, remove, cancel, retry, duplicate - State machine (6): waiting→active→completed/failed, retry→delayed→waiting - Backoff (4): exponential, fixed, jitter range, attempts_made=0 edge - Stall detection (3): detect stalled, counter increment, max→dead - Dependencies (5): parent waits, fail_parent, continue, remove_dep, orphan - Worker lifecycle (5): register, start-without-handlers, claim+execute, non-Error throws, UnrecoverableError bypass - Lock management (3): renewal, token mismatch, claim sets lock fields - Claim mechanics (4): empty queue, priority ordering, name filtering, delayed promotion timing - Cancel & retry (2): cancel active, retry dead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions CLI commands and MCP operations Wire Minions into the GBrain CLI and MCP layer: CLI (gbrain jobs): submit <name> [--params JSON] [--follow] [--dry-run] list [--status S] [--queue Q] [--limit N] get <id> — detailed view with attempt history cancel/retry/delete <id> prune [--older-than 30d] stats — job health dashboard work [--queue Q] [--concurrency N] — Postgres-only worker daemon 6 MCP operations (contract-first, auto-exposed via MCP server): submit_job, get_job, list_jobs, cancel_job, retry_job, get_job_progress Built-in handlers: sync, embed, lint, import. --follow runs inline. Worker daemon blocked on PGLite (exclusive file lock). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update project documentation for Minions job queue CLAUDE.md: added Minions files to key files, updated operation count (36), BrainEngine method count (38), test file count (45), added jobs CLI commands. CHANGELOG.md: added Minions entry to v0.10.0 (background jobs, retry, stall detection, worker daemon). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: Minions v2 — agent orchestration primitives (pause/resume, inbox, tokens, replay) Adds the foundation for Minions as universal agent orchestration infrastructure. GBrain's Postgres-native job queue now supports durable, observable, steerable background agents. The OpenClaw plugin (separate repo) will consume these via library import, not MCP, for zero-latency local integration. ## New capabilities - **Concurrent worker** — Promise pool replaces sequential loop. Per-job AbortController for cooperative cancellation. Graceful shutdown waits for all in-flight jobs via Promise.allSettled. - **Pause/resume** — pauseJob clears the lock and fires AbortSignal on active jobs. Handlers check ctx.signal.aborted and exit cleanly. resumeJob returns paused jobs to waiting. Catch block skips failJob when signal.aborted. - **Inbox (separate table)** — minion_inbox table for sidechannel messages. sendMessage with sender validation (parent job or admin). readInbox is token-fenced and marks read_at atomically. Separate table avoids row bloat from rewriting JSONB on every send. - **Token accounting** — tokens_input/tokens_output/tokens_cache_read columns. updateTokens accumulates; completeJob rolls child tokens up to parent. USD cost computed at read time (no cost_usd column — pricing too volatile). - **Job replay** — replayJob clones a terminal job with optional data overrides. New job, fresh attempts, no parent link. ## Handler contract additions MinionJobContext now provides: - `signal: AbortSignal` — cooperative cancellation - `updateTokens(tokens)` — accumulate token usage - `readInbox()` — check for sidechannel messages - `log()` — now accepts string or TranscriptEntry ## MCP operations added pause_job, resume_job, replay_job, send_job_message — all auto-generate CLI commands and MCP server endpoints. ## Library exports package.json exports map adds ./minions and ./engine-factory paths so plugins can `import { MinionQueue } from 'gbrain/minions'` for direct library use. ## Instruction layer (the teaching) - skills/minion-orchestrator/SKILL.md — when/how to use Minions, decision matrix, lifecycle management, anti-patterns - skills/conventions/subagent-routing.md — cross-cutting rule: all background work goes through Minions - RESOLVER.md — trigger entries for agent orchestration - manifest.json — registered ## Schema migration v6 Additive: 3 token columns, paused status, minion_inbox table with unread index. Full Postgres + PGLite support. No backfill needed. ## Tests 65 tests (was 43): pause/resume (5), inbox (6), tokens (4), replay (4), concurrent worker context (3), plus all existing coverage. ## What's NOT in this commit Deferred to follow-up PRs: - LISTEN/NOTIFY subscribe (needs real Postgres E2E) - Resource governor (depends on concurrent worker stress testing) - Routing eval harness (needs API keys + benchmark data) - OpenClaw plugin (separate @gbrain/openclaw-minions-plugin repo) See docs/designs/MINIONS_AGENT_ORCHESTRATION.md for full CEO-approved design. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(minions): migration v7 — agent_parity_layer schema Adds columns on minion_jobs (depth, max_children, timeout_ms, timeout_at, remove_on_complete, remove_on_fail, idempotency_key) plus the new minion_attachments table. Three partial indexes for bounded scans: idx_minion_jobs_timeout, idx_minion_jobs_parent_status, and uniq_minion_jobs_idempotency. Check constraints enforce non-negative depth and positive child cap / timeout. Additive migration — existing installs pick it up via ensureSchema on next use. No user action required. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): extend types for v7 parity layer Extends MinionJob with depth/max_children/timeout_ms/timeout_at/ remove_on_complete/remove_on_fail/idempotency_key. Extends MinionJobInput with the same options plus max_spawn_depth override. Adds MinionQueueOpts (maxSpawnDepth default 5, maxAttachmentBytes default 5 MiB). Adds AttachmentInput/Attachment shapes and ChildDoneMessage in the InboxMessage union. rowToMinionJob updated to pick up the new columns. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): attachments validator New module validateAttachment() gates every attachment write. Rejects empty filenames, path traversal (.., /, \), null bytes, oversized content (5 MiB default, per-queue override), invalid base64, and implausible content_type headers. Returns normalized { filename, content_type, content (Buffer), sha256, size } on success. The DB also enforces UNIQUE (job_id, filename) as defense-in-depth for concurrent addAttachment races — JS-only checks are not sufficient. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): queue v7 — depth, child cap, timeouts, cascade, idempotency, child_done Wraps completeJob and failJob in engine.transaction() so parent hook invocations (resolveParent, failParent, removeChildDependency) fold into the same transaction as the child update. A process crash between child and parent can't strand the parent in waiting-children anymore. Adds v7 behaviors: - Depth tracking. add() computes depth = parent.depth + 1 and rejects past maxSpawnDepth (default 5). - Per-parent child cap. add() takes SELECT ... FOR UPDATE on the parent, counts non-terminal children, rejects when count >= max_children. NULL max_children = no cap. - Per-job wall-clock timeout. claim() populates timeout_at when timeout_ms is set. New handleTimeouts() dead-letters expired rows with error_text='timeout exceeded'. Terminal — no retry. - Cascade cancel. cancelJob() walks descendants via recursive CTE with depth-100 runaway cap. Returns the root row. Re-parented descendants (parent_job_id NULL) are naturally excluded. - Idempotency. add() uses INSERT ... ON CONFLICT (idempotency_key) DO NOTHING RETURNING; falls back to SELECT when RETURNING is empty. Same key always yields the same job id. - child_done inbox. completeJob inserts {type:'child_done', child_id, job_name, result} into the parent's inbox in the same transaction as the token rollup, guarded by EXISTS so terminal/deleted parents skip without FK violation. New readChildCompletions(parent_id, lock_token, since?) helper; token-fenced like readInbox. - removeOnComplete / removeOnFail. Deletes the row after the parent hook fires, so parent policy sees consistent state. - Attachment methods. addAttachment validates via validateAttachment then INSERTs; UNIQUE (job_id, filename) backs the JS dup check. listAttachments, getAttachment, deleteAttachment round out the API. Fixes pre-existing inverted status bug: add() now puts children in waiting/delayed (not waiting-children) and atomically flips the parent to waiting-children in the same transaction. Tests no longer need manual UPDATE workarounds. Two correctness fixes: - Sibling completion race. Under READ COMMITTED, two grandchildren completing concurrently each saw the other as still-active in the pre-commit snapshot and neither flipped the parent. Fixed by taking SELECT ... FOR UPDATE on the parent row at the start of completeJob and failJob transactions, serializing siblings on the parent lock. - JSONB double-encode. postgres.js conn.unsafe(sql, params) auto- JSON-encodes parameters. Calling JSON.stringify(obj) first stored a JSON string literal (jsonb_typeof=string) and broke payload->>'key' queries silently. Removed JSON.stringify from three call sites (child_done inbox post, updateProgress, sendMessage). PGLite tolerated both forms so unit tests missed it — real-PG E2E caught it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat(minions): worker — timeout safety net + handleTimeouts tick Worker tick now calls handleStalled() first, then handleTimeouts() — stall requeue wins over timeout dead-letter when both could fire in the same cycle. handleTimeouts() guards on lock_until > now() so stalled jobs take the retryable path. launchJob schedules a per-job setTimeout(timeout_ms) that fires ctx.signal as a best-effort handler interrupt. The timer is always cleared in .finally so process exit isn't delayed by a dangling timer. Handlers that respect AbortSignal stop cleanly; handlers that ignore it still get dead-lettered by the DB-side handleTimeouts. Removed post-completeJob and post-failJob parent-hook calls from the worker — those are now inside the queue method transactions. Worker becomes simpler and crash-safer. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(minions): 33 new unit tests for v7 parity layer Covers depth cap, per-parent child cap, timeout dead-letter, cascade cancel (including the re-parent edge case), removeOnComplete / removeOnFail, idempotency (single + concurrent), child_done inbox (posted in txn + survives child removeOnComplete + since cursor), attachment validation (oversize, path traversal, null byte, duplicates, base64), AbortSignal firing on pause mid-handler, catch-block skipping failJob when aborted, worker in-flight bookkeeping, token-rollup guard when parent already terminal, and setTimeout safety-net cleanup. Existing tests updated to remove the inverted-status manual UPDATE workarounds that the add() fix made obsolete. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(e2e): Minions v7 concurrency + OpenClaw resilience coverage minions-concurrency.test.ts spins two MinionWorker instances against the test Postgres, submits 20 jobs, and asserts zero double-claims (every job runs exactly once). This is the only test that actually proves FOR UPDATE SKIP LOCKED under real concurrency — PGLite runs on a single connection and can't exercise the race. minions-resilience.test.ts covers the six OpenClaw daily pains: 1. Spawn storm caps enforce under concurrent submit. 2. Agent stall → handleStalled() requeues; handleTimeouts() skips (lock_until guard). 3. Forgotten dispatches recoverable via child_done inbox. 4. Cascade cancel stops grandchildren mid-flight. 5. Deep tree fan-in (parent → 3 children → 2 grandchildren each) completes with the full inbox chain. 6. Parent crash/recovery resumes from persisted state. helpers.ts extends ALL_TABLES with minion_attachments, minion_inbox, and minion_jobs (FK dependents first) so E2E teardown doesn't leak rows between runs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore: release v0.11.0 — Minions v7 agent orchestration primitives Bumps VERSION / package.json to 0.11.0. Adds CHANGELOG entry covering depth tracking, max_children, per-job timeouts, cascade cancel, idempotency keys, child_done inbox, removeOnComplete/Fail, attachments, migration v7, plus the two correctness fixes (sibling completion race and JSONB double-encode). TODOS.md captures the four v7 follow-ups: per-queue rate limiting, repeat/cron scheduler, worker event emitter, and waitForChildren convenience helpers. 1066 unit + 105 E2E = 1171 tests passing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(minions): unify JSONB inserts, tighten nullish coalescing Three non-blocker cleanups from post-ship review of v0.11.0: - queue.ts add() and completeJob(): pre-stringifying with JSON.stringify while other sites pass raw objects with $n::jsonb casts. postgres.js double-encodes if you stringify first — works on PGLite (text→JSONB auto-cast), fails silently on real PG. Unify on raw object + explicit $n::jsonb cast. - queue.ts readChildCompletions: since clause used sent_at > $2 relying on PG's implicit text→TIMESTAMPTZ coercion. Explicit $2::timestamptz is safer and clearer. - types.ts rowToMinionJob: parent_job_id used || which coerces 0 to null. Harmless today (SERIAL IDs start at 1) but ?? is semantically correct. All 110 unit tests pass. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(minions): updateProgress missed $1::jsonb cast in unification Residual from |
||
|
|
ce15062694 |
feat: GBrain v0.7.0 — Integration Recipes + SKILLPACK Breakout (#39)
* docs: break SKILLPACK into 17 individual guides The 1,281-line SKILLPACK monolith is now 17 individually linkable guides in docs/guides/, organized by category: core patterns, data pipelines, operations, search, and administration. GBRAIN_SKILLPACK.md becomes a structured index with categorized tables linking to each guide. The URL stays stable for backward compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add integration guides, architecture docs, and ethos New documentation directories: - docs/integrations/ — "Getting Data In" landing page, credential gateway, meeting webhooks. Includes recipe format documentation. - docs/architecture/ — Infrastructure layer doc (import, chunk, embed, search) - docs/ethos/ — "Thin Harness, Fat Skills" essay with agent decision guide - docs/designs/ — "Homebrew for Personal AI" 10-star vision document Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add gbrain integrations command + voice-to-brain recipe New CLI command: gbrain integrations (list/show/status/doctor/stats/test) - Standalone command, no database connection needed - Uses gray-matter directly for recipe parsing (not parseMarkdown) - --json flag on every subcommand for agent-parseable output - Bare command shows senses/reflexes dashboard - Health heartbeat via ~/.gbrain/integrations/<id>/heartbeat.jsonl First recipe: recipes/twilio-voice-brain.md - Phone calls create brain pages via Twilio + OpenAI Realtime - Opinionated defaults: caller screening, brain-first lookup, quiet hours - Outbound call smoke test (GBrain calls the user to prove it works) - Validate-as-you-go credential testing - Twilio signature validation for webhook security Migration file for v0.7.0 with agent-readable changelog. 13 unit tests covering parseRecipe, CLI routing, and recipe validation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Getting Data In to README, update CLAUDE.md and manifest README: voice calls in intro bullet list, new "Getting Data In" section with integration table (voice, email, X, calendar) and recipe philosophy. CLAUDE.md: reference new files (integrations.ts, recipes/, docs/guides/, docs/integrations/, docs/architecture/, docs/ethos/). manifest.json: bump to v0.7.0, add recipes_dir field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: v0.7.0 CHANGELOG, TODOS, VERSION bump CHANGELOG: v0.7.0 entry covering integration recipes, voice-to-brain, gbrain integrations command, SKILLPACK breakout, and new documentation. TODOS: 3 new items from CEO/DX reviews (constrained health_check DSL, community recipe submission, always-on deployment recipes). VERSION + package.json: bump to 0.7.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite voice recipe with agent instructions and verified links Major improvements to recipes/twilio-voice-brain.md: - Agent preamble: explains WHY sequential execution matters (each step depends on the previous), defines 4 stop points where the agent MUST pause and verify, tells agent to never say "something went wrong" but instead explain the exact error and fix - User actions are now specific: exact URLs for every credential (Twilio console, OpenAI API keys page, ngrok dashboard), what buttons to click, what fields to copy, common failure modes - All URLs verified via web search against current 2026 documentation: Twilio SID/token at twilio.com/console, OpenAI keys at platform.openai.com/api-keys, ngrok token at dashboard.ngrok.com/get-started/your-authtoken - Cost estimate corrected: OpenAI Realtime is $0.06/min input + $0.24/min output (was understated), total ~$20-22/mo for 100 min - Validate-as-you-go: each credential tested immediately with exact curl commands, failure messages explain what went wrong and how to fix - Smoke test flow: tells user exactly what to say, verifies ALL three outputs (messaging notification + brain page + search result) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add "Homebrew for Personal AI" essay (markdown is code) New essay at docs/ethos/MARKDOWN_SKILLS_AS_RECIPES.md — the distribution corollary to "Thin Harness, Fat Skills." Argues that markdown skill files are simultaneously documentation, specification, package, and source code. The agent is the package manager. The git repo is the app store. Referenced from SKILLPACK index and CLAUDE.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: rewrite agent instructions as command language, promote skills The OpenClaw/Hermes install block is now a drill sergeant, not a tour guide. Every step is an imperative command with exact verification criteria and explicit stop-on-failure behavior. No FYI, no suggestions, just rails. Key changes: - 11-step setup with STOP points after each step - Exact user instructions for Supabase connection string (what to click, what NOT to give the agent, what the string looks like) - "Verify: run X. You must see Y. If not: Z" after every step - Skills table now links to both skill files AND guide docs - Integration recipes table simplified (no "coming soon" placeholders) - Docs section reorganized: for agents / for humans / reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: 4 codex findings + add email-to-brain recipe Codex review found 4 issues, all fixed: 1. getStatus() returned "configured" if ANY secret was set (e.g. just OPENAI_API_KEY). Now requires ALL required secrets before marking configured. Prevents false "configured" status and spurious doctor runs. 2. Twilio health check hit unauthenticated endpoint (always 401). Now uses authenticated curl with SID:token, matching the setup validation. 3. README anchor docs/GBRAIN_SKILLPACK.md#the-dream-cycle broken after SKILLPACK rewrite. Updated to point to docs/guides/cron-schedule.md. 4. Compiled binary can't find recipes/ via import.meta.dir. Added GBRAIN_RECIPES_DIR env var override + global bun install path fallback. Also adds recipes/email-to-brain.md: Gmail deterministic collector pattern with ClawVisor credential gateway, validate-as-you-go, agent instructions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add email, X, calendar, and meeting sync recipes Four new integration recipes extracted from production wintermute patterns: - recipes/email-to-brain.md: Gmail via ClawVisor, deterministic collector pattern (code pulls emails with baked-in links, agent does judgment), noise filtering, signature detection, digest generation - recipes/x-to-brain.md: X API v2, timeline + mentions + keyword search, deletion detection (diffs previous run, verifies 404), engagement velocity tracking, rate limit awareness - recipes/calendar-to-brain.md: Google Calendar via ClawVisor, historical backfill (years of data), daily markdown files with attendees + locations, attendee enrichment for brain pages - recipes/meeting-sync.md: Circleback API, transcript import with speaker labels, attendee detection + filtering, entity propagation to people/ company pages, action item extraction, idempotent by source_id All recipes follow the same format: agent preamble with sequential execution rules, validate-as-you-go credentials, exact URLs for API key setup, stop-on-failure verification, and heartbeat logging. Updated README, SKILLPACK index, and integrations landing page with all 5 recipes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add Google OAuth as alternative to ClawVisor in email + calendar recipes Both recipes now offer two auth options: - Option A: ClawVisor (recommended, handles OAuth + token refresh) - Option B: Google OAuth2 directly (no extra service, you manage tokens) Option B includes step-by-step instructions for Google Cloud Console: exact URLs, which buttons to click, which scopes to add, how to enable the API, and the OAuth flow for token exchange. This removes ClawVisor as a hard dependency for getting started. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add implementation guides with pseudocode and test suggestions Every recipe now includes an "Implementation Guide" section with: - Production-tested pseudocode the agent can follow to build each collector - Edge cases and failure modes discovered in real deployment - Non-obvious implementation details (why the 48h staleness heuristic, why Gmail links need authuser, why SSE responses need double-parsing) - Test suggestions: what the agent should verify after setup email-to-brain: noise filtering algorithm, signature detection patterns, Gmail link generation (authuser is critical), sent-mail dedup x-to-brain: deletion detection with 3 heuristics (7-day, 48h staleness, API verification), engagement velocity thresholds (50 min for 2x, 100 absolute jump), atomic writes, stdout contract, rate limit handling calendar-to-brain: smart chunking (monthly for sparse years, weekly for dense), attendee filtering (rooms, groups, distros), merge-with-existing (only replace ## Calendar section), date/time parsing edge cases meeting-sync: SSE double-JSON parsing, idempotency double-check (grep + filename), auto-tagging from meeting names, git commit after sync Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: 6 new guides from production patterns (wintermute extraction) New guides extracted and generalized from production deployment: - repo-architecture.md: Two-repo pattern (agent behavior vs world knowledge). Strict boundary rules, decision tree, hard rule: never write knowledge to the agent repo. - sub-agent-routing.md: Model routing table by task type. Signal detector pattern (spawn Sonnet on every message). Research pipeline pattern (Opus plans, DeepSeek executes, Opus synthesizes). Cost optimization. - skill-development.md: 5-step cycle (concept, prototype, evaluate, codify, cron). MECE discipline (no overlapping skills). Quality bar checklist. "If you ask twice, it should already be a skill." - idea-capture.md: Originality distribution rating (0-100 across 4 populations). Depth test ("could someone unfamiliar understand WHY?"). Deep cross-linking mandate. Notability filtering. - quiet-hours.md: Hold notifications 11pm-8am local time. Held messages directory pattern. Timezone-aware delivery. Morning briefing pickup. - diligence-ingestion.md: 9-step pipeline for data room materials. Detection patterns (PDF filenames, spreadsheet tabs, user language). Index.md template with bull/bear case. Company page enrichment. All PII scrubbed. Patterns generalized for any user. SKILLPACK index updated with 6 new entries. CLAUDE.md references added. All 37 SKILLPACK links verified. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: upgrade all guides to operational playbooks with pseudocode Every guide now follows the playbook structure: - Goal: one sentence, what this achieves - What the User Gets: without this / with this - Implementation: pseudocode with actual gbrain commands - Tricky Spots: production-tested gotchas - How to Verify: test steps the agent runs after setup Guides upgraded (15 files): - brain-agent-loop: on_message() loop with read/write/sync pseudocode - brain-first-lookup: 4-step lookup cascade with exact commands - brain-vs-memory: routing algorithm for 3 knowledge layers - compiled-truth: page structure + rewrite vs append rules - content-media: 3 ingest patterns (YouTube, social, PDFs) - cron-schedule: full schedule table + dream cycle pseudocode - enrichment-pipeline: 7-step protocol with tier classification - entity-detection: spawn pattern + detection prompt + notability filter - executive-assistant: 3 workflow algorithms (triage, prep, post-inbox) - meeting-ingestion: 6-step transcript-to-brain flow - operational-disciplines: 5 executable discipline blocks - originals-folder: detection + exact-phrasing capture + cross-linking - search-modes: decision tree for keyword vs hybrid vs direct - source-attribution: citation format + hierarchy + conflict resolution - Plus Goal/What User Gets headers on 6 newer guides Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add WebRTC to voice recipe + ngrok Hobby setup guide Voice recipe updates: - Added WebRTC endpoint (POST /session, GET /call, POST /tool) for browser-based calling with RNNoise noise suppression - WebRTC pseudocode with the 4 non-obvious gotchas from production (voice under audio.output.voice, no turn_detection, no session.update on connect, trigger greeting via data channel) - Recommend ngrok Hobby ($8/mo) for fixed domain instead of free tier - Fixed domain means URLs never change, Twilio never breaks New guide: docs/mcp/NGROK_SETUP.md - How to set up ngrok Hobby for both MCP and voice agent - Fixed domain setup, watchdog pattern, AI client configuration - Claude Desktop requires Settings > Integrations (not JSON config) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add dependency graph + ngrok-tunnel + credential-gateway recipes Recipes now have real dependencies via the `requires` field: - voice-to-brain requires ngrok-tunnel (needs public URL for Twilio) - email-to-brain requires credential-gateway (needs Gmail access) - calendar-to-brain requires credential-gateway (needs Calendar access) - x-to-brain and meeting-sync are standalone (direct API keys) Two new infrastructure recipes: - ngrok-tunnel: fixed public URL for MCP + voice. Recommends Hobby ($8/mo) for a domain that never changes. Includes watchdog pattern. - credential-gateway: secure Google service access via ClawVisor (recommended) or direct OAuth2. One setup, all Google recipes use it. Moved ngrok from docs/mcp/ to recipes/ — it's shared infrastructure, not MCP-specific. README and integrations landing page show dependency chains. When agent installs voice-to-brain, it sets up ngrok-tunnel first. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add infra category, fix dashboard alignment, show dependencies DX audit found two bugs in gbrain integrations dashboard: 1. Column alignment broken — IDs > 18 chars ran into descriptions with no space. Fixed: pad to 22 chars. 2. ngrok-tunnel and credential-gateway showed as SENSES but they're infrastructure. Added 'infra' category. Dashboard now shows three sections: INFRASTRUCTURE (set up first), SENSES, REFLEXES. 3. Dependencies now shown inline: "AVAILABLE (needs credential-gateway)" Also added 'requires' field to JSON output for agent consumption. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add frontier model requirement disclaimer to README GBrain's markdown-is-code approach requires models capable of interpreting intent and implementing from architecture descriptions. Tested with Claude Opus 4.6 and GPT-5.4 Thinking. Smaller models will struggle with the recipe format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add PGLite → Supabase upgrade path to README Clarify the database progression: start with PGLite (Postgres as WASM, zero infrastructure, pgvector built in, nothing to install). Graduate to Supabase or self-hosted Postgres when you need connection pooling, concurrency, and remote MCP access from Claude Desktop, Cowork, ChatGPT, Perplexity Computer, or any MCP-compatible agent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: revert PGLite mention (coming in next branch) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: make all 23 guides consistent (Goal/Impl/Tricky/Verify) Every guide now has exactly these sections in this order: - ## Goal (one sentence) - ## What the User Gets (without this / with this) - ## Implementation (pseudocode with gbrain commands) - ## Tricky Spots (3-5 numbered gotchas) - ## How to Verify (3-5 numbered test steps) 11 guides restructured from non-standard headings: - deterministic-collectors, live-sync, upgrades-auto-update (full rewrites) - entity-detection, diligence-ingestion, idea-capture, quiet-hours, repo-architecture, skill-development, sub-agent-routing (restructured) 23/23 guides now pass consistency audit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: restructure README around the #1 blocker (getting data in) The README was leading with Postgres and database architecture. Most users are stuck at step zero: "I have an agent but it doesn't know anything about my life." New structure: 1. The Problem — your agent doesn't know your life 2. Getting Data In — integration recipes, front and center 3. The Compounding Thesis — why this matters 4. How this happened — credibility, origin story 5. When you need Postgres — scale, not starting point Postgres is de-emphasized from a full section to two paragraphs: "You don't need Postgres to start" and "When you need Postgres" (1,000+ files, remote MCP access, multiple AI clients). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move Install to top of README, remove duplicate section Install now appears right after Getting Data In (line 38), not buried at line 295. The user sees: Problem → Getting Data In → Install. Removed the duplicate Install section (262 lines) that was lower in the README. The agent instructions block, CLI quickstart, and all content is now in the single Install section near the top. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: move agent install block to first thing in README "Start here: paste this into your agent" is now the first section, right after the one-line pitch. No scrolling, no context, no preamble. User opens the README, sees the paste block, copies it into OpenClaw or Hermes, and the agent takes over. Flow: pitch → paste block → Getting Data In → Compounding Thesis → origin story Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: compress install block from 11 steps to 5 The agent install block was 102 lines and 11 steps. Now it's 40 lines and 5 steps. Same coverage, half the text. Changes: - Merged "prove keyword search" + "embed" + "prove hybrid search" into one SEARCH step (the user doesn't care about the intermediate) - Merged skillpack, sync, auto-update, integrations, verification into one GO LIVE step with sub-items (post-install polish, not install) - Shortened database instructions (one line instead of 5 sub-steps) - Removed redundant preamble ("YOU MUST COMPLETE EVERY STEP" is now just "Do not skip steps. Verify each step.") The 5 steps: INSTALL → DATABASE → IMPORT → SEARCH → GO LIVE Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: gitignore all .env files, not just specific ones CSO audit found .gitignore covered .env.testing and .env.production but not bare .env. A user creating .env with database credentials could accidentally commit it. Fix: .env and .env.* are now gitignored. .env.*.example files are explicitly un-ignored so templates remain tracked. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * security: scrub PII from essay and recipe examples - 510-MY-GARRY phone mnemonic → "Your Phone Number" - "Garry → Authenticated Mode" → "Owner → Authenticated Mode" - "Telegram" → "secure channel" in auth example - @garrytan → @yourhandle in X recipe example Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |