admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-28 14:59:47 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
fix/backlog-c129
gbrain/docs/RELEASING.md
T
Garry TanGitHubClaude Fable 5
 a81f7e05e8 v0.42.43.0 feat(context): push-based context (#2095) + teardown-exit hardening (#2084) (#2175) 
* fix(cli): exit deliberately after bounded teardown instead of riding the 10s backstop (#2084)

Root cause: bounded teardown (endPoolBounded, #2015) RESOLVES, but lingering
sockets — embedding-provider fetch keep-alive, PgBouncer txn-mode sockets the
bound raced past — keep Bun's event loop alive, so every `gbrain query` paid
a flat 10s tax exiting via the hard-deadline force-exit banner.

Three changes, one contract:

- flushStdoutThenExit (cli-force-exit.ts): when main() resolves and the
  command is not a daemon, exit deliberately — after stdout AND stderr drain
  (writableLength===0, 'drain'-event + poll loop, 2s unref'd guard for a
  blocked pipe). Incident #1959 (force-exit truncating piped stdout) is the
  regression class; pinned by a 256KB real-pipe subprocess test.

- drainThenDisconnect (cli.ts): ONE owner-disconnect helper at all 8 sites
  (op-dispatch, CLI_ONLY fall-through, search dashboard, doctor remediation
  x3, ze-switch, dream, read-only timeout path). Drains the background-work
  registry, then disconnect (best-effort), bounded by the 10s unref'd
  hard-deadline — which is now armed around the TEARDOWN window only, not
  before the op handler (the old placement would have force-killed any op
  slower than 10s). Closes the filed TODOS P3 drain-hoist: six sites
  previously skipped the drain entirely and had no hang timer at all.

- Inner process.exit sweep: mid-handler exits in engine-owning/output-bearing
  paths (status, friction, claw-test, smoke-test, eval cross-modal /
  takes-quality replay / conversation-parser / whoknows-thin, status-thin)
  become process.exitCode + return so they flow through the drains and the
  flush-exit. Pre-engine usage/parse/refusal exits stay as-is.

BrainRegistry.disconnectAll deliberately unchanged: zero production callers
in src/, per-engine disconnects already bounded, and the kernel reclaims
sockets on exit (src/core/timeout.ts doctrine).

DAEMON_COMMANDS gains 'watch' ahead of the #2095 push transport.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(e2e): PgBouncer transaction-mode pooler in CI + teardown e2e (#2084)

Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs
verified only against one production deployment — CI had no transaction-mode
pooler and could never see the class. Now it can:

- docker-compose.ci.yml: `pgbouncer` service (transaction pooling) fronting
  postgres-1, mirroring the production split-pool topology (direct :5432 +
  pooled :6543). AUTH_TYPE=plain (pg16 SCRAM verifiers need the plaintext
  password in the userlist) + IGNORE_STARTUP_PARAMETERS for the
  statement_timeout/idle_in_transaction_session_timeout startup params
  gbrain's client sets (the Supabase pooler whitelists the same).
- test/e2e/pgbouncer-teardown.test.ts: schema + fixture via the DIRECT url
  into a dedicated `gbrain_pgbouncer` database (never races shard TRUNCATEs),
  then spawns the real CLI against the POOLED url and asserts: exit 0,
  stdout intact (the #1959 truncation class), and NO
  "did not return within 10000ms — force-exiting" banner (pre-#2084 it
  printed on 100% of query-shaped ops on this topology). Class bound, not
  exact timing. Skips gracefully without GBRAIN_PGBOUNCER_URL.
- scripts/ci-local.sh: threads GBRAIN_PGBOUNCER_URL +
  GBRAIN_PGBOUNCER_DIRECT_URL into all three e2e phases.

Verified live: both tests green against pgbouncer 1.25.2 in transaction mode.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(schema): context_volunteer_events table (v116) — push-context feedback log (#2095)

One row per page the brain volunteers (op / reflex / watch channels).
"Used" is DERIVED, never written: pages.last_retrieved_at > volunteered_at
(the existing bumpLastRetrievedAt write-back is the open/cite signal), so
there is no second tracking path. session_id/turn are nullable
caller-supplied attribution; rationale is a deterministic template string,
never raw conversation text.

- Migration v116 (idempotent) + mirrors in src/schema.sql +
  src/core/pglite-schema.ts + regenerated schema-embedded.ts (regen also
  folds in pre-existing comment-only drift from the v114 links edits).
- src/core/context/volunteer-events.ts: insertVolunteerEvents (ONE
  multi-row parameterized INSERT — never per-row awaited round-trips) +
  purgeStaleVolunteerEvents (90-day GC, returns 0 on pre-v116 brains).
- Dream cycle purge phase prunes stale events alongside op_checkpoints /
  brainstorm checkpoints / batch-retry audit files.
- RLS on Postgres comes from the v35 auto_rls_on_create_table event
  trigger (the same mechanism that covered v110 page_aliases and v115
  op_checkpoint_paths); the volunteer Postgres e2e pins it.
- No ::jsonb anywhere; no bootstrap probe needed (nothing references the
  table pre-creation; writers guard with try/catch).

Tests: v116 shape + columns + indexes + live insert/purge round-trip on
PGLite (test/migrate.test.ts, 161 pass); schema-bootstrap-coverage green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(context): multi-turn window extraction + confidence-scored volunteer core (#2095)

- entity-salience.ts: extractCandidatesFromWindow(turns) — runs the existing
  per-turn extractor across the last N turns (oldest→newest), merges by the
  normalizeAlias form with occurrence/newest-turn/user-mention metadata, and
  orders by salience (recency > frequency > user-role) so the MAX_CANDIDATES
  cap drops stale assistant chatter, not the entity the user just named.
  Closes the filed assistant-introduced-entities recall TODO; true pronoun
  coreference (never-named antecedents) stays out of scope.

- retrieval-reflex.ts: ReflexPointer gains source_id + arm + confidence +
  matchedNorm. ARM_CONFIDENCE (alias 0.9 / title 0.8 / slug-suffix 0.6)
  lives next to the arm definitions so identity and score can't drift.
  Arm-2 provenance is classified in JS (codex D8 — the combined OR can't
  report which predicate matched). Federated sourceIds[] scope (alias arm
  loops per source; arm 2 uses source_id = ANY — no engine-interface
  change). Suppression gains 'slug-only' mode (codex D7, REQUIRED for
  windowing): the legacy title-whole-word rule would suppress every entity
  merely MENTIONED in a prior window turn, breaking the feature by
  construction — slugs only enter context when a pointer/page was actually
  surfaced. Default stays 'slug-and-title' for the window=1 legacy path.

- volunteer.ts (new): parseWindow (lenient user:/assistant: prefixes, CRLF,
  unprefixed → one user turn), volunteerContext (zero-LLM: extract →
  resolve → +0.05 multi-turn/newest-turn boost → min_confidence 0.7 gate →
  cap 3/5; deterministic rationale strings, never raw conversation text),
  and volunteerUsageStats (per-arm/channel precision from the
  last_retrieved_at join, labeled approximate — 5-min throttle false
  negatives, unrelated-read false positives; codex D9).

Tests: 35 green across volunteer-context (window parsing, pronoun follow-up
via assistant-introduced entity, confidence gating, slug-only suppression,
takes-fence privacy, multi-source scope, caps, stats join math) +
retrieval-reflex back-compat + resolve-ipc.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ops): volunteer_context op — CLI (stdin) + MCP, drained event sink (#2095)

New read-scope op on the contract surface (CLI `gbrain volunteer-context`
with stdin → window, MCP tool for free): takes a rolling conversation
window, returns confidence-gated page pointers with rationales + synopses.
`window` is optional-unless-stats (validated in the handler, codex D9);
`stats: true` returns the volunteered-vs-used precision summary, labeled
APPROXIMATE (the 5-min last-retrieved throttle and unrelated reads both
bias the join). Source scope threads through sourceScopeOpts — federated
grants narrow the volunteer to the granted sources.

Event logging is fire-and-forget through a new `volunteer-events`
background-work sink (volunteer-events.ts, mirrors last-retrieved: tracked
dangling promise set + bounded drain + snapshot-drop on timeout so a
long-lived process never accumulates ghosts). ONE batched INSERT per call,
drained on every exit path by the commit-1 drain hoist; failure never
fails the op (pinned by an injected failing-engine test).

cli formatResult renders both shapes (pointer lines with confidence/arm/
rationale; the stats summary with per-arm precision).

Tests: op contract surface, window-required validation, sink round-trip
with session_id/turn attribution, failing-engine fail-open, federated
grant scoping, stats mode (26 green on PGLite) + a real-Postgres e2e
proving the op + sink + stats join AND that context_volunteer_events has
RLS enabled (keeps the auto-RLS event-trigger mechanism honest for v116).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(context): reflex consumes the rolling window + ambient-channel logging (#2095)

The default-on retrieval reflex now extracts entities from the last N turns
(retrieval_reflex_window_turns, default 4; env
GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; window=1 reproduces the legacy
current-turn-only behavior exactly). assemble() passes the recent
user/assistant turns (hard cap 12); the reflex slices to the configured
window. Assistant-introduced entities and "what did she invest in?"
follow-ups whose antecedent was NAMED in the window now surface pointers —
the issue's "zero agent-initiated queries" success criterion on the
ambient path.

Under windowing, suppression switches to slug-only (codex D7): the legacy
title-whole-word rule would suppress every entity merely MENTIONED in a
prior window turn, breaking the feature by construction. Slugs only enter
prior context when a pointer/page was actually surfaced, so
already-surfaced pages still suppress. The suppression mode flows through
all three resolver rungs (host opts, serve IPC request, direct Postgres).

Ambient-channel feedback (codex D11): the server-side resolver paths
(serve IPC + direct Postgres) log volunteered pointers with
channel: 'reflex' through the drained volunteer-events sink, so
`gbrain volunteer-context --stats` measures the default-on path where most
volunteering happens. Host-injected resolvers (no gbrain engine) can't
log — documented gap. Precision gates, 1.5s ceiling, fail-open, and the
pointer cap are unchanged.

Tests: prev-assistant-turn entity fires; window=1 legacy parity; slug-only
vs already-surfaced suppression; throwing resolver stays fail-open
(16 green).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(cli): gbrain watch — push transport over stdin (#2095)

The issue's headline: the brain volunteers pages as the conversation flows,
instead of waiting to be asked. `some-transcript-feed | gbrain watch` reads
turns line-by-line ('user:'/'assistant:' prefixes set the role; unprefixed
lines are user turns), keeps a rolling window (--window-turns, default 4),
and streams confidence-gated pointers with rationales to stdout (--json for
JSONL). Session dedupe rides the core's slug-only suppression — a slug is
volunteered at most once per session. Events log on channel 'watch' with
session_id + turn through the drained sink.

Lifecycle: watch BLOCKS in the stdin iteration (like `jobs work`) — an
interactive TTY stays alive until Ctrl-C/Ctrl-D, piped input ends at EOF —
so it is deliberately NOT in DAEMON_COMMANDS (reverts the commit-1
placeholder): when main() resolves the work is over, the CLI_ONLY finally
drains volunteer events via drainThenDisconnect, and the entrypoint
flush-exit ends the process. Keeping it in the daemon set would have made
the piped EOF path hang on lingering sockets — the exact #2084 class.
SIGINT closes the stream and flows through the same drain path instead of
killing mid-write. Per-turn resolution failures are fail-open (the stream
never dies on a transient DB error).

Full wiring (eng-review D12): CLI_ONLY + CLI_ONLY_SELF_HELP (WATCH_HELP) +
THIN_CLIENT_REFUSED_COMMANDS (thin clients use the volunteer_context MCP
op) + main --help entry.

Tests: 18 green — help, per-turn volunteering + clean EOF return, rolling
window via assistant-introduced entity, session dedupe, --json shape with
turn attribution, channel-watch event rows, --min-confidence gate, CRLF/
blank tolerance, daemon-gate semantics. Live smoke: piped `gbrain watch`
on a fresh PGLite brain exits 0 at EOF with no force-exit banner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: KEY_FILES + push-context guide + TODOS for the #2084/#2095 wave

- docs/architecture/KEY_FILES.md (current-state): context entries gain the
  window extractor, arm provenance/confidence, suppression modes, volunteer
  + volunteer-events modules; background-work entry now lists FIVE sinks and
  the drainThenDisconnect owner-disconnect contract; new entries for
  src/core/cli-force-exit.ts (the exit contract) and src/commands/watch.ts.
- docs/guides/push-context.md (new): the three channels (reflex/op/watch),
  the confidence model, CLI usage, config keys, and the approximate-stats
  caveat. Linked from CLAUDE.md's reference map.
- CLAUDE.md: ops line mentions volunteer_context + the guide link;
  bun run build:llms regenerated in the same commit (freshness test green).
- TODOS.md: #2095 deferrals filed (SSE push channel, policy skill + doctor
  check, structured messages[] param); the #1981 entity-detection TODO
  narrowed (window extraction covered assistant-introduced entities +
  named-antecedent follow-ups); the drain-hoist P3 marked DONE by the
  #2084 wave.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(e2e): truncate context_volunteer_events in setupDB (#2095)

The new feedback-log table wasn't in ALL_TABLES, so volunteered-event rows
persisted across e2e runs on a reused database and poisoned count/stats
assertions in volunteer-context-postgres on the second run. No FK to pages
(slug join), so position before pages is for hygiene only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cli): own the exit verdict — never trust ambient process.exitCode (#2084)

Caught by the full unit suite: `gbrain apply-migrations` on PGLite started
exiting 99. Root cause: PGLite's Emscripten runtime writes the WASM
backend's proc_exit status into process.exitCode (initdb at create-time,
the postmaster at close-time — `exitCode=status` in pglite's dist), and
the writes land ASYNCHRONOUSLY, outside any snapshot/restore window around
create/close (a guarded attempt verified this). The pre-#2084 success path
never read process.exitCode, so the pollution was invisible; the new
deliberate flush-exit propagated it faithfully.

Fix: gbrain records its own verdict. setCliExitCode(n)/getCliExitCode() in
cli-force-exit.ts — every gbrain-owned exit-code assignment routes through
the setter (still mirrored to process.exitCode for outside readers), and
both exit paths (entrypoint flushStdoutThenExit + the drainThenDisconnect
hard-deadline backstop) read the getter. Swept all assignment sites:
cli.ts (op error, friction, claw-test, smoke-test, eval runners, status,
import errors) + reindex/transcripts/brainstorm/frontmatter/autopilot.

Also updates the v0.42.20 structural pins to the drainThenDisconnect shape
(ordering invariant asserted INSIDE the helper + >=8 helper call sites,
superseding the two-inline-pairs assertion).

Verified: apply-migrations spawn test green; `init --migrate-only` exits 0;
an errored op still exits 1.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: re-pin the teardown-arming invariant at its post-#2084 home

Master's v0.42.41.0 triage wave and the #2084 wave fixed the same
pre-armed-timer bug independently; the merge keeps #2084's shape (arming
inside the shared drainThenDisconnect helper, covering all 8 exit paths).
The structural pin now asserts the same invariant — no pre-try arming;
gated, unref'd, before-drain, cleared — at the helper.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: coverage for ambient reflex-channel logging + watch window/cap flags

Ship coverage audit (85%, gate PASS) named five gaps; the two substantive
cheap ones close here: the codex-D11 logChannel='reflex' path now has a
behavioral pin (events land on channel 'reflex' through the drained sink;
no logChannel → no events), and gbrain watch's --window-turns / --max-pages
flags are exercised (turn-1 attribution under window=1; cap to one page).
Remaining flagged-not-blocking: the wallclock-timeout branch (untestable
without >10s real-clock flake — same rationale as the arming pin),
formatResult's volunteer case (module-private), and the cycle purge wiring.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: close the remaining plan-audit gaps — formatResult rendering + watch SIGINT

formatResult exported for tests (same import-safety contract as cliAliases);
test/cli-format-volunteer.test.ts pins the pointer lines, empty-gate message,
and approximate stats summary. test/watch-command.test.ts gains a real
subprocess SIGINT test: piped stdin that never reaches EOF, SIGINT mid-stream,
assert exit 0 with no force-exit banner — the drain-then-exit lifecycle under
the actual signal, not just the shared exit path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix: doctor's FAIL verdict was zeroed by the owned exit — sweep stragglers + class pin

The merged-state suite caught it: doctor --fast --json reported FAIL but
exited 0. Master's v0.42.41.0 brought raw `process.exitCode =` writes
(doctor.ts hasFail ternary, extract.ts) that the #2084 verdict-owning exit
silently zeroes — getCliExitCode() deliberately never reads ambient
process.exitCode (the PGLite-Emscripten pollution defense), so any setter
that bypasses setCliExitCode reports success on failure.

Swept both sites and added the structural class pin: a test greps src/ for
raw `process.exitCode =` outside cli-force-exit.ts, so the next merge that
introduces one fails loudly instead of lying about exit codes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.43.0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test: quarantine the watch SIGINT subprocess test to the serial lane

The parallel unit shards flake on concurrent CLI subprocess spawns (failed
at 7ms in-suite, green solo) — same isolation rationale as
apply-migrations-pglite-spawn.serial.test.ts and #2141's R3 quarantine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: update project documentation for v0.42.43.0

Post-ship doc verification against the release diff (#2095 push-based
context + #2084 superset hardening), with a cross-model doc review:

- push-context.md: version tag corrected to v0.42.43.0; per-call knobs
  now cover prior_context/days and watch's flag surface accurately;
  feedback-log writes described as best-effort; synopsis fence-strip
  described as unconditional.
- CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context
  release reference corrected to v0.42.43.0.
- KEY_FILES.md: ci-local entry rewritten to current topology (4-shard
  parallel default, four Postgres services, transaction-mode PgBouncer
  + GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts
  dropped from the selector entry.
- TESTING.md: inventory entries for the new #2084 structural pins
  (cli-exit-verdict-pin, cli-pipe-truncation), the push-context test
  suite (volunteer-context, watch-command, watch-sigint.serial,
  cli-format-volunteer), migrate v117 coverage, and the two new E2E
  files (pgbouncer-teardown env gating, volunteer-context-postgres RLS
  pin); check:all row corrected (not a superset of verify).
- AGENTS.md + RELEASING.md: ci:local descriptions updated to the
  sharded + pooler topology.
- CHANGELOG (wording only, entry preserved): "retrieved" instead of
  "opened" for the used-signal, pooler scoped to the local CI gate,
  feedback log labeled best-effort.
- llms-config.ts: index the new push-context guide; bundles
  regenerated (build:llms) and freshness test green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(test): correct the v116 reference — the table shipped as migration v117

* fix: pre-landing review hardening — federated alias parallelism, trust-boundary clamps, shared protocol helpers (#2095)

Five specialist reviewers (testing/maintainability/security/performance/
data-migration) on the reconciled diff; every finding applied:

Performance: the alias arm now resolves all granted sources CONCURRENTLY
(a federated caller paid M sequential RTTs per turn — ~355ms at 5 sources
cross-region, inside the reflex's 1.5s budget); watch's session dedupe is
O(1) Set membership instead of a monotonically growing priorContext string
(O(T²) over a long-lived session); getWindowTurns iterates from the tail
(per-turn cost no longer grows with session length); the resolver's
provenance maps fold into the existing candidate pass.

Security: volunteer_context clamps caller-supplied attribution at the trust
boundary — session_id capped at 256 chars (a read-scoped token could bank
~1MiB TEXT per request, retained 90 days), turn logged only when a safe
integer (a non-integer threw inside the batched INSERT and silently dropped
the whole batch). The privacy comments now state precisely what rationale
may contain (the matched entity's surface form — which by construction
resolved to an existing alias/title/slug — never free conversation text).

Maintainability: TURN_PREFIX_RE + formatVolunteeredPage exported from
volunteer.ts and shared by watch/cli (the two surfaces can no longer
drift); volunteerEventRowsFrom is the single VolunteerEventRow assembly
site for all three channels; watch's window default now honors the same
retrieval_reflex_window_turns config knob the reflex reads; the stale
pre-v116 comments swept to pre-v117.

Testing: the two flake-class CRITICALs fixed (pipe test asserts the
backstop banner instead of a cold-CI-hostile 9s wall bound; the SIGINT test
waits on watch's new machine-readable ready line instead of a fixed 15s
sleep — 2.5s and deterministic now); new coverage for the sink's timeout
branch + ghost-reference drop, watch per-turn fail-open, untrusted knob
clamps (min_confidence/max_pages/days), window-cap ordering (newest user
mention survives), serve-IPC suppression passthrough + channel=reflex
logging, windowTurnCount edge semantics, and structural pins for the sink
registration + cycle purge wiring. The exit-verdict pin's grep is now
operator/whitespace-tolerant.

Deferred with TODOs: resolver index shapes for the per-turn query;
batched first-prune after a long dream-cycle gap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(context): red-team hardening — pre-cap dedupe, delivery-side reflex logging, window clamp

Four red-team findings on the #2095 push-context surface:

- RT1 starvation: watch's session-dedupe Set filtered AFTER volunteerContext's
  cap, so a recurring already-pushed entity burned cap slots every turn and
  starved fresh pages behind it. VolunteerOpts.excludeSlugs now skips inside
  the pointer loop BEFORE the confidence gate and the cap.
- RT3 honest stats: reflex-channel event logging moved from inside the
  resolver to the DELIVERY point — serve's resolve-IPC onDelivered hook fires
  only after the response write succeeds, and buildReflexAddition logs only
  after the per-turn timeout admits the block. A block the client's 250ms
  budget abandoned was never injected and no longer counts as volunteered.
  (logChannel resolver opt removed; logDeliveredReflexPointers is the seam.)
- RT5 unbounded window: --window-turns is clamped to [1, 64] so a config typo
  can't reintroduce the re-scan-everything-per-turn cost class.
- RT2/RT4 documented + filed: PGLite watch connection monopoly (WATCH_HELP,
  push-context guide, TODO to route watch via serve IPC); host-resolver
  suppression contract at ResolveEntitiesFn (TODO for a capability gate).

Tests: starvation guard (watch + volunteerContext unit), window clamp floor +
ceiling, delivery-side logging (helper writes channel=reflex through the
drained sink; bare resolver writes nothing; empty list no-op), IPC wiring test
rewired to onDelivered. KEY_FILES.md + push-context.md updated; build:llms run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(context): env-plane window knob works config-less; harden two gateway-state-leak victims

Three CI-only check failures, two root causes:

1. windowTurnCount ignored GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS when
   loadConfig() returned null (no config file AND no DATABASE_URL — a clean
   CI shard with no brain). loadConfig drops its env→config mapping in that
   case, so the documented escape hatch silently died and the window fell
   back to 4 → windowed extraction widened when the test set window=1 →
   prior-turn entity leaked. Fixed: read the env var directly in
   windowTurnCount, mirroring reflexEnabled's direct process.env read. This
   is a real product bug, not just a test artifact — any config-less host
   using the env hatch was affected. Regression test pins it.

2. sync-cost-preview + doctor-federation-health failed only IN-SHARD: a
   sibling test configured a non-legacy (ZeroEntropy 1280-d / $0.05) gateway
   and never reset it. The legacy-embedding preload only restores the
   OpenAI/1536 default when the gateway slot is EMPTY, so a non-empty foreign
   config survives into the next file — and a file's beforeAll runs BEFORE
   the preload's restoring beforeEach, so federation-health built a
   vector(1280) column and its 1536-d fixture hit CheckExpectedDim. My new
   test files reshuffled the deterministic file→shard assignment, exposing
   this latent ordering bug. Hardened both victims to establish the gateway
   state they assert (sync-cost-preview resets to the unconfigured fallback;
   federation-health pins legacy 1536 before initSchema) so they're
   order-independent. Verified against a simulated leaker run before them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(context): use withEnv() in the window env-hatch test (test-isolation guard)

The regression test added in 82cc7fff mutated process.env directly, which
check:test-isolation (R1) forbids — use the withEnv() helper that restores on
exit, same as the rest of this file. Behavior identical; guard green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-06-14 09:32:58 -07:00

436 lines
22 KiB
Markdown
Raw Permalink Blame History

# Releasing & contributing (gbrain)
The full release + contributor process. CLAUDE.md keeps the ship-critical IRON RULES
inline (the Version-locations table, branch=workspace, post-ship `/document-release`,
the Privacy + Responsible-disclosure rules, PR-title-version-first, never-hand-roll-ship)
and points here for everything else. **Before any ship, read this in full. Use `/ship` —
never hand-roll a release.**
## Pre-ship requirements
Before shipping (/ship) or reviewing (/review), always run the full test suite.
Two equivalent paths:
**Path A — local CI gate (recommended, v0.23.1+):**
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host),
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
containers plus a transaction-mode PgBouncer service (unit phase keeps
`DATABASE_URL` unset; `--no-shard` for the legacy sequential flow). Stronger
than PR CI's 2-file Tier 1 set; closer to what nightly Tier 1 catches. Spins
up + tears down postgres automatically via `docker-compose.ci.yml`. Override
the host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
- `bun run ci:local:diff` runs only the E2E files matched by the diff selector
(`scripts/select-e2e.ts`), falling back to ALL E2E files on unmapped src/
paths or schema/skills/package.json changes. Fast iteration during a focused
branch.
**Path B — manual lifecycle (still supported):**
- `bun test` — unit tests (no database required)
- Follow the "E2E test DB lifecycle" steps above to spin up the test DB,
run `bun run test:e2e`, then tear it down.
Both must pass. Do not ship with failing E2E tests. Do not skip E2E tests.
**Always run typecheck before pushing.** `bun test` (the bun runner)
skips TypeScript type checking — it only enforces runtime behavior.
Three ways to actually gate on types:
1. `bun run test` (npm script in `package.json`) — includes `bun run typecheck`
plus the four shell pre-checks (`check-jsonb-pattern.sh`,
`check-progress-to-stdout.sh`, `check-trailing-newline.sh`,
`check-wasm-embedded.sh`) before the runner. Use this mid-branch.
2. `bun run typecheck` — `tsc --noEmit` standalone. Fast (~5s on this repo).
3. `bun run ci:local` — the full local CI gate from Path A.
The trap is: writing a new test, running `bun test test/foo.test.ts`,
seeing it pass, pushing — and CI's separate typecheck stage rejects an
invalid type literal that the runner accepted. Caught one of these
shipping the v0.23.2 round-trip E2E (`type: 'reflection'` is not a
member of `PageType`). Run `bun run typecheck` once before push, even
when only test files changed.
## CHANGELOG + VERSION are branch-scoped
**VERSION and CHANGELOG describe what THIS branch adds vs master, not how we got
here.** Every feature branch that ships gets its own version bump and CHANGELOG
entry. The entry is product release notes for users; it is not a log of internal
decisions, review rounds, or codex findings.
**Write the CHANGELOG entry at /ship time, not during development.** Mid-branch
iterations, review rounds (CEO/Eng/Codex/DX), and implementation detours belong
in the plan file at `~/.claude/plans/`, not in the CHANGELOG. One unified entry
per branch, covering what the branch added vs the base branch.
**Never edit a CHANGELOG entry that already landed on master.** If master has
v0.18.2 and your branch adds features, bump to the next version (v0.19.0, not
editing master's v0.18.2). When merging master into your branch, master may
bring new CHANGELOG entries above yours — push your entry above master's
latest and verify:
- Does CHANGELOG have your branch's own entry separate from master's entries?
- Is VERSION higher than master's VERSION?
- Is your entry the topmost `## [X.Y.Z]` entry?
- `grep "^## \[" CHANGELOG.md` shows a contiguous version sequence?
If any answer is no, fix it before continuing.
**CHANGELOG is for users, not contributors.** Write like product release notes:
- Lead with what the user can now **do** that they couldn't before. Sell the capability.
- Plain language, not implementation details. "You can now..." not "Refactored the..."
- **Never mention internal artifacts**: plan file IDs, decision tags (D-CX-#, F-ENG-#),
review rounds, codex findings, subcontractor credits. These are invisible to users.
- Put contributor-facing changes in a separate `### For contributors` section at the bottom.
- Every entry should make someone think "oh nice, I want to try that."
**What to omit:**
- "Codex caught X that the CEO review missed" — private process detail.
- "D-CX-3 split errors/warnings" — tag is meaningless to users; name the feature instead.
- "Fix-wave PR #N supersedes #M" — supersede chains belong in PR bodies, not release notes.
- "215 new cases, 3 decisions applied, 7 reviews cleared" — these are planning-mode metrics.
**What to keep:**
- The user-facing change: what commands exist now, what flag was added, what behavior fixed.
- Numbers that mean something to the user: TTHW, commands that timed out before, detection counts.
- Upgrade instructions: `gbrain upgrade` + any manual step if needed.
- Credit to external contributors when a community PR was incorporated.
## CHANGELOG voice + release-summary format
**IRON RULE: the CHANGELOG describes what the user gets, not how the work
happened.** Nobody reading release notes cares that codex caught a bug, that
the plan went through CEO + eng review, that the migration was originally
numbered v68 and renumbered to v79 during master merge, or that two
review rounds caught architectural mistakes. The reader cares what
`gbrain brainstorm` does and how to use it. If a fact only exists because
of the development process, it does NOT belong in the CHANGELOG.
**Specifically forbidden in CHANGELOG entries:**
- Any mention of review processes (CEO review, eng review, codex review,
plan-eng-review, outside voice, adversarial review, autoplan, /review).
- "What we caught and fixed before merging" sections. Bugs found pre-merge
are not changes — they're things that didn't ship.
- Plan file references, plan IDs, plan decision tags (D1, D14, D-CDX-3).
- Migration version drama ("originally v68", "renumbered to v77", "claimed
by parallel waves") — just say "Migration v79 adds X." If the user
cares about migration ordering, they read the diff.
- Round counts, finding counts, decision counts ("25 findings across 2
rounds", "8 architectural decisions", "5/6 expansions accepted").
- Names of internal collaborators ("codex caught", "the reviewer flagged",
"Claude noticed").
- "Plan + reviews" summary bullets. The plan lives in `~/.claude/plans/`;
if a future reader wants the backstory they can grep there.
- Any wording that frames a shipped feature as a *recovery* from a planning
mistake ("the first plan was wrong", "we corrected the approach", "the
shipped version supersedes the original design").
**Smell test:** read the entry as a stranger who has never touched gbrain.
If any sentence makes them think "why are you telling me this?", cut it.
Every sentence in the release-summary AND in the itemized changes must
answer one of three questions: *What can I now do? How do I use it? What
should I watch for after I upgrade?*
Every version entry in `CHANGELOG.md` MUST start with a release-summary section in
the GStack/Garry voice — one viewport's worth of prose + tables that lands like a
verdict, not marketing. The itemized changelog (subsections, bullets, files) goes
BELOW that summary, separated by a `### Itemized changes` header.
The release-summary section gets read by humans, by the auto-update agent, and by
anyone deciding whether to upgrade. The itemized list is for agents that need to
know exactly what changed.
### Release-summary template
**Iron rule: lead ELI10, get precise after.** The first ~150 words of every entry
must be readable by someone who does NOT know gbrain's internals. No file paths,
no function names, no internal constants, no acronyms (no "RRF", no "knobsHash",
no "MODE_BUNDLES", no "CDX-4"), no jargon that requires reading the codebase to
parse. Lead with the user-visible behavior change, in everyday English, like
you're explaining it to a smart engineer who has never opened the repo.
THEN, once the reader knows what shipped and why they'd care, drill into the
precise details: real file paths, real function names, real config keys, real
numbers. The precision part is required (the entry is also the technical record
of what changed), but it lives AFTER the plain-English lead, never before it.
The shape:
1. **One-line bold headline.** What changed for the user, in human English. No
jargon. No internal terms. Example good: "Your search stops boosting weak
pages just because they have a lot of links pointing at them." Example bad:
"PostFusionOpts gains floorRatio; KNOBS_HASH_VERSION bumped 2→3."
2. **Plain-English opener** (~3-5 sentences). Describe the problem this fixes in
everyday terms. Pretend the reader has a brain full of meeting notes and
people pages and wants to know if this release helps them. Concrete example
beats abstract description.
3. **A "How to turn it on" or "How to use it" section** with paste-ready
commands. Real flags, real config keys. This is where precision starts.
4. **A "What you'd see in a concrete example" or "The X numbers that matter"
section** with a table. Use everyday-language column headers ("Page",
"Match quality", "Has many backlinks?") even when the underlying mechanism
is technical. The table teaches what the feature does without requiring the
reader to understand how.
5. **A "What's safe to know about" or "Things to watch" section** for caveats,
side effects, cache invalidation, mid-deploy notes. Still in plain language.
6. **A "What we caught and fixed before merging" section** if the work went
through review (CEO/eng/codex/outside-voice). Translate review findings into
plain English. "We caught a stale-cache bug" beats "knobsHash() did not
include floorRatio in the v=2 hash input."
7. **`### Itemized changes`** (precision lives here). File paths, function
names, types, constants, line numbers. This section is for engineers who
need to know exactly what moved.
Voice rules (apply throughout):
- No em dashes (use commas, periods, "...").
- No AI vocabulary (delve, robust, comprehensive, nuanced, fundamental, etc.) or
banned phrases ("here's the kicker", "the bottom line", etc.).
- Real numbers, real file names, real commands AFTER the ELI10 lead. Not "fast"
but "~30s on 30K pages." In the ELI10 lead, "fast enough that you won't
notice" or "~30 seconds even on a big brain."
- Short paragraphs, mix one-sentence punches with 2-3 sentence runs.
- Connect to user outcomes: "the agent does ~3x less reading" beats "improved
precision."
- Be direct about quality. "Well-designed" or "this is a mess." No dancing.
**The smell test:** if someone who has never opened gbrain reads the first 150
words and walks away knowing what shipped and whether they care, the entry
passes. If they need to grep the codebase to follow along, rewrite the lead.
**Canonical examples in this CHANGELOG:** v0.35.6.0 (floor-ratio gate, written
ELI10-lead-first), v0.34.4.0 (embed stale fix wave). Use those shapes when in
doubt. Avoid the shape of entries that lead with internal constants or release
mechanics; those exist in older history but should not be the model for new
work.
Source material to pull from:
- CHANGELOG.md previous entry for prior context
- Latest `gbrain-evals/docs/benchmarks/[latest].md` for headline numbers (sibling repo)
- Recent commits (`git log <prev-version>..HEAD --oneline`) for what shipped
- Don't make up numbers. If a metric isn't in a benchmark or production data, don't
include it. Say "no measurement yet" if asked.
Target length: ~250-350 words for the summary. Should render as one viewport.
### "To take advantage of v[version]" block (required, v0.13+)
After the release-summary and BEFORE `### Itemized changes`, every `## [X.Y.Z]`
entry MUST include a human-readable self-repair block under the heading
`## To take advantage of v[version]`.
Why: `gbrain upgrade` runs `gbrain post-upgrade` which runs `gbrain apply-migrations`.
This chain has a known weak link — `upgrade.ts` catches post-upgrade failures as
best-effort (so the binary still works). When that chain silently fails, users end
up with half-upgraded brains. The self-repair block gives them a paste-ready
recovery path; the v0.13+ `~/.gbrain/upgrade-errors.jsonl` trail + `gbrain doctor`
integration close the loop.
Template (adapt the verify commands per release):
```markdown
## To take advantage of v[version]
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor`
warns about a partial migration:
1. **Run the orchestrator manually:**
```bash
gbrain apply-migrations --yes
```
2. **Your agent reads `skills/migrations/v[version].md` the next time you interact with it.**
[One sentence on whether headless agents need manual action, or whether the
orchestrator already handled the mechanical side.]
3. **Verify the outcome:**
```bash
[release-specific verify commands, e.g. `gbrain graph ... --depth 2`]
gbrain stats
```
4. **If any step fails or the numbers look wrong,** please file an issue:
https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
- which step broke
This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you.
```
**Skip this block** for patches that are pure bug fixes with zero user-facing action
(rare). If the release has a schema migration, data backfill, or new feature the
user needs to verify, the block is required.
The v0.13.0 entry in CHANGELOG.md is the canonical example.
### Itemized changes (the existing rules)
Below the release summary, write `### Itemized changes` and continue with the
detailed subsections (Knowledge Graph Layer, Schema migrations, Security hardening,
Tests, etc.). Same rules as before:
- Lead with what the user can now DO that they couldn't before
- Frame as benefits and capabilities, not files changed or code written
- Make the user think "hell yeah, I want that"
- Bad: "Added GBRAIN_VERIFY.md installation verification runbook"
- Good: "Your agent now verifies the entire GBrain installation end-to-end, catching
silent sync failures and stale embeddings before they bite you"
- Bad: "Setup skill Phase H and Phase I added"
- Good: "New installs automatically set up live sync so your brain never falls behind"
- **Always credit community contributions.** When a CHANGELOG entry includes work from
a community PR, name the contributor with `Contributed by @username`. Contributors
did real work. Thank them publicly every time, no exceptions.
### Reference: v0.12.0 entry as canonical example
The v0.12.0 entry in CHANGELOG.md is the canonical example of the format. Match its
structure for every future version: bold headline, lead paragraph, "numbers that
matter" with BrainBench-style before/after table, "what this means" closer, then
`### Itemized changes` with the detailed sections below.
## Version migrations
Create a migration file at `skills/migrations/v[version].md` when a release
includes changes that existing users need to act on. The auto-update agent
reads these files post-upgrade (Section 17, Step 4) and executes them.
**You need a migration file when:**
- New setup step that existing installs don't have (e.g., v0.5.0 added live sync,
existing users need to set it up, not just new installs)
- New SKILLPACK section with a MUST ADD setup requirement
- Schema changes that require `gbrain init` or manual SQL
- Changed defaults that affect existing behavior
- Deprecated commands or flags that need replacement
- New verification steps that should run on existing installs
- New cron jobs or background processes that should be registered
**You do NOT need a migration file when:**
- Bug fixes with no behavior changes
- Documentation-only improvements (the agent re-reads docs automatically)
- New optional features that don't affect existing setups
- Performance improvements that are transparent
**The key test:** if an existing user upgrades and does nothing else, will their
brain work worse than before? If yes, migration file. If no, skip it.
Write migration files as agent instructions, not technical notes. Tell the agent
what to do, step by step, with exact commands. See `skills/migrations/v0.5.0.md`
for the pattern.
## Migration is canonical, not advisory
GBrain's job is to deliver a canonical, working setup to every user on upgrade.
Anything that looks like a "host-repo change" — AGENTS.md, cron manifests,
launchctl units, config files outside `~/.gbrain/` — is a GBrain migration
step, not a nudge we leave for the host-repo maintainer. Migrations edit host
files (with backups) to make the canonical setup real. Exceptions: changes
that require human judgment (content edits, renames that break semantics,
host-specific handler registration where shell-exec would be an RCE surface).
Everything mechanical ships in the migration.
**Test:** if shipping a feature requires a sentence that starts with "in
your AGENTS.md, add…" or "in your cron/jobs.json, rewrite…", the migration
orchestrator should be doing that edit, not the user.
**The exception is host-specific code.** For custom Minion handlers
(host-specific integrations like inbox sweeps or third-party API scanners), shipping them as a
data file the worker would exec is an RCE surface. Those get registered in
the host's own repo via the plugin contract (`docs/guides/plugin-handlers.md`);
the migration orchestrator emits a structured TODO to
`~/.gbrain/migrations/pending-host-work.jsonl` + the host agent walks the
TODOs using `skills/migrations/v0.11.0.md` — stays host-agnostic, still
canonical.
## Schema state tracking
`~/.gbrain/update-state.json` tracks which recommended schema directories the user
adopted, declined, or added custom. The auto-update agent (SKILLPACK Section 17)
reads this during upgrades to suggest new schema additions without re-suggesting
things the user already declined. The setup skill writes the initial state during
Phase C/E. Never modify a user's custom directories or re-suggest declined ones.
## GitHub Actions SHA maintenance
All GitHub Actions in `.github/workflows/` are pinned to commit SHAs. Before shipping
(`/ship`) or reviewing (`/review`), check for stale pins and update them:
```bash
for action in actions/checkout oven-sh/setup-bun actions/upload-artifact actions/download-artifact softprops/action-gh-release gitleaks/gitleaks-action; do
tag=$(grep -r "$action@" .github/workflows/ | head -1 | grep -o '#.*' | tr -d '# ')
[ -n "$tag" ] && echo "$action@$tag: $(gh api repos/$action/git/ref/tags/$tag --jq .object.sha 2>/dev/null)"
done
```
If any SHA differs from what's in the workflow files, update the pin and version comment.
## PR descriptions cover the whole branch
Pull request titles and bodies must describe **everything in the PR diff against the
base branch**, not just the most recent commit you made. When you open or update a
PR, walk the full commit range with `git log --oneline <base>..<head>` and write the
body to cover all of it. Group by feature area (schema, code, tests, docs) — not
chronologically by commit.
This matters because reviewers read the PR body to understand what's shipping. If
the body only covers your last commit, they miss everything else and can't review
properly. A 7-commit PR with a body that describes commit 7 is worse than no body
at all — it actively misleads.
When in doubt, run `gh pr view <N> --json commits --jq '[.commits[].messageHeadline]'`
to see what's actually in the PR before writing the body.
## Community PR wave process
Never merge external PRs directly into master. Instead, use the "fix wave" workflow:
1. **Categorize** — group PRs by theme (bug fixes, features, infra, docs)
2. **Deduplicate** — if two PRs fix the same thing, pick the one that changes fewer
lines. Close the other with a note pointing to the winner.
3. **Collector branch** — create a feature branch (e.g. `garrytan/fix-wave-N`), cherry-pick
or manually re-implement the best fixes from each PR. Do NOT merge PR branches directly —
read the diff, understand the fix, and write it yourself if needed.
4. **Test the wave** — verify with `bun test && bun run test:e2e` (full E2E lifecycle).
Every fix in the wave must have test coverage.
5. **Close with context** — every closed PR gets a comment explaining why and what (if
anything) supersedes it. Contributors did real work; respect that with clear communication
and thank them.
6. **Ship as one PR** — single PR to master with all attributions preserved via
`Co-Authored-By:` trailers. Include a summary of what merged and what closed.
**Community PR guardrails:**
- Always AskUserQuestion before accepting commits that touch voice, tone, or
promotional material (README intro, CHANGELOG voice, skill templates).
- Never auto-merge PRs that remove YC references or "neutralize" the founder perspective.
- Preserve contributor attribution in commit messages.
## Checking out PRs from garrytan-agents
`garrytan-agents` is the AI-authored PR account and is NOT a collaborator on
this repo. Its PRs live in a fork, so GitHub Actions triggered by
`pull_request` events on those PRs do not receive base-repo secrets. Any CI
job that needs `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or similar will fail
with empty-env auth errors, regardless of what's set on the base repo. This
is a GitHub security default, not a config bug.
When the user says "check out <PR link>" and the PR is from `garrytan-agents`
(or any other non-collaborator fork), move the branch into the base repo
before running CI:
1. `gh pr checkout <N>` — pull down the fork's branch. Note the PR number and
head branch name (`gh pr view <N> --json headRefName --jq .headRefName`).
2. `git push origin HEAD:<branch-name>` — push the same branch to the base
repo (origin points at `garrytan/gbrain`, not the fork). This is the move
that gives CI access to secrets.
3. `gh pr close <N> --comment "moving to base-repo branch for secret access"`
— close the fork PR so the queue stays clean.
4. `gh pr create --base master --head <branch-name>` — open the replacement
PR from the base-repo branch. **Preserve the original PR's title and body
verbatim** (`gh pr view <N> --json title,body`); contributor attribution
moves to a `Co-Authored-By:` trailer if needed.
Why this over alternatives: adding `garrytan-agents` as a collaborator, or
flipping the repo-wide "send secrets to fork PRs" toggle, both broaden
secret distribution to every fork PR from that account or any fork. Moving
the branch keeps secret scope tight to just the one PR being shipped.
Reference in New Issue View Git Blame Copy Permalink