From 2d3847da00f4c6e7a1d413b7b91864022dfe7f39 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 12 Jun 2026 10:38:25 -0700 Subject: [PATCH] docs: update project documentation for v0.42.43.0 Post-ship doc verification against the release diff (#2095 push-based context + #2084 superset hardening), with a cross-model doc review: - push-context.md: version tag corrected to v0.42.43.0; per-call knobs now cover prior_context/days and watch's flag surface accurately; feedback-log writes described as best-effort; synopsis fence-strip described as unconditional. - CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context release reference corrected to v0.42.43.0. - KEY_FILES.md: ci-local entry rewritten to current topology (4-shard parallel default, four Postgres services, transaction-mode PgBouncer + GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts dropped from the selector entry. - TESTING.md: inventory entries for the new #2084 structural pins (cli-exit-verdict-pin, cli-pipe-truncation), the push-context test suite (volunteer-context, watch-command, watch-sigint.serial, cli-format-volunteer), migrate v117 coverage, and the two new E2E files (pgbouncer-teardown env gating, volunteer-context-postgres RLS pin); check:all row corrected (not a superset of verify). - AGENTS.md + RELEASING.md: ci:local descriptions updated to the sharded + pooler topology. - CHANGELOG (wording only, entry preserved): "retrieved" instead of "opened" for the used-signal, pooler scoped to the local CI gate, feedback log labeled best-effort. - llms-config.ts: index the new push-context guide; bundles regenerated (build:llms) and freshness test green. Co-Authored-By: Claude Fable 5 --- AGENTS.md | 5 +- CHANGELOG.md | 8 ++-- CLAUDE.md | 2 +- docs/RELEASING.md | 18 +++---- docs/TESTING.md | 12 ++++- docs/architecture/KEY_FILES.md | 4 +- docs/guides/push-context.md | 18 ++++--- llms-full.txt | 85 ++++++++++++++++++++++++++++++++-- llms.txt | 1 + scripts/llms-config.ts | 6 +++ 10 files changed, 131 insertions(+), 28 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9efaca672..846a04ce4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -104,8 +104,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont ## Before shipping Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks, -unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a -fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the +guards + typecheck, then 4-shard parallel unit + E2E against four pgvector +containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL` +unset) and tears down. Use `bun run ci:local:diff` for the diff-aware subset during fast iteration on a focused branch. Requires Docker (Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`). diff --git a/CHANGELOG.md b/CHANGELOG.md index 85f07c6cc..846c79d67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,16 +6,16 @@ All notable changes to GBrain will be documented in this file. **The brain now volunteers relevant pages instead of waiting to be asked (gbrain#2095).** Retrieval used to be pull-only: a deep session could run for hours with zero brain contributions — not because the brain had nothing, but because nothing prompted the agent to ask, and pages stored under coined names were missed by literal-string queries. Push-based context inverts that, on three channels sharing one zero-LLM, confidence-gated core: the ambient retrieval reflex now reads the last few conversation turns (an entity your assistant introduced two turns ago resolves on the "what did she invest in?" follow-up), a new `volunteer_context` operation gives any agent a per-turn volunteer surface over CLI stdin or MCP, and `gbrain watch` streams volunteered pages as a transcript flows through it. -Every volunteered page carries an honest confidence (alias match 0.9, exact title 0.8, slug-suffix 0.6, small boosts for repeated or newest-turn mentions; default gate 0.7) and a one-line rationale. A feedback loop closes the tuning circle: volunteered pages are logged, "used" is derived from whether the page actually got opened afterwards, and `gbrain volunteer-context --stats` reports per-arm precision (labeled approximate, because the open signal is throttled). Suppression learned the difference between a page that was actually surfaced and one merely mentioned — under windowing only a surfaced page is held back, so prior-turn mentions can't silence themselves. +Every volunteered page carries an honest confidence (alias match 0.9, exact title 0.8, slug-suffix 0.6, small boosts for repeated or newest-turn mentions; default gate 0.7) and a one-line rationale. A feedback loop closes the tuning circle: volunteered pages are logged, "used" is derived from whether the page actually got retrieved afterwards, and `gbrain volunteer-context --stats` reports per-arm precision (labeled approximate, because the retrieval signal is throttled). Suppression learned the difference between a page that was actually surfaced and one merely mentioned — under windowing only a surfaced page is held back, so prior-turn mentions can't silence themselves. -This release also lands on top of v0.42.42.0's exit-contract work as a strict superset: the transaction-mode pooler topology behind three consecutive teardown waves is now reproduced in CI (a real pooler service + an end-to-end teardown test), the exit-verdict sweep is completed across every command surface (notably `gbrain doctor`, whose FAIL verdict could still report exit 0), and a structural guard makes the next raw exit-code write fail in CI instead of silently reporting success on failure. +This release also lands on top of v0.42.42.0's exit-contract work as a strict superset: the transaction-mode pooler topology behind three consecutive teardown waves is now reproduced in the local CI gate (a real pooler service + an end-to-end teardown test), the exit-verdict sweep is completed across every command surface (notably `gbrain doctor`, whose FAIL verdict could still report exit 0), and a structural guard makes the next raw exit-code write fail in CI instead of silently reporting success on failure. ### Added - **`volunteer_context` operation** (CLI: `gbrain volunteer-context`, MCP tool) — pipe recent turns in (`user:` / `assistant:` prefixed lines, or plain text), get confidence-gated page pointers with rationales and synopses out. `--stats` returns the volunteered-vs-used precision summary. Per-call knobs: `max_pages`, `min_confidence`, `session_id`/`turn` attribution. - **`gbrain watch`** — the streaming push transport: feed a transcript on stdin, volunteered pages stream out (`--json` for JSONL), each slug at most once per session. Piped input exits cleanly at end-of-input; interactive sessions run until Ctrl-C. - **Rolling-window retrieval reflex** — the ambient channel extracts entities from the last 4 turns (configurable via `retrieval_reflex_window_turns` / `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`; 1 restores the previous single-turn behavior). Assistant-introduced entities and named-antecedent follow-ups now surface pointers with zero agent-initiated queries. -- **Volunteered-context feedback log** — every volunteered page is recorded (channel, arm, confidence, optional session/turn) with 90-day retention handled by the nightly cycle; rationales are deterministic templates, never raw conversation text. Synopses pass through the same privacy boundary as `get_page`. -- **Transaction-mode pooler in CI** — the local CI gate now runs a real transaction-pooling service in front of Postgres with an end-to-end teardown test, so the bug class behind gbrain#1972/#2015/#2084 is reproducible before it ships, not after. +- **Volunteered-context feedback log** — volunteered pages are recorded best-effort (channel, arm, confidence, optional session/turn) with 90-day retention handled by the nightly cycle; rationales are deterministic templates, never raw conversation text. Synopses always strip the takes/facts privacy fences before reaching a prompt. +- **Transaction-mode pooler in the local CI gate** — `bun run ci:local` now runs a real transaction-pooling service in front of Postgres with an end-to-end teardown test, so the bug class behind gbrain#1972/#2015/#2084 is reproducible before it ships, not after. ### Fixed - **`gbrain doctor` exits 1 on FAIL again on every engine.** Its verdict write predated the v0.42.42.0 exit-verdict channel and was being silently zeroed; swept, plus a structural test that fails CI on the next raw exit-code write anywhere in the CLI. diff --git a/CLAUDE.md b/CLAUDE.md index d63e9b132..592b5e1dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.43 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP +Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 14ba6765f..3cff19ddc 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -12,15 +12,17 @@ Before shipping (/ship) or reviewing (/review), always run the full test suite. Two equivalent paths: **Path A — local CI gate (recommended, v0.23.1+):** -- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit - tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a - fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to - what nightly Tier 1 catches. Spins up + tears down postgres automatically via - `docker-compose.ci.yml`. Override the host port with - `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides. +- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), + guards + typecheck, then 4-shard parallel unit + E2E against four pgvector + containers plus a transaction-mode PgBouncer service (unit phase keeps + `DATABASE_URL` unset; `--no-shard` for the legacy sequential flow). Stronger + than PR CI's 2-file Tier 1 set; closer to what nightly Tier 1 catches. Spins + up + tears down postgres automatically via `docker-compose.ci.yml`. Override + the host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides. - `bun run ci:local:diff` runs only the E2E files matched by the diff selector - (`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or - schema/skills/package.json changes. Fast iteration during a focused branch. + (`scripts/select-e2e.ts`), falling back to ALL E2E files on unmapped src/ + paths or schema/skills/package.json changes. Fast iteration during a focused + branch. **Path B — manual lifecycle (still supported):** - `bun test` — unit tests (no database required) diff --git a/docs/TESTING.md b/docs/TESTING.md index d39384b98..93b33252c 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -15,7 +15,7 @@ Seven test command tiers, each with a clear scope: | `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. | | `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. | | `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. | -| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. | +| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. | ### CI vs local: intentionally divergent file sets @@ -126,6 +126,12 @@ Unit tests and what they cover: - `test/cli-finish-teardown.test.ts` — the #2084 teardown contract: `computeTeardownDeadlineMs` formula/floor/live-registry scaling + `GBRAIN_TEARDOWN_DEADLINE_MS` override (garbage/zero/negative values fall back to the formula); `finishCliTeardown` clean path (drain BEFORE disconnect, no exit, no warn), backstop on hung drain or disconnect (honors an errored op's exit code), throwing drain/disconnect warned + swallowed; the gbrain-owned verdict channel is immune to PGLite WASM `process.exitCode` writes; `flushThenExit` unit coverage with mocked streams (exits once after both stream callbacks, non-TTY aliveness grace, blocked-pipe guard, EPIPE-safe, `GBRAIN_FLUSH_GRACE_MS` override). - `test/flush-then-exit-harness.test.ts` — real spawned-Bun pipe semantics for `flushThenExit` (fixture: `test/fixtures/flush-then-exit-harness.ts`): a 4MB piped stdout payload arrives byte-complete with the exit code even with a late reader, small output survives exit with a concurrent reader, and the fence resolves promptly (wall time well under the guard + grace ceiling). - `test/cli-should-force-exit.test.ts` — `shouldForceExitAfterMain` daemon-survival gate: `serve` (stdio and `--http`) never force-exits, including with preceding global flags; op commands / empty / flag-only argv do; the #2084 case that space-separated global-flag VALUES can't fake a command (`--timeout 30s serve` resolves to the `serve` daemon, not a `30s` command). +- `test/cli-exit-verdict-pin.test.ts` — #2084 structural class pin: greps `src/` so the NEXT raw `process.exitCode =` write fails CI (a raw write bypasses the gbrain-owned verdict channel and gets silently zeroed by the deliberate flush-exit — the bug that made doctor's FAIL path exit 0). Runtime variants live in `test/cli-finish-teardown.test.ts`; this is the review-time guard. +- `test/cli-pipe-truncation.test.ts` — real-CLI pipe completeness (the #1959 incident class), implementation-agnostic: the actual CLI run the way agents run it (piped stdout) produces complete, parseable, byte-stable `--tools-json` output and exits deliberately, well under the teardown backstop. Synthetic flush-mechanism coverage stays in `test/flush-then-exit-harness.test.ts`. +- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join. +- `test/watch-command.test.ts` — `gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin). +- `test/watch-sigint.serial.test.ts` — `gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`). +- `test/cli-format-volunteer.test.ts` — `formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary. - `test/config.test.ts` — config redaction. - `test/files.test.ts` — MIME/hash. - `test/import-file.test.ts` — import pipeline. @@ -133,7 +139,7 @@ Unit tests and what they cover: - `test/file-migration.test.ts` — file migration. - `test/file-resolver.test.ts` — file resolution. - `test/import-resume.test.ts` — import checkpoints. -- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion. +- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion; v117 `context_volunteer_events` (named + idempotent entry, documented columns + both source-scoped indexes after `initSchema`, insert + 90-day `purgeStaleVolunteerEvents` round-trip). - `test/bootstrap.test.ts` — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on a simulated legacy brain, fresh-install regression guard, legacy `links` shape coverage. - `test/schema-bootstrap-coverage.test.ts` — CI guard. `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in `PGLITE_SCHEMA_SQL`; the test fails loudly if `applyForwardReferenceBootstrap` skips one (extend both arrays when adding a column-with-index to the embedded schema blob). Also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`) and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies — catching the column-only forward-reference class (e.g. `sources.archived`, `oauth_clients.source_id`) that a CREATE INDEX parser alone can't see. `parseBaseTableColumns` strips SQL line + block comments before identifying column names so commented-out lines don't hide adjacent columns. - `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each, snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via `PGLITE_SCHEMA_SQL` or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skips without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`. @@ -223,6 +229,8 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D - `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required). - `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use. - `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner. +- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972→#2015→#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures. +- `test/e2e/volunteer-context-postgres.test.ts` — `volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated. - `test/e2e/openclaw-reference-compat.test.ts` — `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape. - `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory. - `test/e2e/search-exclude.test.ts` — `test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths. diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index c9f51cb57..0819ef4c1 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -323,8 +323,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/markdown.ts` — Frontmatter parsing + body splitter. `coerceFrontmatterString(v)` coerces a non-string `title`/`slug`/`type` to a deterministic string at parse time so a YAML-typed value never reaches `.toLowerCase()` and throws (the #1939 wedge: `title: 2024-06-01` parsed as a `Date`, `title: 1458` as a number, and the throw blocked the sync bookmark from advancing); a `Date` becomes its UTC ISO date (`2024-06-01`, machine-independent and matching the on-disk token, unlike `String(date)`), `null`/`undefined` become `''`, everything else uses `String()`. `splitBody` requires an explicit timeline sentinel (``, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus existing people/companies/deals/etc heuristics). - `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (must be DEFAULT 5 to preserve SIGKILL-rescue). Wired into `bun test`. - `scripts/check-source-id-projection.sh` — CI grep guard for the multi-source bug class. Greps `src/core/postgres-engine.ts` + `src/core/pglite-engine.ts` for `SELECT.*FROM pages` projections matching the `rowToPage` feeder shape (id + slug + type + title) and fails if `source_id` is missing. `Page.source_id` is required at the type level; a projection dropping the column produces `Page` rows with `source_id: undefined` while TypeScript's `: string` lies about it. Wired into `bun run verify` + `bun run check:all`. -- `docker-compose.ci.yml` + `scripts/ci-local.sh` — Local CI gate. `bun run ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` with named volumes (`gbrain-ci-pg-data`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs unit tests with `DATABASE_URL` unset, then runs all 29 E2E files sequentially. `--diff` swaps in the diff-aware selector; `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434; override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than PR CI's 2-file Tier 1 set. -- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed: EMPTY → all 29 files; DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout; SRC → escape-hatch paths (schema, package.json, skills/) trigger all, else the hand-tuned `E2E_TEST_MAP` glob narrows, and an unmapped src/ change still emits ALL files (never silently nothing). Pure-function exports `selectTests`, `classify`, `matchGlob`. `bun run ci:select-e2e` prints the current selection on stdout. `test/select-e2e.test.ts` covers all 4 branches plus 3 regression guards (skills/, untracked files, unmapped src/) — 24 cases. +- `docker-compose.ci.yml` + `scripts/ci-local.sh` — Local CI gate. `bun run ci:local` spins up four `pgvector/pgvector:pg16` services (postgres-1..4) + `oven/bun:1` with named volumes (`gbrain-ci-pg-data-{1..4}`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs guards + typecheck, then the Tier 1 default: 4-shard parallel unit + E2E (`xargs -P4`, one Postgres per shard; unit phase keeps `DATABASE_URL` unset). `--no-shard` falls back to the legacy unsharded sequential flow (debug aid); `--diff` runs the diff-aware selector unsharded. Also runs a `pgbouncer` service (`edoburu/pgbouncer`, `POOL_MODE: transaction`, `AUTH_TYPE: plain` — pg16 stores SCRAM verifiers, so the userlist must hold the plaintext password; `IGNORE_STARTUP_PARAMETERS` whitelists gbrain's `statement_timeout`/`idle_in_transaction_session_timeout` startup params the way the Supabase pooler does) fronting postgres-1 on host port `GBRAIN_CI_PGBOUNCER_PORT` (default 6543); every E2E invocation exports `GBRAIN_PGBOUNCER_URL` (pooled; dedicated `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures) + `GBRAIN_PGBOUNCER_DIRECT_URL`, consumed by `test/e2e/pgbouncer-teardown.test.ts` — the transaction-mode teardown bug class (#1972/#2015/#2084) reproduced in the local gate instead of only in production. `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434; override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than PR CI's 2-file Tier 1 set. +- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed: EMPTY → all files; DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout; SRC → escape-hatch paths (schema, package.json, skills/) trigger all, else the hand-tuned `E2E_TEST_MAP` glob narrows, and an unmapped src/ change still emits ALL files (never silently nothing). Pure-function exports `selectTests`, `classify`, `matchGlob`. `bun run ci:select-e2e` prints the current selection on stdout. `test/select-e2e.test.ts` covers all 4 branches plus 3 regression guards (skills/, untracked files, unmapped src/) — 24 cases. - `scripts/run-e2e.sh` — Sequential E2E runner. Accepts an optional argv-driven file list (used by `ci:local:diff`) and a `--dry-run-list` flag that prints the resolved file list and exits (used by `ci-local.sh`'s startup smoke-test). Falls back to `test/e2e/*.test.ts` when invoked with no args. - `scripts/llms-config.ts` + `scripts/build-llms.ts` — Generator for `llms.txt` (llmstxt.org-spec web index) + `llms-full.txt` (inlined single-fetch bundle). Curated config drives both. Run `bun run build:llms` after adding a new doc. `LLMS_REPO_BASE` env lets forks regenerate with their own URL base. `FULL_SIZE_BUDGET` (600KB) caps the inline bundle; generator WARNs if exceeded. Committed output has no runtime consumer; committed for GitHub browsing and fork-safe fetching. - `AGENTS.md` — Local-clone entry point for non-Claude agents (Codex, Cursor, OpenClaw, Aider). Mirrors `CLAUDE.md` intent via relative links. Claude Code keeps using `CLAUDE.md`. diff --git a/docs/guides/push-context.md b/docs/guides/push-context.md index 5dd23b87e..801ac5a28 100644 --- a/docs/guides/push-context.md +++ b/docs/guides/push-context.md @@ -1,4 +1,4 @@ -# Push-based context (#2095, v0.43) +# Push-based context (#2095, v0.42.43.0) Retrieval used to be pull-only: the agent had to *know to ask* before the brain contributed anything. Push-based context inverts that — the brain volunteers @@ -53,13 +53,19 @@ precision to tune `min_confidence`, not as an exact metric. | `retrieval_reflex` | true | the ambient channel's master switch | | `retrieval_reflex_max_pointers` | 3 | pointer cap per turn | -Per-call knobs on the op/watch: `max_pages`, `min_confidence`, `session_id`, -`turn` (attribution columns in the feedback log). +Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch` +(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch); +on the op only: `prior_context` (text whose already-surfaced slugs are suppressed), +`session_id` / `turn` attribution params (watch stamps its own per-session id and +turn numbers in the feedback log), and `days` to size the `--stats` window. ## Storage + privacy Volunteered pages log to `context_volunteer_events` (migration v117): slug, arm, confidence, channel, optional session/turn — the rationale is a -deterministic template string, never raw conversation text. Rows are pruned -after 90 days by the dream cycle's purge phase. Synopses pass through the same -takes/facts-fence privacy boundary as `get_page`. +deterministic template string, never raw conversation text. Event writes are +best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal, +not an audit trail. Rows are pruned after 90 days by the dream cycle's purge +phase. Synopses always strip the takes/facts fences — the same strip `get_page` +applies to untrusted callers, applied unconditionally here so private fence rows +never reach a prompt regardless of caller trust. diff --git a/llms-full.txt b/llms-full.txt index 4a2b5a38f..389fd1b6a 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -117,8 +117,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont ## Before shipping Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks, -unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a -fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the +guards + typecheck, then 4-shard parallel unit + E2E against four pgvector +containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL` +unset) and tears down. Use `bun run ci:local:diff` for the diff-aware subset during fast iteration on a focused branch. Requires Docker (Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`). @@ -186,7 +187,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.43 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP +Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). @@ -3373,6 +3374,84 @@ the bundled resolver lives at [`skills/RESOLVER.md`](../../skills/RESOLVER.md). --- +## docs/guides/push-context.md + +Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md + +# Push-based context (#2095, v0.42.43.0) + +Retrieval used to be pull-only: the agent had to *know to ask* before the brain +contributed anything. Push-based context inverts that — the brain volunteers +relevant pages from the recent conversation, confidence-gated so push noise +never becomes worse than pull silence. + +Three channels share one zero-LLM core (`src/core/context/volunteer.ts`): + +| Channel | Surface | When to use | +|---|---|---| +| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call | +| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn | +| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out | + +## How it decides + +1. **Extract** entities across the last N turns (capitalized runs, `@handles`), + merged with recency / frequency / user-role salience. Assistant-introduced + entities and "what did she invest in?" follow-ups whose antecedent was named + in the window now resolve. +2. **Resolve** through the alias table, exact titles, and slug suffixes — each + arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6, + +0.05 when mentioned in ≥2 turns or the newest turn. +3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an + explicit lower gate), suppress pages already surfaced (slug-presence only), + cap at 3 pages (hard cap 5). + +## CLI + +```bash +# one-shot: pipe recent turns (oldest → newest) +printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \ + | gbrain volunteer-context + +# streaming: volunteered pages print as the transcript flows +some-transcript-feed | gbrain watch --json + +# the feedback loop: how often were volunteered pages actually opened? +gbrain volunteer-context --stats +``` + +Stats are **approximate** by design: "used" means `pages.last_retrieved_at > +volunteered_at` — the 5-minute last-retrieved throttle causes false negatives +and unrelated reads of the same page cause false positives. Use the per-arm +precision to tune `min_confidence`, not as an exact metric. + +## Config + +| Key | Default | What it does | +|---|---|---| +| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) | +| `retrieval_reflex` | true | the ambient channel's master switch | +| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn | + +Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch` +(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch); +on the op only: `prior_context` (text whose already-surfaced slugs are suppressed), +`session_id` / `turn` attribution params (watch stamps its own per-session id and +turn numbers in the feedback log), and `days` to size the `--stats` window. + +## Storage + privacy + +Volunteered pages log to `context_volunteer_events` (migration v117): slug, +arm, confidence, channel, optional session/turn — the rationale is a +deterministic template string, never raw conversation text. Event writes are +best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal, +not an audit trail. Rows are pruned after 90 days by the dream cycle's purge +phase. Synopses always strip the takes/facts fences — the same strip `get_page` +applies to untrusted callers, applied unconditionally here so private fence rows +never reach a prompt regardless of caller trust. + +--- + ## docs/mcp/DEPLOY.md Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md diff --git a/llms.txt b/llms.txt index 8ca28d88a..34d94b6d4 100644 --- a/llms.txt +++ b/llms.txt @@ -25,6 +25,7 @@ Repo: https://github.com/garrytan/gbrain - [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist. - [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery. - [docs/guides/scaling-skills.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/scaling-skills.md): Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss. +- [docs/guides/push-context.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md): Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop. - [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment. ## AI providers diff --git a/scripts/llms-config.ts b/scripts/llms-config.ts index e81154c6a..b2b338a86 100644 --- a/scripts/llms-config.ts +++ b/scripts/llms-config.ts @@ -151,6 +151,12 @@ export const SECTIONS: DocSection[] = [ "Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.", path: "docs/guides/scaling-skills.md", }, + { + title: "docs/guides/push-context.md", + description: + "Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.", + path: "docs/guides/push-context.md", + }, { title: "docs/mcp/DEPLOY.md", description: "MCP server deployment.",