mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
docs: update project documentation for v0.42.43.0
Post-ship doc verification against the release diff (#2095 push-based context + #2084 superset hardening), with a cross-model doc review: - push-context.md: version tag corrected to v0.42.43.0; per-call knobs now cover prior_context/days and watch's flag surface accurately; feedback-log writes described as best-effort; synopsis fence-strip described as unconditional. - CLAUDE.md: stale operation count (~47 -> ~90); volunteer_context release reference corrected to v0.42.43.0. - KEY_FILES.md: ci-local entry rewritten to current topology (4-shard parallel default, four Postgres services, transaction-mode PgBouncer + GBRAIN_PGBOUNCER_URL/_DIRECT_URL exports); stale E2E file counts dropped from the selector entry. - TESTING.md: inventory entries for the new #2084 structural pins (cli-exit-verdict-pin, cli-pipe-truncation), the push-context test suite (volunteer-context, watch-command, watch-sigint.serial, cli-format-volunteer), migrate v117 coverage, and the two new E2E files (pgbouncer-teardown env gating, volunteer-context-postgres RLS pin); check:all row corrected (not a superset of verify). - AGENTS.md + RELEASING.md: ci:local descriptions updated to the sharded + pooler topology. - CHANGELOG (wording only, entry preserved): "retrieved" instead of "opened" for the used-signal, pooler scoped to the local CI gate, feedback log labeled best-effort. - llms-config.ts: index the new push-context guide; bundles regenerated (build:llms) and freshness test green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Fable 5
Claude Fable 5
parent
ce794970b7
commit
2d3847da00
@@ -104,8 +104,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
## Before shipping
|
||||
|
||||
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
|
||||
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
|
||||
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
|
||||
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
|
||||
containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL`
|
||||
unset) and tears down. Use `bun run ci:local:diff` for the
|
||||
diff-aware subset during fast iteration on a focused branch. Requires Docker
|
||||
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
|
||||
|
||||
|
||||
+4
-4
@@ -6,16 +6,16 @@ All notable changes to GBrain will be documented in this file.
|
||||
|
||||
**The brain now volunteers relevant pages instead of waiting to be asked (gbrain#2095).** Retrieval used to be pull-only: a deep session could run for hours with zero brain contributions — not because the brain had nothing, but because nothing prompted the agent to ask, and pages stored under coined names were missed by literal-string queries. Push-based context inverts that, on three channels sharing one zero-LLM, confidence-gated core: the ambient retrieval reflex now reads the last few conversation turns (an entity your assistant introduced two turns ago resolves on the "what did she invest in?" follow-up), a new `volunteer_context` operation gives any agent a per-turn volunteer surface over CLI stdin or MCP, and `gbrain watch` streams volunteered pages as a transcript flows through it.
|
||||
|
||||
Every volunteered page carries an honest confidence (alias match 0.9, exact title 0.8, slug-suffix 0.6, small boosts for repeated or newest-turn mentions; default gate 0.7) and a one-line rationale. A feedback loop closes the tuning circle: volunteered pages are logged, "used" is derived from whether the page actually got opened afterwards, and `gbrain volunteer-context --stats` reports per-arm precision (labeled approximate, because the open signal is throttled). Suppression learned the difference between a page that was actually surfaced and one merely mentioned — under windowing only a surfaced page is held back, so prior-turn mentions can't silence themselves.
|
||||
Every volunteered page carries an honest confidence (alias match 0.9, exact title 0.8, slug-suffix 0.6, small boosts for repeated or newest-turn mentions; default gate 0.7) and a one-line rationale. A feedback loop closes the tuning circle: volunteered pages are logged, "used" is derived from whether the page actually got retrieved afterwards, and `gbrain volunteer-context --stats` reports per-arm precision (labeled approximate, because the retrieval signal is throttled). Suppression learned the difference between a page that was actually surfaced and one merely mentioned — under windowing only a surfaced page is held back, so prior-turn mentions can't silence themselves.
|
||||
|
||||
This release also lands on top of v0.42.42.0's exit-contract work as a strict superset: the transaction-mode pooler topology behind three consecutive teardown waves is now reproduced in CI (a real pooler service + an end-to-end teardown test), the exit-verdict sweep is completed across every command surface (notably `gbrain doctor`, whose FAIL verdict could still report exit 0), and a structural guard makes the next raw exit-code write fail in CI instead of silently reporting success on failure.
|
||||
This release also lands on top of v0.42.42.0's exit-contract work as a strict superset: the transaction-mode pooler topology behind three consecutive teardown waves is now reproduced in the local CI gate (a real pooler service + an end-to-end teardown test), the exit-verdict sweep is completed across every command surface (notably `gbrain doctor`, whose FAIL verdict could still report exit 0), and a structural guard makes the next raw exit-code write fail in CI instead of silently reporting success on failure.
|
||||
|
||||
### Added
|
||||
- **`volunteer_context` operation** (CLI: `gbrain volunteer-context`, MCP tool) — pipe recent turns in (`user:` / `assistant:` prefixed lines, or plain text), get confidence-gated page pointers with rationales and synopses out. `--stats` returns the volunteered-vs-used precision summary. Per-call knobs: `max_pages`, `min_confidence`, `session_id`/`turn` attribution.
|
||||
- **`gbrain watch`** — the streaming push transport: feed a transcript on stdin, volunteered pages stream out (`--json` for JSONL), each slug at most once per session. Piped input exits cleanly at end-of-input; interactive sessions run until Ctrl-C.
|
||||
- **Rolling-window retrieval reflex** — the ambient channel extracts entities from the last 4 turns (configurable via `retrieval_reflex_window_turns` / `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`; 1 restores the previous single-turn behavior). Assistant-introduced entities and named-antecedent follow-ups now surface pointers with zero agent-initiated queries.
|
||||
- **Volunteered-context feedback log** — every volunteered page is recorded (channel, arm, confidence, optional session/turn) with 90-day retention handled by the nightly cycle; rationales are deterministic templates, never raw conversation text. Synopses pass through the same privacy boundary as `get_page`.
|
||||
- **Transaction-mode pooler in CI** — the local CI gate now runs a real transaction-pooling service in front of Postgres with an end-to-end teardown test, so the bug class behind gbrain#1972/#2015/#2084 is reproducible before it ships, not after.
|
||||
- **Volunteered-context feedback log** — volunteered pages are recorded best-effort (channel, arm, confidence, optional session/turn) with 90-day retention handled by the nightly cycle; rationales are deterministic templates, never raw conversation text. Synopses always strip the takes/facts privacy fences before reaching a prompt.
|
||||
- **Transaction-mode pooler in the local CI gate** — `bun run ci:local` now runs a real transaction-pooling service in front of Postgres with an end-to-end teardown test, so the bug class behind gbrain#1972/#2015/#2084 is reproducible before it ships, not after.
|
||||
|
||||
### Fixed
|
||||
- **`gbrain doctor` exits 1 on FAIL again on every engine.** Its verdict write predated the v0.42.42.0 exit-verdict channel and was being silently zeroed; swept, plus a structural test that fails CI on the next raw exit-code write anywhere in the CLI.
|
||||
|
||||
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
|
||||
|
||||
## Architecture
|
||||
|
||||
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.43 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
|
||||
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
|
||||
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
|
||||
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
|
||||
markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
|
||||
+10
-8
@@ -12,15 +12,17 @@ Before shipping (/ship) or reviewing (/review), always run the full test suite.
|
||||
Two equivalent paths:
|
||||
|
||||
**Path A — local CI gate (recommended, v0.23.1+):**
|
||||
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit
|
||||
tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a
|
||||
fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to
|
||||
what nightly Tier 1 catches. Spins up + tears down postgres automatically via
|
||||
`docker-compose.ci.yml`. Override the host port with
|
||||
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
|
||||
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host),
|
||||
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
|
||||
containers plus a transaction-mode PgBouncer service (unit phase keeps
|
||||
`DATABASE_URL` unset; `--no-shard` for the legacy sequential flow). Stronger
|
||||
than PR CI's 2-file Tier 1 set; closer to what nightly Tier 1 catches. Spins
|
||||
up + tears down postgres automatically via `docker-compose.ci.yml`. Override
|
||||
the host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
|
||||
- `bun run ci:local:diff` runs only the E2E files matched by the diff selector
|
||||
(`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or
|
||||
schema/skills/package.json changes. Fast iteration during a focused branch.
|
||||
(`scripts/select-e2e.ts`), falling back to ALL E2E files on unmapped src/
|
||||
paths or schema/skills/package.json changes. Fast iteration during a focused
|
||||
branch.
|
||||
|
||||
**Path B — manual lifecycle (still supported):**
|
||||
- `bun test` — unit tests (no database required)
|
||||
|
||||
+10
-2
@@ -15,7 +15,7 @@ Seven test command tiers, each with a clear scope:
|
||||
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
|
||||
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
|
||||
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
|
||||
| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. |
|
||||
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
|
||||
|
||||
### CI vs local: intentionally divergent file sets
|
||||
|
||||
@@ -126,6 +126,12 @@ Unit tests and what they cover:
|
||||
- `test/cli-finish-teardown.test.ts` — the #2084 teardown contract: `computeTeardownDeadlineMs` formula/floor/live-registry scaling + `GBRAIN_TEARDOWN_DEADLINE_MS` override (garbage/zero/negative values fall back to the formula); `finishCliTeardown` clean path (drain BEFORE disconnect, no exit, no warn), backstop on hung drain or disconnect (honors an errored op's exit code), throwing drain/disconnect warned + swallowed; the gbrain-owned verdict channel is immune to PGLite WASM `process.exitCode` writes; `flushThenExit` unit coverage with mocked streams (exits once after both stream callbacks, non-TTY aliveness grace, blocked-pipe guard, EPIPE-safe, `GBRAIN_FLUSH_GRACE_MS` override).
|
||||
- `test/flush-then-exit-harness.test.ts` — real spawned-Bun pipe semantics for `flushThenExit` (fixture: `test/fixtures/flush-then-exit-harness.ts`): a 4MB piped stdout payload arrives byte-complete with the exit code even with a late reader, small output survives exit with a concurrent reader, and the fence resolves promptly (wall time well under the guard + grace ceiling).
|
||||
- `test/cli-should-force-exit.test.ts` — `shouldForceExitAfterMain` daemon-survival gate: `serve` (stdio and `--http`) never force-exits, including with preceding global flags; op commands / empty / flag-only argv do; the #2084 case that space-separated global-flag VALUES can't fake a command (`--timeout 30s serve` resolves to the `serve` daemon, not a `30s` command).
|
||||
- `test/cli-exit-verdict-pin.test.ts` — #2084 structural class pin: greps `src/` so the NEXT raw `process.exitCode =` write fails CI (a raw write bypasses the gbrain-owned verdict channel and gets silently zeroed by the deliberate flush-exit — the bug that made doctor's FAIL path exit 0). Runtime variants live in `test/cli-finish-teardown.test.ts`; this is the review-time guard.
|
||||
- `test/cli-pipe-truncation.test.ts` — real-CLI pipe completeness (the #1959 incident class), implementation-agnostic: the actual CLI run the way agents run it (piped stdout) produces complete, parseable, byte-stable `--tools-json` output and exits deliberately, well under the teardown backstop. Synthetic flush-mechanism coverage stays in `test/flush-then-exit-harness.test.ts`.
|
||||
- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join.
|
||||
- `test/watch-command.test.ts` — `gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin).
|
||||
- `test/watch-sigint.serial.test.ts` — `gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`).
|
||||
- `test/cli-format-volunteer.test.ts` — `formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary.
|
||||
- `test/config.test.ts` — config redaction.
|
||||
- `test/files.test.ts` — MIME/hash.
|
||||
- `test/import-file.test.ts` — import pipeline.
|
||||
@@ -133,7 +139,7 @@ Unit tests and what they cover:
|
||||
- `test/file-migration.test.ts` — file migration.
|
||||
- `test/file-resolver.test.ts` — file resolution.
|
||||
- `test/import-resume.test.ts` — import checkpoints.
|
||||
- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion.
|
||||
- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion; v117 `context_volunteer_events` (named + idempotent entry, documented columns + both source-scoped indexes after `initSchema`, insert + 90-day `purgeStaleVolunteerEvents` round-trip).
|
||||
- `test/bootstrap.test.ts` — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on a simulated legacy brain, fresh-install regression guard, legacy `links` shape coverage.
|
||||
- `test/schema-bootstrap-coverage.test.ts` — CI guard. `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in `PGLITE_SCHEMA_SQL`; the test fails loudly if `applyForwardReferenceBootstrap` skips one (extend both arrays when adding a column-with-index to the embedded schema blob). Also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`) and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies — catching the column-only forward-reference class (e.g. `sources.archived`, `oauth_clients.source_id`) that a CREATE INDEX parser alone can't see. `parseBaseTableColumns` strips SQL line + block comments before identifying column names so commented-out lines don't hide adjacent columns.
|
||||
- `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each, snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via `PGLITE_SCHEMA_SQL` or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skips without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`.
|
||||
@@ -223,6 +229,8 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
|
||||
- `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required).
|
||||
- `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use.
|
||||
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
|
||||
- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972→#2015→#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures.
|
||||
- `test/e2e/volunteer-context-postgres.test.ts` — `volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated.
|
||||
- `test/e2e/openclaw-reference-compat.test.ts` — `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
|
||||
- `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `<fork>/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
|
||||
- `test/e2e/search-exclude.test.ts` — `test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths.
|
||||
|
||||
@@ -323,8 +323,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/markdown.ts` — Frontmatter parsing + body splitter. `coerceFrontmatterString(v)` coerces a non-string `title`/`slug`/`type` to a deterministic string at parse time so a YAML-typed value never reaches `.toLowerCase()` and throws (the #1939 wedge: `title: 2024-06-01` parsed as a `Date`, `title: 1458` as a number, and the throw blocked the sync bookmark from advancing); a `Date` becomes its UTC ISO date (`2024-06-01`, machine-independent and matching the on-disk token, unlike `String(date)`), `null`/`undefined` become `''`, everything else uses `String()`. `splitBody` requires an explicit timeline sentinel (`<!-- timeline -->`, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus existing people/companies/deals/etc heuristics).
|
||||
- `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (must be DEFAULT 5 to preserve SIGKILL-rescue). Wired into `bun test`.
|
||||
- `scripts/check-source-id-projection.sh` — CI grep guard for the multi-source bug class. Greps `src/core/postgres-engine.ts` + `src/core/pglite-engine.ts` for `SELECT.*FROM pages` projections matching the `rowToPage` feeder shape (id + slug + type + title) and fails if `source_id` is missing. `Page.source_id` is required at the type level; a projection dropping the column produces `Page` rows with `source_id: undefined` while TypeScript's `: string` lies about it. Wired into `bun run verify` + `bun run check:all`.
|
||||
- `docker-compose.ci.yml` + `scripts/ci-local.sh` — Local CI gate. `bun run ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` with named volumes (`gbrain-ci-pg-data`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs unit tests with `DATABASE_URL` unset, then runs all 29 E2E files sequentially. `--diff` swaps in the diff-aware selector; `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434; override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than PR CI's 2-file Tier 1 set.
|
||||
- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed: EMPTY → all 29 files; DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout; SRC → escape-hatch paths (schema, package.json, skills/) trigger all, else the hand-tuned `E2E_TEST_MAP` glob narrows, and an unmapped src/ change still emits ALL files (never silently nothing). Pure-function exports `selectTests`, `classify`, `matchGlob`. `bun run ci:select-e2e` prints the current selection on stdout. `test/select-e2e.test.ts` covers all 4 branches plus 3 regression guards (skills/, untracked files, unmapped src/) — 24 cases.
|
||||
- `docker-compose.ci.yml` + `scripts/ci-local.sh` — Local CI gate. `bun run ci:local` spins up four `pgvector/pgvector:pg16` services (postgres-1..4) + `oven/bun:1` with named volumes (`gbrain-ci-pg-data-{1..4}`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs guards + typecheck, then the Tier 1 default: 4-shard parallel unit + E2E (`xargs -P4`, one Postgres per shard; unit phase keeps `DATABASE_URL` unset). `--no-shard` falls back to the legacy unsharded sequential flow (debug aid); `--diff` runs the diff-aware selector unsharded. Also runs a `pgbouncer` service (`edoburu/pgbouncer`, `POOL_MODE: transaction`, `AUTH_TYPE: plain` — pg16 stores SCRAM verifiers, so the userlist must hold the plaintext password; `IGNORE_STARTUP_PARAMETERS` whitelists gbrain's `statement_timeout`/`idle_in_transaction_session_timeout` startup params the way the Supabase pooler does) fronting postgres-1 on host port `GBRAIN_CI_PGBOUNCER_PORT` (default 6543); every E2E invocation exports `GBRAIN_PGBOUNCER_URL` (pooled; dedicated `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures) + `GBRAIN_PGBOUNCER_DIRECT_URL`, consumed by `test/e2e/pgbouncer-teardown.test.ts` — the transaction-mode teardown bug class (#1972/#2015/#2084) reproduced in the local gate instead of only in production. `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434; override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than PR CI's 2-file Tier 1 set.
|
||||
- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed: EMPTY → all files; DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout; SRC → escape-hatch paths (schema, package.json, skills/) trigger all, else the hand-tuned `E2E_TEST_MAP` glob narrows, and an unmapped src/ change still emits ALL files (never silently nothing). Pure-function exports `selectTests`, `classify`, `matchGlob`. `bun run ci:select-e2e` prints the current selection on stdout. `test/select-e2e.test.ts` covers all 4 branches plus 3 regression guards (skills/, untracked files, unmapped src/) — 24 cases.
|
||||
- `scripts/run-e2e.sh` — Sequential E2E runner. Accepts an optional argv-driven file list (used by `ci:local:diff`) and a `--dry-run-list` flag that prints the resolved file list and exits (used by `ci-local.sh`'s startup smoke-test). Falls back to `test/e2e/*.test.ts` when invoked with no args.
|
||||
- `scripts/llms-config.ts` + `scripts/build-llms.ts` — Generator for `llms.txt` (llmstxt.org-spec web index) + `llms-full.txt` (inlined single-fetch bundle). Curated config drives both. Run `bun run build:llms` after adding a new doc. `LLMS_REPO_BASE` env lets forks regenerate with their own URL base. `FULL_SIZE_BUDGET` (600KB) caps the inline bundle; generator WARNs if exceeded. Committed output has no runtime consumer; committed for GitHub browsing and fork-safe fetching.
|
||||
- `AGENTS.md` — Local-clone entry point for non-Claude agents (Codex, Cursor, OpenClaw, Aider). Mirrors `CLAUDE.md` intent via relative links. Claude Code keeps using `CLAUDE.md`.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Push-based context (#2095, v0.43)
|
||||
# Push-based context (#2095, v0.42.43.0)
|
||||
|
||||
Retrieval used to be pull-only: the agent had to *know to ask* before the brain
|
||||
contributed anything. Push-based context inverts that — the brain volunteers
|
||||
@@ -53,13 +53,19 @@ precision to tune `min_confidence`, not as an exact metric.
|
||||
| `retrieval_reflex` | true | the ambient channel's master switch |
|
||||
| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn |
|
||||
|
||||
Per-call knobs on the op/watch: `max_pages`, `min_confidence`, `session_id`,
|
||||
`turn` (attribution columns in the feedback log).
|
||||
Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch`
|
||||
(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch);
|
||||
on the op only: `prior_context` (text whose already-surfaced slugs are suppressed),
|
||||
`session_id` / `turn` attribution params (watch stamps its own per-session id and
|
||||
turn numbers in the feedback log), and `days` to size the `--stats` window.
|
||||
|
||||
## Storage + privacy
|
||||
|
||||
Volunteered pages log to `context_volunteer_events` (migration v117): slug,
|
||||
arm, confidence, channel, optional session/turn — the rationale is a
|
||||
deterministic template string, never raw conversation text. Rows are pruned
|
||||
after 90 days by the dream cycle's purge phase. Synopses pass through the same
|
||||
takes/facts-fence privacy boundary as `get_page`.
|
||||
deterministic template string, never raw conversation text. Event writes are
|
||||
best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal,
|
||||
not an audit trail. Rows are pruned after 90 days by the dream cycle's purge
|
||||
phase. Synopses always strip the takes/facts fences — the same strip `get_page`
|
||||
applies to untrusted callers, applied unconditionally here so private fence rows
|
||||
never reach a prompt regardless of caller trust.
|
||||
|
||||
+82
-3
@@ -117,8 +117,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
|
||||
## Before shipping
|
||||
|
||||
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
|
||||
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
|
||||
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
|
||||
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
|
||||
containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL`
|
||||
unset) and tears down. Use `bun run ci:local:diff` for the
|
||||
diff-aware subset during fast iteration on a focused branch. Requires Docker
|
||||
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
|
||||
|
||||
@@ -186,7 +187,7 @@ mount, CEO-class with multiple team brains) and
|
||||
|
||||
## Architecture
|
||||
|
||||
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.43 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
|
||||
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
|
||||
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
|
||||
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
|
||||
markdown files (tool-agnostic, work with both CLI and plugin contexts).
|
||||
@@ -3373,6 +3374,84 @@ the bundled resolver lives at [`skills/RESOLVER.md`](../../skills/RESOLVER.md).
|
||||
|
||||
---
|
||||
|
||||
## docs/guides/push-context.md
|
||||
|
||||
Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md
|
||||
|
||||
# Push-based context (#2095, v0.42.43.0)
|
||||
|
||||
Retrieval used to be pull-only: the agent had to *know to ask* before the brain
|
||||
contributed anything. Push-based context inverts that — the brain volunteers
|
||||
relevant pages from the recent conversation, confidence-gated so push noise
|
||||
never becomes worse than pull silence.
|
||||
|
||||
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
|
||||
|
||||
| Channel | Surface | When to use |
|
||||
|---|---|---|
|
||||
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
|
||||
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
|
||||
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
|
||||
|
||||
## How it decides
|
||||
|
||||
1. **Extract** entities across the last N turns (capitalized runs, `@handles`),
|
||||
merged with recency / frequency / user-role salience. Assistant-introduced
|
||||
entities and "what did she invest in?" follow-ups whose antecedent was named
|
||||
in the window now resolve.
|
||||
2. **Resolve** through the alias table, exact titles, and slug suffixes — each
|
||||
arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6,
|
||||
+0.05 when mentioned in ≥2 turns or the newest turn.
|
||||
3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an
|
||||
explicit lower gate), suppress pages already surfaced (slug-presence only),
|
||||
cap at 3 pages (hard cap 5).
|
||||
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
# one-shot: pipe recent turns (oldest → newest)
|
||||
printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \
|
||||
| gbrain volunteer-context
|
||||
|
||||
# streaming: volunteered pages print as the transcript flows
|
||||
some-transcript-feed | gbrain watch --json
|
||||
|
||||
# the feedback loop: how often were volunteered pages actually opened?
|
||||
gbrain volunteer-context --stats
|
||||
```
|
||||
|
||||
Stats are **approximate** by design: "used" means `pages.last_retrieved_at >
|
||||
volunteered_at` — the 5-minute last-retrieved throttle causes false negatives
|
||||
and unrelated reads of the same page cause false positives. Use the per-arm
|
||||
precision to tune `min_confidence`, not as an exact metric.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | What it does |
|
||||
|---|---|---|
|
||||
| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) |
|
||||
| `retrieval_reflex` | true | the ambient channel's master switch |
|
||||
| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn |
|
||||
|
||||
Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch`
|
||||
(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch);
|
||||
on the op only: `prior_context` (text whose already-surfaced slugs are suppressed),
|
||||
`session_id` / `turn` attribution params (watch stamps its own per-session id and
|
||||
turn numbers in the feedback log), and `days` to size the `--stats` window.
|
||||
|
||||
## Storage + privacy
|
||||
|
||||
Volunteered pages log to `context_volunteer_events` (migration v117): slug,
|
||||
arm, confidence, channel, optional session/turn — the rationale is a
|
||||
deterministic template string, never raw conversation text. Event writes are
|
||||
best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal,
|
||||
not an audit trail. Rows are pruned after 90 days by the dream cycle's purge
|
||||
phase. Synopses always strip the takes/facts fences — the same strip `get_page`
|
||||
applies to untrusted callers, applied unconditionally here so private fence rows
|
||||
never reach a prompt regardless of caller trust.
|
||||
|
||||
---
|
||||
|
||||
## docs/mcp/DEPLOY.md
|
||||
|
||||
Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md
|
||||
|
||||
@@ -25,6 +25,7 @@ Repo: https://github.com/garrytan/gbrain
|
||||
- [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist.
|
||||
- [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery.
|
||||
- [docs/guides/scaling-skills.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/scaling-skills.md): Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.
|
||||
- [docs/guides/push-context.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md): Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.
|
||||
- [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment.
|
||||
|
||||
## AI providers
|
||||
|
||||
@@ -151,6 +151,12 @@ export const SECTIONS: DocSection[] = [
|
||||
"Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.",
|
||||
path: "docs/guides/scaling-skills.md",
|
||||
},
|
||||
{
|
||||
title: "docs/guides/push-context.md",
|
||||
description:
|
||||
"Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.",
|
||||
path: "docs/guides/push-context.md",
|
||||
},
|
||||
{
|
||||
title: "docs/mcp/DEPLOY.md",
|
||||
description: "MCP server deployment.",
|
||||
|
||||
Reference in New Issue
Block a user