v0.42.48.0 feat(durability): auto-harden brain repos for git durability on PAT+URL (#2241)

* feat(git): divergence-safe pull, push-probe, default-branch detection for brain durability

Add GIT_ENV_AUTH + divergenceSafePull (skip-on-dirty, conflict-abort-clean,
never-mid-rebase), detectDefaultBranch, pushProbe, and an env-gated
GBRAIN_GIT_ALLOW_FILE_TRANSPORT escape hatch. Export GIT_ENV. pullRepo's
--ff-only contract is unchanged.

* feat(durability): brain-repo hardening core (hook, helper, cron, PAT, AGENTS rules)

hardenBrainRepo/unhardenBrainRepo: local untracked post-commit hook + committed
brain-commit-push.sh (one shared push-retry template), repo-scoped credential
with existing-helper reuse, push-probe verify, active-resolver-file rules with
taxonomy from _brain-filing-rules.json, minimal DB-free pull cron. PAT redaction
via redactSecretsInText.

* feat(sources): harden/pull/unharden commands + auto-harden on add --url

sources harden/pull/unharden subcommands; --pat-file/--no-harden on add;
auto-harden managed clones on add; unharden-before-remove. cli.ts pre-connect
early-exit for DB-free 'sources pull --path' (the cron entry, never opens PGLite).

* test(durability): unit + integration coverage for brain-repo durability

git helpers, core harden/unharden, hook+helper E2E (real background push),
cron generators. 41 tests across 4 files.

* chore: bump version and changelog (v0.42.48.0)

Brain-repo git durability: auto-harden a brain's working tree (local auto-push
hook, committed commit-push helper, always-on agent rules, DB-free pull cron,
repo-scoped credential, push-probe verify) the moment gbrain gets a PAT + URL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sources): route harden exit code through setCliExitVerdict

A raw process.exitCode write is zeroed by the owned-verdict flush-exit
(#2084 PGLite-Emscripten pollution defense); cli-exit-verdict-pin guard
caught it. Use setCliExitVerdict(3) so 'sources harden' actually reports
needs-attention to cron/automation.

* docs: document brain-repo durability (KEY_FILES + multi-source guide)

KEY_FILES: extend git-remote.ts entry (divergenceSafePull, pushProbe,
detectDefaultBranch, GIT_ENV_AUTH, GBRAIN_GIT_ALLOW_FILE_TRANSPORT) + add
brain-repo-durability.ts/sources-harden.ts entry. multi-source-brains.md:
add a Durability (auto-harden) how-to covering sources harden/pull/unharden,
--pat-file, the guarantees, and the security posture.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-17 06:41:37 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 9d88680a51
commit 7ea92d602c
15 changed files with 1801 additions and 4 deletions
+2 -1
View File
@@ -49,7 +49,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/core/disk-walk.ts``walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
- `src/core/git-head.ts` — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true also requires a clean working tree (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). `requireCleanWorkingTree` is `boolean | 'ignore-untracked'` — in `'ignore-untracked'` mode the clean probe runs `git status --porcelain --untracked-files=no` so a quiet repo with stray untracked dirs (`?? companies/`, `?? media/`) is still "unchanged" (sync's incremental path keys off the commit diff and never imports untracked files); `GitCleanProbe` gains an `ignoreUntracked?` second arg. Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) keep unit tests R2-compliant (no `mock.module`). Uses `execFileSync` with array args so shell metachars in `local_path` cannot escape to a shell (the regression test runs real `execFileSync` against `'/nonexistent/$(touch <sentinel>)/repo'` and asserts the sentinel is never created). Fail-open on every error (missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → false) preserving the caller's prior time-based behavior. The chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Pinned by `test/core/git-head.test.ts` (incl. the shell-injection regression guard).
- `src/core/source-health.ts` — per-source health metrics for `gbrain sources status` + doctor's `federation_health`. Commit-relative staleness: `newestCommitMs(localPath)` = HEAD committer time via `git log -1 --format=%ct` (fail-open null; NO working-tree mtime parsing — committed content only, robust against the porcelain-mtime bug farm); pure `lagFromContentMs(contentMs|null, lastSyncMs|null, nowMs)` = remote/column comparator (null lastSync → null; negative wall-clock → skew passthrough; `contentMs <= lastSync` → 0; else/null-content → wall-clock). `computeAllSourceMetrics(engine, sources, {probeContent?})`: LOCAL (`probeContent:true`, `gbrain sources status`) → `isSourceUnchangedSinceSync(..., {requireCleanWorkingTree:'ignore-untracked'}) ? 0 : wall-clock` (live commit-hash catches HEAD moving to an old-dated commit a timestamp compare would miss); REMOTE (default, `federation_health` on the HTTP MCP path) → `lagFromContentMs(row.newest_content_at, ...)`, NO git subprocess (trust boundary). `commitTimeMs(localPath, sha)` is the `newestCommitMs` sibling pinned to an arbitrary commit (committer time via `git show -s --format=%ct <sha>`, fail-open null, execFileSync array args) — the resumable sync stamps `newest_content_at` against its pinned target commit, not whatever HEAD raced to. Pinned by `test/source-health.test.ts`.
- `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo`, `pullRepo`, and `fetchRemote(repoPath, branch)` (the last added for the sync cost-estimator's fetch-first path, #2139, so a cost preview / dry-run fetches through the same hardened flags + `GIT_TERMINAL_PROMPT=0` as real sync rather than a less-protected route). Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (a combined array would spread `--no-recurse-submodules` before the verb where real git rejects it exit 129). `cloneRepo` argv: `git <GIT_SSRF_FLAGS> clone <GIT_SSRF_SUBCOMMAND_FLAGS> --depth=1 [--branch X] -- <url> <dir>`. `pullRepo` argv: `git <GIT_SSRF_FLAGS> -C <dir> pull <GIT_SSRF_SUBCOMMAND_FLAGS> --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`).
- `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo`, `pullRepo`, and `fetchRemote(repoPath, branch)` (the last added for the sync cost-estimator's fetch-first path, #2139, so a cost preview / dry-run fetches through the same hardened flags + `GIT_TERMINAL_PROMPT=0` as real sync rather than a less-protected route). Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (a combined array would spread `--no-recurse-submodules` before the verb where real git rejects it exit 129). `cloneRepo` argv: `git <GIT_SSRF_FLAGS> clone <GIT_SSRF_SUBCOMMAND_FLAGS> --depth=1 [--branch X] -- <url> <dir>`. `pullRepo` argv: `git <GIT_SSRF_FLAGS> -C <dir> pull <GIT_SSRF_SUBCOMMAND_FLAGS> --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). Also exports the durability-side helpers that power `gbrain sources harden/pull`: `GIT_ENV_AUTH` (the no-prompt env minus the askpass `/bin/false` overrides, so an auth'd push/fetch can consult the repo's configured credential helper while `GIT_TERMINAL_PROMPT=0` still fails fast on a missing credential), `divergenceSafePull(repoPath, branch)` (fetch + `pull --rebase`; returns `skipped_dirty` on a dirty tree, `conflict_aborted` on a rebase conflict after `rebase --abort` so the tree is never left mid-rebase, else `up_to_date`/`advanced`), `detectDefaultBranch` (origin/HEAD → current branch → `main`), `pushProbe(repoPath, branch)` (authenticated `push --dry-run` that proves push access and classifies `auth`/`protected`/`unreachable`), and `isWorkingTreeDirty`. These auth'd paths route their `protocol.file.allow` through `GBRAIN_GIT_ALLOW_FILE_TRANSPORT` (default `never`; set `=1` for self-hosted filesystem remotes), unlike clone/pull which stay strict.
- `src/core/brain-repo-durability.ts` + `src/commands/sources-harden.ts` — brain-repo git durability. `hardenBrainRepo(opts)` makes a brain's working tree durable, idempotently: divergence-safe pull, a LOCAL untracked `.git/hooks/post-commit` auto-push safety net (never committed — a pulled commit can't rewrite executed code next to the token; installed into the active `core.hooksPath` dir and excluded via `.git/info/exclude` when that dir is tracked), a committed `scripts/brain-commit-push.sh` that refuses to exit 0 without a confirmed push (hook + helper render from ONE bash push-retry template — DRY at the TS source, not by the hook sourcing a repo-controlled script), durability rules patched into the active resolver file (`findResolverFile` → RESOLVER.md > AGENTS.md; taxonomy rendered from the bundled `_brain-filing-rules.json`), a minimal DB-free pull cron (launchd/crontab running `gbrain sources pull --path <dir>` so it never opens the PGLite single-writer lock), and a push-probe verify (no heartbeat commit). Credential is REPO-scoped (`acceptPat` from `--pat-file`/`GBRAIN_GITHUB_PAT`, warns on loose perms; reuses an existing repo-local `credential.helper`, else a `0600` store wired via repo-local config); the token is redacted everywhere via `redactSecretsInText` and never enters the repo, remote URL, logs, or `DurabilityReport`. `unhardenBrainRepo` removes the cron/hook/credential wiring (ownership-fingerprinted) and runs before `sources remove`. CLI: `gbrain sources harden <id|--all>` / `pull <id>|--path <dir>` / `unharden <id>`; auto-harden fires on `sources add --url ... --pat-file` for managed clones (`--no-harden` opts out). `sources pull --path` is dispatched in `src/cli.ts` BEFORE `connectEngine` so the cron stays DB-free. CLI-only (writes executables + an OS cron + a credential helper on the host); never exposed over MCP. Tests: `test/brain-repo-durability.serial.test.ts`, `test/git-remote-durable.serial.test.ts`, `test/brain-durability-hook.serial.test.ts`, `test/durability-cron.test.ts`.
- `src/commands/storage.ts``gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time.
- `gbrain.yml` (brain repo root) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database.
- `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check).
+52
View File
@@ -155,6 +155,58 @@ Reads span federated sources by default. Writes require a resolved
source (explicit, inferred, or default). The resolver never picks a
source silently when ambiguous — it errors with a clear fix.
## Durability: keep a brain repo in sync (auto-harden)
A long-lived agent that writes to a knowledge-wiki git repo needs three
things to never lose work: pull before it edits, push every write, and not
go stale while it sits idle. `gbrain sources harden` installs all of that,
idempotently. The moment you add a brain repo with a token, it runs
automatically:
```bash
# Clone + register a GitHub repo, then auto-harden it for durability.
# Use a fine-grained PAT scoped to just this repo.
gbrain sources add wiki --url https://github.com/you/brain-wiki.git --pat-file ~/.secrets/wiki-pat
# → clones, then installs: local auto-push hook, scripts/brain-commit-push.sh,
# always-on durability rules in AGENTS.md/RESOLVER.md, a 30-min pull cron,
# and a repo-scoped credential. Verifies push works before declaring done.
# Run the same audit on an existing source any time (idempotent):
gbrain sources harden wiki --pat-file ~/.secrets/wiki-pat
# Pull on demand (the cron calls the --path form, which never opens the DB):
gbrain sources pull wiki
# Remove the durability scaffolding (also runs automatically on `sources remove`):
gbrain sources unharden wiki
```
What hardening guarantees:
- **Pull-first, conflict-safe.** Every pull is a divergence-safe rebase. A
dirty working tree is skipped (your in-progress edits are never touched); a
rebase conflict is aborted cleanly and flagged for attention, never left
half-applied.
- **Push is never deferred.** `scripts/brain-commit-push.sh "<msg>" <path>`
commits and pushes atomically and refuses to report success without a
confirmed push. The post-commit hook is a best-effort background fallback;
the helper is the guarantee.
- **No silent staleness.** A 30-minute background pull keeps an idle session
current. It runs DB-free, so it never contends with a live brain for the
PGLite single-writer lock.
Flags: `--no-cron` skips the scheduled pull, `--no-verify` skips the push
probe, `--dry-run` reports what would change, `--json` emits a machine
report, `--all` hardens every source with a remote (same-account only).
`--no-harden` on `sources add` opts out of auto-harden.
Security: the push automation is installed locally per machine (never
committed into the repo), the token is wired per-repo (an existing
credential helper is reused when present), and it never appears in the repo,
the remote URL, logs, or the JSON report. For a self-hosted git server
reachable only over a filesystem path, set `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1`
(default is HTTPS-only).
## Upgrading an existing brain
`gbrain upgrade` runs the v16 + v17 migrations automatically. Your