diff --git a/CHANGELOG.md b/CHANGELOG.md index e6fef5a6e..6bec6e4c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,39 @@ Every scoreboard row carries a `seam` label: the `openclaw` row exercises the sh - CLI exit codes for the new command route through the shared write-fence + aliveness-grace exit seam, so PGLite's WASM exit-code stomping and Bun's exit-time stdout discard can't corrupt the CI contract. To take advantage of v0.44.0.0: run `gbrain eval brainbench` — no setup, no keys, no brain required. If it ever reports something broken after an upgrade, `bun evals/brainbench/generator/gen.ts` rebuilds the corpus byte-identically and `gbrain eval brainbench --update-baseline` re-derives the baseline from an actual run; both are safe to re-run any time. +## [0.42.49.0] - 2026-06-16 + +**Big embed backfills and syncs now throttle themselves when the database gets busy, so clearing a backlog can't starve the job queue — no more external babysitter scripts.** A naive `gbrain embed --stale` or large `gbrain sync` against a PgBouncer transaction-mode pooler could saturate it and starve the minion supervisor's lock renewals, cascading `lock-renewal-failed` into dead jobs. The field workaround was an external wrapper that SIGSTOP/SIGCONT'd the process off a side-pool latency probe. That approach was blind (the side pool read low latency while the pool that mattered starved), unsafe (SIGSTOP can freeze a process mid-transaction holding locks), and couldn't touch peak pressure. gbrain now does this natively, and better. + +Pacing is **opt-in** (default `off`) and built on one composable primitive: it caps simultaneous in-flight DB writes (the real lever against pooler-slot starvation), measures the work's own query latency in-band (so it can never be blind), and sleeps cooperatively between safe points (never mid-transaction, so the lock heartbeat keeps firing). Turn it on per-run with `gbrain embed --stale --pace`, or set `pace.mode` in config to pace every embed path plus the production embed-backfill job automatically. `GBRAIN_PACE_*` env vars override config as an incident escape hatch. + +### Added +- **`--pace[=mode]` for `gbrain embed`** — `off`/`gentle`/`balanced`/`aggressive` bundles (bare `--pace` = balanced), plus `--pace-max-concurrency=N`. `--background` carries the explicit override into the queued `embed` job; the handler re-resolves env > config > bundle at execution. +- **`pace.mode` config + `GBRAIN_PACE_*` env** — config paces every `runEmbedCore` caller (cycle embed, catch-up, sync-auto-embed) and the prod `embed-backfill` job automatically; env beats config for incident response. +- **Composable `db-pacer` primitive** (`src/core/db-pacer.ts`) + named bundles (`src/core/pace-mode.ts`) — concurrency permit + in-band EWMA + jittered cooperative sleep, abort-throwing, fail-open. `sync` uses the shared permit across its parallel worker engines. +- **Pacing telemetry** — `EmbedResult.pacing` (cap, samples, EWMA latency, slept ms) in `--json`, plus a one-line stderr summary. + +### Changed +- **`gbrain embed --stale` now single-flights per source** using the same lock the `embed-backfill` job holds, so a hand-run backfill and a queued job can't grind the same source concurrently. Paced runs add a bounded end-of-run rescan (catches rows that landed behind the cursor during a longer run), and the embed time budget excludes paced-sleep time so a contended DB still converges instead of exiting early. + +### To take advantage of v0.42.49.0 +`gbrain upgrade`. Pacing is off by default — nothing changes until you opt in. To clear a big embed backlog safely on a busy pooler: `gbrain embed --stale --pace` (or `--pace=gentle` to be extra conservative). To pace the background embed-backfill job and every embed path automatically: `gbrain config set pace.mode balanced`. During an incident you can override without a redeploy: `GBRAIN_PACE_MODE=gentle` or `GBRAIN_PACE_MAX_CONCURRENCY=4`. +## [0.42.48.0] - 2026-06-16 + +**Brain repos harden themselves for durability the moment gbrain is given a PAT and a GitHub URL.** Fresh agents kept drifting out of sync with their knowledge-wiki git repos: writes sat local-only and never pushed, long-lived sessions edited a stale tree, and scratch output landed outside the repo and vanished. Now `gbrain sources add --url --pat-file

` auto-hardens the managed clone, and `gbrain sources harden ` runs the same audit idempotently against any source. Hardening is six always-on guarantees: it pulls current state (divergence-safe rebase that skips a dirty tree and never leaves a half-rebase), installs a local auto-push safety net, ships a committed `scripts/brain-commit-push.sh` that refuses to report success without a confirmed push, writes always-on durability rules into the agent's context file (deterministic filing from the canonical taxonomy, commit-and-push-never-deferred, pull-before-each-write-batch), registers a 30-minute background pull so an idle session can't go stale, and verifies push access up front. + +This is gbrain's first push path and first credential storage, built secure by default. The push automation is installed locally per machine rather than committed into the repo, the GitHub token is wired per-repo (least-privilege; an existing credential helper is reused when present rather than writing a new one), and the token never enters the repo, the tracked remote URL, logs, or the run report. Hardening proves push works with a dry-run probe before declaring done, so a read-only token or a protected branch surfaces immediately instead of silently dropping writes later. `gbrain sources unharden ` cleanly removes everything it installed and runs automatically before `sources remove`. + +### Added +- `gbrain sources harden [--pat-file

] [--branch ] [--no-cron] [--no-verify] [--dry-run] [--json]` — idempotent brain-repo durability hardening. +- `gbrain sources pull | --path

[--branch ]` — divergence-safe rebase-pull; `--path` runs DB-free so the 30-minute cron never contends for the local engine lock. +- `gbrain sources unharden ` — remove the durability cron, hook, and credential wiring. +- `--pat-file` / `--no-harden` on `gbrain sources add`; managed clones added with a PAT auto-harden. +- `git-remote.ts`: `divergenceSafePull`, `detectDefaultBranch`, `pushProbe`, and an env-gated `GBRAIN_GIT_ALLOW_FILE_TRANSPORT` escape hatch for self-hosted filesystem remotes. + +### To take advantage of v0.42.48.0 +`gbrain upgrade`. Add a brain repo with `gbrain sources add --url --pat-file ` and it hardens automatically; or run `gbrain sources harden --pat-file ` on an existing source. Use a fine-grained PAT scoped to just that repo. Existing brains are untouched until you opt in. + ## [0.42.47.0] - 2026-06-16 **A brain now travels with its own operating manual, and gbrain finally tells you how to run it better (gbrain#2180).** Two long-standing gaps closed. First: a brain repo can carry its own skillpack — skills authored for and versioned with that specific brain — and any harness that connects is offered it. Connect a fresh Claude Code or a thin client to a mature brain and it learns, on the spot, which meeting-ingestion or diligence protocol the brain expects, instead of starting blind. Second: gbrain stops being purely passive. `gbrain advisor` reads the brain's own state and hands back a ranked, read-only list of high-leverage actions — pending migrations, version drift, stalled backfills, low embedding coverage, setup smells — each with the exact command to fix it. It never acts on its own; it shows you and asks. diff --git a/CLAUDE.md b/CLAUDE.md index e27e4b981..30c36e6df 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -401,6 +401,57 @@ hatches — no config-dashboard surface by design): | `GBRAIN_SYNC_YIELD_EVERY` | 64 | Yield the event loop (`setTimeout(0)`, NOT `setImmediate` — Bun starves the timers phase under a tight setImmediate loop) every N files so the lock-refresh `setInterval` heartbeat fires mid-import. | | `GBRAIN_LOCK_STEAL_GRACE_SECONDS` | derived (~600 at 30min TTL) | A holder that refreshed within this window is NOT stolen even if its TTL lapsed (starved-but-alive). Dead holders stop refreshing, age past the grace, and become stealable; TTL stays the backstop. | +## Pace Mode (DB-contention-aware backfill pacing) + +A naive `gbrain embed --stale` / large `sync` can saturate a PgBouncer +transaction-mode pooler and starve the minion supervisor's lock renewals +(`lock-renewal-failed` → dead jobs). Pacing is the native, composable fix — it +replaces external SIGSTOP/SIGCONT wrapper scripts. **Opt-in: default mode `off`.** + +The composable primitive is `src/core/db-pacer.ts` (`createDbPacer`): +- **Concurrency cap is the real lever** (caps simultaneous in-flight DB writes = + pooler slots held). Embed paths set their worker count to `maxConcurrency` + (single pool, no permit); `sync` uses the shared `acquire()` **permit** because + each parallel worker owns a separate engine (one budget must span pools). +- **In-band signal** (`observe(ms)` EWMA from the work's own queries — never + blind the way an out-of-band probe pool was). **No probe loop, no + `probeLatency` engine method.** +- **Cooperative `pace()` sleep** on `setTimeout` (keeps the lock heartbeat + firing), jittered to avoid a thundering-herd resume. `acquire()`/`pace()` throw + `AbortError` on cancel; everything else is fail-open (a pacer bug never kills a + backfill, never throws an unhandledRejection). + +Named bundles resolve through `src/core/pace-mode.ts` (`resolvePaceMode`), mirror +of the search-mode pattern but with **env ABOVE config** (incident escape hatch): + + per-call flag → GBRAIN_PACE_* env → config (pace.*) → PACE_BUNDLES[mode] → off + +| Knob | off | gentle | balanced | aggressive | +|---|---|---|---|---| +| `maxConcurrency` | (off) | 4 | 8 | 16 | +| `paceAtMs` (EWMA → sleep) | — | 250 | 500 | 1000 | +| `maxSleepMs` (jittered cap) | — | 2000 | 1500 | 1000 | + +**Surfaces.** `gbrain embed --stale --pace[=mode]` (bare `--pace` = balanced), +`--pace-max-concurrency=N`. `--background` carries explicit pace OVERRIDES (not +the resolved bundle) into the `embed` job payload; the handler re-resolves +env>config>bundle at execution so `GBRAIN_PACE_*` still wins (CX5). Config-level +`pace.mode` paces EVERY `runEmbedCore` caller (cycle embed, embed-catch-up, +sync-auto-embed) and the prod `embed-backfill` job automatically. `sync` reads +env/config. PGLite / mode `off` → no-op pacer. + +**Correctness fixes pacing bundles** (longer paced runs widen these): CLI +`embed --stale` single-flights via the SAME per-source lock key as the +`embed-backfill` handler (`src/core/embed-backfill-lock.ts`; all-source runs lock +every source in sorted order) so a hand-run backfill and a queued job can't race +the NULL→non-NULL upsert (`TODOS:2299`); a **bounded** end-of-run keyset re-entry +(max 3 + forward-progress, paced runs only) catches rows inserted behind the +cursor (`TODOS:2301`); and the embed wall-clock budget timer is re-armed around +`pace()` sleeps so paced time doesn't burn the work budget. + +`EmbedResult.pacing` carries the end-of-run telemetry (cap, samples, EWMA, slept +ms, max waiters) for `--json`; a one-line summary prints to stderr. + ## Build `bun build --compile --outfile bin/gbrain src/cli.ts` diff --git a/TODOS.md b/TODOS.md index cbf734069..07100cd4e 100644 --- a/TODOS.md +++ b/TODOS.md @@ -16,6 +16,56 @@ Deferred from the BrainBench wave (eng-reviewed; plan + GSTACK REVIEW REPORT at - [ ] **Periodic re-baselining (the ratchet doesn't auto-tighten).** Improvements aren't banked into master's baseline until a PR updates it, so a regression back to a stale baseline level passes. Documented as an accepted residual in `docs/eval/BRAINBENCH.md`; the fix is an operator habit or a scheduled job that re-runs `--update-baseline` after metric-improving merges. Priority: P3. - [ ] **Hermetic-ize the 7 env-sensitive LLM-availability tests.** `test/think-gateway-adapter.test.ts`, `test/conversation-parser/llm-base.test.ts`/`llm-fallback.test.ts`, `test/doctor-ze-checks.test.ts` assert behavior "when ANTHROPIC_API_KEY is unset" by reading the live process env — they fail on any dev shell that exports provider keys (verified failing on clean master in such a shell; green in keyless CI). Stub/save-restore the env per test so local runs match CI. Priority: P2. +## Pace Mode follow-ups (filed v0.42.49.0) + +Deferred from the paced-backfill wave (CEO + eng review CLEARED). Core shipped: +`db-pacer` + `pace-mode` wired into embed (CLI + shared core + `embed-backfill` +job) and sync. See CLAUDE.md "Pace Mode". + +- [ ] **P2 — `doctor` pacing check (E2).** Detect a txn-mode pooler (port 6543) + running unpaced bulk and recommend `--pace`; optionally correlate recent + `minion_jobs` deaths with backfill windows. Where: `src/commands/doctor.ts`. +- [ ] **P2 — `--pace=auto` autotuned thresholds (E3).** Derive `paceAtMs`/cap from + observed baseline latency (rolling median) instead of fixed bundle values, + mirroring `gbrain search tune`. Needs a baseline window + cold-start default + + config persistence — not a small add. Where: `src/core/pace-mode.ts` + + `src/core/db-pacer.ts`. +- [ ] **P3 — First-class pacing in more minion job handlers (E5).** `embed-backfill` + is paced; extend to `extract`/`embed-catch-up`/contextual-reindex handlers with + supervisor-detection downgrade. Today these inherit config/env pacing only when + they call `runEmbedCore`. +- [ ] **P1-companion — Supervisor concurrency 3→2 + job-kind slot fairness (E7).** + The daemon-side root cause the external wrapper's probe was blind to: + `embed-backfill`/`autopilot-cycle` jobs can occupy all supervisor slots + (`:215` below). Pacing makes backfills safe; this fixes the residual death rate. + Where: `src/core/minions/supervisor.ts` + queue slot accounting. +- [ ] **P3 — `gbrain sync --pace` CLI flag.** Sync reads env/config pacing today; + add a per-run `--pace[=mode]` flag for symmetry with `embed`. Where: + `src/commands/sync.ts` arg parsing. +- [ ] **P3 — Real-PG e2e for pacing.** Gated on `DATABASE_URL`: paced + `embed --stale --pace --progress-json` caps concurrency + emits telemetry; + single-flight rejects a 2nd concurrent run; lock heartbeat advances during a + paced sleep (short-TTL). Unit coverage (`db-pacer`/`pace-mode`) already ships. +## brain-repo durability follow-ups (filed v0.42.48.0) + +- [ ] **P3 — gbrain write-path calls commit-push synchronously when durability is on.** + v0.42.48.0 ships the synchronous `brain-commit-push.sh` as the guarantee and a local + post-commit hook as a best-effort fallback. The strongest durability (codex outside-voice + D13-C) is to have gbrain's own write-through path call the commit-push helper synchronously + when a source is hardened — that also covers writes that never get committed by an agent. + Deferred because it touches the write path; the hook + mandated helper cover the + agent-driven case today. + - **Where to start:** `src/core/write-through.ts:writePageThrough` + a per-source "hardened" + flag to gate the synchronous push. + +- [ ] **P3 — Unify the durability pull cron with autopilot's OS-scheduler.** + v0.42.48.0 ships a minimal launchd/crontab installer inside `brain-repo-durability.ts` + (D12: minimal-now to keep the diff off the load-bearing autopilot feature). Extract a shared + `os-scheduler.ts` (`installPeriodic`/`removePeriodic`) and have both autopilot and brain-pull + call it, so there's one OS-cron path. + - **Where to start:** `src/commands/autopilot.ts` (`installLaunchd`/`installSystemd`/ + `installCrontab`/`writeWrapperScript`) + `brain-repo-durability.ts:installDurabilityCron`. + ## gbrain#2200 federated-read follow-ups (filed v0.42.46.0) - [ ] **P1 — Close the federated-read scope on the remaining same-class by-slug read ops.** diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 61eafd04f..d0e0b884c 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -49,7 +49,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/disk-walk.ts` — `walkBrainRepo(repoPath)` returns `Map` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens). - `src/core/git-head.ts` — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true also requires a clean working tree (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). `requireCleanWorkingTree` is `boolean | 'ignore-untracked'` — in `'ignore-untracked'` mode the clean probe runs `git status --porcelain --untracked-files=no` so a quiet repo with stray untracked dirs (`?? companies/`, `?? media/`) is still "unchanged" (sync's incremental path keys off the commit diff and never imports untracked files); `GitCleanProbe` gains an `ignoreUntracked?` second arg. Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) keep unit tests R2-compliant (no `mock.module`). Uses `execFileSync` with array args so shell metachars in `local_path` cannot escape to a shell (the regression test runs real `execFileSync` against `'/nonexistent/$(touch )/repo'` and asserts the sentinel is never created). Fail-open on every error (missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → false) preserving the caller's prior time-based behavior. The chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Pinned by `test/core/git-head.test.ts` (incl. the shell-injection regression guard). - `src/core/source-health.ts` — per-source health metrics for `gbrain sources status` + doctor's `federation_health`. Commit-relative staleness: `newestCommitMs(localPath)` = HEAD committer time via `git log -1 --format=%ct` (fail-open null; NO working-tree mtime parsing — committed content only, robust against the porcelain-mtime bug farm); pure `lagFromContentMs(contentMs|null, lastSyncMs|null, nowMs)` = remote/column comparator (null lastSync → null; negative wall-clock → skew passthrough; `contentMs <= lastSync` → 0; else/null-content → wall-clock). `computeAllSourceMetrics(engine, sources, {probeContent?})`: LOCAL (`probeContent:true`, `gbrain sources status`) → `isSourceUnchangedSinceSync(..., {requireCleanWorkingTree:'ignore-untracked'}) ? 0 : wall-clock` (live commit-hash catches HEAD moving to an old-dated commit a timestamp compare would miss); REMOTE (default, `federation_health` on the HTTP MCP path) → `lagFromContentMs(row.newest_content_at, ...)`, NO git subprocess (trust boundary). `commitTimeMs(localPath, sha)` is the `newestCommitMs` sibling pinned to an arbitrary commit (committer time via `git show -s --format=%ct `, fail-open null, execFileSync array args) — the resumable sync stamps `newest_content_at` against its pinned target commit, not whatever HEAD raced to. Pinned by `test/source-health.test.ts`. -- `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo`, `pullRepo`, and `fetchRemote(repoPath, branch)` (the last added for the sync cost-estimator's fetch-first path, #2139, so a cost preview / dry-run fetches through the same hardened flags + `GIT_TERMINAL_PROMPT=0` as real sync rather than a less-protected route). Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (a combined array would spread `--no-recurse-submodules` before the verb where real git rejects it exit 129). `cloneRepo` argv: `git clone --depth=1 [--branch X] -- `. `pullRepo` argv: `git -C pull --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). +- `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo`, `pullRepo`, and `fetchRemote(repoPath, branch)` (the last added for the sync cost-estimator's fetch-first path, #2139, so a cost preview / dry-run fetches through the same hardened flags + `GIT_TERMINAL_PROMPT=0` as real sync rather than a less-protected route). Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (a combined array would spread `--no-recurse-submodules` before the verb where real git rejects it exit 129). `cloneRepo` argv: `git clone --depth=1 [--branch X] -- `. `pullRepo` argv: `git -C pull --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). Also exports the durability-side helpers that power `gbrain sources harden/pull`: `GIT_ENV_AUTH` (the no-prompt env minus the askpass `/bin/false` overrides, so an auth'd push/fetch can consult the repo's configured credential helper while `GIT_TERMINAL_PROMPT=0` still fails fast on a missing credential), `divergenceSafePull(repoPath, branch)` (fetch + `pull --rebase`; returns `skipped_dirty` on a dirty tree, `conflict_aborted` on a rebase conflict after `rebase --abort` so the tree is never left mid-rebase, else `up_to_date`/`advanced`), `detectDefaultBranch` (origin/HEAD → current branch → `main`), `pushProbe(repoPath, branch)` (authenticated `push --dry-run` that proves push access and classifies `auth`/`protected`/`unreachable`), and `isWorkingTreeDirty`. These auth'd paths route their `protocol.file.allow` through `GBRAIN_GIT_ALLOW_FILE_TRANSPORT` (default `never`; set `=1` for self-hosted filesystem remotes), unlike clone/pull which stay strict. +- `src/core/brain-repo-durability.ts` + `src/commands/sources-harden.ts` — brain-repo git durability. `hardenBrainRepo(opts)` makes a brain's working tree durable, idempotently: divergence-safe pull, a LOCAL untracked `.git/hooks/post-commit` auto-push safety net (never committed — a pulled commit can't rewrite executed code next to the token; installed into the active `core.hooksPath` dir and excluded via `.git/info/exclude` when that dir is tracked), a committed `scripts/brain-commit-push.sh` that refuses to exit 0 without a confirmed push (hook + helper render from ONE bash push-retry template — DRY at the TS source, not by the hook sourcing a repo-controlled script), durability rules patched into the active resolver file (`findResolverFile` → RESOLVER.md > AGENTS.md; taxonomy rendered from the bundled `_brain-filing-rules.json`), a minimal DB-free pull cron (launchd/crontab running `gbrain sources pull --path ` so it never opens the PGLite single-writer lock), and a push-probe verify (no heartbeat commit). Credential is REPO-scoped (`acceptPat` from `--pat-file`/`GBRAIN_GITHUB_PAT`, warns on loose perms; reuses an existing repo-local `credential.helper`, else a `0600` store wired via repo-local config); the token is redacted everywhere via `redactSecretsInText` and never enters the repo, remote URL, logs, or `DurabilityReport`. `unhardenBrainRepo` removes the cron/hook/credential wiring (ownership-fingerprinted) and runs before `sources remove`. CLI: `gbrain sources harden ` / `pull |--path ` / `unharden `; auto-harden fires on `sources add --url ... --pat-file` for managed clones (`--no-harden` opts out). `sources pull --path` is dispatched in `src/cli.ts` BEFORE `connectEngine` so the cron stays DB-free. CLI-only (writes executables + an OS cron + a credential helper on the host); never exposed over MCP. Tests: `test/brain-repo-durability.serial.test.ts`, `test/git-remote-durable.serial.test.ts`, `test/brain-durability-hook.serial.test.ts`, `test/durability-cron.test.ts`. - `src/commands/storage.ts` — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time. - `gbrain.yml` (brain repo root) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database. - `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check). diff --git a/docs/guides/multi-source-brains.md b/docs/guides/multi-source-brains.md index 05da8f042..084605598 100644 --- a/docs/guides/multi-source-brains.md +++ b/docs/guides/multi-source-brains.md @@ -155,6 +155,58 @@ Reads span federated sources by default. Writes require a resolved source (explicit, inferred, or default). The resolver never picks a source silently when ambiguous — it errors with a clear fix. +## Durability: keep a brain repo in sync (auto-harden) + +A long-lived agent that writes to a knowledge-wiki git repo needs three +things to never lose work: pull before it edits, push every write, and not +go stale while it sits idle. `gbrain sources harden` installs all of that, +idempotently. The moment you add a brain repo with a token, it runs +automatically: + +```bash +# Clone + register a GitHub repo, then auto-harden it for durability. +# Use a fine-grained PAT scoped to just this repo. +gbrain sources add wiki --url https://github.com/you/brain-wiki.git --pat-file ~/.secrets/wiki-pat +# → clones, then installs: local auto-push hook, scripts/brain-commit-push.sh, +# always-on durability rules in AGENTS.md/RESOLVER.md, a 30-min pull cron, +# and a repo-scoped credential. Verifies push works before declaring done. + +# Run the same audit on an existing source any time (idempotent): +gbrain sources harden wiki --pat-file ~/.secrets/wiki-pat + +# Pull on demand (the cron calls the --path form, which never opens the DB): +gbrain sources pull wiki + +# Remove the durability scaffolding (also runs automatically on `sources remove`): +gbrain sources unharden wiki +``` + +What hardening guarantees: + +- **Pull-first, conflict-safe.** Every pull is a divergence-safe rebase. A + dirty working tree is skipped (your in-progress edits are never touched); a + rebase conflict is aborted cleanly and flagged for attention, never left + half-applied. +- **Push is never deferred.** `scripts/brain-commit-push.sh "" ` + commits and pushes atomically and refuses to report success without a + confirmed push. The post-commit hook is a best-effort background fallback; + the helper is the guarantee. +- **No silent staleness.** A 30-minute background pull keeps an idle session + current. It runs DB-free, so it never contends with a live brain for the + PGLite single-writer lock. + +Flags: `--no-cron` skips the scheduled pull, `--no-verify` skips the push +probe, `--dry-run` reports what would change, `--json` emits a machine +report, `--all` hardens every source with a remote (same-account only). +`--no-harden` on `sources add` opts out of auto-harden. + +Security: the push automation is installed locally per machine (never +committed into the repo), the token is wired per-repo (an existing +credential helper is reused when present), and it never appears in the repo, +the remote URL, logs, or the JSON report. For a self-hosted git server +reachable only over a filesystem path, set `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1` +(default is HTTPS-only). + ## Upgrading an existing brain `gbrain upgrade` runs the v16 + v17 migrations automatically. Your diff --git a/llms-full.txt b/llms-full.txt index 281f87b6a..cfb976455 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -550,6 +550,57 @@ hatches — no config-dashboard surface by design): | `GBRAIN_SYNC_YIELD_EVERY` | 64 | Yield the event loop (`setTimeout(0)`, NOT `setImmediate` — Bun starves the timers phase under a tight setImmediate loop) every N files so the lock-refresh `setInterval` heartbeat fires mid-import. | | `GBRAIN_LOCK_STEAL_GRACE_SECONDS` | derived (~600 at 30min TTL) | A holder that refreshed within this window is NOT stolen even if its TTL lapsed (starved-but-alive). Dead holders stop refreshing, age past the grace, and become stealable; TTL stays the backstop. | +## Pace Mode (DB-contention-aware backfill pacing) + +A naive `gbrain embed --stale` / large `sync` can saturate a PgBouncer +transaction-mode pooler and starve the minion supervisor's lock renewals +(`lock-renewal-failed` → dead jobs). Pacing is the native, composable fix — it +replaces external SIGSTOP/SIGCONT wrapper scripts. **Opt-in: default mode `off`.** + +The composable primitive is `src/core/db-pacer.ts` (`createDbPacer`): +- **Concurrency cap is the real lever** (caps simultaneous in-flight DB writes = + pooler slots held). Embed paths set their worker count to `maxConcurrency` + (single pool, no permit); `sync` uses the shared `acquire()` **permit** because + each parallel worker owns a separate engine (one budget must span pools). +- **In-band signal** (`observe(ms)` EWMA from the work's own queries — never + blind the way an out-of-band probe pool was). **No probe loop, no + `probeLatency` engine method.** +- **Cooperative `pace()` sleep** on `setTimeout` (keeps the lock heartbeat + firing), jittered to avoid a thundering-herd resume. `acquire()`/`pace()` throw + `AbortError` on cancel; everything else is fail-open (a pacer bug never kills a + backfill, never throws an unhandledRejection). + +Named bundles resolve through `src/core/pace-mode.ts` (`resolvePaceMode`), mirror +of the search-mode pattern but with **env ABOVE config** (incident escape hatch): + + per-call flag → GBRAIN_PACE_* env → config (pace.*) → PACE_BUNDLES[mode] → off + +| Knob | off | gentle | balanced | aggressive | +|---|---|---|---|---| +| `maxConcurrency` | (off) | 4 | 8 | 16 | +| `paceAtMs` (EWMA → sleep) | — | 250 | 500 | 1000 | +| `maxSleepMs` (jittered cap) | — | 2000 | 1500 | 1000 | + +**Surfaces.** `gbrain embed --stale --pace[=mode]` (bare `--pace` = balanced), +`--pace-max-concurrency=N`. `--background` carries explicit pace OVERRIDES (not +the resolved bundle) into the `embed` job payload; the handler re-resolves +env>config>bundle at execution so `GBRAIN_PACE_*` still wins (CX5). Config-level +`pace.mode` paces EVERY `runEmbedCore` caller (cycle embed, embed-catch-up, +sync-auto-embed) and the prod `embed-backfill` job automatically. `sync` reads +env/config. PGLite / mode `off` → no-op pacer. + +**Correctness fixes pacing bundles** (longer paced runs widen these): CLI +`embed --stale` single-flights via the SAME per-source lock key as the +`embed-backfill` handler (`src/core/embed-backfill-lock.ts`; all-source runs lock +every source in sorted order) so a hand-run backfill and a queued job can't race +the NULL→non-NULL upsert (`TODOS:2299`); a **bounded** end-of-run keyset re-entry +(max 3 + forward-progress, paced runs only) catches rows inserted behind the +cursor (`TODOS:2301`); and the embed wall-clock budget timer is re-armed around +`pace()` sleeps so paced time doesn't burn the work budget. + +`EmbedResult.pacing` carries the end-of-run telemetry (cap, samples, EWMA, slept +ms, max waiters) for `--json`; a one-line summary prints to stderr. + ## Build `bun build --compile --outfile bin/gbrain src/cli.ts` diff --git a/src/cli.ts b/src/cli.ts index 0d352428d..dc6cb7c78 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -288,6 +288,17 @@ async function main() { } } + // DB-free durability pull (v0.42.44 D2): the harden cron calls + // `gbrain sources pull --path ` every ~30 min. It must NOT open PGLite + // (a live long-lived session holds the single-writer lock), so handle it + // BEFORE connectEngine. The `sources pull ` form (no --path) still routes + // through handleCliOnly → runSources with an engine. + if (command === 'sources' && subArgs[0] === 'pull' && subArgs.includes('--path')) { + const { runPull } = await import('./commands/sources-harden.ts'); + await runPull(null, subArgs.slice(1)); + return; + } + // CLI-only commands if (CLI_ONLY.has(command)) { await handleCliOnly(command, subArgs); diff --git a/src/commands/embed.ts b/src/commands/embed.ts index 3afefab03..6d8816a6e 100644 --- a/src/commands/embed.ts +++ b/src/commands/embed.ts @@ -9,7 +9,16 @@ import { loadConfig } from '../core/config.ts'; import { slog, serr } from '../core/console-prefix.ts'; import { filterOutEmbedSkipped } from '../core/embed-skip.ts'; import { runSlidingPool } from '../core/worker-pool.ts'; -import { isAborted, anySignal } from '../core/abort-check.ts'; +import { isAborted, anySignal, AbortError } from '../core/abort-check.ts'; +import { type DbPacer, createDbPacer, createNoopPacer, observed } from '../core/db-pacer.ts'; +import { + resolvePaceMode, + loadPaceModeConfig, + readPaceEnv, + type PaceKeyOverrides, +} from '../core/pace-mode.ts'; +import { tryAcquireDbLock, type DbLockHandle } from '../core/db-lock.ts'; +import { embedBackfillLockId } from '../core/embed-backfill-lock.ts'; export interface EmbedOpts { /** Embed ALL pages (every chunk). */ @@ -71,6 +80,33 @@ export interface EmbedOpts { * with the internal wall-clock budget timer via `anySignal`. */ signal?: AbortSignal; + /** + * DB-contention pacing (paced-backfill). Raw inputs resolved in + * runEmbedCore via env > config > bundle (env beats config = incident + * escape hatch). `perCallMode` is from `--pace[=mode]`; `perCall` from + * `--pace-max-concurrency` etc. Absent ⇒ resolves from env/config (so a + * queued job paced by config alone still throttles). Mode `off` ⇒ no-op. + */ + pace?: { + perCallMode?: string; + perCall?: PaceKeyOverrides; + }; + /** + * When the pace overrides were SERIALIZED from a background-job payload (not + * typed at an interactive CLI), resolve them at the config tier so + * `GBRAIN_PACE_*` on the worker still overrides at execution (Codex P2). Set + * by the `embed` job handler; unset for interactive CLI runs. + */ + paceFromBackground?: boolean; + /** + * E-2 (paced-backfill): single-flight the stale run by taking the SAME + * per-source lock the `embed-backfill` minion handler uses, so a hand-run CLI + * backfill and a queued job can't grind the same source at once (closing the + * NULL→non-NULL upsert race window that paced — longer — runs widen). Set + * ONLY by the CLI (`runEmbed`); the minion path already locks. All-source + * runs lock every source in sorted order. dryRun skips it. + */ + singleFlight?: boolean; } /** @@ -94,6 +130,24 @@ export interface EmbedResult { pages_processed: number; /** True if this run was a dry-run. */ dryRun: boolean; + /** + * E1 (paced-backfill): end-of-run pacing telemetry. Present ONLY when pacing + * was active (enabled bundle). The number the operator could not get from an + * external wrapper ("zero pauses" ≠ "queue safe"). + */ + pacing?: { + maxConcurrency: number; + /** In-band latency samples folded into the EWMA. */ + samples: number; + /** Final EWMA of observed DB-op latency (ms), or null if no samples. */ + ewmaMs: number | null; + /** Cumulative cooperative-sleep time (ms). */ + totalSleptMs: number; + /** Number of cooperative sleeps. */ + sleeps: number; + /** High-water mark of acquirers blocked on the permit (sync path). */ + maxWaiters: number; + }; } /** @@ -207,11 +261,118 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis return result; } if (opts.all || opts.stale) { - await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, { - batchSize: opts.batchSize, - priority: opts.priority, - catchUp: opts.catchUp, - }, opts.signal); + // E-2 (paced-backfill): CLI single-flight. Take the SAME per-source lock as + // the embed-backfill minion handler so a hand-run backfill and a queued job + // are mutually exclusive per source. All-source runs lock every source in + // sorted (deterministic) order to avoid acquire-order deadlock. Released in + // the finally below. Skipped for dryRun and when the caller didn't opt in + // (cycle / catch-up / sync-auto-embed callers never single-flight). + const sfLocks: DbLockHandle[] = []; + if (opts.singleFlight && opts.stale && !opts.dryRun) { + let lockSourceIds: string[]; + if (opts.sourceId) { + lockSourceIds = [opts.sourceId]; + } else { + try { + const rows = await engine.listAllSources(); + lockSourceIds = rows.map((r) => r.id).sort(); + } catch { + lockSourceIds = []; + } + } + for (const sid of lockSourceIds) { + let lock: DbLockHandle | null = null; + try { + lock = await tryAcquireDbLock(engine, embedBackfillLockId(sid), 60); + } catch { + // Fail-open: a lock-subsystem error must not crash a backfill. Drop + // single-flight for this run (release what we took) and proceed. + for (const h of sfLocks) { + try { await h.release(); } catch { /* best-effort */ } + } + sfLocks.length = 0; + break; + } + if (!lock) { + // Another backfill (CLI or job) holds this source. Release what we + // took and bail cleanly rather than racing the upsert path. + for (const h of sfLocks) { + try { await h.release(); } catch { /* best-effort */ } + } + serr(` [embed] another backfill is already running for source "${sid}"; skipping (single-flight).`); + return result; + } + sfLocks.push(lock); + } + } + + // Resolve DB-contention pacing (env > config > bundle; env is the + // incident escape hatch). dryRun skips it — no writes to pace. A + // disabled bundle yields a no-op pacer (zero overhead on the hot path). + let pacer: DbPacer = createNoopPacer(); + let paceMaxConcurrency: number | undefined; + if (!opts.dryRun) { + try { + const cfg = await loadPaceModeConfig(engine); + const { envMode, envOverrides } = readPaceEnv(); + // Codex P2: an interactive CLI flag (--pace) is the most immediate + // intent and sits at the per-call tier (beats env). But a flag + // SERIALIZED into a background job payload must sit at the CONFIG tier + // so GBRAIN_PACE_* on the worker can still override it at execution + // (incident escape hatch). paceFromBackground distinguishes the two. + const fromBg = !!opts.paceFromBackground; + const knobs = resolvePaceMode({ + mode: fromBg ? (opts.pace?.perCallMode ?? cfg.mode) : cfg.mode, + configOverrides: fromBg + ? { ...cfg.configOverrides, ...(opts.pace?.perCall ?? {}) } + : cfg.configOverrides, + envMode, + envOverrides, + perCallMode: fromBg ? undefined : opts.pace?.perCallMode, + perCall: fromBg ? undefined : opts.pace?.perCall, + }); + if (knobs.enabled) { + pacer = createDbPacer({ bundle: knobs }); + paceMaxConcurrency = knobs.maxConcurrency; + } + } catch { + // Fail-open: pacing must never break a backfill. + pacer = createNoopPacer(); + } + } + try { + await embedAll(engine, !!opts.stale, !!opts.dryRun, result, opts.onProgress, opts.sourceId, { + batchSize: opts.batchSize, + priority: opts.priority, + catchUp: opts.catchUp, + pacer, + paceMaxConcurrency, + }, opts.signal); + } finally { + // E1: surface pacing telemetry (human + structured) when pacing was on. + const snap = pacer.snapshot(); + if (snap.enabled) { + result.pacing = { + maxConcurrency: snap.maxConcurrency, + samples: snap.sampleCount, + ewmaMs: snap.ewmaMs, + totalSleptMs: snap.totalSleptMs, + sleeps: snap.sleepCount, + maxWaiters: snap.maxWaiters, + }; + serr( + ` [embed] pacing: cap=${snap.maxConcurrency} samples=${snap.sampleCount} ` + + `ewma=${snap.ewmaMs === null ? 'n/a' : Math.round(snap.ewmaMs) + 'ms'} ` + + `slept=${snap.totalSleptMs}ms/${snap.sleepCount}`, + ); + } + pacer.dispose(); + // E-2: release single-flight locks (reverse order). Best-effort; the + // lock TTL is the backstop if a release fails. + for (const h of sfLocks.reverse()) { + try { await h.release(); } catch { /* best-effort; TTL covers it */ } + } + } return result; } if (opts.slug) { @@ -221,6 +382,39 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis throw new Error('No embed target specified. Pass { slug }, { slugs }, { all }, or { stale }.'); } +/** + * Parse the `--pace` family from a CLI arg list. Returns ONLY the explicit + * overrides (CX5: never the full resolved bundle) so they can be serialized + * into a background-job payload and re-resolved (env > config > bundle) at + * execution. Returns undefined when no pace flag is present. + * + * Recognized: `--pace` (bare ⇒ balanced), `--pace=`, + * `--pace-max-concurrency=` / `--pace-max-concurrency `. + */ +export function parsePaceArgs( + args: string[], +): { perCallMode?: string; perCall?: PaceKeyOverrides } | undefined { + let perCallMode: string | undefined; + let perCall: PaceKeyOverrides | undefined; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--pace') { + perCallMode = 'balanced'; + } else if (a.startsWith('--pace=')) { + perCallMode = a.slice('--pace='.length) || 'balanced'; + } else if (a.startsWith('--pace-max-concurrency=')) { + const n = parseInt(a.slice('--pace-max-concurrency='.length), 10); + if (Number.isFinite(n) && n >= 1) (perCall ??= {}).maxConcurrency = n; + } else if (a === '--pace-max-concurrency') { + const n = parseInt(args[i + 1] ?? '', 10); + if (Number.isFinite(n) && n >= 1) (perCall ??= {}).maxConcurrency = n; + i++; // consume the value token so positional parsing can't read it as a slug (Codex P2) + } + } + if (perCallMode === undefined && perCall === undefined) return undefined; + return { ...(perCallMode !== undefined && { perCallMode }), ...(perCall && { perCall }) }; +} + export async function runEmbed(engine: BrainEngine, args: string[]): Promise { // v0.36+ T7: --background submits via Minion queue, returns job_id to // stdout, exits. Same semantics in TTY and cron (D9). @@ -239,6 +433,10 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise= 0 ? cleanArgs.slice(slugsI + 1).filter(a => !a.startsWith('--')) : undefined, sourceId: srcI >= 0 ? cleanArgs[srcI + 1] : undefined, + // CX1+CX5: carry explicit pace overrides into the `embed` job payload + // (the job name CLI --background actually submits). The handler + // re-resolves env > config > bundle at execution. + ...(parsePaceArgs(cleanArgs) && { pace: parsePaceArgs(cleanArgs) }), }; }, source: 'cli', @@ -262,12 +460,14 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise= 0 ? args[priorityIdx + 1] : undefined; const priority = priorityRaw === 'recent' ? 'recent' as const : undefined; const catchUp = args.includes('--catch-up'); + const pace = parsePaceArgs(args); let opts: EmbedOpts; if (slugsIdx >= 0) { opts = { slugs: args.slice(slugsIdx + 1).filter(a => !a.startsWith('--')), dryRun, sourceId, batchSize, priority, catchUp }; } else if (all || stale) { - opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp }; + // E-2: CLI-only single-flight for stale runs (the minion path locks itself). + opts = { all, stale, dryRun, sourceId, batchSize, priority, catchUp, ...(pace && { pace }), ...(stale && { singleFlight: true }) }; } else { const slug = args.find(a => !a.startsWith('--')); if (!slug) { @@ -416,6 +616,10 @@ async function embedAll( batchSize?: number; priority?: 'recent'; catchUp?: boolean; + /** DB-contention pacer (paced-backfill); no-op when pacing is off. */ + pacer?: DbPacer; + /** Resolved concurrency cap (E-1: the worker count, no separate permit). */ + paceMaxConcurrency?: number; }, signal?: AbortSignal, ) { @@ -443,6 +647,10 @@ async function embedAll( return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature, signal); } + // --all path: pacer (no-op when off). E-1: lower the worker count to the + // resolved cap instead of adding a separate permit. + const pacer = staleOpts?.pacer ?? createNoopPacer(); + // v0.31.12: when sourceId is set, scope listPages to that source. // v0.41 (D8 + Codex r2 #11): apply embed-skip filter via the shared // helper so the `--all` path honors `frontmatter.embed_skip` the same @@ -466,7 +674,13 @@ async function embedAll( // (3000+/min for tier 1 = 50+/sec, 20 parallel is safely below) and // avoids overwhelming postgres connection pools. Users can tune via // GBRAIN_EMBED_CONCURRENCY env var based on their tier/infra. - const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10); + // Paced runs lower this to the resolved cap (the real lever vs pooler-slot + // starvation); unpaced keeps the env/default 20. Codex P2: only ever LOWER — + // never raise above an operator's existing env cap. + const BASE_CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10); + const CONCURRENCY = staleOpts?.paceMaxConcurrency + ? Math.min(BASE_CONCURRENCY, staleOpts.paceMaxConcurrency) + : BASE_CONCURRENCY; async function embedOnePage(page: typeof pages[number]) { // #1737: bail before doing any work for this page if the run was aborted. @@ -475,7 +689,7 @@ async function embedAll( // target the correct (source_id, slug) row, not the 'default' source. const pageSourceId = page.source_id; const pageOpts = pageSourceId ? { sourceId: pageSourceId } : undefined; - const chunks = await engine.getChunks(page.slug, pageOpts); + const chunks = await observed(pacer, () => engine.getChunks(page.slug, pageOpts)); const toEmbed = chunks; // staleOnly path handled above via embedAllStale result.total_chunks += chunks.length; @@ -511,10 +725,12 @@ async function embedAll( embedding: embeddingMap.get(c.chunk_index) ?? undefined, token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), })); - await engine.upsertChunks(page.slug, updated, pageOpts); + await observed(pacer, () => engine.upsertChunks(page.slug, updated, pageOpts)); // v0.41.31: stamp embedding provenance so a later model swap is // detectable as stale. - await engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }); + await observed(pacer, () => + engine.setPageEmbeddingSignature(page.slug, { sourceId: pageSourceId, signature }), + ); result.embedded += toEmbed.length; } catch (e: unknown) { serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`); @@ -523,6 +739,12 @@ async function embedAll( processed++; result.pages_processed++; onProgress?.(processed, pages.length, result.embedded); + // Cooperative DB-contention pace between pages (no-op when unpaced). + try { + await pacer.pace(signal); + } catch (e) { + if (!(e instanceof AbortError)) throw e; + } } // v0.41.15.0: sliding worker pool extracted into src/core/worker-pool.ts. @@ -576,6 +798,10 @@ async function embedAllStale( batchSize?: number; priority?: 'recent'; catchUp?: boolean; + /** DB-contention pacer (paced-backfill); no-op when pacing is off. */ + pacer?: DbPacer; + /** Resolved concurrency cap (E-1: the worker count, no separate permit). */ + paceMaxConcurrency?: number; }, signature?: string, externalSignal?: AbortSignal, @@ -626,7 +852,14 @@ async function embedAllStale( // (page_id, chunk_index). Each query finishes in <1s. // v0.41.18.0 (A13): --batch-size N CLI flag overrides hardcoded 2000 default. const PAGE_SIZE = staleOpts?.batchSize ?? 2000; - const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10); + // Paced runs lower concurrency to the resolved cap (E-1: worker count IS the + // lever on this single pool, no separate permit). Codex P2: pacing only ever + // LOWERS concurrency — never raise above an operator's existing env cap. + const BASE_CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10); + const CONCURRENCY = staleOpts?.paceMaxConcurrency + ? Math.min(BASE_CONCURRENCY, staleOpts.paceMaxConcurrency) + : BASE_CONCURRENCY; + const pacer = staleOpts?.pacer ?? createNoopPacer(); // D3 + D3a + D8: wall-clock budget. 30 min default; env override. // #1946: --catch-up removes the wall-clock cap. The prior code set BUDGET_MS = @@ -640,9 +873,22 @@ async function embedAllStale( ? null : parseInt(process.env.GBRAIN_EMBED_TIME_BUDGET_MS || `${30 * 60 * 1000}`, 10); const budgetController = new AbortController(); - const budgetTimer = BUDGET_MS != null + const budgetStart = Date.now(); + let budgetTimer = BUDGET_MS != null ? setTimeout(() => budgetController.abort(), BUDGET_MS) : undefined; + // E-4 (paced-backfill): the budget measures WORK, not waiting. After each + // batch, re-arm the timer to fire at start + BUDGET + total-paced-sleep, so a + // contended DB that spends time in pace() sleeps converges instead of exiting + // having embedded little. No-op when unpaced (totalSleptMs stays 0) or in + // catch-up (no budget timer). + const rearmBudgetForPacing = (): void => { + if (BUDGET_MS == null) return; + const slept = pacer.snapshot().totalSleptMs; + if (budgetTimer) clearTimeout(budgetTimer); + const fireInMs = budgetStart + BUDGET_MS + slept - Date.now(); + budgetTimer = setTimeout(() => budgetController.abort(), Math.max(0, fireInMs)); + }; const budgetSignal = budgetController.signal; // #1737: the effective signal fires when EITHER the internal wall-clock // budget OR the caller's abort (worker timeout / lock loss / SIGTERM) fires. @@ -670,6 +916,33 @@ async function embedAllStale( // surfaces that loudly instead of looking like a clean run. let embedFailures = 0; + // E-3 (paced-backfill): bounded end-of-run re-entry. A longer paced run gives + // a live writer (sync / put_page) more time to insert NEW stale rows BEHIND + // the keyset cursor (TODOS:2301). When the cursor exhausts, re-scan from the + // start — capped at MAX_REENTRIES AND requiring forward progress (a pass that + // embeds 0 while count>0 stops) so a writer outrunning embed can't spin + // forever. + const MAX_REENTRIES = 3; + let reentries = 0; + let lastReentryEmbedded = 0; + const maybeReenter = async (): Promise => { + // Scoped to PACED runs: pacing lengthens the run, which is what widens the + // behind-cursor window. Unpaced runs keep prior (single-pass) behavior. + if (!pacer.snapshot().enabled) return false; + if (effectiveSignal.aborted) return false; + if (reentries >= MAX_REENTRIES) return false; + const remaining = await engine.countStaleChunks(sourceOpt); + if (remaining === 0) return false; + if (result.embedded === lastReentryEmbedded) return false; // no forward progress + lastReentryEmbedded = result.embedded; + reentries++; + afterPageId = 0; + afterChunkIndex = -1; + afterUpdatedAt = null; + serr(`\n [embed] re-entry ${reentries}/${MAX_REENTRIES}: ${remaining} stale chunk(s) appeared during the run; rescanning from start.`); + return true; + }; + try { // eslint-disable-next-line no-constant-condition while (true) { @@ -684,17 +957,22 @@ async function embedAllStale( break; } - const batch = await engine.listStaleChunks({ - batchSize: PAGE_SIZE, - afterPageId, - afterChunkIndex, - ...(orderBy === 'updated_desc' && { - orderBy, - afterUpdatedAt, + const batch = await observed(pacer, () => + engine.listStaleChunks({ + batchSize: PAGE_SIZE, + afterPageId, + afterChunkIndex, + ...(orderBy === 'updated_desc' && { + orderBy, + afterUpdatedAt, + }), + ...(sourceId && { sourceId }), }), - ...(sourceId && { sourceId }), - }); - if (batch.length === 0) break; + ); + if (batch.length === 0) { + if (await maybeReenter()) continue; + break; + } totalChunksLoaded += batch.length; // Advance cursor to last row in this batch. @@ -729,7 +1007,7 @@ async function embedAllStale( try { const embeddings = await embedBatchWithBackoff(stale.map(c => c.chunk_text), { abortSignal: effectiveSignal }); // Re-fetch existing chunks and merge to avoid deleting non-stale chunks. - const existing = await engine.getChunks(slug, { sourceId: keySourceId }); + const existing = await observed(pacer, () => engine.getChunks(slug, { sourceId: keySourceId })); const staleIdxToEmbedding = new Map(); for (let j = 0; j < stale.length; j++) { staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]); @@ -741,14 +1019,16 @@ async function embedAllStale( embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined, token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), })); - await engine.upsertChunks(slug, merged, { sourceId: keySourceId }); + await observed(pacer, () => engine.upsertChunks(slug, merged, { sourceId: keySourceId })); // v0.41.31: stamp provenance after the page's chunks are embedded — // but only when EVERY chunk was stale (fully re-embedded this pass). // A partially-stale page keeps preserved chunks of unknown/old // provenance, so don't claim it's current. (After invalidate, a // signature-drifted page IS fully stale → this stamps it.) if (signature && stale.length === existing.length) { - await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }); + await observed(pacer, () => + engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }), + ); } result.embedded += stale.length; } catch (e: unknown) { @@ -763,6 +1043,17 @@ async function embedAllStale( // Use staleCount as the estimated total for progress (not exact after // pagination starts, but directionally correct). onProgress?.(totalProcessedPages, Math.ceil(staleCount / PAGE_SIZE) * keys.length, result.embedded); + // Cooperative DB-contention pace between keys (no-op when unpaced). + // E-4 (Codex P1): pace() is subject to the EXTERNAL abort only, NOT the + // wall-clock budget — a contended DB's sleep must not be cut by the + // budget timer before its time is credited. Re-arm the budget right + // after each sleep so accrued sleep never eats into work time. + try { + await pacer.pace(externalSignal); + rearmBudgetForPacing(); + } catch (e) { + if (!(e instanceof AbortError)) throw e; + } } // v0.41.15.0: migrated to shared runSlidingPool. The pool checks @@ -778,8 +1069,14 @@ async function embedAllStale( failureLabel: (key) => key, }); + // E-4: extend the work budget by any paced-sleep time accrued this batch. + rearmBudgetForPacing(); + // If we got fewer rows than PAGE_SIZE, we've reached the end. - if (batch.length < PAGE_SIZE) break; + if (batch.length < PAGE_SIZE) { + if (await maybeReenter()) continue; + break; + } } } finally { if (budgetTimer) clearTimeout(budgetTimer); diff --git a/src/commands/jobs.ts b/src/commands/jobs.ts index a2d7db0a6..029b0c167 100644 --- a/src/commands/jobs.ts +++ b/src/commands/jobs.ts @@ -8,6 +8,7 @@ import { MinionQueue } from '../core/minions/queue.ts'; import { MinionWorker } from '../core/minions/worker.ts'; import { WORKER_EXIT_RSS_WATCHDOG } from '../core/minions/worker-exit-codes.ts'; import type { MinionJob, MinionJobStatus } from '../core/minions/types.ts'; +import type { PaceKeyOverrides } from '../core/pace-mode.ts'; import { loadConfig, isThinClient } from '../core/config.ts'; import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts'; import { parseNiceValue, applyNiceness, getEffectiveNiceness, formatNice } from '../core/minions/niceness.ts'; @@ -1398,6 +1399,18 @@ export async function registerBuiltinHandlers( slugs: Array.isArray(job.data.slugs) ? (job.data.slugs as string[]) : undefined, all: !!job.data.all, stale: job.data.all ? false : (job.data.stale !== false), + sourceId: typeof job.data.sourceId === 'string' ? job.data.sourceId : undefined, + // CX1+CX5: pace overrides ride in the job payload as explicit overrides + // only; runEmbedCore re-resolves env > config > bundle at execution so + // GBRAIN_PACE_* still wins during an incident. + ...(job.data.pace && typeof job.data.pace === 'object' + ? { + pace: job.data.pace as { perCallMode?: string; perCall?: PaceKeyOverrides }, + // Serialized from the queued payload → config tier so GBRAIN_PACE_* + // on the worker still wins at execution (Codex P2 escape hatch). + paceFromBackground: true, + } + : {}), onProgress: (done, total, embedded) => { // Fire-and-forget: progress updates are best-effort and must not // block the worker loop. diff --git a/src/commands/sources-harden.ts b/src/commands/sources-harden.ts new file mode 100644 index 000000000..60bc91d21 --- /dev/null +++ b/src/commands/sources-harden.ts @@ -0,0 +1,179 @@ +/** + * gbrain sources harden / pull / unharden — brain-repo git durability (v0.42.44). + * + * gbrain sources harden [--pat-file

] [--branch ] + * [--no-cron] [--no-verify] [--dry-run] [--json] + * gbrain sources pull | --path

[--branch ] + * gbrain sources unharden + * + * `harden`/`unharden` write executables, an OS cron, and a credential helper on + * the host → CLI-only (never MCP). `pull --path` is DB-free (the cron's entry): + * cli.ts dispatches it BEFORE connectEngine so a live PGLite session keeps its + * single-writer lock. + */ + +import type { BrainEngine } from '../core/engine.ts'; +import { + hardenBrainRepo, unhardenBrainRepo, acceptPat, + type DurabilityReport, +} from '../core/brain-repo-durability.ts'; +import { divergenceSafePull, detectDefaultBranch } from '../core/git-remote.ts'; +import { setCliExitVerdict } from '../core/cli-force-exit.ts'; +import { existsSync } from 'fs'; +import { join } from 'path'; + +interface SourceRow { id: string; local_path: string | null; config: unknown; } + +function flagVal(args: string[], name: string): string | undefined { + const i = args.indexOf(name); + if (i !== -1 && i + 1 < args.length) return args[i + 1]; + const pref = `${name}=`; + const hit = args.find(a => a.startsWith(pref)); + return hit ? hit.slice(pref.length) : undefined; +} + +function configHost(config: unknown): string | null { + try { + const url = (config as Record)?.remote_url; + if (typeof url === 'string' && url) return new URL(url).hostname; + } catch { /* */ } + return null; +} + +async function loadSourceRows(engine: BrainEngine, id: string | undefined, all: boolean): Promise { + if (all) { + return engine.executeRaw(`SELECT id, local_path, config FROM sources WHERE local_path IS NOT NULL ORDER BY id`); + } + if (!id) throw new Error('Usage: gbrain sources harden [--pat-file

] [--branch ] [--no-cron] [--no-verify] [--dry-run] [--json]'); + return engine.executeRaw(`SELECT id, local_path, config FROM sources WHERE id = $1`, [id]); +} + +// ── harden ────────────────────────────────────────────────────────────────── + +export async function runHarden(engine: BrainEngine, args: string[]): Promise { + const all = args.includes('--all'); + const id = all ? undefined : args.find(a => !a.startsWith('--') + && a !== flagVal(args, '--pat-file') && a !== flagVal(args, '--branch')); + const json = args.includes('--json'); + const dryRun = args.includes('--dry-run'); + const installCron = !args.includes('--no-cron'); + const verify = !args.includes('--no-verify'); + const branch = flagVal(args, '--branch'); + const patFile = flagVal(args, '--pat-file'); + + const pat = acceptPat({ patFile }); + for (const w of pat?.warnings ?? []) console.error(`[gbrain] ${w}`); + + const rows = await loadSourceRows(engine, id, all); + if (rows.length === 0) { + console.error(all ? 'No sources with a local_path to harden.' : `Source "${id}" not found.`); + process.exit(1); + } + + // --all guard (codex): one PAT must not silently span multiple hosts/accounts. + if (all && pat) { + const hosts = new Set(rows.map(r => configHost(r.config)).filter(Boolean)); + if (hosts.size > 1) { + console.error(`[gbrain] Refusing --all with one PAT across multiple hosts (${[...hosts].join(', ')}). Harden each source with its own --pat-file.`); + process.exit(2); + } + } + + const reports: DurabilityReport[] = []; + for (const row of rows) { + if (!row.local_path || !existsSync(join(row.local_path, '.git'))) { + console.error(`[${row.id}] skipped — no local git repo at ${row.local_path ?? '(none)'}`); + continue; + } + const report = await hardenBrainRepo({ + repoPath: row.local_path, sourceId: row.id, branch, + pat: pat?.token, installCron, verify, dryRun, + logger: json ? undefined : (l) => console.error(` ${l}`), + }); + reports.push(report); + if (!json) renderReport(report); + } + + if (json) console.log(JSON.stringify({ reports }, null, 2)); + + // Non-zero exit if any source needs attention, so cron/automation notices. + // Route through setCliExitVerdict — a raw process.exitCode write is zeroed by + // the owned-verdict flush-exit (#2084 / PGLite-Emscripten pollution defense). + if (reports.some(r => r.needs_attention.length > 0)) setCliExitVerdict(3); +} + +function renderReport(r: DurabilityReport): void { + console.log(`\n[${r.source_id}] durability — ${r.repo_path} (branch ${r.branch})`); + for (const s of r.steps) { + const mark = s.status === 'ok' ? '✓' : s.status === 'fixed' ? '+' : s.status === 'skipped' ? '·' : '⚠'; + console.log(` ${mark} ${s.step.padEnd(11)} ${s.detail}`); + } + if (r.needs_attention.length) { + console.log(` NEEDS ATTENTION:`); + for (const n of r.needs_attention) console.log(` - ${n}`); + } + console.log(` clean against origin: ${r.clean_against_origin ? 'yes' : 'no'}`); +} + +// ── pull (DB-free when --path is given) ───────────────────────────────────── + +export async function runPull(engine: BrainEngine | null, args: string[]): Promise { + const path = flagVal(args, '--path'); + const branchFlag = flagVal(args, '--branch'); + + let repoPath: string; + if (path) { + repoPath = path; + } else { + const id = args.find(a => !a.startsWith('--') && a !== branchFlag); + if (!engine || !id) { + console.error('Usage: gbrain sources pull | --path

[--branch ]'); + process.exit(2); + } + const rows = await engine.executeRaw(`SELECT id, local_path, config FROM sources WHERE id = $1`, [id]); + if (rows.length === 0 || !rows[0].local_path) { + console.error(`Source "${id}" not found or has no local_path.`); + process.exit(1); + } + repoPath = rows[0].local_path; + } + + if (!existsSync(join(repoPath, '.git'))) { + console.error(`[gbrain] not a git repo: ${repoPath}`); + process.exit(1); + } + const branch = branchFlag || detectDefaultBranch(repoPath); + const outcome = divergenceSafePull(repoPath, branch); + switch (outcome.status) { + case 'up_to_date': console.log(`up to date (${branch})`); break; + case 'advanced': console.log(`advanced ${outcome.from.slice(0, 7)}→${outcome.to.slice(0, 7)} (${branch})`); break; + case 'skipped_dirty': console.log(`skipped — working tree dirty (${branch})`); break; + case 'conflict_aborted': + console.error(`[gbrain] ${outcome.detail}`); + process.exit(3); + } +} + +// ── unharden ──────────────────────────────────────────────────────────────── + +export async function runUnharden(engine: BrainEngine, args: string[]): Promise { + const id = args.find(a => !a.startsWith('--')); + if (!id) { + console.error('Usage: gbrain sources unharden '); + process.exit(2); + } + const rows = await engine.executeRaw(`SELECT id, local_path, config FROM sources WHERE id = $1`, [id]); + if (rows.length === 0) { + console.error(`Source "${id}" not found.`); + process.exit(1); + } + const steps = await unhardenBrainRepo({ + repoPath: rows[0].local_path ?? '', + sourceId: rows[0].id, + logger: (l) => console.error(l), + }); + for (const s of steps) { + const mark = s.status === 'fixed' ? '+' : '·'; + console.log(` ${mark} ${s.step}: ${s.detail}`); + } +} diff --git a/src/commands/sources.ts b/src/commands/sources.ts index 352ccc863..e6bac3b37 100644 --- a/src/commands/sources.ts +++ b/src/commands/sources.ts @@ -130,6 +130,8 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise { let displayName: string | undefined; let federated: boolean | null = null; let cloneDir: string | undefined; + let patFile: string | undefined; + let noHarden = false; for (let i = 1; i < args.length; i++) { const a = args[i]; @@ -139,6 +141,8 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise { if (a === '--federated') { federated = true; continue; } if (a === '--no-federated') { federated = false; continue; } if (a === '--clone-dir') { cloneDir = args[++i]; continue; } + if (a === '--pat-file') { patFile = args[++i]; continue; } + if (a === '--no-harden') { noHarden = true; continue; } console.error(`Unknown flag: ${a}`); process.exit(2); } @@ -181,6 +185,36 @@ async function runAdd(engine: BrainEngine, args: string[]): Promise { console.log( ` federated: ${fed}${fed ? ' — appears in cross-source default search' : ' — only searched when explicitly named via --source'}`, ); + + // v0.42.44 — auto-harden managed clones for git durability the moment a brain + // repo is added with a PAT. Best-effort: NEVER fail `add` if hardening fails. + // Only managed clones (gbrain owns the working tree); --path repos are unowned. + if (finalRemoteUrl && created.local_path && !noHarden) { + try { + const { hardenBrainRepo, acceptPat } = await import('../core/brain-repo-durability.ts'); + const pat = acceptPat({ patFile }); + for (const w of pat?.warnings ?? []) console.error(`[gbrain] ${w}`); + if (!pat) { + console.error('[gbrain] No PAT provided (--pat-file or GBRAIN_GITHUB_PAT) — skipping durability hardening.'); + console.error(` Run \`gbrain sources harden ${id} --pat-file

\` later to enable auto-push.`); + } else { + console.error('[gbrain] Hardening brain repo for durability…'); + const report = await hardenBrainRepo({ + repoPath: created.local_path, sourceId: id, pat: pat.token, + logger: (l) => console.error(` ${l}`), + }); + if (report.needs_attention.length) { + console.error('[gbrain] Durability hardened with warnings:'); + for (const n of report.needs_attention) console.error(` - ${n}`); + } else { + console.error('[gbrain] Durability hardened ✓'); + } + } + } catch (e) { + console.error(`[gbrain] Durability hardening skipped (non-fatal): ${(e as Error).message}`); + console.error(` Run \`gbrain sources harden ${id}\` to retry.`); + } + } } /** @@ -366,6 +400,16 @@ async function runRemove(engine: BrainEngine, args: string[]): Promise { } } + // v0.42.44 — tear down durability scaffolding BEFORE the row is deleted (we + // need the path/label while it still exists). Best-effort; tolerates missing + // repo/cron/credential independently. + try { + const { unhardenBrainRepo } = await import('../core/brain-repo-durability.ts'); + await unhardenBrainRepo({ repoPath: src.local_path ?? '', sourceId: id, logger: (l) => console.error(l) }); + } catch (e) { + console.error(`[gbrain] durability teardown skipped (non-fatal): ${(e as Error).message}`); + } + await engine.executeRaw(`DELETE FROM sources WHERE id = $1`, [id]); const pageCount = impact?.pageCount ?? 0; console.log(`Removed source "${id}" (${pageCount} pages + dependent rows cascaded).`); @@ -1265,6 +1309,10 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise [--pat-file

] [--branch ] [--no-cron] [--no-verify] [--dry-run] [--json] + v0.42.44 — make a brain repo durable: local + auto-push hook, committed commit-push helper, + always-on agent rules, 30-min pull cron, and + repo-scoped credential. Idempotent. + pull | --path

[--branch ] + Divergence-safe rebase-pull (skip-on-dirty). + --path is DB-free (the harden cron's entry). + unharden Remove durability cron/hook/credential wiring. Source id: [a-z0-9-]{1,32}. Immutable citation key. diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 09f8bafa9..d788cd201 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -83,6 +83,9 @@ import { type OpCheckpointKey, } from '../core/op-checkpoint.ts'; import { registerCleanup } from '../core/process-cleanup.ts'; +import { type DbPacer, createDbPacer, createNoopPacer, observed } from '../core/db-pacer.ts'; +import { resolvePaceMode, loadPaceModeConfig, readPaceEnv } from '../core/pace-mode.ts'; +import { AbortError } from '../core/abort-check.ts'; /** * v0.42.x (#1794) -- resumable incremental sync checkpoint. @@ -2366,6 +2369,27 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise config > bundle (env = incident escape + // hatch); fail-open so pacing never breaks a sync. + let pacer: DbPacer = createNoopPacer(); + try { + const pcfg = await loadPaceModeConfig(engine); + const { envMode, envOverrides } = readPaceEnv(); + const knobs = resolvePaceMode({ + mode: pcfg.mode, + configOverrides: pcfg.configOverrides, + envMode, + envOverrides, + }); + if (knobs.enabled) pacer = createDbPacer({ bundle: knobs }); + } catch { + pacer = createNoopPacer(); + } + async function importOnePath(eng: BrainEngine, path: string): Promise { const filePath = join(syncRepoPath, path); if (!existsSync(filePath)) { @@ -2396,13 +2420,24 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise1 / --all the stuck file is the // begin-line with no matching completion in the in-flight set. if (process.env.GBRAIN_SYNC_TRACE) serr(`[sync] begin import: ${path}`); + // paced-backfill: acquire a DB-write permit (caps total concurrent writes + // across all worker engines). Throws AbortError on cancel while waiting — + // treat as a clean skip; the worker loop sees signal.aborted next tick. + let permit; + try { + permit = await pacer.acquire(opts.signal); + } catch (e) { + if (e instanceof AbortError) return; + throw e; + } try { // v0.18.0+ multi-source: thread `opts.sourceId` so per-page tx writes // (putPage / getTags / addTag / removeTag / deleteChunks / upsertChunks // / addLink) target (sourceId, slug). Pre-fix the schema DEFAULT // 'default' was applied even for non-default sources, fabricating // duplicate rows that crashed bare-slug subqueries with Postgres 21000. - const result = await importFile(eng, filePath, path, { noEmbed, sourceId: opts.sourceId, activePack: syncActivePack }); + const result = await observed(pacer, () => + importFile(eng, filePath, path, { noEmbed, sourceId: opts.sourceId, activePack: syncActivePack })); if (result.status === 'imported') { chunksCreated += result.chunks; pagesAffected.push(result.slug); @@ -2427,12 +2462,23 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise AGENTS.md) + * 6. a DB-free pull cron (every 30 min) + * 7. verify by authenticated push-probe (proves push auth; no heartbeat) + * + * Trust boundary (this is gbrain's FIRST push path + FIRST secret storage): + * - The hook is LOCAL + untracked so a pulled commit can't rewrite executed + * code next to the PAT. Both hook and helper render from ONE bash template + * (PUSH_RETRY) — DRY at the TS source level, NOT by the hook sourcing a + * repo-controlled script. + * - Credential is repo-scoped (local git config), token redacted everywhere + * via shell-redact's exact-value scrubber, store file 0600. + * + * CLI-only by design (writes executables + an OS cron + a credential helper on + * the host): never exposed over MCP. + */ + +import { + existsSync, readFileSync, writeFileSync, mkdirSync, chmodSync, rmSync, statSync, appendFileSync, +} from 'fs'; +import { join, dirname, relative, isAbsolute } from 'path'; +import { execFileSync, execSync } from 'child_process'; +import { + GIT_ENV, GIT_ENV_AUTH, divergenceSafePull, detectDefaultBranch, pushProbe, + type PullOutcome, type PushProbeResult, +} from './git-remote.ts'; +import { findResolverFile, RESOLVER_FILENAMES } from './resolver-filenames.ts'; +import { redactSecretsInText } from './minions/handlers/shell-redact.ts'; +// Static import → bundled into the --compile binary so the taxonomy never drifts +// and needs no runtime skills/ directory. +import filingRulesDoc from '../../skills/_brain-filing-rules.json'; + +// ── Types ─────────────────────────────────────────────────────────────────── + +export type StepName = + | 'pull' | 'credential' | 'hook' | 'helper' | 'agents' | 'cron' | 'verify' | 'commit'; +export type StepStatus = 'ok' | 'fixed' | 'skipped' | 'needs_attention'; + +export interface DurabilityStep { + step: StepName; + status: StepStatus; + detail: string; // ALWAYS redacted — never contains the PAT +} + +export interface DurabilityReport { + source_id: string; + repo_path: string; + branch: string; + steps: DurabilityStep[]; + missing: string[]; // what was missing on entry + fixed: string[]; // what this run changed + needs_attention: string[]; + clean_against_origin: boolean; +} + +export interface HardenOpts { + repoPath: string; + sourceId: string; + branch?: string; // default: detectDefaultBranch + pat?: string; // already-loaded token; never logged + installCron?: boolean; // default true + verify?: boolean; // default true + dryRun?: boolean; + intervalSec?: number; // cron cadence; default 1800 + logger?: (line: string) => void; +} + +export interface UnhardenOpts { + repoPath: string; + sourceId: string; + logger?: (line: string) => void; +} + +// ── Banners / markers (idempotency keys) ──────────────────────────────────── + +const HOOK_BANNER = '# gbrain brain-durability post-commit hook (v0.42.44+)'; +const HELPER_BANNER = '# gbrain brain-commit-push helper (v0.42.44+)'; +const AGENTS_BEGIN = ''; +const AGENTS_END = ''; +const HELPER_REL = 'scripts/brain-commit-push.sh'; +const CRED_MANAGED_KEY = 'gbrain.durability.managedcredential'; + +function gbrainHome(): string { + return process.env.GBRAIN_HOME || join(process.env.HOME || '', '.gbrain'); +} + +/** Resolve the gbrain CLI path for the cron wrapper (inlined to avoid a + * core→commands import). which gbrain → process.execPath → argv[1] → "gbrain". */ +function resolveGbrainCliPath(): string { + try { + const which = execSync('which gbrain', { encoding: 'utf-8', stdio: ['ignore', 'pipe', 'ignore'] }).trim(); + if (which) return which; + } catch { /* not on PATH */ } + const exec = process.execPath ?? ''; + if (exec.endsWith('/gbrain') || exec.endsWith('\\gbrain.exe')) return exec; + const arg1 = process.argv[1] ?? ''; + if (arg1.endsWith('/gbrain') || arg1.endsWith('\\gbrain.exe')) return arg1; + return 'gbrain'; +} +function credStoreFile(): string { + return join(gbrainHome(), 'git-credentials'); +} +function pushLogPath(): string { + return join(gbrainHome(), 'brain-push.log'); +} + +// ── Shared bash push-retry template (DRY at the TS source — D7) ────────────── +// Rendered into BOTH the (committed) helper and the (local, untracked) hook so +// there is one source of truth without the hook executing repo-controlled code. +const PUSH_RETRY = `# --- gbrain durability push-retry (generated; one source of truth) --- +brain_push() { + _branch="$1" + _log="\${GBRAIN_HOME:-$HOME/.gbrain}/brain-push.log" + mkdir -p "$(dirname "$_log")" 2>/dev/null || true + _gd="$(git rev-parse --git-dir 2>/dev/null || echo .git)" + # Serialize concurrent pushes (commit bursts) so they coalesce instead of a + # rebase-retry herd. No-op if flock is unavailable. + if command -v flock >/dev/null 2>&1; then + exec 9>"$_gd/gbrain-push.lock" + flock -w 30 9 || { echo "$(date -u +%FT%TZ) [push] lock-timeout $_branch" >>"$_log"; return 0; } + fi + if git push origin "HEAD:$_branch" >>"$_log" 2>&1; then + echo "$(date -u +%FT%TZ) [push] ok $_branch $(git rev-parse --short HEAD 2>/dev/null)" >>"$_log"; return 0 + fi + echo "$(date -u +%FT%TZ) [push] rejected; rebase-pull $_branch" >>"$_log" + if git pull --rebase origin "$_branch" >>"$_log" 2>&1 && git push origin "HEAD:$_branch" >>"$_log" 2>&1; then + echo "$(date -u +%FT%TZ) [push] ok-after-rebase $_branch $(git rev-parse --short HEAD 2>/dev/null)" >>"$_log"; return 0 + fi + git rebase --abort >/dev/null 2>&1 || true + echo "$(date -u +%FT%TZ) [push] LOCAL-ONLY, NEEDS ATTENTION: $_branch @ $(git rev-parse --short HEAD 2>/dev/null) could not reach origin. Run: gbrain sources pull && git push" >>"$_log" + return 1 +}`; + +function renderPostCommitHook(): string { + return `#!/usr/bin/env bash +${HOOK_BANNER} +# LOCAL + untracked — NEVER commit this file. Best-effort background auto-push so +# agent writes don't sit local-only. The real guarantee is ${HELPER_REL}. +# Bypass: git commit --no-verify. +set -euo pipefail + +_branch="$(git rev-parse --abbrev-ref HEAD 2>/dev/null || echo HEAD)" +if [ "$_branch" = "HEAD" ]; then + echo "$(date -u +%FT%TZ) [push] detached HEAD; skip" >> "\${GBRAIN_HOME:-$HOME/.gbrain}/brain-push.log" 2>/dev/null || true + exit 0 +fi + +${PUSH_RETRY} + +# Detach so the commit returns instantly; all output goes to the log. +( brain_push "$_branch" ) /dev/null 2>&1 & +disown 2>/dev/null || true +exit 0 +`; +} + +function renderCommitPushHelper(): string { + return `#!/usr/bin/env bash +${HELPER_BANNER} +# THE DURABILITY GUARANTEE: add -> commit -> push, atomically. Refuses to exit 0 +# without a confirmed push. Usage: +# scripts/brain-commit-push.sh "message" [path ...] +# scripts/brain-commit-push.sh --push-only [branch] +set -euo pipefail + +${PUSH_RETRY} + +_branch="$(git rev-parse --abbrev-ref HEAD)" +if [ "\${1:-}" = "--push-only" ]; then + brain_push "\${2:-$_branch}"; exit $? +fi + +_msg="\${1:?usage: brain-commit-push.sh [paths...]}"; shift || true +# Pull first so the local tree is current before we stage. +git fetch origin >/dev/null 2>&1 || true +git pull --rebase origin "$_branch" || { git rebase --abort >/dev/null 2>&1 || true; echo "rebase conflict: manual attention needed" >&2; exit 3; } + +# EXPLICIT paths only — never a blind 'git add -A' (would risk committing +# secrets, temp files, or unrelated edits). +if [ "$#" -eq 0 ]; then + echo "refusing blind 'git add -A' — pass explicit path(s) to commit" >&2; exit 2 +fi +git add -- "$@" +if git diff --cached --quiet; then echo "nothing to commit"; exit 0; fi +git commit -m "$_msg" + +if brain_push "$_branch"; then exit 0; fi +echo "PUSH FAILED — commit is local-only, NEEDS ATTENTION (see ${'$'}{GBRAIN_HOME:-$HOME/.gbrain}/brain-push.log)" >&2 +exit 4 +`; +} + +// ── Managed AGENTS/RESOLVER block (taxonomy from filing rules; no drift) ───── + +function renderTaxonomyLines(): string { + const seen = new Set(); + const lines: string[] = []; + for (const r of (filingRulesDoc as any).rules ?? []) { + const dir = String(r.directory || '').trim(); + if (!dir || seen.has(dir)) continue; + seen.add(dir); + lines.push(` - \`${dir}\` — ${r.kind}`); + } + return lines.join('\n'); +} + +function renderManagedBlock(): string { + return `${AGENTS_BEGIN} + +## Brain durability rules (always on) + +1. **Deterministic filing — never use /tmp as storage.** Every persistent output + goes to its taxonomy path (canonical, from \`skills/_brain-filing-rules.json\`): +${renderTaxonomyLines()} + Writing to /tmp, scratch dirs, or outside the repo is forbidden for anything + meant to persist. + +2. **Every write is committed AND pushed — push is never deferred.** After any + persistent write, run \`scripts/brain-commit-push.sh "" \` (it commits, + pushes, and FAILS LOUDLY if the push doesn't land), then confirm links resolve + with \`gbrain check-resolvable\`. Do not move on until the push succeeded. The + post-commit hook is only a best-effort fallback — the helper is the guarantee. + +3. **Pull before you touch anything.** Run \`git fetch && git pull --rebase\` at + session start and again before each batch of writes, so a long-lived session + never edits a stale tree (a cron also pulls every ~30 min). +${AGENTS_END}`; +} + +/** Patch the active resolver file with the managed block (idempotent). */ +function patchResolverFile(repoPath: string, dryRun: boolean): { status: StepStatus; detail: string } { + const existing = findResolverFile(repoPath); + const target = existing ?? join(repoPath, RESOLVER_FILENAMES[1]); // default AGENTS.md + const block = renderManagedBlock(); + const name = relative(repoPath, target) || target; + + let current = ''; + if (existsSync(target)) current = readFileSync(target, 'utf-8'); + + let next: string; + const b = current.indexOf(AGENTS_BEGIN); + const e = current.indexOf(AGENTS_END); + if (b !== -1 && e !== -1 && e > b) { + const before = current.slice(0, b); + const after = current.slice(e + AGENTS_END.length); + next = before + block + after; + if (next === current) return { status: 'ok', detail: `${name}: durability rules already current` }; + } else if (current.trim().length === 0) { + next = block + '\n'; + } else { + next = current.replace(/\s*$/, '') + '\n\n' + block + '\n'; + } + + if (dryRun) return { status: 'fixed', detail: `${name}: would write durability rules (dry-run)` }; + writeFileSync(target, next); + return { status: 'fixed', detail: `${name}: durability rules written` }; +} + +// ── Local untracked post-commit hook (D9) ─────────────────────────────────── + +/** Resolve the active hooks dir (honors a pre-existing core.hooksPath). */ +function resolveHooksDir(repoPath: string): { dir: string; tracked: boolean } { + let hooksPath = ''; + try { + hooksPath = execFileSync('git', ['-C', repoPath, 'config', '--get', 'core.hooksPath'], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + } catch { /* unset — normal */ } + if (hooksPath) { + const dir = isAbsolute(hooksPath) ? hooksPath : join(repoPath, hooksPath); + // A hooksPath outside .git/ (e.g. .githooks) is a TRACKED location. + const tracked = !dir.includes(`${join('.git', '')}`) && !dir.endsWith('.git/hooks'); + return { dir, tracked }; + } + return { dir: join(repoPath, '.git', 'hooks'), tracked: false }; +} + +/** Ensure a repo-relative path is in .git/info/exclude so our hook stays untracked. */ +function ensureExcluded(repoPath: string, relPath: string): void { + const exclude = join(repoPath, '.git', 'info', 'exclude'); + try { + mkdirSync(dirname(exclude), { recursive: true }); + let body = existsSync(exclude) ? readFileSync(exclude, 'utf-8') : ''; + if (!body.split('\n').some(l => l.trim() === relPath)) { + if (body.length && !body.endsWith('\n')) body += '\n'; + body += `${relPath}\n`; + writeFileSync(exclude, body); + } + } catch { /* best-effort */ } +} + +function installLocalHook(repoPath: string, dryRun: boolean): { status: StepStatus; detail: string } { + const { dir, tracked } = resolveHooksDir(repoPath); + const hookPath = join(dir, 'post-commit'); + const script = renderPostCommitHook(); + + if (existsSync(hookPath)) { + const cur = readFileSync(hookPath, 'utf-8'); + if (cur.includes(HOOK_BANNER)) { + if (cur === script) return { status: 'ok', detail: `${relative(repoPath, hookPath)} already current` }; + if (dryRun) return { status: 'fixed', detail: `would refresh ${relative(repoPath, hookPath)} (dry-run)` }; + writeFileSync(hookPath, script); chmodSync(hookPath, 0o755); + return { status: 'fixed', detail: `refreshed ${relative(repoPath, hookPath)}` }; + } + // Foreign post-commit hook present — back it up, then install ours. + if (!dryRun) writeFileSync(hookPath + '.bak', cur); + } + if (dryRun) return { status: 'fixed', detail: `would install ${relative(repoPath, hookPath)} (dry-run)` }; + mkdirSync(dir, { recursive: true }); + writeFileSync(hookPath, script); chmodSync(hookPath, 0o755); + // If the hooks dir is a tracked location (.githooks via frontmatter), keep OUR + // hook untracked so it never becomes repo-controlled code (D9). + if (tracked) ensureExcluded(repoPath, relative(repoPath, hookPath)); + return { status: 'fixed', detail: `installed local untracked ${relative(repoPath, hookPath)}` }; +} + +function uninstallLocalHook(repoPath: string): boolean { + const { dir } = resolveHooksDir(repoPath); + const hookPath = join(dir, 'post-commit'); + if (!existsSync(hookPath)) return false; + if (!readFileSync(hookPath, 'utf-8').includes(HOOK_BANNER)) return false; + rmSync(hookPath); + if (existsSync(hookPath + '.bak')) { writeFileSync(hookPath, readFileSync(hookPath + '.bak')); rmSync(hookPath + '.bak'); } + return true; +} + +// ── Committed helper ──────────────────────────────────────────────────────── + +function installHelper(repoPath: string, dryRun: boolean): { status: StepStatus; detail: string } { + const helperPath = join(repoPath, HELPER_REL); + const script = renderCommitPushHelper(); + if (existsSync(helperPath) && readFileSync(helperPath, 'utf-8') === script) { + // Ensure exec bit even when content is current. + try { chmodSync(helperPath, 0o755); } catch { /* */ } + return { status: 'ok', detail: `${HELPER_REL} already current` }; + } + if (dryRun) return { status: 'fixed', detail: `would write ${HELPER_REL} (dry-run)` }; + mkdirSync(dirname(helperPath), { recursive: true }); + writeFileSync(helperPath, script); chmodSync(helperPath, 0o755); + return { status: 'fixed', detail: `wrote ${HELPER_REL}` }; +} + +// ── Repo-scoped credential wiring (D11) ───────────────────────────────────── + +function gitConfigGet(repoPath: string, key: string, localOnly = false): string { + try { + const scope = localOnly ? ['--local'] : []; + return execFileSync('git', ['-C', repoPath, 'config', ...scope, '--get', key], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + } catch { return ''; } +} +function gitConfigSet(repoPath: string, key: string, value: string): void { + execFileSync('git', ['-C', repoPath, 'config', key, value], { + stdio: 'ignore', timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }); +} +function gitConfigUnset(repoPath: string, key: string): void { + try { + execFileSync('git', ['-C', repoPath, 'config', '--unset-all', key], { + stdio: 'ignore', timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }); + } catch { /* not set */ } +} + +function remoteHost(repoPath: string): string { + try { + const url = execFileSync('git', ['-C', repoPath, 'remote', 'get-url', 'origin'], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + return new URL(url).hostname || 'github.com'; + } catch { return 'github.com'; } +} + +/** + * Wire a repo-scoped credential. If a working helper is already configured, + * reuse it (no plaintext write). Otherwise fall back to a 0600 store file wired + * via the repo's LOCAL config only (least-privilege — not every github.com + * remote under the account). The token is never returned or logged. + */ +function wireRepoCredential(repoPath: string, pat: string, dryRun: boolean): { status: StepStatus; detail: string } { + // Only a REPO-LOCAL helper triggers reuse. A global helper (e.g. the macOS + // osxkeychain default) must NOT block wiring the explicitly-provided PAT — + // the user gave us a token expressly to use for this repo (D11). + const existing = gitConfigGet(repoPath, 'credential.helper', /*localOnly*/ true); + const ours = gitConfigGet(repoPath, CRED_MANAGED_KEY, true) === 'true'; + if (existing && !ours) { + return { status: 'ok', detail: `reusing repo-local credential.helper (no plaintext store written)` }; + } + + const host = remoteHost(repoPath); + const store = credStoreFile(); + const line = `https://x-access-token:${pat}@${host}`; + // Already fully wired by us with this credential present → idempotent no-op. + if (ours && existing && existsSync(store) && readFileSync(store, 'utf-8').split('\n').includes(line)) { + return { status: 'ok', detail: `repo-scoped credential already wired for ${host}` }; + } + if (dryRun) return { status: 'fixed', detail: 'would wire repo-scoped credential (dry-run)' }; + + mkdirSync(dirname(store), { recursive: true, mode: 0o700 }); + try { chmodSync(gbrainHome(), 0o700); } catch { /* */ } + let body = existsSync(store) ? readFileSync(store, 'utf-8') : ''; + if (!body.split('\n').some(l => l === line)) { + if (body.length && !body.endsWith('\n')) body += '\n'; + body += `${line}\n`; + writeFileSync(store, body, { mode: 0o600 }); + } + try { chmodSync(store, 0o600); } catch { /* */ } + // Repo-LOCAL wiring → only this repo uses the store (D11). + gitConfigSet(repoPath, 'credential.helper', `store --file ${store}`); + gitConfigSet(repoPath, CRED_MANAGED_KEY, 'true'); + return { status: 'fixed', detail: `wired repo-scoped credential for ${host} (store 0600)` }; +} + +function removeCredentialWiring(repoPath: string): boolean { + if (gitConfigGet(repoPath, CRED_MANAGED_KEY, true) !== 'true') return false; // only what we created + gitConfigUnset(repoPath, 'credential.helper'); + gitConfigUnset(repoPath, CRED_MANAGED_KEY); + return true; +} + +// ── Minimal DB-free pull cron (D2 + D12) ──────────────────────────────────── + +function cronLabel(sourceId: string): string { + return `com.gbrain.brain-pull.${sourceId.replace(/[^A-Za-z0-9._-]/g, '_')}`; +} +function cronWrapperPath(sourceId: string): string { + return join(gbrainHome(), `brain-pull-${sourceId.replace(/[^A-Za-z0-9._-]/g, '_')}.sh`); +} +function launchdPlistPath(sourceId: string): string { + return join(process.env.HOME || '', 'Library', 'LaunchAgents', `${cronLabel(sourceId)}.plist`); +} + +/** Pure cron-wrapper renderer (DB-free pull; secret-free — sources the shell + * profile rather than baking keys in). Exported for tests. */ +export function renderCronWrapper(sourceId: string, repoPath: string, branch: string, cli: string, logPath: string): string { + const q = (s: string) => s.replace(/'/g, "'\\''"); + return `#!/bin/bash +# Auto-generated by gbrain sources harden — DB-free durability pull (${sourceId}). +# Sources the shell profile for secrets, then runs the hardened, DB-free pull. +[ -f ~/.zshenv ] && source ~/.zshenv 2>/dev/null +source ~/.zshrc 2>/dev/null || source ~/.bashrc 2>/dev/null || true +# Self-disable if the captured checkout is gone (rename/relocation). +if [ ! -d '${q(repoPath)}/.git' ]; then + echo "$(date -u +%FT%TZ) [cron] path gone, skipping: ${q(repoPath)}" >> "${q(logPath)}" 2>/dev/null || true + exit 0 +fi +exec '${q(cli)}' sources pull --path '${q(repoPath)}' --branch '${q(branch)}' +`; +} + +function writeCronWrapper(sourceId: string, repoPath: string, branch: string): string { + const wrapper = cronWrapperPath(sourceId); + const body = renderCronWrapper(sourceId, repoPath, branch, resolveGbrainCliPath(), pushLogPath()); + mkdirSync(dirname(wrapper), { recursive: true }); + writeFileSync(wrapper, body, { mode: 0o755 }); + return wrapper; +} + +export function generateBrainPullPlist(label: string, wrapperPath: string, home: string, intervalSec: number): string { + const esc = (s: string) => s.replace(/&/g, '&').replace(//g, '>'); + return ` + + + + Label${esc(label)} + ProgramArguments${esc(wrapperPath)} + StartInterval${intervalSec} + StandardOutPath${esc(home)}/.gbrain/brain-pull.log + StandardErrorPath${esc(home)}/.gbrain/brain-pull.err + +`; +} + +function installDurabilityCron(sourceId: string, repoPath: string, branch: string, intervalSec: number, dryRun: boolean): { status: StepStatus; detail: string } { + const wrapper = dryRun ? cronWrapperPath(sourceId) : writeCronWrapper(sourceId, repoPath, branch); + const home = process.env.HOME || ''; + if (process.platform === 'darwin') { + const plistPath = launchdPlistPath(sourceId); + if (dryRun) return { status: 'fixed', detail: `would install launchd ${cronLabel(sourceId)} every ${intervalSec}s (dry-run)` }; + mkdirSync(dirname(plistPath), { recursive: true }); + writeFileSync(plistPath, generateBrainPullPlist(cronLabel(sourceId), wrapper, home, intervalSec)); + try { execSync(`launchctl unload "${plistPath}" 2>/dev/null`, { stdio: 'ignore' }); } catch { /* */ } + try { execSync(`launchctl load "${plistPath}"`, { stdio: 'ignore' }); } catch { /* loaded best-effort */ } + return { status: 'fixed', detail: `launchd ${cronLabel(sourceId)} every ${intervalSec}s` }; + } + // Linux: crontab line, deduped on the label marker. + const minutes = Math.max(1, Math.round(intervalSec / 60)); + const marker = `# ${cronLabel(sourceId)}`; + const cronLine = `*/${minutes} * * * * ${wrapper} ${marker}`; + if (dryRun) return { status: 'fixed', detail: `would install crontab (every ${minutes}m) (dry-run)` }; + let existingCron = ''; + try { existingCron = execSync('crontab -l 2>/dev/null', { encoding: 'utf-8' }); } catch { /* none */ } + const kept = existingCron.split('\n').filter(l => l && !l.includes(marker)); + const next = [...kept, cronLine, ''].join('\n'); + try { + execSync('crontab -', { input: next, stdio: ['pipe', 'ignore', 'ignore'] }); + return { status: 'fixed', detail: `crontab every ${minutes}m` }; + } catch (e) { + return { status: 'needs_attention', detail: `crontab install failed: ${(e as Error).message.slice(0, 120)}` }; + } +} + +function removeDurabilityCron(sourceId: string): boolean { + let removed = false; + if (process.platform === 'darwin') { + const plistPath = launchdPlistPath(sourceId); + if (existsSync(plistPath)) { + try { execSync(`launchctl unload "${plistPath}" 2>/dev/null`, { stdio: 'ignore' }); } catch { /* */ } + rmSync(plistPath); removed = true; + } + } else { + const marker = `# ${cronLabel(sourceId)}`; + try { + const cur = execSync('crontab -l 2>/dev/null', { encoding: 'utf-8' }); + if (cur.includes(marker)) { + const next = cur.split('\n').filter(l => l && !l.includes(marker)).join('\n') + '\n'; + execSync('crontab -', { input: next, stdio: ['pipe', 'ignore', 'ignore'] }); + removed = true; + } + } catch { /* none */ } + } + const wrapper = cronWrapperPath(sourceId); + if (existsSync(wrapper)) { rmSync(wrapper); removed = true; } + return removed; +} + +// ── PAT acceptance (D8) ───────────────────────────────────────────────────── + +export interface AcceptPatResult { token: string; source: string; warnings: string[]; } + +/** + * Resolve a PAT: --pat-file (preferred) > GBRAIN_GITHUB_PAT env. Never a bare CLI + * arg (process-listing leak). Validates non-empty; WARNs loudly on loose perms + * but continues (mirrors GBRAIN_ALLOW_PRIVATE_REMOTES). Returns null if none. + */ +export function acceptPat(opts: { patFile?: string }): AcceptPatResult | null { + const warnings: string[] = []; + if (opts.patFile) { + if (!existsSync(opts.patFile)) throw new Error(`--pat-file not found: ${opts.patFile}`); + try { + const mode = statSync(opts.patFile).mode; + if (mode & 0o077) warnings.push(`WARN: PAT file ${opts.patFile} is group/other-readable (mode ${(mode & 0o777).toString(8)}); chmod 600 it`); + } catch { /* */ } + const token = readFileSync(opts.patFile, 'utf-8').trim(); + if (!token) throw new Error(`--pat-file is empty: ${opts.patFile}`); + return { token, source: 'pat-file', warnings }; + } + const env = (process.env.GBRAIN_GITHUB_PAT || '').trim(); + if (env) return { token: env, source: 'env:GBRAIN_GITHUB_PAT', warnings }; + return null; +} + +// ── Orchestration ─────────────────────────────────────────────────────────── + +function isGitRepo(repoPath: string): boolean { + return existsSync(join(repoPath, '.git')); +} + +function currentBranch(repoPath: string): string { + try { + return execFileSync('git', ['-C', repoPath, 'rev-parse', '--abbrev-ref', 'HEAD'], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + } catch { return 'HEAD'; } +} + +function headSha(repoPath: string): string { + try { + return execFileSync('git', ['-C', repoPath, 'rev-parse', 'HEAD'], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + } catch { return ''; } +} + +function pullDetail(o: PullOutcome): { status: StepStatus; detail: string } { + switch (o.status) { + case 'up_to_date': return { status: 'ok', detail: 'already up to date with origin' }; + case 'advanced': return { status: 'fixed', detail: `advanced ${o.from.slice(0, 7)}→${o.to.slice(0, 7)}` }; + case 'skipped_dirty': return { status: 'skipped', detail: 'working tree dirty — pull skipped (in-progress edits preserved)' }; + case 'conflict_aborted': return { status: 'needs_attention', detail: o.detail }; + } +} + +/** + * Harden a brain repo for durability. Idempotent: a second run on an + * already-hardened repo produces all ok/skipped and NO new commit. + */ +export async function hardenBrainRepo(opts: HardenOpts): Promise { + const { repoPath, sourceId } = opts; + const dryRun = !!opts.dryRun; + const installCron = opts.installCron !== false; + const verify = opts.verify !== false; + const intervalSec = opts.intervalSec ?? 1800; + const redact = opts.pat ? (s: string) => redactSecretsInText(s, new Map([['github_pat', opts.pat!]])) : (s: string) => s; + const log = (l: string) => opts.logger?.(redact(l)); + + if (!isGitRepo(repoPath)) throw new Error(`not a git repo: ${repoPath}`); + + const branch = opts.branch || detectDefaultBranch(repoPath); + const steps: DurabilityStep[] = []; + const push = (step: StepName, r: { status: StepStatus; detail: string }) => { + const s: DurabilityStep = { step, status: r.status, detail: redact(r.detail) }; + steps.push(s); log(`[${step}] ${s.status}: ${s.detail}`); + return s; + }; + + // Refuse on detached HEAD — pushing to a wrong ref is worse than not pushing. + if (currentBranch(repoPath) === 'HEAD') { + push('pull', { status: 'needs_attention', detail: 'detached HEAD — checkout a branch before hardening' }); + } else { + // 1. pull current state + try { push('pull', pullDetail(divergenceSafePull(repoPath, branch))); } + catch (e) { push('pull', { status: 'needs_attention', detail: `fetch/pull failed: ${(e as Error).message.slice(0, 140)}` }); } + } + + // 2. credential + if (opts.pat) push('credential', wireRepoCredential(repoPath, opts.pat, dryRun)); + else push('credential', { status: 'skipped', detail: 'no PAT provided — relying on existing git auth' }); + + // 3. local untracked hook + push('hook', installLocalHook(repoPath, dryRun)); + // 4. committed helper + push('helper', installHelper(repoPath, dryRun)); + // 5. resolver/AGENTS rules + push('agents', patchResolverFile(repoPath, dryRun)); + // 6. cron + if (installCron) push('cron', installDurabilityCron(sourceId, repoPath, branch, intervalSec, dryRun)); + else push('cron', { status: 'skipped', detail: '--no-cron' }); + + // 7. verify (push-probe) + commit scaffolding if push works + let clean = false; + if (verify && !dryRun) { + const probe: PushProbeResult = pushProbe(repoPath, branch, { redactDetail: redact }); + if (!probe.ok) { + push('verify', { status: 'needs_attention', detail: `push-probe failed (${probe.reason}): ${probe.detail}` }); + } else { + push('verify', { status: 'ok', detail: 'push-probe ok — push auth confirmed' }); + // Commit the durability scaffolding (helper + rules) — real content, the + // genuine end-to-end proof (no synthetic heartbeat). No-op when unchanged. + const committed = commitScaffolding(repoPath, branch, redact); + if (committed) push('commit', committed); + clean = headMatchesOrigin(repoPath, branch); + } + } else if (dryRun) { + push('verify', { status: 'skipped', detail: 'dry-run' }); + } else { + push('verify', { status: 'skipped', detail: '--no-verify' }); + } + + const missing = steps.filter(s => s.status === 'fixed').map(s => s.step); + const fixed = missing; + const needs_attention = steps.filter(s => s.status === 'needs_attention').map(s => `${s.step}: ${s.detail}`); + return { source_id: sourceId, repo_path: repoPath, branch, steps, missing, fixed, needs_attention, clean_against_origin: clean }; +} + +function commitScaffolding(repoPath: string, branch: string, redact: (s: string) => string): { status: StepStatus; detail: string } | null { + // Stage only the durability artifacts we manage — never a blind add. + const paths: string[] = [HELPER_REL]; + const resolver = findResolverFile(repoPath); + if (resolver) paths.push(relative(repoPath, resolver)); + try { + execFileSync('git', ['-C', repoPath, 'add', '--', ...paths], { stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV } }); + const staged = execFileSync('git', ['-C', repoPath, 'diff', '--cached', '--name-only'], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + if (!staged) return { status: 'ok', detail: 'scaffolding already committed' }; + execFileSync('git', ['-C', repoPath, 'commit', '-m', 'chore(gbrain): install brain durability scaffolding'], { + stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV }, + }); + execFileSync('git', ['-C', repoPath, ...['-c', 'http.followRedirects=false'], 'push', 'origin', `HEAD:${branch}`], { + stdio: ['ignore', 'pipe', 'pipe'], timeout: 120_000, env: { ...process.env, ...GIT_ENV_AUTH }, + }); + return { status: 'fixed', detail: 'committed + pushed durability scaffolding' }; + } catch (e) { + return { status: 'needs_attention', detail: redact(`scaffolding commit/push failed: ${(e as Error).message.slice(0, 140)}`) }; + } +} + +function headMatchesOrigin(repoPath: string, branch: string): boolean { + try { + const local = headSha(repoPath); + const remote = execFileSync('git', ['-C', repoPath, 'rev-parse', `origin/${branch}`], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + return !!local && local === remote; + } catch { return false; } +} + +/** Remove durability scaffolding: cron, local hook, credential wiring. Leaves + * committed content (helper, resolver block) intact. Idempotent. */ +export async function unhardenBrainRepo(opts: UnhardenOpts): Promise { + const { repoPath, sourceId } = opts; + const steps: DurabilityStep[] = []; + const cronRemoved = removeDurabilityCron(sourceId); + steps.push({ step: 'cron', status: cronRemoved ? 'fixed' : 'skipped', detail: cronRemoved ? 'cron removed' : 'no cron' }); + const hookRemoved = isGitRepo(repoPath) ? uninstallLocalHook(repoPath) : false; + steps.push({ step: 'hook', status: hookRemoved ? 'fixed' : 'skipped', detail: hookRemoved ? 'hook removed' : 'no gbrain hook' }); + const credRemoved = isGitRepo(repoPath) ? removeCredentialWiring(repoPath) : false; + steps.push({ step: 'credential', status: credRemoved ? 'fixed' : 'skipped', detail: credRemoved ? 'credential wiring removed' : 'no gbrain credential wiring' }); + opts.logger?.(steps.map(s => `[${s.step}] ${s.status}: ${s.detail}`).join('\n')); + return steps; +} diff --git a/src/core/db-pacer.ts b/src/core/db-pacer.ts new file mode 100644 index 000000000..fa9573ff4 --- /dev/null +++ b/src/core/db-pacer.ts @@ -0,0 +1,305 @@ +/** + * db-pacer.ts — composable DB-contention pacing primitive (paced-backfill internals). + * + * gbrain's bulk write paths (embed --stale, sync) can saturate a PgBouncer + * transaction-mode pooler and starve the minion supervisor's lock renewals. + * An operator's external SIGSTOP/SIGCONT wrapper proved the idea but was blind + * (out-of-band probe pool), unsafe (froze the child mid-transaction), and + * couldn't touch PEAK pressure (it paused between batches while N workers were + * already in flight). + * + * This primitive fixes all three: + * - CONCURRENCY CAP is the real lever. `acquire()` is a counting semaphore + * that bounds simultaneous in-flight DB writes — directly limiting pooler + * slots held at once. (Single-pool callers like embed instead lower their + * worker count to `maxConcurrency`; the permit exists for the MULTI-pool + * case — sync's separate engine per worker — where one budget must span + * pools a single worker-count can't.) + * - IN-BAND latency is the signal. `observe(ms)` feeds an EWMA from the work's + * OWN queries — it can never be blind the way a separate probe pool was. + * - COOPERATIVE sleep, never SIGSTOP. `pace()` sleeps between safe points on + * `setTimeout` (timers phase), so the lock-heartbeat `setInterval` keeps + * firing. Per-call jitter prevents a 20-worker thundering-herd resume. + * + * Contracts: + * - `acquire()` / `pace()` THROW `AbortError` if the signal fires while + * waiting — a cancel can never fall through into a DB call. + * - FAIL-OPEN: any unexpected internal error degrades to a no-op; a pacer bug + * must never kill a backfill, and nothing here throws an unhandledRejection + * (the failure class behind the prior #1972 lock-renewal crash). + * - `off` mode / PGLite → `createNoopPacer()`: unbounded permits, zero sleeps. + */ + +import { AbortError, anySignal } from './abort-check.ts'; +import type { PaceBundle } from './pace-mode.ts'; + +/** Snapshot of pacer state for telemetry + tests. */ +export interface PaceSnapshot { + enabled: boolean; + maxConcurrency: number; + /** Permits currently held. */ + active: number; + /** EWMA of observed DB-op latency (ms); null until the first sample. */ + ewmaMs: number | null; + /** Cumulative ms spent in cooperative sleeps. */ + totalSleptMs: number; + /** Number of `pace()` calls that actually slept. */ + sleepCount: number; + /** High-water mark of waiters blocked in `acquire()`. */ + maxWaiters: number; + /** Count of `observe()` samples folded into the EWMA. */ + sampleCount: number; +} + +/** A held permit. `release()` is idempotent. */ +export interface Permit { + release(): void; +} + +export interface DbPacer { + /** + * Acquire a DB-write permit (caps simultaneous in-flight writes to + * `maxConcurrency`). THROWS `AbortError` if `signal` fires while waiting. + * On a disabled pacer, resolves immediately with a no-op permit. + */ + acquire(signal?: AbortSignal): Promise; + /** Feed an observed DB-op latency sample (ms). NaN / negative is ignored. */ + observe(latencyMs: number): void; + /** + * Cooperative sleep when recent latency is high (EWMA > `paceAtMs`), jittered + * per call and capped at `maxSleepMs`. No-op when latency is fine. THROWS + * `AbortError` if `signal` fires during the sleep. + */ + pace(signal?: AbortSignal): Promise; + snapshot(): PaceSnapshot; + /** Stop accepting waiters and release any blocked acquirers. Idempotent. */ + dispose(): void; +} + +/** Test/impl seams. Production defaults read real timers + Math.random. */ +export interface DbPacerSeams { + /** Sleep `ms`, rejecting `AbortError` if `signal` fires first. */ + sleep?: (ms: number, signal?: AbortSignal) => Promise; + /** Jitter source in [0, 1). */ + rng?: () => number; +} + +export interface CreateDbPacerOpts extends DbPacerSeams { + bundle: PaceBundle; +} + +interface Waiter { + resolve: (p: Permit) => void; + reject: (e: unknown) => void; + onAbort?: () => void; + cleanup?: () => void; + signal?: AbortSignal; +} + +/** + * Default abortable sleep. Resolves after `ms`; rejects `AbortError` if the + * signal fires first. `ms <= 0` resolves on the next microtask. + */ +function defaultSleep(ms: number, signal?: AbortSignal): Promise { + if (signal?.aborted) return Promise.reject(new AbortError(abortReason(signal))); + if (!(ms > 0)) return Promise.resolve(); + return new Promise((resolve, reject) => { + const timer = setTimeout(() => { + cleanup(); + resolve(); + }, ms); + // NOTE: do NOT unref() this timer. The bulk loop is awaiting it, so if it + // were unref'd and no other referenced handle existed, the process could + // exit mid-sleep and truncate the backfill (Codex P1). + const onAbort = () => { + cleanup(); + reject(new AbortError(abortReason(signal))); + }; + const cleanup = () => { + clearTimeout(timer); + signal?.removeEventListener('abort', onAbort); + }; + signal?.addEventListener('abort', onAbort, { once: true }); + }); +} + +function abortReason(signal?: AbortSignal | null): string { + const r = signal?.reason; + if (r instanceof Error) return r.message; + return String(r ?? 'aborted'); +} + +const NOOP_PERMIT: Permit = { release() {} }; + +/** No-op pacer: unbounded concurrency, zero sleeps. For `off` mode / PGLite. */ +export function createNoopPacer(): DbPacer { + return { + acquire: () => Promise.resolve(NOOP_PERMIT), + observe: () => {}, + pace: () => Promise.resolve(), + snapshot: () => ({ + enabled: false, + maxConcurrency: 0, + active: 0, + ewmaMs: null, + totalSleptMs: 0, + sleepCount: 0, + maxWaiters: 0, + sampleCount: 0, + }), + dispose: () => {}, + }; +} + +export function createDbPacer(opts: CreateDbPacerOpts): DbPacer { + const { bundle } = opts; + if (!bundle.enabled) return createNoopPacer(); + + const max = Math.max(1, Math.floor(bundle.maxConcurrency)); + const paceAtMs = Math.max(0, bundle.paceAtMs); + const maxSleepMs = Math.max(0, bundle.maxSleepMs); + const alpha = bundle.ewmaAlpha > 0 && bundle.ewmaAlpha <= 1 ? bundle.ewmaAlpha : 0.3; + const sleep = opts.sleep ?? defaultSleep; + const rng = opts.rng ?? Math.random; + + let active = 0; + let disposed = false; + let ewma: number | null = null; + let totalSleptMs = 0; + let sleepCount = 0; + let maxWaiters = 0; + let sampleCount = 0; + const waiters: Waiter[] = []; + + function makePermit(): Permit { + let released = false; + return { + release() { + if (released) return; + released = true; + // Hand the slot to the next waiter without dropping `active`, so the + // cap is never momentarily exceeded. + const next = waiters.shift(); + if (next) { + next.cleanup?.(); + next.resolve(makePermit()); + } else { + active = Math.max(0, active - 1); + } + }, + }; + } + + async function acquire(signal?: AbortSignal): Promise { + try { + if (signal?.aborted) throw new AbortError(abortReason(signal)); + if (disposed) return NOOP_PERMIT; + if (active < max) { + active++; + return makePermit(); + } + return await new Promise((resolve, reject) => { + const waiter: Waiter = { resolve, reject, signal }; + const onAbort = () => { + const i = waiters.indexOf(waiter); + if (i >= 0) waiters.splice(i, 1); + reject(new AbortError(abortReason(signal))); + }; + waiter.onAbort = onAbort; + waiter.cleanup = () => signal?.removeEventListener('abort', onAbort); + signal?.addEventListener('abort', onAbort, { once: true }); + waiters.push(waiter); + if (waiters.length > maxWaiters) maxWaiters = waiters.length; + }); + } catch (err) { + // Abort must propagate so the loop can't fall into a DB call. + if (err instanceof AbortError) throw err; + // Anything else: fail-open (a pacer bug must not kill the backfill). + return NOOP_PERMIT; + } + } + + function observe(latencyMs: number): void { + try { + if (typeof latencyMs !== 'number' || !Number.isFinite(latencyMs) || latencyMs < 0) return; + ewma = ewma === null ? latencyMs : alpha * latencyMs + (1 - alpha) * ewma; + sampleCount++; + } catch { + /* fail-open */ + } + } + + async function pace(signal?: AbortSignal): Promise { + let ms = 0; + try { + if (ewma === null || ewma <= paceAtMs || maxSleepMs <= 0) return; + const base = Math.min(maxSleepMs, ewma); + // Jitter to 50-100% of base so N workers don't resume in lockstep. + const jitter = 0.5 + 0.5 * clamp01(rng()); + ms = Math.round(base * jitter); + if (ms <= 0) return; + } catch { + return; // fail-open: never let knob math kill the loop + } + try { + await sleep(ms, signal); + totalSleptMs += ms; + sleepCount++; + } catch (err) { + // Abort must propagate (a cancel can't be swallowed); any other sleep + // failure is fail-open — a pacer bug never kills the backfill. + if (err instanceof AbortError) throw err; + } + } + + function snapshot(): PaceSnapshot { + return { + enabled: true, + maxConcurrency: max, + active, + ewmaMs: ewma, + totalSleptMs, + sleepCount, + maxWaiters, + sampleCount, + }; + } + + function dispose(): void { + if (disposed) return; + disposed = true; + // Release blocked acquirers with no-op permits so their loops end cleanly + // rather than hanging forever. + while (waiters.length > 0) { + const w = waiters.shift(); + w?.cleanup?.(); + w?.resolve(NOOP_PERMIT); + } + } + + return { acquire, observe, pace, snapshot, dispose }; +} + +function clamp01(n: number): number { + if (!Number.isFinite(n)) return 0; + if (n < 0) return 0; + if (n > 1) return 1; + return n; +} + +/** + * Convenience: time a DB op and feed its latency to the pacer. Keeps call sites + * a one-liner — `await observed(pacer, () => engine.upsertChunks(...))`. + */ +export async function observed(pacer: DbPacer, fn: () => Promise): Promise { + const t0 = Date.now(); + try { + return await fn(); + } finally { + pacer.observe(Date.now() - t0); + } +} + +// anySignal is re-exported intentionally so call sites that compose a budget +// signal with the pacer can import from one place. +export { anySignal }; diff --git a/src/core/embed-backfill-lock.ts b/src/core/embed-backfill-lock.ts new file mode 100644 index 000000000..af9ceabc6 --- /dev/null +++ b/src/core/embed-backfill-lock.ts @@ -0,0 +1,17 @@ +/** + * Shared lock identity for embed backfills (paced-backfill E-2). + * + * The per-source lock key + TTL live here, in a zero-dependency module, so BOTH + * the `embed-backfill` minion handler AND the CLI `embed --stale` single-flight + * take the SAME key — a hand-run backfill and a queued job are then mutually + * exclusive per source. Kept dependency-free to avoid an import cycle between + * embed.ts, embed-stale.ts, and the handler. + */ + +/** Per-source embed-backfill lock id, namespaced like sync's. */ +export function embedBackfillLockId(sourceId: string): string { + return `gbrain-embed-backfill:${sourceId}`; +} + +/** Lock TTL (minutes) for embed backfills. */ +export const EMBED_BACKFILL_LOCK_TTL_MIN = 60; diff --git a/src/core/embed-stale.ts b/src/core/embed-stale.ts index 5ca95cd78..484f3ea55 100644 --- a/src/core/embed-stale.ts +++ b/src/core/embed-stale.ts @@ -20,6 +20,8 @@ import type { BrainEngine } from './engine.ts'; import type { ChunkInput } from './types.ts'; import { embedBatchWithBackoff } from '../commands/embed.ts'; +import { type DbPacer, createNoopPacer, observed } from './db-pacer.ts'; +import { AbortError } from './abort-check.ts'; /** Last visited (page_id, chunk_index) for keyset-resume across runs. */ export interface StaleCursor { @@ -59,6 +61,15 @@ export interface EmbedStaleOpts { * Omit to keep the legacy `embedding IS NULL`-only behavior. */ embeddingSignature?: string; + /** + * DB-contention pacer (paced-backfill). When enabled it (a) supplies the + * worker count via the caller passing `concurrency = bundle.maxConcurrency` + * — E-1: no separate permit on this single-pool path — and (b) the loop + * `observe()`s its DB-op latency and `pace()`s between keys. Omit (or pass a + * disabled bundle) for a no-op. The pacer is NEVER used to acquire permits + * here; concurrency is the worker count. + */ + pacer?: DbPacer; } export interface EmbedStaleResult { @@ -105,6 +116,9 @@ export async function embedStaleForSource( const signal = opts.signal; const embedFn = opts.embedFn ?? ((texts, fnOpts) => embedBatchWithBackoff(texts, { abortSignal: fnOpts.abortSignal })); + // Defaulted no-op when pacing is off, so the observe()/pace() call sites + // below are unconditional and cost ~nothing on the unpaced path. + const pacer = opts.pacer ?? createNoopPacer(); let afterPageId = opts.cursor?.afterPageId ?? 0; let afterChunkIndex = opts.cursor?.afterChunkIndex ?? -1; @@ -136,12 +150,14 @@ export async function embedStaleForSource( return result; } - const batch = await engine.listStaleChunks({ - batchSize, - afterPageId, - afterChunkIndex, - sourceId, - }); + const batch = await observed(pacer, () => + engine.listStaleChunks({ + batchSize, + afterPageId, + afterChunkIndex, + sourceId, + }), + ); if (batch.length === 0) { result.done = true; return result; @@ -177,7 +193,9 @@ export async function embedStaleForSource( stale.map((c) => c.chunk_text), { abortSignal: signal }, ); - const existing = await engine.getChunks(slug, { sourceId: keySourceId }); + const existing = await observed(pacer, () => + engine.getChunks(slug, { sourceId: keySourceId }), + ); const staleIdxToEmbedding = new Map(); for (let j = 0; j < stale.length; j++) { staleIdxToEmbedding.set(stale[j].chunk_index, embeddings[j]); @@ -189,13 +207,15 @@ export async function embedStaleForSource( embedding: staleIdxToEmbedding.get(c.chunk_index) ?? undefined, token_count: c.token_count || Math.ceil(c.chunk_text.length / 4), })); - await engine.upsertChunks(slug, merged, { sourceId: keySourceId }); + await observed(pacer, () => engine.upsertChunks(slug, merged, { sourceId: keySourceId })); // v0.41.31: stamp provenance only when EVERY chunk was stale (fully // re-embedded this pass) — a partially-stale page keeps preserved // chunks of unknown provenance, so don't claim current. After the // invalidate pass above, signature-drifted pages ARE fully stale. if (signature && stale.length === existing.length) { - await engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }); + await observed(pacer, () => + engine.setPageEmbeddingSignature(slug, { sourceId: keySourceId, signature }), + ); } result.embedded += stale.length; result.pagesProcessed += 1; @@ -215,6 +235,15 @@ export async function embedStaleForSource( while (nextIdx < keys.length && !signal?.aborted) { const idx = nextIdx++; await embedOneKey(keys[idx]); + // Cooperative DB-contention pace between keys (no-op when unpaced). + // pace() throws AbortError on cancel — treat as graceful worker exit; + // the for(;;) loop sees signal.aborted and returns aborted next tick. + try { + await pacer.pace(signal); + } catch (e) { + if (e instanceof AbortError) return; + throw e; + } } } diff --git a/src/core/git-remote.ts b/src/core/git-remote.ts index 43f8515eb..d3673c29b 100644 --- a/src/core/git-remote.ts +++ b/src/core/git-remote.ts @@ -140,7 +140,7 @@ export class GitOperationError extends Error { } } -const GIT_ENV = { +export const GIT_ENV = { // Confine to the gbrain SSRF model — no credential helpers, no SSH askpass, // no GUI prompts. Inherit PATH so git itself is findable. GIT_TERMINAL_PROMPT: '0', @@ -149,6 +149,21 @@ const GIT_ENV = { SSH_ASKPASS: '/bin/false', } as const; +/** + * Auth-capable git env for the durability push/probe paths (v0.42.44). + * + * Read-only clone/pull keep the strict GIT_ENV (askpass=/bin/false) so they can + * never prompt. But push, push-probe, and the durability cron's authed fetch + * MUST be able to consult the repo's configured credential helper (repo-scoped + * `store`/`osxkeychain`) — a `/bin/false` askpass would defeat that. We drop the + * askpass overrides but KEEP `GIT_TERMINAL_PROMPT=0` so a *missing* credential + * fails fast instead of hanging a non-interactive cron forever. + */ +export const GIT_ENV_AUTH = { + GIT_TERMINAL_PROMPT: '0', + GCM_INTERACTIVE: 'never', +} as const; + /** * Clone a remote git repo with SSRF-defensive flags. * - destDir must NOT exist or must be empty. @@ -287,3 +302,190 @@ export function validateRepoState( } return 'healthy'; } + +// ── Durability helpers (v0.42.44) ─────────────────────────────────────────── +// Used by the brain-repo durability feature (`gbrain sources harden/pull`) and +// the DB-free pull cron. These are the auth-capable, rebase-aware counterparts +// to the strict read-only `pullRepo` (which stays `--ff-only` for `sync.ts`). + +/** + * Global SSRF flags for the durability fetch/pull/push paths. Identical to + * GIT_SSRF_FLAGS except `protocol.file.allow` honors the env escape hatch + * `GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1` (mirrors GBRAIN_ALLOW_PRIVATE_REMOTES) so + * self-hosted local-filesystem remotes — and the test suite — can use the file + * transport. Default stays `never`. These ops act on an ALREADY-validated origin + * (set + checked at clone time); `http.followRedirects=false` is the live guard. + */ +function durableSsrfFlags(): string[] { + const fileAllow = process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT === '1' ? 'always' : 'never'; + return [ + '-c', 'http.followRedirects=false', + '-c', `protocol.file.allow=${fileAllow}`, + '-c', 'protocol.ext.allow=never', + ]; +} + +/** Run a git subcommand, returning trimmed stdout. Throws GitOperationError. */ +function runGit( + repoPath: string, + globalFlags: readonly string[], + subcommand: string, + subArgs: readonly string[], + op: GitOperationError['op'], + opts: { timeoutMs?: number; env?: Record } = {}, +): string { + try { + const out = execFileSync( + 'git', + ['-C', repoPath, ...globalFlags, subcommand, ...subArgs], + { + stdio: ['ignore', 'pipe', 'pipe'], + timeout: opts.timeoutMs ?? 120_000, + env: { ...process.env, ...(opts.env ?? GIT_ENV) }, + }, + ); + return out.toString().trim(); + } catch (e) { + throw new GitOperationError(op, `git ${subcommand} failed in ${repoPath}: ${(e as Error).message}`, e); + } +} + +/** True if the working tree has staged or unstaged changes (untracked too). */ +export function isWorkingTreeDirty(repoPath: string): boolean { + const out = runGit(repoPath, [], 'status', ['--porcelain'], 'pull', { timeoutMs: 30_000 }); + return out.length > 0; +} + +/** + * Resolve the repo's default branch, local-only (no network): + * origin/HEAD symbolic-ref → current branch (if not detached) → 'main'. + */ +export function detectDefaultBranch(repoPath: string): string { + try { + const sym = execFileSync('git', ['-C', repoPath, 'symbolic-ref', '--short', 'refs/remotes/origin/HEAD'], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + if (sym.startsWith('origin/')) return sym.slice('origin/'.length); + if (sym) return sym; + } catch { /* origin/HEAD not set — fall through */ } + try { + const cur = execFileSync('git', ['-C', repoPath, 'rev-parse', '--abbrev-ref', 'HEAD'], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + if (cur && cur !== 'HEAD') return cur; + } catch { /* detached or no commits */ } + return 'main'; +} + +/** True if a rebase is mid-flight (rebase-merge or rebase-apply state dir exists). */ +function rebaseInProgress(repoPath: string): boolean { + for (const name of ['rebase-merge', 'rebase-apply']) { + try { + const p = execFileSync('git', ['-C', repoPath, 'rev-parse', '--git-path', name], { + stdio: ['ignore', 'pipe', 'ignore'], timeout: 10_000, env: { ...process.env, ...GIT_ENV }, + }).toString().trim(); + const abs = p.startsWith('/') ? p : join(repoPath, p); + if (existsSync(abs)) return true; + } catch { /* ignore */ } + } + return false; +} + +export type PullOutcome = + | { status: 'up_to_date' } + | { status: 'advanced'; from: string; to: string } + | { status: 'skipped_dirty' } + | { status: 'conflict_aborted'; detail: string }; + +/** + * Divergence-safe pull: `fetch` + `pull --rebase`, never leaving a mid-rebase. + * + * - Dirty working tree → `skipped_dirty` (NORMAL mid-session state, not an + * error; never auto-stashes, never touches in-progress edits). + * - Rebase conflict → `git rebase --abort`, verify no rebase state remains, + * return `conflict_aborted` ("manual attention needed"). Never throws past + * this — the repo is always left clean (possibly un-advanced). + * + * Auth-capable (GIT_ENV_AUTH) so it works against private remotes via the + * repo's configured credential helper. SSRF flags applied on every call. + */ +export function divergenceSafePull( + repoPath: string, + branch: string, + opts: { timeoutMs?: number } = {}, +): PullOutcome { + const timeoutMs = opts.timeoutMs ?? 300_000; + + if (isWorkingTreeDirty(repoPath)) return { status: 'skipped_dirty' }; + + const before = runGit(repoPath, [], 'rev-parse', ['HEAD'], 'pull', { timeoutMs: 10_000 }); + const ssrf = durableSsrfFlags(); + + runGit(repoPath, ssrf, 'fetch', [...GIT_SSRF_SUBCOMMAND_FLAGS, 'origin', branch], 'pull', { + timeoutMs, env: { ...GIT_ENV_AUTH }, + }); + + try { + runGit(repoPath, ssrf, 'pull', [...GIT_SSRF_SUBCOMMAND_FLAGS, '--rebase', 'origin', branch], 'pull', { + timeoutMs, env: { ...GIT_ENV_AUTH }, + }); + } catch (e) { + // Abort any half-applied rebase so the tree is never left mid-rebase. + try { + execFileSync('git', ['-C', repoPath, 'rebase', '--abort'], { + stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV }, + }); + } catch { /* best-effort */ } + // If state STILL remains, try once more, then report regardless. + if (rebaseInProgress(repoPath)) { + try { + execFileSync('git', ['-C', repoPath, 'rebase', '--abort'], { + stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV }, + }); + } catch { /* best-effort */ } + } + return { + status: 'conflict_aborted', + detail: `pull --rebase on ${branch} conflicted; rebase aborted — manual attention needed (${(e as Error).message.slice(0, 120)})`, + }; + } + + const after = runGit(repoPath, [], 'rev-parse', ['HEAD'], 'pull', { timeoutMs: 10_000 }); + return before === after ? { status: 'up_to_date' } : { status: 'advanced', from: before, to: after }; +} + +export type PushProbeResult = + | { ok: true } + | { ok: false; reason: 'auth' | 'protected' | 'unreachable' | 'other'; detail: string }; + +/** + * Authenticated `git push --dry-run` against origin/. Proves push auth + * works AND surfaces read-only PATs / branch protection BEFORE harden declares + * "hardened" — with zero history pollution (no commit). Auth-capable env. + * + * `redactDetail` (e.g. shell-redact's value scrubber bound to the PAT) is + * applied to the captured stderr so a token echoed by git never reaches a log. + */ +export function pushProbe( + repoPath: string, + branch: string, + opts: { timeoutMs?: number; redactDetail?: (s: string) => string } = {}, +): PushProbeResult { + const redact = opts.redactDetail ?? ((s: string) => s); + try { + execFileSync( + 'git', + ['-C', repoPath, ...durableSsrfFlags(), 'push', ...GIT_SSRF_SUBCOMMAND_FLAGS, '--dry-run', 'origin', `HEAD:${branch}`], + { stdio: ['ignore', 'pipe', 'pipe'], timeout: opts.timeoutMs ?? 60_000, env: { ...process.env, ...GIT_ENV_AUTH } }, + ); + return { ok: true }; + } catch (e) { + const raw = redact((e as Error).message || ''); + const low = raw.toLowerCase(); + let reason: 'auth' | 'protected' | 'unreachable' | 'other' = 'other'; + if (low.includes('authentication') || low.includes('403') || low.includes('permission') || low.includes('could not read')) reason = 'auth'; + else if (low.includes('protected') || low.includes('pre-receive') || low.includes('hook declined')) reason = 'protected'; + else if (low.includes('could not resolve') || low.includes('unable to access') || low.includes('timed out') || low.includes('network')) reason = 'unreachable'; + return { ok: false, reason, detail: raw.slice(0, 200) }; + } +} diff --git a/src/core/minions/handlers/embed-backfill.ts b/src/core/minions/handlers/embed-backfill.ts index 8fb6d52ac..30ffd6ed5 100644 --- a/src/core/minions/handlers/embed-backfill.ts +++ b/src/core/minions/handlers/embed-backfill.ts @@ -36,12 +36,15 @@ import { BudgetTracker, BudgetExhausted } from '../../budget/budget-tracker.ts'; import { withBudgetTracker } from '../../ai/gateway.ts'; import { embedStaleForSource } from '../../embed-stale.ts'; import { currentEmbeddingSignature } from '../../embedding.ts'; +import { type DbPacer, createDbPacer, createNoopPacer } from '../../db-pacer.ts'; +import { resolvePaceMode, loadPaceModeConfig, readPaceEnv } from '../../pace-mode.ts'; import type { BrainEngine } from '../../engine.ts'; import type { MinionJobContext } from '../types.ts'; import { parseUsdLimit, usdLimitToCap, resolveSpendPosture } from '../../spend-posture.ts'; +import { embedBackfillLockId, EMBED_BACKFILL_LOCK_TTL_MIN } from '../../embed-backfill-lock.ts'; + const DEFAULT_MAX_USD_PER_JOB = 10; -const EMBED_BACKFILL_LOCK_TTL_MIN = 60; export interface EmbedBackfillJobData { sourceId: string; @@ -62,9 +65,36 @@ export interface EmbedBackfillResult { budgetCapUsd?: number; } -/** Compose the lock id for embed-backfill, namespaced like sync's. */ -function embedBackfillLockId(sourceId: string): string { - return `gbrain-embed-backfill:${sourceId}`; +/** + * Resolve a DB-contention pacer from env > config > bundle for the prod + * backfill path. Fail-open: any error → no-op pacer (pacing never breaks a + * backfill). Returns the pacer + the resolved concurrency cap (E-1: the + * worker count for embedStaleForSource's single pool, no separate permit). + */ +async function resolveBackfillPacer( + engine: BrainEngine, + jobData: Record, +): Promise<{ pacer: DbPacer; concurrency?: number }> { + try { + const cfg = await loadPaceModeConfig(engine); + const { envMode, envOverrides } = readPaceEnv(); + const jobPace = (jobData.pace && typeof jobData.pace === 'object' + ? jobData.pace + : {}) as { perCallMode?: string; perCall?: Record }; + // Codex P2: serialized job pace sits at the CONFIG tier (job choice beats + // standing config, but env beats both) so GBRAIN_PACE_* on the worker is a + // real incident escape hatch for an already-queued job. + const knobs = resolvePaceMode({ + mode: jobPace.perCallMode ?? cfg.mode, + configOverrides: { ...cfg.configOverrides, ...(jobPace.perCall ?? {}) }, + envMode, + envOverrides, + }); + if (!knobs.enabled) return { pacer: createNoopPacer() }; + return { pacer: createDbPacer({ bundle: knobs }), concurrency: knobs.maxConcurrency }; + } catch { + return { pacer: createNoopPacer() }; + } } /** @@ -129,11 +159,18 @@ export function makeEmbedBackfillHandler(engine: BrainEngine) { label: `embed-backfill:${sourceId}`, }); + // paced-backfill: resolve env > config > bundle (env = incident escape + // hatch). No-op when off. This is the prod path that originally starved + // the supervisor, so pacing it is the headline win. + const { pacer, concurrency } = await resolveBackfillPacer(engine, job.data); + try { const result = await withBudgetTracker(tracker, async () => embedStaleForSource(engine, sourceId, { batchSize, signal: job.signal, + pacer, + ...(concurrency !== undefined && { concurrency }), // v0.41.31: re-embed pages whose model signature drifted + stamp // provenance as chunks land. embeddingSignature: currentEmbeddingSignature(), @@ -184,6 +221,7 @@ export function makeEmbedBackfillHandler(engine: BrainEngine) { } throw err; } finally { + pacer.dispose(); // ALWAYS release. Aborts, throws, budget-exhaust — all paths unwind here. try { await lock.release(); diff --git a/src/core/pace-mode.ts b/src/core/pace-mode.ts new file mode 100644 index 000000000..7daf1da33 --- /dev/null +++ b/src/core/pace-mode.ts @@ -0,0 +1,286 @@ +/** + * DB-contention pacing mode bundles (paced-backfill internals). + * + * Named modes that bundle the DB-pacer knobs into a single config key so an + * operator picks once and stops thinking about it. Mirrors the v0.32.3 + * search-mode pattern at `src/core/search/mode.ts` (named bundle + per-key + * overrides + per-call opts), with ONE deliberate difference: the resolution + * chain puts ENV ABOVE CONFIG so `GBRAIN_PACE_*` is a real incident escape + * hatch — an operator can override bad production config without a redeploy. + * + * per-call flag → GBRAIN_PACE_* env → config (pace.*) → MODE_BUNDLES[mode] → off + * + * `DEFAULT_PACE_MODE` is `'off'`: pacing is strictly opt-in and never changes + * default behavior. `off` resolves to `enabled: false`, which the DB-pacer + * turns into a no-op (unbounded concurrency, zero sleeps). + * + * This module is PURE — no DB calls, no env reads inside `resolvePaceMode`. + * The caller pre-loads config (`loadPaceModeConfig`) and env (`readPaceEnv`) + * and passes them in, so the resolver is trivially testable. + */ + +export type PaceMode = 'off' | 'gentle' | 'balanced' | 'aggressive'; + +export const PACE_MODES: ReadonlyArray = Object.freeze([ + 'off', + 'gentle', + 'balanced', + 'aggressive', +]); + +export const DEFAULT_PACE_MODE: PaceMode = 'off'; + +/** + * A complete knob set for one pace mode. Every field is required so the bundle + * is self-contained and per-key overrides are obvious diffs. + */ +export interface PaceBundle { + /** Master switch. `false` ⇒ the DB-pacer is a no-op (off mode / PGLite). */ + enabled: boolean; + /** + * Primary lever: cap on simultaneous in-flight DB writes. For single-pool + * paths (embed) this becomes the worker count; for multi-pool paths (sync) + * it's enforced by the shared `acquire()` permit. The real defense against + * pooler-slot starvation. + */ + maxConcurrency: number; + /** EWMA of observed DB-op latency (ms) above which `pace()` starts sleeping. */ + paceAtMs: number; + /** Ceiling on a single cooperative sleep (ms). Jittered downward per call. */ + maxSleepMs: number; + /** EWMA smoothing factor in (0, 1]. Higher = reacts faster to recent latency. */ + ewmaAlpha: number; +} + +/** + * The four bundles. Frozen at import so a typo can't redefine "balanced" to + * mean different things on different installs. `off` is the disabled sentinel; + * its numeric fields are inert (the pacer short-circuits on `enabled: false`). + * + * Concurrency picks are deliberately well below the default embed concurrency + * (`GBRAIN_EMBED_CONCURRENCY`, 20) — the whole point is to hold fewer pooler + * slots than an unpaced run. + */ +export const PACE_BUNDLES: Readonly>> = Object.freeze({ + off: Object.freeze({ + enabled: false, + maxConcurrency: 0, + paceAtMs: 0, + maxSleepMs: 0, + ewmaAlpha: 0, + }), + gentle: Object.freeze({ + enabled: true, + maxConcurrency: 4, + paceAtMs: 250, + maxSleepMs: 2000, + ewmaAlpha: 0.3, + }), + balanced: Object.freeze({ + enabled: true, + maxConcurrency: 8, + paceAtMs: 500, + maxSleepMs: 1500, + ewmaAlpha: 0.3, + }), + aggressive: Object.freeze({ + enabled: true, + maxConcurrency: 16, + paceAtMs: 1000, + maxSleepMs: 1000, + ewmaAlpha: 0.3, + }), +}); + +export function isPaceMode(x: unknown): x is PaceMode { + return typeof x === 'string' && (PACE_MODES as ReadonlyArray).includes(x); +} + +/** + * Per-key overrides (from the config table OR from env). Every field optional; + * undefined ⇒ fall through to the next precedence layer. + */ +export interface PaceKeyOverrides { + enabled?: boolean; + maxConcurrency?: number; + paceAtMs?: number; + maxSleepMs?: number; + ewmaAlpha?: number; +} + +/** + * Resolve the active pace knob set. Pure: the caller supplies the resolved + * config + env layers. + * + * Mode precedence: perCallMode → envMode → mode(config) → DEFAULT_PACE_MODE + * Knob precedence: perCall → env → config → MODE_BUNDLES[mode] + * + * Env above config is intentional (incident escape hatch — see file header). + */ +export interface ResolvePaceModeInput { + /** `config.pace.mode`. */ + mode?: string; + /** `GBRAIN_PACE_MODE`. */ + envMode?: string; + /** `--pace=` (or bare `--pace` resolved to 'balanced' by the caller). */ + perCallMode?: string; + /** Per-key overrides from the config table. */ + configOverrides?: PaceKeyOverrides; + /** Per-key overrides from `GBRAIN_PACE_*` env. */ + envOverrides?: PaceKeyOverrides; + /** Per-call overrides (e.g. `--pace-max-concurrency`). */ + perCall?: PaceKeyOverrides; +} + +export interface ResolvedPaceKnobs extends PaceBundle { + /** Which bundle supplied the defaults (after fallback). */ + resolved_mode: PaceMode; + /** True if the resolved mode string was a recognized PaceMode. */ + mode_valid: boolean; +} + +export function resolvePaceMode(input: ResolvePaceModeInput): ResolvedPaceKnobs { + const rawMode = + firstString(input.perCallMode) ?? firstString(input.envMode) ?? firstString(input.mode); + const normalized = rawMode ? rawMode.trim().toLowerCase() : ''; + const valid = isPaceMode(normalized); + const resolved_mode: PaceMode = valid ? (normalized as PaceMode) : DEFAULT_PACE_MODE; + const bundle = PACE_BUNDLES[resolved_mode]; + + const pc = input.perCall ?? {}; + const env = input.envOverrides ?? {}; + const cfg = input.configOverrides ?? {}; + + const pick = (key: K): PaceBundle[K] => { + if (pc[key] !== undefined) return pc[key] as PaceBundle[K]; + if (env[key] !== undefined) return env[key] as PaceBundle[K]; + if (cfg[key] !== undefined) return cfg[key] as PaceBundle[K]; + return bundle[key]; + }; + + const enabled = pick('enabled'); + // Clamp to safe ranges so a fat-fingered override can't disable the cap + // (maxConcurrency must be >= 1 when enabled) or wedge the EWMA. + let maxConcurrency = pick('maxConcurrency'); + if (enabled && (!Number.isFinite(maxConcurrency) || maxConcurrency < 1)) { + maxConcurrency = bundle.enabled ? bundle.maxConcurrency : 8; + } + let ewmaAlpha = pick('ewmaAlpha'); + if (!Number.isFinite(ewmaAlpha) || ewmaAlpha <= 0 || ewmaAlpha > 1) { + ewmaAlpha = bundle.enabled ? bundle.ewmaAlpha : 0.3; + } + const paceAtMs = Math.max(0, pick('paceAtMs')); + const maxSleepMs = Math.max(0, pick('maxSleepMs')); + + return { + enabled, + maxConcurrency, + paceAtMs, + maxSleepMs, + ewmaAlpha, + resolved_mode, + mode_valid: valid, + }; +} + +function firstString(v: unknown): string | undefined { + return typeof v === 'string' && v.trim().length > 0 ? v : undefined; +} + +/** Config-table key for the mode selection (separate from the override keys). */ +export const PACE_MODE_KEY = 'pace.mode'; + +/** Per-knob config keys this module reads. Used by a future `--reset`. */ +export const PACE_MODE_CONFIG_KEYS: ReadonlyArray = Object.freeze([ + 'pace.enabled', + 'pace.max_concurrency', + 'pace.pace_at_ms', + 'pace.max_sleep_ms', + 'pace.ewma_alpha', +]); + +/** Build PaceKeyOverrides from a flat config-table snapshot (sparse). */ +export function loadOverridesFromConfig( + configMap: Record, +): PaceKeyOverrides { + return parseOverrides((k) => configMap[k], { + enabled: 'pace.enabled', + maxConcurrency: 'pace.max_concurrency', + paceAtMs: 'pace.pace_at_ms', + maxSleepMs: 'pace.max_sleep_ms', + ewmaAlpha: 'pace.ewma_alpha', + }); +} + +/** Build PaceKeyOverrides from `GBRAIN_PACE_*` env (incident escape hatch). */ +export function readPaceEnv(env: Record = process.env): { + envMode?: string; + envOverrides: PaceKeyOverrides; +} { + const overrides = parseOverrides((k) => env[k], { + enabled: 'GBRAIN_PACE_ENABLED', + maxConcurrency: 'GBRAIN_PACE_MAX_CONCURRENCY', + paceAtMs: 'GBRAIN_PACE_AT_MS', + maxSleepMs: 'GBRAIN_PACE_MAX_SLEEP_MS', + ewmaAlpha: 'GBRAIN_PACE_EWMA_ALPHA', + }); + return { envMode: env.GBRAIN_PACE_MODE, envOverrides: overrides }; +} + +function parseOverrides( + get: (k: string) => string | undefined, + keys: Record, +): PaceKeyOverrides { + const out: PaceKeyOverrides = {}; + const en = get(keys.enabled); + if (en !== undefined) out.enabled = en === '1' || en.toLowerCase() === 'true'; + const mc = get(keys.maxConcurrency); + if (mc !== undefined) { + const n = parseInt(mc, 10); + if (Number.isFinite(n) && n >= 1 && n <= 256) out.maxConcurrency = n; + } + const pa = get(keys.paceAtMs); + if (pa !== undefined) { + const n = parseInt(pa, 10); + if (Number.isFinite(n) && n >= 0) out.paceAtMs = n; + } + const ms = get(keys.maxSleepMs); + if (ms !== undefined) { + const n = parseInt(ms, 10); + if (Number.isFinite(n) && n >= 0) out.maxSleepMs = n; + } + const ea = get(keys.ewmaAlpha); + if (ea !== undefined) { + const n = parseFloat(ea); + if (Number.isFinite(n) && n > 0 && n <= 1) out.ewmaAlpha = n; + } + return out; +} + +/** + * Load the live pace config (mode + per-key overrides) from the brain engine. + * Errors swallowed → mode-bundle defaults (the config table may predate this + * feature on old brains). Does NOT read env — the caller layers `readPaceEnv` + * on top so env can beat config. + */ +export async function loadPaceModeConfig(engine: { + getConfig(key: string): Promise; +}): Promise<{ mode?: string; configOverrides: PaceKeyOverrides }> { + const safeGet = async (k: string): Promise => { + try { + const v = await engine.getConfig(k); + return typeof v === 'string' ? v : undefined; + } catch { + return undefined; + } + }; + const [mode, ...vals] = await Promise.all([ + safeGet(PACE_MODE_KEY), + ...PACE_MODE_CONFIG_KEYS.map(safeGet), + ]); + const configMap: Record = {}; + PACE_MODE_CONFIG_KEYS.forEach((key, i) => { + if (vals[i] !== undefined) configMap[key] = vals[i]; + }); + return { mode, configOverrides: loadOverridesFromConfig(configMap) }; +} diff --git a/test/brain-durability-hook.serial.test.ts b/test/brain-durability-hook.serial.test.ts new file mode 100644 index 000000000..b676b5eff --- /dev/null +++ b/test/brain-durability-hook.serial.test.ts @@ -0,0 +1,123 @@ +/** + * End-to-end durability hook + helper (v0.42.44): the generated bash actually + * pushes. Real git, local bare remote. Validates the D13 guarantee (helper), + * the D9 self-contained local hook, and the D7 "one push-retry template" claim + * (the hook works even with the committed helper deleted). + */ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync, existsSync, readFileSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { execFileSync } from 'child_process'; +import { hardenBrainRepo } from '../src/core/brain-repo-durability.ts'; + +function git(cwd: string, ...args: string[]): string { + return execFileSync('git', ['-C', cwd, '-c', 'protocol.file.allow=always', ...args], { + stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8', + }).trim(); +} +function originHead(bare: string): string { + return git(bare, 'rev-parse', 'refs/heads/main'); +} +async function waitForOrigin(bare: string, expectSha: string, ms = 8000): Promise { + const deadline = Date.now() + ms; + while (Date.now() < deadline) { + try { if (originHead(bare) === expectSha) return true; } catch { /* */ } + await new Promise(r => setTimeout(r, 150)); + } + return false; +} + +let root: string, work: string, bare: string; +let oldHome: string | undefined, oldGbrainHome: string | undefined; + +beforeEach(async () => { + root = mkdtempSync(join(tmpdir(), 'bdh-')); + oldHome = process.env.HOME; oldGbrainHome = process.env.GBRAIN_HOME; + process.env.HOME = mkdtempSync(join(root, 'home-')); + process.env.GBRAIN_HOME = join(process.env.HOME, '.gbrain'); + process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT = '1'; + bare = mkdtempSync(join(root, 'origin-')) + '.git'; + execFileSync('git', ['init', '-q', '--bare', '-b', 'main', bare], { stdio: 'ignore' }); + work = mkdtempSync(join(root, 'work-')); + execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, work], { stdio: 'ignore' }); + git(work, 'config', 'user.email', 't@t.t'); git(work, 'config', 'user.name', 'tester'); + writeFileSync(join(work, 'README.md'), 'init\n'); + git(work, 'add', 'README.md'); git(work, 'commit', '-qm', 'init'); git(work, 'push', '-q', 'origin', 'main'); + git(work, 'remote', 'set-head', 'origin', 'main'); + await hardenBrainRepo({ repoPath: work, sourceId: 'wiki', pat: 'ghp_x', installCron: false }); +}); +afterEach(() => { + if (oldHome === undefined) delete process.env.HOME; else process.env.HOME = oldHome; + if (oldGbrainHome === undefined) delete process.env.GBRAIN_HOME; else process.env.GBRAIN_HOME = oldGbrainHome; + delete process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT; + rmSync(root, { recursive: true, force: true }); +}); + +describe('brain-commit-push.sh (D13 guarantee)', () => { + test('add → commit → push lands on origin', () => { + mkdirSync(join(work, 'people'), { recursive: true }); + writeFileSync(join(work, 'people', 'alice.md'), '# alice\n'); + // helper requires explicit path; stages people/alice.md + execFileSync('bash', [join(work, 'scripts', 'brain-commit-push.sh'), 'add alice', 'people/alice.md'], { + cwd: work, stdio: ['ignore', 'pipe', 'pipe'], env: process.env, + }); + expect(originHead(bare)).toBe(git(work, 'rev-parse', 'HEAD')); + // origin actually has the file + const verify = mkdtempSync(join(root, 'verify-')); + execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, verify], { stdio: 'ignore' }); + expect(existsSync(join(verify, 'people', 'alice.md'))).toBe(true); + }); + + test('refuses success when the push cannot land (exit non-zero)', () => { + git(work, 'remote', 'set-url', 'origin', join(root, 'gone.git')); + writeFileSync(join(work, 'x.md'), 'x\n'); + let code = 0; + try { + execFileSync('bash', [join(work, 'scripts', 'brain-commit-push.sh'), 'msg', 'x.md'], { + cwd: work, stdio: ['ignore', 'pipe', 'pipe'], env: process.env, + }); + } catch (e: any) { code = e.status ?? 1; } + expect(code).not.toBe(0); // committed but push failed → loud failure + }); + + test('refuses a blind add (no explicit path)', () => { + let code = 0; + try { + execFileSync('bash', [join(work, 'scripts', 'brain-commit-push.sh'), 'msg'], { + cwd: work, stdio: ['ignore', 'pipe', 'pipe'], env: process.env, + }); + } catch (e: any) { code = e.status ?? 1; } + expect(code).toBe(2); + }); +}); + +describe('post-commit hook (D9 local, D7 self-contained)', () => { + test('a direct commit auto-pushes in the background', async () => { + writeFileSync(join(work, 'note.md'), 'note\n'); + git(work, 'add', 'note.md'); git(work, 'commit', '-qm', 'note'); // fires .git/hooks/post-commit + const head = git(work, 'rev-parse', 'HEAD'); + expect(await waitForOrigin(bare, head)).toBe(true); + }); + + test('the hook works even with the committed helper deleted (self-contained)', async () => { + rmSync(join(work, 'scripts', 'brain-commit-push.sh')); + git(work, 'add', '-A'); git(work, 'commit', '-qm', 'remove helper'); + const head = git(work, 'rev-parse', 'HEAD'); + expect(await waitForOrigin(bare, head)).toBe(true); + }); + + test('logs a clear LOCAL-ONLY line when origin is unreachable', async () => { + git(work, 'remote', 'set-url', 'origin', join(root, 'gone2.git')); + writeFileSync(join(work, 'orphan.md'), 'o\n'); + git(work, 'add', 'orphan.md'); git(work, 'commit', '-qm', 'orphan'); + const log = join(process.env.GBRAIN_HOME!, 'brain-push.log'); + const deadline = Date.now() + 8000; + let found = false; + while (Date.now() < deadline) { + if (existsSync(log) && readFileSync(log, 'utf-8').includes('NEEDS ATTENTION')) { found = true; break; } + await new Promise(r => setTimeout(r, 150)); + } + expect(found).toBe(true); + }); +}); diff --git a/test/brain-repo-durability.serial.test.ts b/test/brain-repo-durability.serial.test.ts new file mode 100644 index 000000000..d381e492b --- /dev/null +++ b/test/brain-repo-durability.serial.test.ts @@ -0,0 +1,223 @@ +/** + * brain-repo-durability core (v0.42.44): hardenBrainRepo / unhardenBrainRepo / + * acceptPat. Real git against a local bare remote. HOME + GBRAIN_HOME are + * redirected to a tmp dir; installCron:false so the suite never touches launchd. + */ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync, readFileSync, existsSync, statSync, chmodSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { execFileSync } from 'child_process'; +import { + hardenBrainRepo, unhardenBrainRepo, acceptPat, +} from '../src/core/brain-repo-durability.ts'; + +const PAT = 'ghp_TESTSECRETTOKEN0123456789abcdef'; + +function git(cwd: string, ...args: string[]): string { + return execFileSync('git', ['-C', cwd, '-c', 'protocol.file.allow=always', ...args], { + stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8', + }).trim(); +} +function commitCount(work: string): number { + return parseInt(git(work, 'rev-list', '--count', 'HEAD'), 10); +} +/** git config read that returns '' instead of throwing when the key is unset. */ +function cfg(work: string, key: string): string { + try { return git(work, 'config', '--local', '--get', key); } catch { return ''; } +} + +let root: string; +let work: string; +let bare: string; +let oldHome: string | undefined; +let oldGbrainHome: string | undefined; + +function makePair(): void { + bare = mkdtempSync(join(root, 'origin-')) + '.git'; + execFileSync('git', ['init', '-q', '--bare', '-b', 'main', bare], { stdio: 'ignore' }); + work = mkdtempSync(join(root, 'work-')); + execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, work], { stdio: 'ignore' }); + git(work, 'config', 'user.email', 't@t.t'); + git(work, 'config', 'user.name', 'tester'); + writeFileSync(join(work, 'README.md'), 'init\n'); + git(work, 'add', 'README.md'); git(work, 'commit', '-qm', 'init'); git(work, 'push', '-q', 'origin', 'main'); + try { git(work, 'remote', 'set-head', 'origin', 'main'); } catch { /* */ } +} + +async function harden(extra: Record = {}) { + return hardenBrainRepo({ repoPath: work, sourceId: 'wiki', pat: PAT, installCron: false, ...extra }); +} + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), 'brd-')); + oldHome = process.env.HOME; oldGbrainHome = process.env.GBRAIN_HOME; + process.env.HOME = mkdtempSync(join(root, 'home-')); + process.env.GBRAIN_HOME = join(process.env.HOME, '.gbrain'); + process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT = '1'; + makePair(); +}); +afterEach(() => { + if (oldHome === undefined) delete process.env.HOME; else process.env.HOME = oldHome; + if (oldGbrainHome === undefined) delete process.env.GBRAIN_HOME; else process.env.GBRAIN_HOME = oldGbrainHome; + delete process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT; + rmSync(root, { recursive: true, force: true }); +}); + +describe('hardenBrainRepo', () => { + test('installs hook (local, untracked, +x), helper, and AGENTS rules', async () => { + const r = await harden(); + // hook + const hookPath = join(work, '.git', 'hooks', 'post-commit'); + expect(existsSync(hookPath)).toBe(true); + expect(readFileSync(hookPath, 'utf-8')).toContain('post-commit hook'); + expect(statSync(hookPath).mode & 0o111).toBeTruthy(); // executable + // helper (committed, +x) + const helperPath = join(work, 'scripts', 'brain-commit-push.sh'); + expect(existsSync(helperPath)).toBe(true); + expect(statSync(helperPath).mode & 0o111).toBeTruthy(); + // AGENTS.md with managed block + taxonomy + const agents = readFileSync(join(work, 'AGENTS.md'), 'utf-8'); + expect(agents).toContain('BEGIN gbrain-brain-durability'); + expect(agents).toContain('people/'); + expect(agents).toContain('brain-commit-push.sh'); + // verify pushed scaffolding → clean against origin + expect(r.clean_against_origin).toBe(true); + expect(r.needs_attention).toEqual([]); + }); + + test('is idempotent — second run adds NO new commit', async () => { + await harden(); + const after1 = commitCount(work); + const r2 = await harden(); + expect(commitCount(work)).toBe(after1); // no churn + // every step is ok/skipped on the second pass (nothing left to fix) + expect(r2.steps.every(s => s.status === 'ok' || s.status === 'skipped')).toBe(true); + }); + + test('the post-commit hook is UNTRACKED (never committed)', async () => { + await harden(); + const tracked = git(work, 'ls-files'); + expect(tracked.includes('post-commit')).toBe(false); + expect(tracked).toContain('scripts/brain-commit-push.sh'); // helper IS tracked + }); + + test('D3 — patches RESOLVER.md when it exists, not AGENTS.md', async () => { + writeFileSync(join(work, 'RESOLVER.md'), '# my resolver\n\nuser content\n'); + git(work, 'add', 'RESOLVER.md'); git(work, 'commit', '-qm', 'resolver'); + await harden(); + expect(readFileSync(join(work, 'RESOLVER.md'), 'utf-8')).toContain('BEGIN gbrain-brain-durability'); + expect(existsSync(join(work, 'AGENTS.md'))).toBe(false); + }); + + test('AGENTS block patch preserves user content above and below', async () => { + writeFileSync(join(work, 'AGENTS.md'), '# Top\n\nkeep above\n\n## footer\nkeep below\n'); + git(work, 'add', 'AGENTS.md'); git(work, 'commit', '-qm', 'agents'); + await harden(); + const body = readFileSync(join(work, 'AGENTS.md'), 'utf-8'); + expect(body).toContain('keep above'); + expect(body).toContain('keep below'); + expect(body).toContain('BEGIN gbrain-brain-durability'); + // patch-in-place: exactly one managed block + expect(body.split('BEGIN gbrain-brain-durability').length - 1).toBe(1); + }); + + test('D11 — writes a repo-scoped credential (0600 store, local config, ownership key)', async () => { + await harden(); + const store = join(process.env.GBRAIN_HOME!, 'git-credentials'); + expect(existsSync(store)).toBe(true); + expect(statSync(store).mode & 0o077).toBe(0); // not group/other readable + expect(git(work, 'config', '--local', '--get', 'credential.helper')).toContain('store --file'); + expect(cfg(work, 'gbrain.durability.managedcredential')).toBe('true'); + }); + + test('D11 — reuses an existing credential.helper (no plaintext store written)', async () => { + git(work, 'config', 'credential.helper', 'osxkeychain'); + await harden(); + const store = join(process.env.GBRAIN_HOME!, 'git-credentials'); + expect(existsSync(store)).toBe(false); + expect(git(work, 'config', '--local', '--get', 'credential.helper')).toBe('osxkeychain'); + }); + + test('PAT never appears in the serialized report', async () => { + const r = await harden(); + expect(JSON.stringify(r).includes(PAT)).toBe(false); + }); + + test('detached HEAD → pull step needs_attention (refuses to push to a wrong ref)', async () => { + const sha = git(work, 'rev-parse', 'HEAD'); + git(work, 'checkout', '-q', sha); // detached + const r = await harden({ verify: false }); + const pull = r.steps.find(s => s.step === 'pull'); + expect(pull?.status).toBe('needs_attention'); + }); + + test('D10 — verify reports needs_attention when push-probe fails (read-only/unreachable)', async () => { + git(work, 'remote', 'set-url', 'origin', join(root, 'unreachable.git')); + const r = await harden(); + const verify = r.steps.find(s => s.step === 'verify'); + expect(verify?.status).toBe('needs_attention'); + expect(r.clean_against_origin).toBe(false); + expect(r.needs_attention.length).toBeGreaterThan(0); + // No scaffolding commit when we can't confirm a push. + expect(r.steps.find(s => s.step === 'commit')).toBeUndefined(); + }); + + test('dry-run makes no commit and writes no files', async () => { + const before = commitCount(work); + await harden({ dryRun: true }); + expect(commitCount(work)).toBe(before); + expect(existsSync(join(work, 'scripts', 'brain-commit-push.sh'))).toBe(false); + }); +}); + +describe('unhardenBrainRepo', () => { + test('removes hook + credential wiring; leaves committed content', async () => { + await harden(); + const steps = await unhardenBrainRepo({ repoPath: work, sourceId: 'wiki' }); + expect(existsSync(join(work, '.git', 'hooks', 'post-commit'))).toBe(false); + expect(cfg(work, 'gbrain.durability.managedcredential')).toBe(''); + // committed helper stays + expect(existsSync(join(work, 'scripts', 'brain-commit-push.sh'))).toBe(true); + expect(steps.find(s => s.step === 'hook')?.status).toBe('fixed'); + }); + + test('idempotent when not hardened (all skipped)', async () => { + const steps = await unhardenBrainRepo({ repoPath: work, sourceId: 'wiki' }); + expect(steps.every(s => s.status === 'skipped')).toBe(true); + }); +}); + +describe('acceptPat (D8)', () => { + test('reads + trims a pat-file', () => { + const p = join(root, 'pat.txt'); + writeFileSync(p, `${PAT}\n`, { mode: 0o600 }); + const r = acceptPat({ patFile: p }); + expect(r?.token).toBe(PAT); + expect(r?.warnings).toEqual([]); + }); + test('throws on a missing pat-file', () => { + expect(() => acceptPat({ patFile: join(root, 'nope.txt') })).toThrow(); + }); + test('throws on an empty pat-file', () => { + const p = join(root, 'empty.txt'); writeFileSync(p, ' \n', { mode: 0o600 }); + expect(() => acceptPat({ patFile: p })).toThrow(); + }); + test('warns (but continues) on loose perms', () => { + const p = join(root, 'loose.txt'); writeFileSync(p, PAT); chmodSync(p, 0o644); + const r = acceptPat({ patFile: p }); + expect(r?.token).toBe(PAT); + expect(r?.warnings.length).toBeGreaterThan(0); + }); + test('falls back to GBRAIN_GITHUB_PAT env', () => { + const old = process.env.GBRAIN_GITHUB_PAT; + process.env.GBRAIN_GITHUB_PAT = PAT; + try { expect(acceptPat({})?.source).toBe('env:GBRAIN_GITHUB_PAT'); } + finally { if (old === undefined) delete process.env.GBRAIN_GITHUB_PAT; else process.env.GBRAIN_GITHUB_PAT = old; } + }); + test('returns null when no PAT is available', () => { + const old = process.env.GBRAIN_GITHUB_PAT; delete process.env.GBRAIN_GITHUB_PAT; + try { expect(acceptPat({})).toBeNull(); } + finally { if (old !== undefined) process.env.GBRAIN_GITHUB_PAT = old; } + }); +}); diff --git a/test/db-pacer.test.ts b/test/db-pacer.test.ts new file mode 100644 index 000000000..c5168d52a --- /dev/null +++ b/test/db-pacer.test.ts @@ -0,0 +1,202 @@ +/** + * Pins the DB-pacer primitive: concurrency cap (the real lever), abort-throws, + * in-band EWMA, jittered cooperative sleep, fail-open, and the no-op path. + * + * Uses a fake sleep + deterministic rng so there's no real wall-clock wait and + * no flakiness — the whole suite runs in microtask time. + */ +import { describe, expect, test } from 'bun:test'; +import { createDbPacer, createNoopPacer, type DbPacer } from '../src/core/db-pacer.ts'; +import { AbortError } from '../src/core/abort-check.ts'; +import { PACE_BUNDLES, type PaceBundle } from '../src/core/pace-mode.ts'; + +const BUNDLE: PaceBundle = { + enabled: true, + maxConcurrency: 2, + paceAtMs: 100, + maxSleepMs: 1000, + ewmaAlpha: 0.5, +}; + +/** A fake sleep that resolves immediately but still honors abort. */ +function fakeSleep(ms: number, signal?: AbortSignal): Promise { + if (signal?.aborted) return Promise.reject(new AbortError('aborted')); + return Promise.resolve(); +} + +function pacer(over: Partial = {}, rng = () => 0.5): DbPacer { + return createDbPacer({ bundle: { ...BUNDLE, ...over }, sleep: fakeSleep, rng }); +} + +describe('concurrency cap (the real lever)', () => { + test('caps simultaneous permits at maxConcurrency; release hands off FIFO', async () => { + const p = pacer({ maxConcurrency: 2 }); + const a = await p.acquire(); + const b = await p.acquire(); + expect(p.snapshot().active).toBe(2); + + let cResolved = false; + const cPromise = p.acquire().then((permit) => { + cResolved = true; + return permit; + }); + // c is blocked: still 2 active, 1 waiter. + await Promise.resolve(); + expect(cResolved).toBe(false); + expect(p.snapshot().active).toBe(2); + expect(p.snapshot().maxWaiters).toBe(1); + + a.release(); // hands the slot to c + const c = await cPromise; + expect(cResolved).toBe(true); + expect(p.snapshot().active).toBe(2); // handoff kept the cap, never 3 + + b.release(); + c.release(); + expect(p.snapshot().active).toBe(0); + }); + + test('double release is idempotent', async () => { + const p = pacer({ maxConcurrency: 1 }); + const a = await p.acquire(); + a.release(); + a.release(); + expect(p.snapshot().active).toBe(0); + }); +}); + +describe('abort throws (never falls into a DB call)', () => { + test('acquire throws AbortError when the signal fires while waiting', async () => { + const p = pacer({ maxConcurrency: 1 }); + await p.acquire(); // exhausts the single slot + const ac = new AbortController(); + const blocked = p.acquire(ac.signal); + ac.abort(); + await expect(blocked).rejects.toBeInstanceOf(AbortError); + expect(p.snapshot().maxWaiters).toBe(1); + }); + + test('acquire throws immediately when already aborted', async () => { + const p = pacer(); + const ac = new AbortController(); + ac.abort(); + await expect(p.acquire(ac.signal)).rejects.toBeInstanceOf(AbortError); + }); + + test('pace throws AbortError when aborted during sleep', async () => { + // sleep that rejects with AbortError when the signal is set. + const p = createDbPacer({ + bundle: BUNDLE, + sleep: (_ms, signal) => + signal?.aborted ? Promise.reject(new AbortError('aborted')) : Promise.resolve(), + rng: () => 0.5, + }); + p.observe(1000); // EWMA well above paceAtMs → will sleep + const ac = new AbortController(); + ac.abort(); + await expect(p.pace(ac.signal)).rejects.toBeInstanceOf(AbortError); + }); +}); + +describe('in-band EWMA + observe guard', () => { + test('first sample seeds EWMA; later samples smooth it', () => { + const p = pacer({ ewmaAlpha: 0.5 }); + expect(p.snapshot().ewmaMs).toBeNull(); + p.observe(200); + expect(p.snapshot().ewmaMs).toBe(200); + p.observe(400); + expect(p.snapshot().ewmaMs).toBe(300); // 0.5*400 + 0.5*200 + expect(p.snapshot().sampleCount).toBe(2); + }); + + test('NaN / negative samples are ignored', () => { + const p = pacer(); + p.observe(Number.NaN); + p.observe(-5); + p.observe(Infinity); + expect(p.snapshot().ewmaMs).toBeNull(); + expect(p.snapshot().sampleCount).toBe(0); + }); +}); + +describe('cooperative sleep', () => { + test('no sleep when EWMA is below paceAtMs', async () => { + const p = pacer({ paceAtMs: 100 }); + p.observe(50); + await p.pace(); + expect(p.snapshot().sleepCount).toBe(0); + expect(p.snapshot().totalSleptMs).toBe(0); + }); + + test('sleeps when EWMA exceeds paceAtMs; jittered to 50-100% of base, capped', async () => { + // rng=0 → jitter factor 0.5 → ms = round(base * 0.5). + const p = pacer({ paceAtMs: 100, maxSleepMs: 1000 }, () => 0); + p.observe(800); // base = min(1000, 800) = 800; ms = 400 + await p.pace(); + expect(p.snapshot().sleepCount).toBe(1); + expect(p.snapshot().totalSleptMs).toBe(400); + }); + + test('sleep base is capped at maxSleepMs', async () => { + const p = pacer({ paceAtMs: 100, maxSleepMs: 300 }, () => 1); // jitter factor 1.0 + p.observe(5000); // base = min(300, 5000) = 300; ms = 300 + await p.pace(); + expect(p.snapshot().totalSleptMs).toBe(300); + }); + + test('different rng values decorrelate sleep durations (anti-thundering-herd)', async () => { + const lo = pacer({ paceAtMs: 100 }, () => 0); // factor 0.5 + const hi = pacer({ paceAtMs: 100 }, () => 1); // factor 1.0 + lo.observe(800); + hi.observe(800); + await lo.pace(); + await hi.pace(); + expect(lo.snapshot().totalSleptMs).toBe(400); + expect(hi.snapshot().totalSleptMs).toBe(800); + }); +}); + +describe('fail-open', () => { + test('an unexpected (non-abort) sleep failure does not throw', async () => { + const p = createDbPacer({ + bundle: BUNDLE, + sleep: () => Promise.reject(new Error('boom')), + rng: () => 0.5, + }); + p.observe(1000); + await expect(p.pace()).resolves.toBeUndefined(); + // The failed sleep is not counted. + expect(p.snapshot().sleepCount).toBe(0); + }); +}); + +describe('no-op pacer (off mode / PGLite)', () => { + test('createNoopPacer: unbounded acquire, zero sleeps', async () => { + const p = createNoopPacer(); + const permits = await Promise.all([p.acquire(), p.acquire(), p.acquire()]); + expect(permits).toHaveLength(3); + p.observe(99999); + await p.pace(); + expect(p.snapshot().enabled).toBe(false); + expect(p.snapshot().totalSleptMs).toBe(0); + }); + + test('an off bundle yields a no-op pacer', async () => { + const p = createDbPacer({ bundle: PACE_BUNDLES.off }); + expect(p.snapshot().enabled).toBe(false); + await p.acquire(); + await p.acquire(); // never blocks despite maxConcurrency 0 + expect(p.snapshot().active).toBe(0); + }); +}); + +describe('dispose', () => { + test('dispose releases blocked acquirers with a no-op permit', async () => { + const p = pacer({ maxConcurrency: 1 }); + await p.acquire(); + const blocked = p.acquire(); + p.dispose(); + const permit = await blocked; // resolves instead of hanging + expect(permit).toBeDefined(); + }); +}); diff --git a/test/durability-cron.test.ts b/test/durability-cron.test.ts new file mode 100644 index 000000000..66b56c511 --- /dev/null +++ b/test/durability-cron.test.ts @@ -0,0 +1,44 @@ +/** + * Durability cron generators (v0.42.44, D2 + D12): pure-string renderers. + * Asserts the cron is DB-free (gbrain sources pull --path, NOT `pull `), + * secret-free, self-disabling, and that the launchd plist is periodic. + */ +import { describe, test, expect } from 'bun:test'; +import { renderCronWrapper, generateBrainPullPlist } from '../src/core/brain-repo-durability.ts'; + +const TOKEN = 'ghp_SHOULD_NEVER_APPEAR'; + +describe('renderCronWrapper (D2 DB-free)', () => { + const w = renderCronWrapper('wiki', '/data/clones/wiki', 'main', '/usr/local/bin/gbrain', '/home/u/.gbrain/brain-push.log'); + + test('calls the DB-free path command, not the engine-opening one', () => { + expect(w).toContain("sources pull --path '/data/clones/wiki'"); + expect(w).toContain("--branch 'main'"); + expect(w).not.toMatch(/sources pull '?wiki'?(\s|$)/); // never `sources pull wiki` + }); + + test('self-disables when the captured checkout is gone', () => { + expect(w).toContain("if [ ! -d '/data/clones/wiki/.git' ]"); + expect(w).toContain('path gone, skipping'); + }); + + test('sources the shell profile (secret-free) and never bakes a token', () => { + expect(w).toContain('source ~/.zshenv'); + expect(w.includes(TOKEN)).toBe(false); + }); +}); + +describe('generateBrainPullPlist (D12 launchd)', () => { + const plist = generateBrainPullPlist('com.gbrain.brain-pull.wiki', '/home/u/.gbrain/brain-pull-wiki.sh', '/home/u', 1800); + + test('is periodic (StartInterval), not a KeepAlive daemon', () => { + expect(plist).toContain('StartInterval1800'); + expect(plist).not.toContain('KeepAlive'); + }); + + test('carries the per-source label and the wrapper path only (no secret)', () => { + expect(plist).toContain('com.gbrain.brain-pull.wiki'); + expect(plist).toContain('/home/u/.gbrain/brain-pull-wiki.sh'); + expect(plist.includes(TOKEN)).toBe(false); + }); +}); diff --git a/test/git-remote-durable.serial.test.ts b/test/git-remote-durable.serial.test.ts new file mode 100644 index 000000000..527346bc3 --- /dev/null +++ b/test/git-remote-durable.serial.test.ts @@ -0,0 +1,153 @@ +/** + * git-remote durability helpers (v0.42.44): divergenceSafePull, detectDefaultBranch, + * pushProbe, isWorkingTreeDirty. Real git against local bare remotes. + * + * Local file transport is enabled via GBRAIN_GIT_ALLOW_FILE_TRANSPORT=1 (the + * documented escape hatch the durability paths honor). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync, existsSync, mkdirSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { execFileSync } from 'child_process'; +import { + divergenceSafePull, detectDefaultBranch, pushProbe, isWorkingTreeDirty, +} from '../src/core/git-remote.ts'; + +function git(cwd: string, ...args: string[]): string { + return execFileSync('git', ['-C', cwd, '-c', 'protocol.file.allow=always', ...args], { + stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8', + }).trim(); +} +function gitIn(cwd: string, ...args: string[]): string { + return execFileSync('git', [...args], { cwd, stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8' }).trim(); +} + +let root: string; +let bare: string; + +/** A bare origin + a working clone with one commit on `main`. */ +function makePair(): { bare: string; work: string } { + const b = mkdtempSync(join(root, 'origin-')) + '.git'; + execFileSync('git', ['init', '-q', '--bare', '-b', 'main', b], { stdio: 'ignore' }); + const w = mkdtempSync(join(root, 'work-')); + execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', b, w], { stdio: 'ignore' }); + git(w, 'config', 'user.email', 't@t.t'); + git(w, 'config', 'user.name', 'tester'); + writeFileSync(join(w, 'README.md'), 'init\n'); + git(w, 'add', 'README.md'); + git(w, 'commit', '-qm', 'init'); + git(w, 'push', '-q', 'origin', 'main'); + // Set origin/HEAD so detectDefaultBranch can resolve it. + try { git(w, 'remote', 'set-head', 'origin', 'main'); } catch { /* */ } + return { bare: b, work: w }; +} + +function secondClone(b: string): string { + const w = mkdtempSync(join(root, 'work2-')); + execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', b, w], { stdio: 'ignore' }); + git(w, 'config', 'user.email', 'u@u.u'); + git(w, 'config', 'user.name', 'tester2'); + return w; +} + +beforeAll(() => { process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT = '1'; }); +afterAll(() => { delete process.env.GBRAIN_GIT_ALLOW_FILE_TRANSPORT; }); +beforeEach(() => { root = mkdtempSync(join(tmpdir(), 'gd-')); }); +afterEach(() => { rmSync(root, { recursive: true, force: true }); }); + +describe('detectDefaultBranch', () => { + test('resolves origin/HEAD', () => { + const { work } = makePair(); + expect(detectDefaultBranch(work)).toBe('main'); + }); + test('falls back to main when nothing resolves', () => { + const empty = mkdtempSync(join(root, 'bare-')); // not a git repo + expect(detectDefaultBranch(empty)).toBe('main'); + }); +}); + +describe('isWorkingTreeDirty', () => { + test('false when clean, true after an edit', () => { + const { work } = makePair(); + expect(isWorkingTreeDirty(work)).toBe(false); + writeFileSync(join(work, 'README.md'), 'changed\n'); + expect(isWorkingTreeDirty(work)).toBe(true); + }); +}); + +describe('divergenceSafePull', () => { + test('up_to_date when already current', () => { + const { work } = makePair(); + expect(divergenceSafePull(work, 'main').status).toBe('up_to_date'); + }); + + test('advanced when origin has a new commit', () => { + const { bare, work } = makePair(); + const other = secondClone(bare); + writeFileSync(join(other, 'b.txt'), 'b\n'); + git(other, 'add', 'b.txt'); git(other, 'commit', '-qm', 'b'); git(other, 'push', '-q', 'origin', 'main'); + const out = divergenceSafePull(work, 'main'); + expect(out.status).toBe('advanced'); + expect(existsSync(join(work, 'b.txt'))).toBe(true); + }); + + test('skipped_dirty when the working tree is dirty', () => { + const { work } = makePair(); + writeFileSync(join(work, 'README.md'), 'local edit\n'); + expect(divergenceSafePull(work, 'main').status).toBe('skipped_dirty'); + }); + + test('rebases local commits over origin', () => { + const { bare, work } = makePair(); + const other = secondClone(bare); + writeFileSync(join(other, 'remote.txt'), 'r\n'); + git(other, 'add', 'remote.txt'); git(other, 'commit', '-qm', 'remote'); git(other, 'push', '-q', 'origin', 'main'); + // local commit on a DIFFERENT file → clean rebase + writeFileSync(join(work, 'local.txt'), 'l\n'); + git(work, 'add', 'local.txt'); git(work, 'commit', '-qm', 'local'); + const out = divergenceSafePull(work, 'main'); + expect(out.status).toBe('advanced'); + expect(existsSync(join(work, 'remote.txt'))).toBe(true); + expect(existsSync(join(work, 'local.txt'))).toBe(true); + }); + + test('conflict_aborted leaves NO rebase state', () => { + const { bare, work } = makePair(); + const other = secondClone(bare); + writeFileSync(join(other, 'README.md'), 'remote version\n'); + git(other, 'add', 'README.md'); git(other, 'commit', '-qm', 'remote'); git(other, 'push', '-q', 'origin', 'main'); + // local commit touching the SAME line → rebase conflict + writeFileSync(join(work, 'README.md'), 'local version\n'); + git(work, 'add', 'README.md'); git(work, 'commit', '-qm', 'local'); + const out = divergenceSafePull(work, 'main'); + expect(out.status).toBe('conflict_aborted'); + // The "never mid-rebase" invariant: + expect(existsSync(join(work, '.git', 'rebase-merge'))).toBe(false); + expect(existsSync(join(work, '.git', 'rebase-apply'))).toBe(false); + // Working tree is usable (HEAD is the local commit, not a conflicted state). + expect(gitIn(work, 'rev-parse', '--abbrev-ref', 'HEAD')).toBe('main'); + }); +}); + +describe('pushProbe', () => { + test('ok against a writable remote', () => { + const { work } = makePair(); + expect(pushProbe(work, 'main')).toEqual({ ok: true }); + }); + + test('not ok when origin is unreachable', () => { + const { work } = makePair(); + git(work, 'remote', 'set-url', 'origin', join(root, 'does-not-exist.git')); + const r = pushProbe(work, 'main'); + expect(r.ok).toBe(false); + }); + + test('redactDetail scrubs a token from the failure detail', () => { + const { work } = makePair(); + git(work, 'remote', 'set-url', 'origin', join(root, 'nope-ghp_SECRETTOKEN.git')); + const r = pushProbe(work, 'main', { redactDetail: (s) => s.replaceAll('ghp_SECRETTOKEN', '***') }); + expect(r.ok).toBe(false); + if (!r.ok) expect(r.detail.includes('ghp_SECRETTOKEN')).toBe(false); + }); +}); diff --git a/test/pace-mode.test.ts b/test/pace-mode.test.ts new file mode 100644 index 000000000..414662d6f --- /dev/null +++ b/test/pace-mode.test.ts @@ -0,0 +1,120 @@ +/** + * Pins the DB-pacing mode bundles + resolution chain. + * The load-bearing claim: env beats config (incident escape hatch), per-call + * beats env, and `off` resolves to a disabled bundle. + */ +import { describe, expect, test } from 'bun:test'; +import { + PACE_BUNDLES, + PACE_MODES, + DEFAULT_PACE_MODE, + isPaceMode, + resolvePaceMode, + loadOverridesFromConfig, + readPaceEnv, +} from '../src/core/pace-mode.ts'; + +describe('pace-mode bundles', () => { + test('off is the default and is disabled', () => { + expect(DEFAULT_PACE_MODE).toBe('off'); + expect(PACE_BUNDLES.off.enabled).toBe(false); + }); + + test('enabled bundles cap concurrency below the unpaced default (20)', () => { + for (const m of ['gentle', 'balanced', 'aggressive'] as const) { + expect(PACE_BUNDLES[m].enabled).toBe(true); + expect(PACE_BUNDLES[m].maxConcurrency).toBeGreaterThanOrEqual(1); + expect(PACE_BUNDLES[m].maxConcurrency).toBeLessThan(20); + } + }); + + test('isPaceMode guards the union', () => { + expect(PACE_MODES.every(isPaceMode)).toBe(true); + expect(isPaceMode('turbo')).toBe(false); + expect(isPaceMode(undefined)).toBe(false); + }); +}); + +describe('resolvePaceMode precedence', () => { + test('unknown mode falls back to off (disabled), mode_valid=false', () => { + const r = resolvePaceMode({ mode: 'turbo' }); + expect(r.resolved_mode).toBe('off'); + expect(r.mode_valid).toBe(false); + expect(r.enabled).toBe(false); + }); + + test('config mode applies its bundle', () => { + const r = resolvePaceMode({ mode: 'balanced' }); + expect(r.resolved_mode).toBe('balanced'); + expect(r.maxConcurrency).toBe(PACE_BUNDLES.balanced.maxConcurrency); + }); + + test('env mode beats config mode (incident escape hatch)', () => { + const r = resolvePaceMode({ mode: 'gentle', envMode: 'aggressive' }); + expect(r.resolved_mode).toBe('aggressive'); + }); + + test('per-call mode beats env and config', () => { + const r = resolvePaceMode({ mode: 'gentle', envMode: 'balanced', perCallMode: 'aggressive' }); + expect(r.resolved_mode).toBe('aggressive'); + }); + + test('env knob override beats config knob override', () => { + const r = resolvePaceMode({ + mode: 'balanced', + configOverrides: { maxConcurrency: 6 }, + envOverrides: { maxConcurrency: 2 }, + }); + expect(r.maxConcurrency).toBe(2); + }); + + test('per-call knob beats env and config', () => { + const r = resolvePaceMode({ + mode: 'balanced', + configOverrides: { maxConcurrency: 6 }, + envOverrides: { maxConcurrency: 2 }, + perCall: { maxConcurrency: 12 }, + }); + expect(r.maxConcurrency).toBe(12); + }); + + test('clamps an out-of-range maxConcurrency back to the bundle when enabled', () => { + const r = resolvePaceMode({ mode: 'balanced', perCall: { maxConcurrency: 0 } }); + expect(r.enabled).toBe(true); + expect(r.maxConcurrency).toBe(PACE_BUNDLES.balanced.maxConcurrency); + }); + + test('clamps a bad ewmaAlpha back to a sane default', () => { + const r = resolvePaceMode({ mode: 'balanced', perCall: { ewmaAlpha: 5 } }); + expect(r.ewmaAlpha).toBeGreaterThan(0); + expect(r.ewmaAlpha).toBeLessThanOrEqual(1); + }); +}); + +describe('config + env parsing', () => { + test('loadOverridesFromConfig parses present keys only', () => { + const ov = loadOverridesFromConfig({ + 'pace.max_concurrency': '5', + 'pace.pace_at_ms': '400', + }); + expect(ov.maxConcurrency).toBe(5); + expect(ov.paceAtMs).toBe(400); + expect(ov.maxSleepMs).toBeUndefined(); + }); + + test('readPaceEnv reads GBRAIN_PACE_* including mode', () => { + const { envMode, envOverrides } = readPaceEnv({ + GBRAIN_PACE_MODE: 'gentle', + GBRAIN_PACE_MAX_CONCURRENCY: '3', + GBRAIN_PACE_ENABLED: 'true', + }); + expect(envMode).toBe('gentle'); + expect(envOverrides.maxConcurrency).toBe(3); + expect(envOverrides.enabled).toBe(true); + }); + + test('rejects out-of-range env concurrency (falls through)', () => { + const { envOverrides } = readPaceEnv({ GBRAIN_PACE_MAX_CONCURRENCY: '99999' }); + expect(envOverrides.maxConcurrency).toBeUndefined(); + }); +});