diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 5484e8c21..062cb1859 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -42,7 +42,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/db.ts` — Connection management, schema initialization. `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT`/`GBRAIN_IDLE_TX_TIMEOUT`/`GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (`setSessionDefaults` kept as a back-compat no-op shim). `connect()` returns `Promise` — `true` iff THIS call created the module singleton, `false` if it joined an existing one; the decision is atomic (no `await` between the `if (sql)` null-check and the synchronous `sql = postgres(...)` assignment), so two concurrent module connects can't both claim creation. `PostgresEngine` stores the return as its `_ownsModuleSingleton` token and only the creating engine may `db.disconnect()` the singleton — a borrower probe engine (lint/doctor config-lift) no-ops its disconnect, so its teardown can't null the connection the long-lived cycle owner is still using (the dream-cycle "connect() has not been called" failure). The module `sql` is only ever nulled by `db.disconnect()` (postgres.js auto-reconnects its own internal pool and never touches our reference). `disconnect()` snapshots + nulls `sql` before awaiting the pool end so a concurrent connect can't join a pool that's already closing. The end routes through `endPoolBounded(pool)` (#1972) — a gbrain-owned `Promise.race` of `pool.end({ timeout: POOL_END_TIMEOUT_SECONDS })` against a hard timer — so a PgBouncer transaction-mode drain that never settles can't hang teardown — the #2084 contract (finishCliTeardown's computed-deadline backstop + flushThenExit's fence-and-grace exit in cli-force-exit.ts) bounds it and delivers pending stdout before exit. `connection-manager.ts` ends its direct + read pools concurrently through the same helper so the per-pool bounds don't stack. - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`). Copies the complete source catalog FIRST (`copyMigrationSources` — every `sources` row incl. archived rows and sync/routing metadata, `ON CONFLICT (id) DO UPDATE`, `default` ordered first) so every page write has a valid `pages.source_id` FK parent and the target preserves per-source behavior; pages copy afterward, tracked in the resume manifest by composite `(source_id, slug)` key. The resume manifest is target-aware: `migrationTargetId(config)` hashes `(engine, locator)` (`database_url` for Postgres, resolved `database_path` for PGLite) and `manifestMatchesTarget` requires `schema_version === 2` plus a matching `target_id` — a legacy engine-only manifest, or one from a DIFFERENT target of the same engine kind, starts fresh instead of skipping "completed" pages the new target never received. Pinned by `test/migrate-engine-resume.test.ts` (manifest identity) + `test/e2e/migrate-engine-sources-postgres.test.ts` (source catalog lands before overlapping-slug pages, PGLite → real Postgres). - `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags). `importFromContent` and `importCodeFile` stamp `pages.embedding_signature` via `setPageEmbeddingSignature(slug, {sourceId, signature: currentEmbeddingSignature()})` when the import actually embedded (not `--no-embed`) so a model/dims swap is detectable as stale; `importCodeFile` only stamps when every chunk was freshly embedded this call (`needsEmbedIndexes.length === chunks.length`), mixed reuse-by-hash pages stay unstamped (`reindex --code --force` / `embed --stale` handle those). `importFromContent`'s tag reconciliation is ADD-ONLY: it only `addTag` (idempotent, ON CONFLICT DO NOTHING). The `tags` table has no provenance column and frontmatter tags are stripped from stored `pages.frontmatter` (markdown.ts:118), so a frontmatter-origin tag can't be distinguished from a DB-enrichment tag (auto-tag / dream synthesize / signal-detector) at re-import — deletion is unsafe (would wipe enrichment under `gbrain reindex --markdown`). Accepted trade-off: removing a tag from frontmatter no longer removes it from the DB on next sync (needs a `tag_source` provenance column). Pinned by `test/reindex-preserve-tags.test.ts` + `test/import-file.test.ts`. -- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). Exported `pruneDir(name: string): boolean` is the single source of truth for descent-time directory exclusion across walkers — blocks `node_modules` (no leading dot, so naive walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars; `isSyncable` applies it per path segment, and `walkMarkdownFiles` in `src/commands/extract.ts` + `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing to save the IO of walking thousands of vendor files (closes #923 + #202). `manageGitignore` worktree discriminator matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) so Conductor worktrees (first-class repos) get `.gitignore` management for storage-tiering (closes #889). The sync-failure ledger (failure store, error classifier, the shared bookmark gate, and the doctor severity rule) lives in `src/core/sync-failure-ledger.ts`; `sync.ts` re-exports `classifyErrorCode`, `summarizeFailuresByCode`, `loadSyncFailures`, `unacknowledgedSyncFailures`, `acknowledgeSyncFailures`, `recordSyncFailures`, `decideSyncFailureSeverity`, `applySyncFailureGate`, and the `SyncFailure` type for backward-compatible imports — see its entry below. +- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). Exported `pruneDir(name: string): boolean` is the single source of truth for descent-time directory exclusion across walkers — blocks `node_modules` (no leading dot, so naive walkers slipped through and inflated MISSING_OPEN counts via vendor packages), `vendor`/`dist`/`build`/`venv`, dot-prefix dirs, and `*.raw` sidecars — NOT `ops/`, which is ordinary user content (#2404; the bundled daily-task-manager stores `ops/tasks` there); `isSyncable` applies it per path segment, and `walkMarkdownFiles` in `src/commands/extract.ts` + `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing to save the IO of walking thousands of vendor files (closes #923 + #202). `manageGitignore` worktree discriminator matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) so Conductor worktrees (first-class repos) get `.gitignore` management for storage-tiering (closes #889). The sync-failure ledger (failure store, error classifier, the shared bookmark gate, and the doctor severity rule) lives in `src/core/sync-failure-ledger.ts`; `sync.ts` re-exports `classifyErrorCode`, `summarizeFailuresByCode`, `loadSyncFailures`, `unacknowledgedSyncFailures`, `acknowledgeSyncFailures`, `recordSyncFailures`, `decideSyncFailureSeverity`, `applySyncFailureGate`, and the `SyncFailure` type for backward-compatible imports — see its entry below. - `src/core/sync-failure-ledger.ts` — the bounded auto-skip sync failure ledger (issue #1939; formerly inline "Bug 9" in `sync.ts`). A LEAF module (imports only fs/path/crypto/config) so `sync.ts` can re-export it without a circular dependency. State lives in `~/.gbrain/sync-failures.jsonl`, one JSON object per line, keyed by `(source_id, path)` with a per-key `attempts` count and a 3-state machine: `open` (fresh/blocking) → `acknowledged` (human resolved via `gbrain sync --skip-failed`) or `auto_skipped` (chronic). `classifyErrorCode(errorMsg)` regex classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` (also recognizes `PAGE_JUNK_PATTERN` from the content-sanity gate); `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`; `MISSING_OPEN`/`MISSING_CLOSE`/`EMPTY_FRONTMATTER` regexes match the `markdown.ts` validator strings, `FILE_TOO_LARGE` covers `import-file.ts:199, 352, 401`, `SYMLINK_NOT_ALLOWED` covers `:347`. All mutations run under `withLedgerLock` (cross-process file lock) with an atomic rename write. The auto-skip threshold resolves via `resolveAutoSkipThreshold()` from `GBRAIN_SYNC_AUTOSKIP_AFTER` (default `DEFAULT_AUTOSKIP_AFTER = 3`; `0` disables the valve = pure fail-closed). Two pure decision functions are the unit-test surface: `decideGateAction({fileFailures, sentinels, attemptsByPath, threshold, skipFailed})` returns `hard_block | block | advance | advance_then_autoskip` (sentinels like `` ALWAYS hard-block, even with `--skip-failed`, so a history rewrite can't auto-skip; any FRESH failure with `attempts < threshold` blocks fail-closed; only when ALL failures are chronic does it `advance_then_autoskip`), and `decideSyncFailureSeverity({entries, nowMs, failHours})` returns the `sync_failures` doctor status (`ok` when zero unresolved; `fail` when ≥10 OPEN-blocking or the oldest OPEN failure has blocked the bookmark past `failHours`; otherwise `warn` — `auto_skipped`-only rows stay WARN-visible regardless of count because the bookmark already advanced). `applySyncFailureGate(input)` is the one orchestrator BOTH sync paths (incremental + full/`runImport`) call: it records/clears ledger rows, runs `decideGateAction`, then executes effects in the crash-safe order (advance the bookmark FIRST via the injected `advance()` callback, THEN auto-skip the chronic set) so a crash can never mark a file skipped while leaving sync wedged. `isSkippablePath` rejects `<…>` sentinels. Pinned by `test/sync-failure-ledger.serial.test.ts` + `test/sync-failures.test.ts`. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local). - `src/core/storage-config.ts` — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked`/`supabase_only`) to canonical (`db_tracked`/`db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Uses a dedicated parser for the `gbrain.yml` shape rather than gray-matter (broken on delimiter-less YAML). @@ -50,7 +50,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/git-head.ts` — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true also requires a clean working tree (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). `requireCleanWorkingTree` is `boolean | 'ignore-untracked'` — in `'ignore-untracked'` mode the clean probe runs `git status --porcelain --untracked-files=no` so a quiet repo with stray untracked dirs (`?? companies/`, `?? media/`) is still "unchanged" (sync's incremental path keys off the commit diff and never imports untracked files); `GitCleanProbe` gains an `ignoreUntracked?` second arg. Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) keep unit tests R2-compliant (no `mock.module`). Uses `execFileSync` with array args so shell metachars in `local_path` cannot escape to a shell (the regression test runs real `execFileSync` against `'/nonexistent/$(touch )/repo'` and asserts the sentinel is never created). Fail-open on every error (missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → false) preserving the caller's prior time-based behavior. The chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Pinned by `test/core/git-head.test.ts` (incl. the shell-injection regression guard). - `src/core/source-health.ts` — per-source health metrics for `gbrain sources status` + doctor's `federation_health`. Commit-relative staleness: `newestCommitMs(localPath)` = HEAD committer time via `git log -1 --format=%ct` (fail-open null; NO working-tree mtime parsing — committed content only, robust against the porcelain-mtime bug farm); pure `lagFromContentMs(contentMs|null, lastSyncMs|null, nowMs)` = remote/column comparator (null lastSync → null; negative wall-clock → skew passthrough; `contentMs <= lastSync` → 0; else/null-content → wall-clock). `computeAllSourceMetrics(engine, sources, {probeContent?})`: LOCAL (`probeContent:true`, `gbrain sources status`) → `isSourceUnchangedSinceSync(..., {requireCleanWorkingTree:'ignore-untracked'}) ? 0 : wall-clock` (live commit-hash catches HEAD moving to an old-dated commit a timestamp compare would miss); REMOTE (default, `federation_health` on the HTTP MCP path) → `lagFromContentMs(row.newest_content_at, ...)`, NO git subprocess (trust boundary). `commitTimeMs(localPath, sha)` is the `newestCommitMs` sibling pinned to an arbitrary commit (committer time via `git show -s --format=%ct `, fail-open null, execFileSync array args) — the resumable sync stamps `newest_content_at` against its pinned target commit, not whatever HEAD raced to. Pinned by `test/source-health.test.ts`. - `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo`, `pullRepo`, and `fetchRemote(repoPath, branch)` (the last added for the sync cost-estimator's fetch-first path, #2139, so a cost preview / dry-run fetches through the same hardened flags + `GIT_TERMINAL_PROMPT=0` as real sync rather than a less-protected route). Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (a combined array would spread `--no-recurse-submodules` before the verb where real git rejects it exit 129). `cloneRepo` argv: `git clone --depth=1 [--branch X] -- `. `pullRepo` argv: `git -C pull --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). Also exports the durability-side helpers that power `gbrain sources harden/pull`: `GIT_ENV_AUTH` (the no-prompt env minus the askpass `/bin/false` overrides, so an auth'd push/fetch can consult the repo's configured credential helper while `GIT_TERMINAL_PROMPT=0` still fails fast on a missing credential), `divergenceSafePull(repoPath, branch)` (fetch + `pull --rebase`; returns `skipped_dirty` on a dirty tree, `conflict_aborted` on a rebase conflict after `rebase --abort` so the tree is never left mid-rebase, else `up_to_date`/`advanced`), `detectDefaultBranch` (origin/HEAD → current branch → `main`), `pushProbe(repoPath, branch)` (authenticated `push --dry-run` that proves push access and classifies `auth`/`protected`/`unreachable`), and `isWorkingTreeDirty`. These auth'd paths route their `protocol.file.allow` through `GBRAIN_GIT_ALLOW_FILE_TRANSPORT` (default `never`; set `=1` for self-hosted filesystem remotes), unlike clone/pull which stay strict. -- `src/core/brain-repo-durability.ts` + `src/commands/sources-harden.ts` — brain-repo git durability. `hardenBrainRepo(opts)` makes a brain's working tree durable, idempotently: divergence-safe pull, a LOCAL untracked `.git/hooks/post-commit` auto-push safety net (never committed — a pulled commit can't rewrite executed code next to the token; installed into the active `core.hooksPath` dir and excluded via `.git/info/exclude` when that dir is tracked), a committed `scripts/brain-commit-push.sh` that refuses to exit 0 without a confirmed push (hook + helper render from ONE bash push-retry template — DRY at the TS source, not by the hook sourcing a repo-controlled script), durability rules patched into the active resolver file (`findResolverFile` → RESOLVER.md > AGENTS.md; taxonomy rendered from the bundled `_brain-filing-rules.json`), a minimal DB-free pull cron (launchd/crontab running `gbrain sources pull --path ` so it never opens the PGLite single-writer lock), and a push-probe verify (no heartbeat commit). Credential is REPO-scoped (`acceptPat` from `--pat-file`/`GBRAIN_GITHUB_PAT`, warns on loose perms; reuses an existing repo-local `credential.helper`, else a `0600` store wired via repo-local config); the token is redacted everywhere via `redactSecretsInText` and never enters the repo, remote URL, logs, or `DurabilityReport`. `unhardenBrainRepo` removes the cron/hook/credential wiring (ownership-fingerprinted) and runs before `sources remove`. CLI: `gbrain sources harden ` / `pull |--path ` / `unharden `; auto-harden fires on `sources add --url ... --pat-file` for managed clones (`--no-harden` opts out). `sources pull --path` is dispatched in `src/cli.ts` BEFORE `connectEngine` so the cron stays DB-free. CLI-only (writes executables + an OS cron + a credential helper on the host); never exposed over MCP. Tests: `test/brain-repo-durability.serial.test.ts`, `test/git-remote-durable.serial.test.ts`, `test/brain-durability-hook.serial.test.ts`, `test/durability-cron.test.ts`. +- `src/core/brain-repo-durability.ts` + `src/commands/sources-harden.ts` — brain-repo git durability. `hardenBrainRepo(opts)` makes a brain's working tree durable, idempotently: divergence-safe pull, a LOCAL untracked `.git/hooks/post-commit` auto-push safety net (never committed — a pulled commit can't rewrite executed code next to the token; installed into the active `core.hooksPath` dir and excluded via `.git/info/exclude` when that dir is tracked), a committed `scripts/brain-commit-push.sh` that refuses to exit 0 without a confirmed push and stages+commits BEFORE any pull so a dirty tree of modified pages (the write-through shape) can still be committed — the push-retry's rebase-on-reject handles a remote that advanced (#2426; hook + helper render from ONE bash push-retry template — DRY at the TS source, not by the hook sourcing a repo-controlled script), durability rules patched into the active resolver file (`findResolverFile` → RESOLVER.md > AGENTS.md; taxonomy rendered from the bundled `_brain-filing-rules.json`), a minimal DB-free pull cron (launchd/crontab running `gbrain sources pull --path ` so it never opens the PGLite single-writer lock), and a push-probe verify (no heartbeat commit). Credential is REPO-scoped (`acceptPat` from `--pat-file`/`GBRAIN_GITHUB_PAT`, warns on loose perms; reuses an existing repo-local `credential.helper`, else a `0600` store wired via repo-local config); the token is redacted everywhere via `redactSecretsInText` and never enters the repo, remote URL, logs, or `DurabilityReport`. `unhardenBrainRepo` removes the cron/hook/credential wiring (ownership-fingerprinted) and runs before `sources remove`. CLI: `gbrain sources harden ` / `pull |--path ` / `unharden `; auto-harden fires on `sources add --url ... --pat-file` for managed clones (`--no-harden` opts out). `sources pull --path` is dispatched in `src/cli.ts` BEFORE `connectEngine` so the cron stays DB-free. CLI-only (writes executables + an OS cron + a credential helper on the host); never exposed over MCP. Tests: `test/brain-repo-durability.serial.test.ts`, `test/git-remote-durable.serial.test.ts`, `test/brain-durability-hook.serial.test.ts`, `test/durability-cron.test.ts`. - `src/commands/storage.ts` — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time. - `gbrain.yml` (brain repo root) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database. - `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check). @@ -197,7 +197,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/sources.ts` — `gbrain sources {list,add,remove,archive,restore,archived,purge,current,status,audit}`. `current [--json]` calls `resolveSourceWithTier()` and prints `source_id`, `tier` (`flag | env | dotfile | local_path | brain_default | seed_default`), and optional `detail` (decision table in `skills/conventions/brain-routing.md`). `status [--json]` — read-only per-source dashboard (last sync, staleness, page count, embedding coverage, unacked failures); thin wrapper around `buildSyncStatusReport` + `printSyncStatusReport` from `src/commands/sync.ts`; `--json` emits stable `{schema_version: 1, sources, ...}` on stdout; filters input to `local_path IS NOT NULL AND archived IS NOT TRUE`. `audit [--json]` — read-only dry-run disk scan for size distribution + would-blocks + junk-pattern hits WITHOUT touching the DB; walks `sources.local_path`, reads each markdown file, runs `assessContent()` from `src/core/content-sanity.ts`, aggregates by verdict (`ok | warn_oversize | hard_block_junk_pattern`). The live `runStatus` health table gains a `BACKFILL` column between `EMBED` and `FAILS` (`active(N)` beats `queued(N)` beats `idle`, from `SourceMetrics.backfill_active` / `backfill_queued` in `src/core/source-health.ts`) so operators see deferred `embed-backfill` minion work after `sync --all` exits 0; `jobCountsBySource` in `source-health.ts` widens its `minion_jobs` SQL with two `COUNT(*) FILTER (WHERE name = 'embed-backfill' AND ...)` aggregates (best-effort, all-0 on pre-minions brains). Pinned by `test/content-sanity.test.ts`, `test/import-file-content-sanity.test.ts`, `test/source-health.test.ts`. - `src/commands/reindex-frontmatter.ts` — `gbrain reindex-frontmatter`. Query path wrapped in the standard `withEngine(...)` lifecycle so `engine.connect()` runs before the first SQL call. Pinned by `test/reindex-frontmatter-connect.test.ts`. - `src/core/source-resolver.ts` — 6-tier source resolution. `resolveSourceWithTier(engine, explicit, cwd)` returns `{ source_id, tier: SourceTier, detail? }` alongside `resolveSourceId()` (unchanged). `SOURCE_TIER_NAMES = ['flag', 'env', 'dotfile', 'local_path', 'sole_non_default', 'brain_default', 'seed_default']` (7 entries; order matches priority). Tier `sole_non_default` slots between `local_path` and `brain_default`: when NO `sources.default` config is set AND exactly one registered source has `local_path` AND isn't `'default'`, auto-route to it; archived sources excluded (try/catch for pre-v34 brains); private `pickSoleNonDefaultSource(engine)` shared by both resolver entry points so they cannot drift. Exported `formatSoleNonDefaultNudge(sourceId): string | null` builds the user-facing stderr nudge (null when `GBRAIN_NO_SOLE_NON_DEFAULT_NUDGE=1`). `src/commands/sync.ts:1497-1519` calls `resolveSourceWithTier` unconditionally so the tier fires; `src/commands/import.ts:96-128` mirrors with the tier-gated nudge. Consumed by `gbrain sources current`, `import --source-id`, `extract --source-id`, and the `source_routing_health` doctor check. Pinned by `test/source-resolver-with-tier.test.ts` (`withEnv()` per test-isolation lint), `test/source-resolver-sole-non-default.test.ts` (14 cases), `test/sync-sole-non-default-routing.test.ts` (3 PGLite cases driving real `runSync`). -- `src/core/sync.ts` extension — `isSyncable` factored through private `classifySync(path, opts): SyncableReason | null`; exported companion `unsyncableReason(path, opts)` returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` is a named export (the four canonical metafile basenames `schema.md`, `index.md`, `log.md`, `README.md`). `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. `src/commands/sync.ts:772` cleanup loop guards on `unsyncableReason(path) === 'metafile'` so previously-indexed metafile pages survive every re-sync. Does NOT cover `manifest.deleted` (the upstream filter already strips metafiles). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases, duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases incl. the renamed `.md → .txt` negative). +- `src/core/sync.ts` extension — `isSyncable` factored through private `classifySync(path, opts): SyncableReason | null`; exported companion `unsyncableReason(path, opts)` returns the same tagged reason or null when syncable. `SYNC_SKIP_FILES` is a named export (the four canonical metafile basenames `schema.md`, `index.md`, `log.md`, `README.md`). `SyncableReason` union: `'metafile' | 'strategy' | 'pruned-dir' | 'include-glob-miss' | 'exclude-glob-hit'`. The `commands/sync.ts` cleanup loop guards on `unsyncableReason(path)` being `'metafile'` OR `'pruned-dir'` (#2404) so previously-indexed metafile pages AND deliberately put-created pages under pruned dirs survive every re-sync. Does NOT cover `manifest.deleted` (the upstream filter already strips metafiles). Pinned by `test/sync-isSyncable-shape.test.ts` (15 cases, duality contract) + `test/sync-metafile-skip.serial.test.ts` (3 PGLite cases incl. the renamed `.md → .txt` negative). - `src/core/import-file.ts` extension — identity-based dedup pre-check at `:427-490`. Calls `engine.findDuplicatePage?.(sourceId, {hash, frontmatterId})` (optional `?` so test doubles compile). Posture: SKIP when `frontmatter.id` matches (true external duplicate from overlapping ingest roots), WARN-ALWAYS on content_hash collision with different/missing `frontmatter.id` (templates and daily logs may legitimately share text), FAIL CLOSED on lookup error, bypass via `--force-rechunk`. Soft-deleted pages excluded at the engine layer so tombstones don't block legitimate re-imports under new slugs. Pinned by `test/import-dedup-frontmatter-id.test.ts` (11 cases). - `src/core/engine.ts` extension — two interface members: (1) optional `findDuplicatePage?(sourceId, {hash, frontmatterId?}): Promise<{slug, id} | null>` (identity precedence is content_hash OR frontmatter->>'id', both with `deleted_at IS NULL`); (2) `resolveSlugs(partial, opts?)` extended with `{sourceId?, sourceIds?}` so the MCP fuzzy `get_page` path scopes by source (field names match `sourceScopeOpts(ctx)` output so handlers spread directly; back-compatible — no opts gives prior behavior). Plus a stable tiebreaker `ORDER BY score DESC, page_id ASC, chunk_id ASC` in `searchVector` in both engines: on a score tie (basis-vector eval fixtures) older `page_id` wins, closing the planner-non-determinism class where a new index on `pages` could flip ranking on tied scores. - `src/core/migrate.ts` v95 — `pages_dedup_partial_index` adds `CREATE INDEX pages_dedup_idx ON pages (source_id, content_hash) WHERE deleted_at IS NULL`. Postgres uses `CREATE INDEX CONCURRENTLY` with `transaction: false` + pre-drops any invalid remnant; PGLite uses plain `CREATE INDEX`. Powers `findDuplicatePage` hot path (O(log n) instead of O(n)). @@ -283,7 +283,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `test/fixtures/whoknows-eval.jsonl` — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries; placeholder uses obviously-example slugs (`wiki/people/example-alice`). Drives `test/e2e/whoknows.test.ts` (seeds a matching synthetic brain, asserts the >=80% gate) and the `whoknows_health` doctor check. - `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904). `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent: validation-gated (median-of-3 + epsilon=0.05), budget-capped (preflight estimator), per-skill DB-locked (`tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (history-intent-first 5-step commit), body-only mutations (frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload`. Two reflect calls per step; rejected-edit buffer LRU-bounded to 100; bundled-skill gate; bootstrap workflow (sentinel + `--bootstrap-reviewed`); D_sel floor (>=5 with `--split` override); audit JSONL via `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Surface: dream-cycle phase wrapper; `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`); `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`); MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly, validates `skill_name` kebab-only + confines caller-supplied benchmark/held-out paths to skillsDir for remote callers); Minion `skillopt` handler + `--background` with `allowProtectedSubmit: true`; write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag); held-out real-user test set via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`). `--bootstrap-from-skill` → `runBootstrapFromSkill` in `src/core/skillopt/bootstrap-benchmark.ts`: reads `SKILL.md` directly (no `routing-eval.jsonl`), makes ONE LLM call emitting a full starter benchmark (tasks + rule judges) as JSONL, parsed line-by-line with skip-bad-line salvage and a min-2-valid-checks-per-task drop; provider/transport errors PROPAGATE (not collapsed to `bootstrap_empty`). `--bootstrap-tasks N` (default 15, capped 50); `maxTokens` scales `min(8000, max(4000, N*220))`. The stderr REVIEW line prints the literal `gbrain skillopt --bootstrap-reviewed --split 1:1:1` — load-bearing because the default `4:1:5` split makes a 15-task starter's `D_sel = floor(15/10) = 1`, below the `>=5` floor, so a 15-task benchmark needs `--split 1:1:1`. Both bootstrap generators share `assertBenchmarkAbsent` + `readSkillBodyOrThrow`; `--bootstrap-from-skill` is mutually exclusive with `--bootstrap-from-routing`/`--benchmark`/`--all`/`--target-models`/`--resume`. Generated rule judges are explicitly WEAK DRAFTS to be strengthened during the review gate. The F11 held-out gate is wired: `--held-out ` is parsed and threaded through every caller (CLI main + `--background` `held_out_path` + batch/fleet `heldOutPath` + the `run_skillopt` `held_out_path` param), running at CHECKPOINT ACCEPTANCE so no-mutate/fleet paths can't promote a held-out-failing candidate. `assertBundledMutationHeldOut` in bundled-skill-gate.ts: bundled + `--allow-mutate-bundled` requires a NON-EMPTY held-out (`MIN_HELD_OUT_SIZE = D_SEL_MIN_SIZE` = 5, derived so they can't desync) or hard-refuses (exit 2), for ALL callers (they funnel through `runSkillOpt`); held-out must be task_id-DISJOINT from the benchmark (overlap rejected — can't catch overfitting). `receipt.baseline_sel_score` populated + a real final-test eval (`test_score` + `baseline_test_score`) scoring best + baseline on `split.test`; shared `scoreSkillOnTasks` primitive (validate-gate.ts) backs baseline/final-test/held-out scoring. `--no-mutate` writes proposed.md via `writeProposed` in version-store.ts. `maxRuntimeMin` ENFORCED (wall-clock deadline between steps → `skillopt_runtime_exceeded` → outcome aborted). Three eval-internal ablation opts on `SkillOptOpts` (NOT on CLI): `reflectMode` (`'both'`/`'failure-only'`), `disableValidationGate` (greedy-accept), `optimizerMode` (`'reflect'`/`'one-shot-rewrite'`), recorded in `RunReceipt` + audit `run_start` for replayability; `ROLLOUT_SUCCESS_THRESHOLD = 0.5` named constant for the partition; one-shot fence-strip is anchored (`^```...```$`) so an embedded code sample isn't truncated. Budget no-pricing fix: Claude Haiku 4.5's dateless canonical id `claude-haiku-4-5` is in `src/core/anthropic-pricing.ts` (a `BudgetTracker`-capped run on Haiku otherwise threw `no_pricing` on the FIRST `chat()` of every rollout); `runValidationGate` (validate-gate.ts) scans settled results for `isMustAbortError(error)` (from `worker-pool.ts`; `BUDGET_EXHAUSTED` is in `MUST_ABORT_ERROR_TAGS`) and re-throws so the caller aborts loudly instead of recording a hollow `selScore:0` — ordinary non-abort rollout errors still fail-open to `score:0` (judge-hiccup posture preserved). Pinned by 152 tests across 18 files (foundation + adversarial + v2 surface + E2E PGLite serial), `test/skillopt/bootstrap-from-skill.test.ts` (20 cases), `test/skillopt/rollout.test.ts`, `test/skillopt/validate-gate-abort.test.ts` (3 cases), held-out ENFORCE + one-shot-rewrite unit cases, and e2e (F11 block/allow, bundled no-mutate, runtime deadline, receipt honesty, held-out disjointness, no-DB-pollution). Drives the Track B SkillOpt benchmark suite in the sibling `gbrain-evals` repo. - `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge rejecting ideas with resistance >4.5 "too obvious", stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2`. `judges.ts` exports `runJudge(config, ideas)` + two configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` cognitive_load 0.50 + inversion rule). Calibration cold-start fallback: when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back in `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch); internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. The fire-and-forget IIFE is tracked in a module-scoped `Set>`; `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns the pending count. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then a fallback `process.exit(0)` fires ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive) — closes the PGLite CLI search/query/get-hang class where the IIFE raced disconnect and PGLite's WASM kept Bun's event loop alive. `pages.last_retrieved_at TIMESTAMPTZ NULL` has a full (NOT partial) B-tree index covering both NULL and range branches; full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output via `isLsdOutput()` in `src/core/cycle/transcript-discovery.ts` short-circuiting `isDreamOutput()`. `gbrain eval brainstorm ` is a three-axis conjunctive gate (distance + usefulness + grounding — distance alone is gameable). `gbrain doctor` has a `brainstorm_health` check (migration applied, `search.track_retrieval` setting, calibration cold-start status). `judges.ts` computes the judge token budget via `computeJudgeMaxTokens(ideaCount, modelId)` (named constants `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`; `ANTHROPIC_OUTPUT_CAPS` map: Opus 4.7 32K, Sonnet 4.6 / Haiku 4.5 64K, legacy Claude 3.5 8K) so a large multi-call judge doesn't truncate mid-JSON; with no `modelOverride` the cap routes through the gateway's actual configured chat model via `getChatModel()`. `--save` for both commands persists through the canonical ingestion path: `persistSavedIdea(engine, {slug, content, provenanceVia})` calls `importFromContent({noEmbed:true, sourcePath})` (chunked + tagged + content_hash so search finds it, no embedding cost at save) THEN renders the saved row to disk via the shared `writePageThrough` helper (file rendered FROM the row so the two sinks can't diverge and `gbrain sync` doesn't churn it). `formatSaveOutcome(outcome, ctx)` returns an honest per-branch message (both-sinks, DB-only when no `sync.repo_path`/repo-not-a-dir, DB-saved-but-file-errored, total-failure → loud `save FAILED … NOT persisted` on stderr + nonzero exit) — closes the silent-false-success class where `--save` printed "Saved" unconditionally even when the DB write failed. `buildIdeaSlug(question, label, nonce?)` adds a random nonce suffix (injectable for tests) so two same-day runs sharing the first 60 slug chars don't clobber. `--json` callers stay DB-only. `buildBrainstormFrontmatterObject(result)` in orchestrator.ts returns the object form for `serializeMarkdown` (string `buildBrainstormFrontmatter` untouched). Pinned by `test/last-retrieved.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir asserts search/get/query exit 0 in <15s + daemon-survival), `test/fix-wave-structural.test.ts` (asserts the drain `await` is textually BEFORE `engine.disconnect`), `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm,judges-maxtokens,save}.test.ts`. Open Collider source: `github.com/CL-ML/open-collider`. -- `src/core/write-through.ts` — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` resolves the disk target from the ASSIGNED source's own working tree (`sources.local_path`), re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown`, and writes the `.md` under that tree's root so the brain has a committable artifact that round-trips through `gbrain sync`. A source with its own `local_path` writes there; a source WITHOUT one falls back to the global `sync.repo_path` ONLY when this is the sole source (then that path is unambiguously this source's tree) and otherwise skips with `source_has_no_local_path` rather than leak into a sibling source's git repo. Rendering FROM the row means file and row cannot diverge. ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync`, cleaning up temp on any failure, so a crash or concurrent `gbrain sync`/autopilot walking the live git tree never reads a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_has_no_local_path' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts`. +- `src/core/write-through.ts` — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` resolves the disk target from the ASSIGNED source's own working tree (`sources.local_path`), re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown`, and writes the `.md` under that tree's root so the brain has a committable artifact that round-trips through `gbrain sync`. A source with its own `local_path` writes there; a source WITHOUT one falls back to the global `sync.repo_path` ONLY when this is the sole source (then that path is unambiguously this source's tree) and otherwise skips with `source_has_no_local_path` rather than leak into a sibling source's git repo. Rendering FROM the row means file and row cannot diverge. ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync`, cleaning up temp on any failure, so a crash or concurrent `gbrain sync`/autopilot walking the live git tree never reads a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_has_no_local_path' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. On a durability-hardened repo (`isDurabilityHardened` — the gbrain post-commit hook is installed, i.e. the user ran `gbrain sources harden`), a successful write is best-effort COMMITTED via `commitWriteThroughFile` (path-limited `git commit -- `, never sweeps unrelated edits; the hook then background-pushes) so write-through content reaches git instead of accumulating uncommitted forever (#2426); result carries `committed?: boolean`. Unhardened repos keep write-only behavior. Consumers: `put_page` op and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts` + `test/write-through-commit.serial.test.ts`. - `src/core/model-id.ts` — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface has no parallel re-implementations of `provider:model` splitting — slash-form ids (`anthropic/claude-sonnet-4-6`) classify correctly instead of falling through to "unknown model". Distinct from the gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`, which throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by `test/model-id.test.ts`. - `src/core/ai/model-resolver.ts:parseModelId` — gateway-side resolver accepts both colon and slash form (`provider:model` and `provider/model`) so a slash-form id resolves to the same recipe at every gateway entry point (chat / embed / rerank) instead of throwing `AIConfigError: model id must be in format provider:model`. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. @@ -302,8 +302,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/embedding-dim-check.ts` extension — facts.embedding dim drift surface. `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (migration v40 falls back to `vector` on pgvector < 0.7); regex ordering is halfvec-before-vector (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (NOT bare REINDEX, which doesn't rewrite the index after a column-type change). `assertFactsEmbeddingDimMatchesConfig(engine)` is the preflight — throws `FactsEmbeddingDimMismatchError` (tagged `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool MUST_ABORT semantics) when configured dim ≠ column width; cached per-engine via `WeakMap`; PGLite engines silently skip. Doctor check `facts_embedding_width_consistency` (registered after `embedding_width_consistency`) reuses the same helpers with an identical ALTER recipe. Pinned by `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). Test seam `__resetFactsEmbeddingCastCacheForTest()` clears the per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). -- `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. Below the valve, stale pages are partitioned by git history via exported `listEverCommittedPaths(repoPath)` (one `git log --all --no-renames --diff-filter=A --name-only` pass; null on non-git dirs → unchanged behavior): a stale path that EVER existed in history was genuinely deleted → reconciled; a path with NO history is DB-only write-through (never committed/pushed, e.g. lost to a fresh clone) → the page is KEPT and its markdown re-exported to the working tree via `writePageThrough`, with a stderr hint to commit it (#2426 — "absent from git" is the symptom of the missing write-through commit, not evidence the content is disposable). Pinned by `test/sync-reconcile-db-only.serial.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). +- `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). `collectSyncableFiles`' shared emit filter `isCollectibleForWalker` applies the SAME segment-level `pruneDir` gate as incremental sync's `classifySync` — load-bearing for the `git ls-files` fast path, which enumerates tracked files under dot-dirs/vendored trees that the FS walk never descends into; without it `sync --full` imported (and resurrected soft-deleted) pages incremental sync excludes (#2607). Pinned by `test/import-git-fastpath-prune.test.ts`. - `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. - `src/core/sort-newest-first.ts` — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line here instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (descending order, mixed prefixes, empty, single-element, in-place-mutation contract). - `src/core/cycle.ts` — brain maintenance cycle primitive (9 phases). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantic order: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans. `synthesize` runs after sync (cross-references see a fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default, so extract is the canonical materialization); `recompute_emotional_weight` sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` incrementally, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). `CycleReport.schema_version: "1"` is stable; `totals` is additive (`pages_emotional_weight_recomputed`, `transcripts_processed`, `synth_pages_written`, `patterns_written`). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon inline path, the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `yieldBetweenPhases` runs between phases; `yieldDuringPhase` is in-phase keepalive (synthesize/patterns renew the cycle-lock TTL during long waits). Engine nullable; lock-skip on read-only phase selections. `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal with `checkAborted()` between every phase. `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult` (threaded to `runPhaseExtract` as the 4th arg) and takes `willRunExtractPhase: boolean` setting `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor (not the drift-prone global `config.sync.last_commit`). `CycleOpts.brainDir` is `string | null`; when null (checkout-less postgres/Supabase brain) the 6 filesystem phases (lint/backlinks/sync/synthesize/extract/patterns) skip with `details.reason: 'no_brain_dir'` and the DB-only phases run; `resolveSourceForDir` is null-tolerant. `cycleSourceId = opts.sourceId ?? resolveSourceForDir(engine, brainDir)` is the canonical per-source scope for `extract_facts`/`extract_atoms`/calibration so `gbrain dream --source repo-a` reconciles repo-a's facts even with no checkout (instead of scoping to `'default'` while stamping repo-a fresh). `deriveStatus` counts `edges_resolved`/`edges_ambiguous` as work so an edges-only cycle reports `ok` not `clean`; the `jobs.ts` `autopilot-cycle` + phase-wrapper handlers pass `null` (not `'.'`) when no repo is configured. The cycle is SPLIT for autopilot fan-out (#2194/#2227): `PHASE_SCOPE` partitions `ALL_PHASES` into `GLOBAL_PHASES` (brain-wide: embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile/synthesize_concepts/skillopt) and `NON_GLOBAL_PHASES` (source + mixed). Per-source `autopilot-cycle` jobs run only `NON_GLOBAL_PHASES` and stamp `last_source_cycle_at`; the single `autopilot-global-maintenance` job runs `GLOBAL_PHASES` (no `sourceId`) and stamps the brain-level `autopilot.last_global_at` config key (`LAST_GLOBAL_AT_KEY`). `last_full_cycle_at` is still written alongside `last_source_cycle_at` on a per-source success for doctor/legacy readers (no longer a gate for the brain-wide phases). Pinned by `test/dream-postgres.serial.test.ts` + `test/jobs-autopilot-cycle-braindir.serial.test.ts` + `test/autopilot-global-maintenance.test.ts`. diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 9b2ff4fbc..21eeaaef5 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -188,7 +188,7 @@ export function walkMarkdownFiles(dir: string): { path: string; relPath: string // Pre-fix, this walker had only an ad-hoc dot-prefix exclusion and didn't // call isSyncable at all — so it descended into `node_modules/`, emitted // markdown files from there, AND ignored the canonical exclusion list - // (`.raw/`, `ops/`, README.md, etc.). Now: pruneDir skips entire vendor + // (`.raw/`, README.md, etc.). Now: pruneDir skips entire vendor // subtrees before recursion (saving IO), and isSyncable filters the emit // set against the canonical markdown-strategy rules. const files: { path: string; relPath: string }[] = []; diff --git a/src/commands/import.ts b/src/commands/import.ts index 241bcff28..a4e494f80 100644 --- a/src/commands/import.ts +++ b/src/commands/import.ts @@ -510,10 +510,21 @@ function isCollectibleForWalker( strategy: SyncStrategy, multimodalOn: boolean, ): boolean { + // #2607: apply the SAME segment-level prune gate as incremental sync's + // `classifySync` (core/sync.ts). The FS walk below prunes at descent time, + // but the git fast path enumerates via `git ls-files` and historically + // filtered only by extension — so `sync --full` imported (and resurrected + // previously-deleted) pages under dot-dirs / vendored trees that incremental + // sync excludes. Full and incremental must agree on the exclusion set. + // (In the FS-walk route `path` is a basename, so this is the same dot-file + // check pruneDir already applied there — no behavior change on that route.) + const segments = path.split('/'); + if (segments.some((seg) => !pruneDir(seg))) return false; + // Metafiles are directory scaffolding (READMEs / index / log / schema / // resolver), not typed brain pages — same exclusion `sync`'s `isSyncable` // applies. Guards both the FS-walk and the git-fast-path collection routes. - const basename = path.split('/').pop() || ''; + const basename = segments[segments.length - 1] || ''; if ((SYNC_SKIP_FILES as readonly string[]).includes(basename)) return false; switch (strategy) { diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 9331a9a2f..cafe34604 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -1897,7 +1897,13 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 0) { + // #2426: a stale page whose source_path was NEVER committed to git is + // DB-only write-through (the file was written into the clone but never + // committed/pushed, then lost — e.g. a fresh clone). "Absent from git" + // is the SYMPTOM of that bug, not evidence the content is disposable. + // Keep those pages and re-export their markdown to the working tree so + // they're file-backed again; only pages whose file once existed in git + // history (i.e. was genuinely deleted) are reconcile-deleted. + const everCommitted = listEverCommittedPaths(repoPath); + const pathBySlug = new Map(rows.map(r => [r.slug, r.source_path])); + let deletableSlugs = plan.staleSlugs; + const dbOnlySlugs: string[] = []; + if (everCommitted) { + deletableSlugs = []; + for (const slug of plan.staleSlugs) { + const sp = pathBySlug.get(slug); + if (sp && !everCommitted.has(sp.replace(/\\/g, '/'))) dbOnlySlugs.push(slug); + else deletableSlugs.push(slug); + } + } + if (dbOnlySlugs.length > 0) { + let reExported = 0; + try { + const { writePageThrough } = await import('../core/write-through.ts'); + for (const slug of dbOnlySlugs) { + const r = await writePageThrough(engine, slug, { sourceId: sid }); + if (r.written) reExported++; + } + } catch { /* best-effort — pages are preserved either way */ } + serr( + `\n Kept ${dbOnlySlugs.length} page(s) whose markdown was never committed to git ` + + `(DB-only write-through — not deleting).` + + (reExported > 0 ? ` Re-exported ${reExported} of them to the working tree.` : '') + + `\n Commit + push them (e.g. scripts/brain-commit-push.sh, or 'gbrain sources harden') ` + + `so the next sync sees them as file-backed.`, + ); + } const deleteScopedOpts = { sourceId: sid }; - for (let i = 0; i < plan.staleSlugs.length; i += DELETE_BATCH_SIZE) { - const batch = plan.staleSlugs.slice(i, i + DELETE_BATCH_SIZE); + for (let i = 0; i < deletableSlugs.length; i += DELETE_BATCH_SIZE) { + const batch = deletableSlugs.slice(i, i + DELETE_BATCH_SIZE); try { const deleted = await engine.deletePages(batch, deleteScopedOpts); reconciledDeletes += deleted.length; @@ -3290,6 +3332,34 @@ export function planReconcileDeletes( return { staleSlugs, reconcilableCount: reconcilable.length, massDelete }; } +/** + * #2426: every repo-relative path that ever appeared as an ADD in git history + * (rename detection off, so a `git mv` destination still counts as an add). + * Used by the full-sync reconcile to distinguish "file was committed and later + * deleted" (genuine delete → reconcile) from "file was NEVER committed" + * (DB-only write-through → preserve). Returns null when `repoPath` isn't a git + * work tree or git is unavailable — callers keep the plain-directory behavior. + * Forward-slash-normalized to match `normalizeReconcilePath` membership tests. + */ +export function listEverCommittedPaths(repoPath: string): Set | null { + let stdout: string; + try { + stdout = execFileSync( + 'git', + ['-C', repoPath, '-c', 'core.quotepath=off', 'log', '--all', '--no-renames', + '--diff-filter=A', '--format=', '--name-only'], + { encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] }, + ); + } catch { + return null; + } + const set = new Set(); + for (const line of stdout.split('\n')) { + if (line) set.add(line.replace(/\\/g, '/')); + } + return set; +} + /** * #2828 escape hatch: `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the pre-valve * behavior for the rare intentional bulk removal. Env-only (an incident-time diff --git a/src/core/brain-repo-durability.ts b/src/core/brain-repo-durability.ts index d4b77c09f..7fa1685aa 100644 --- a/src/core/brain-repo-durability.ts +++ b/src/core/brain-repo-durability.ts @@ -183,15 +183,17 @@ if [ "\${1:-}" = "--push-only" ]; then fi _msg="\${1:?usage: brain-commit-push.sh [paths...]}"; shift || true -# Pull first so the local tree is current before we stage. -git fetch origin >/dev/null 2>&1 || true -git pull --rebase origin "$_branch" || { git rebase --abort >/dev/null 2>&1 || true; echo "rebase conflict: manual attention needed" >&2; exit 3; } # EXPLICIT paths only — never a blind 'git add -A' (would risk committing # secrets, temp files, or unrelated edits). if [ "$#" -eq 0 ]; then echo "refusing blind 'git add -A' — pass explicit path(s) to commit" >&2; exit 2 fi +# COMMIT BEFORE PULL (#2426): the old order (fetch + pull --rebase, THEN stage) +# aborted on any dirty tree — 'cannot pull with rebase: You have unstaged +# changes' — so the helper could never commit a MODIFIED page (exactly the +# write-through case). Stage + commit first; brain_push below already handles +# a remote that advanced (push -> rejected -> pull --rebase -> push). git add -- "$@" if git diff --cached --quiet; then echo "nothing to commit"; exit 0; fi git commit -m "$_msg" @@ -337,6 +339,47 @@ function uninstallLocalHook(repoPath: string): boolean { return true; } +/** + * True when the gbrain durability post-commit hook is installed — i.e. the + * user opted this repo into push-durability via `gbrain sources harden`. + * Cheap (one git-config read + one file read); used as the gate for + * write-through auto-commit (#2426). + */ +export function isDurabilityHardened(repoPath: string): boolean { + try { + const { dir } = resolveHooksDir(repoPath); + const hookPath = join(dir, 'post-commit'); + return existsSync(hookPath) && readFileSync(hookPath, 'utf-8').includes(HOOK_BANNER); + } catch { + return false; + } +} + +/** + * #2426: best-effort commit of a single write-through artifact so DB writes + * reach git (the post-commit hook then background-pushes). Pre-fix, + * write-through `.md` accumulated uncommitted forever: it never reached the + * remote, froze `last_sync_at` (HEAD never moved), and a later `sync --full` + * delete-reconcile treated the never-committed pages as disposable. + * + * Path-limited (`git commit -- `) so unrelated staged/dirty edits are + * never swept into the commit. Never throws; returns false on any failure + * (index.lock contention, nothing changed, detached states) — the DB row and + * the on-disk file remain the durable sinks either way. + */ +export function commitWriteThroughFile(repoPath: string, absPath: string, slug: string): boolean { + try { + const rel = relative(repoPath, absPath); + if (!rel || rel.startsWith('..') || isAbsolute(rel)) return false; + const gitOpts = { stdio: 'ignore', timeout: 30_000, env: { ...process.env, ...GIT_ENV } } as const; + execFileSync('git', ['-C', repoPath, 'add', '--', rel], gitOpts); + execFileSync('git', ['-C', repoPath, 'commit', '-m', `gbrain: write-through ${slug}`, '--', rel], gitOpts); + return true; + } catch { + return false; + } +} + // ── Committed helper ──────────────────────────────────────────────────────── function installHelper(repoPath: string, dryRun: boolean): { status: StepStatus; detail: string } { diff --git a/src/core/sync.ts b/src/core/sync.ts index a0d4856f0..3bbf4633d 100644 --- a/src/core/sync.ts +++ b/src/core/sync.ts @@ -255,7 +255,12 @@ const PRUNE_DIR_NAMES = new Set([ // with the first-sync walker in commands/import.ts. 'venv', '.raw', - 'ops', + // NOTE (#2404): `'ops'` used to be in this list (a v0.2.0-era carve-out for + // one brain layout). Matching the bare segment pruned EVERY user `ops/` + // directory at any depth — sync silently deleted `ops/*` pages and never + // imported `ops/*` files, while the bundled daily-task-manager skill + // prescribes `ops/tasks` as its canonical page. `ops/` is ordinary content; + // do NOT re-add it. Only generated/vendored trees belong here. ]); /** @@ -352,8 +357,8 @@ function classifySync(path: string, opts: SyncableOptions = {}): SyncableReason if (!isAllowedByStrategy(path, strategy)) return 'strategy'; // Skip every path segment that pruneDir would block walkers from descending - // into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars, - // `node_modules/` (latent bug fix), and `ops/` at any depth. + // into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars, and + // vendor/generated trees (`node_modules/`, `vendor/`, …) at any depth. const segments = path.split('/'); if (segments.some(p => !pruneDir(p))) return 'pruned-dir'; diff --git a/src/core/write-through.ts b/src/core/write-through.ts index 5ca2b565c..02f792a3f 100644 --- a/src/core/write-through.ts +++ b/src/core/write-through.ts @@ -27,6 +27,7 @@ import { randomBytes } from 'crypto'; import type { BrainEngine } from './engine.ts'; import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts'; import { isWriteTargetContained } from './path-confine.ts'; +import { isDurabilityHardened, commitWriteThroughFile } from './brain-repo-durability.ts'; /** Minimal logger surface — structurally compatible with operations.ts `Logger`. */ export interface WriteThroughLogger { @@ -36,6 +37,13 @@ export interface WriteThroughLogger { export interface WriteThroughResult { written: boolean; path?: string; + /** + * True when the write was also committed to git (#2426). Only attempted on + * repos hardened via `gbrain sources harden` (durability hook installed); + * the hook then background-pushes the commit. Best-effort — a false/absent + * value never blocks the write. + */ + committed?: boolean; /** * Non-error reasons the file was not written: * - no_repo_configured: the resolved target (source `local_path` or, for a @@ -157,7 +165,20 @@ export async function writePageThrough( throw writeErr; } - return { written: true, path: filePath }; + // #2426: on a durability-hardened repo (user ran `gbrain sources harden`), + // commit the artifact so it reaches git — pre-fix, write-through content + // stayed uncommitted forever: never pushed, `last_sync_at` frozen, and + // silently deleted by a later `sync --full` delete-reconcile. The local + // post-commit hook background-pushes the commit. Best-effort: a commit + // failure never fails the write (the DB row + file are the durable sinks). + let committed = false; + try { + if (isDurabilityHardened(writeRoot)) { + committed = commitWriteThroughFile(writeRoot, filePath, slug); + } + } catch { /* best-effort */ } + + return { written: true, path: filePath, ...(committed ? { committed } : {}) }; } catch (e) { const msg = e instanceof Error ? e.message : String(e); opts.logger?.warn(`[write-through] failed for ${slug}: ${msg}`); diff --git a/test/brain-durability-hook.serial.test.ts b/test/brain-durability-hook.serial.test.ts index b676b5eff..38aff4df9 100644 --- a/test/brain-durability-hook.serial.test.ts +++ b/test/brain-durability-hook.serial.test.ts @@ -90,6 +90,36 @@ describe('brain-commit-push.sh (D13 guarantee)', () => { } catch (e: any) { code = e.status ?? 1; } expect(code).toBe(2); }); + + test('#2426 — commits a MODIFIED tracked file even when the remote advanced (commit before pull)', () => { + // Pre-fix, the helper ran `git pull --rebase` BEFORE staging, so any dirty + // tree (a modified/enriched page — exactly the write-through case) aborted + // with 'cannot pull with rebase: You have unstaged changes' (exit 3). The + // helper could only ever commit untracked-NEW files, never modifications. + // Remove the post-commit hook so its background push can't race the + // helper's own push (macOS has no flock to serialize them) — this test + // targets the HELPER's ordering; hook behavior is covered below. + rmSync(join(work, '.git', 'hooks', 'post-commit')); + // Advance the remote from a second clone so a pull is genuinely needed. + const other = mkdtempSync(join(root, 'other-')); + execFileSync('git', ['-c', 'protocol.file.allow=always', 'clone', '-q', bare, other], { stdio: 'ignore' }); + git(other, 'config', 'user.email', 'o@o.o'); git(other, 'config', 'user.name', 'other'); + writeFileSync(join(other, 'remote.md'), 'from other\n'); + git(other, 'add', 'remote.md'); git(other, 'commit', '-qm', 'remote change'); git(other, 'push', '-q', 'origin', 'main'); + + // Dirty MODIFICATION of a tracked file in the hardened clone (write-through shape). + writeFileSync(join(work, 'README.md'), 'modified by write-through\n'); + execFileSync('bash', [join(work, 'scripts', 'brain-commit-push.sh'), 'wt: README', 'README.md'], { + cwd: work, stdio: ['ignore', 'pipe', 'pipe'], env: process.env, + }); + + // Both the remote's commit and ours are on origin/main. + const subjects = git(bare, 'log', '--format=%s', 'main'); + expect(subjects).toContain('wt: README'); + expect(subjects).toContain('remote change'); + // Working tree is clean — the modification was committed, not stranded. + expect(git(work, 'status', '--porcelain', 'README.md')).toBe(''); + }); }); describe('post-commit hook (D9 local, D7 self-contained)', () => { diff --git a/test/brain-writer-walk-prune.test.ts b/test/brain-writer-walk-prune.test.ts index 1cd3540a0..9b455ca25 100644 --- a/test/brain-writer-walk-prune.test.ts +++ b/test/brain-writer-walk-prune.test.ts @@ -43,8 +43,10 @@ beforeAll(() => { writeFileSync(join(root, '.obsidian', 'workspace.json'), '{}'); mkdirSync(join(root, 'people', 'pedro.raw'), { recursive: true }); writeFileSync(join(root, 'people', 'pedro.raw', 'source.md'), '---\ntitle: should not visit\n---\n'); + // ops/ is ORDINARY content (#2404) — walker MUST descend (it used to be + // wrongly pruned, silently excluding user runbooks / ops/tasks). mkdirSync(join(root, 'ops', 'logs'), { recursive: true }); - writeFileSync(join(root, 'ops', 'logs', 'run.md'), '# nope\n'); + writeFileSync(join(root, 'ops', 'logs', 'run.md'), '---\ntitle: Run\n---\n\nbody\n'); // Nested node_modules — must also be pruned, not just at the root. mkdirSync(join(root, 'people', 'tools', 'node_modules', 'inner'), { recursive: true }); writeFileSync(join(root, 'people', 'tools', 'node_modules', 'inner', 'a.md'), '---\ntitle: nope\n---\n'); @@ -95,8 +97,11 @@ describe('walkDir (brain-writer.ts) — descent-time pruning', () => { walkDir(root, (f) => { files.push(f); }, (dir) => visited.push(dir)); expect(visited.some(d => d.endsWith('/people'))).toBe(true); expect(visited.some(d => d.endsWith('/concepts/subdir'))).toBe(true); + // ops/ is ordinary content — descended, not pruned (#2404). + expect(visited.some(d => d.endsWith('/ops/logs'))).toBe(true); expect(files.some(f => f.endsWith('/people/alice.md'))).toBe(true); expect(files.some(f => f.endsWith('/concepts/subdir/thing.md'))).toBe(true); + expect(files.some(f => f.endsWith('/ops/logs/run.md'))).toBe(true); // And explicitly does NOT visit the file under node_modules. expect(files.some(f => f.includes('/node_modules/'))).toBe(false); }); @@ -107,7 +112,7 @@ describe('walkDir (brain-writer.ts) — descent-time pruning', () => { // visitDir would be called with node_modules paths. const descents: string[] = []; walkDir(root, () => {}, (d) => descents.push(d)); - const vendor = descents.filter(d => /\/(node_modules|\.git|\.obsidian|ops)(\/|$)/.test(d) || /\.raw$/.test(d)); + const vendor = descents.filter(d => /\/(node_modules|\.git|\.obsidian)(\/|$)/.test(d) || /\.raw$/.test(d)); expect(vendor).toEqual([]); }); }); @@ -119,13 +124,20 @@ describe('collectFiles (frontmatter.ts) — descent-time pruning parity', () => expect(visited.some(d => d.includes('/node_modules'))).toBe(false); }); - test('does NOT descend into .git, .obsidian, *.raw, or ops', () => { + test('does NOT descend into .git, .obsidian, or *.raw', () => { const visited: string[] = []; collectFiles(root, (dir) => visited.push(dir)); expect(visited.some(d => d.includes('/.git'))).toBe(false); expect(visited.some(d => d.includes('/.obsidian'))).toBe(false); expect(visited.some(d => d.endsWith('.raw'))).toBe(false); - expect(visited.some(d => d.endsWith('/ops') || d.includes('/ops/'))).toBe(false); + }); + + test('DOES descend into ops/ — ordinary content, not a vendor tree (#2404)', () => { + const visited: string[] = []; + collectFiles(root, (dir) => visited.push(dir)); + expect(visited.some(d => d.endsWith('/ops') || d.includes('/ops/'))).toBe(true); + const files = collectFiles(root); + expect(files.some(f => f.endsWith('/ops/logs/run.md'))).toBe(true); }); test('does NOT descend into git submodule directories', () => { diff --git a/test/e2e/sync.test.ts b/test/e2e/sync.test.ts index e8cc62631..3f02767c2 100644 --- a/test/e2e/sync.test.ts +++ b/test/e2e/sync.test.ts @@ -224,7 +224,7 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => { expect(bob).toBeNull(); }); - test('sync skips non-syncable files (README, hidden, .raw)', async () => { + test('sync skips non-syncable files (README, hidden, .raw) but imports ops/ (#2404)', async () => { const { performSync } = await import('../../src/commands/sync.ts'); const engine = getEngine(); @@ -249,8 +249,9 @@ describeE2E('E2E: Git-to-DB Sync Pipeline', () => { const raw = await engine.getPage('.raw/data'); expect(raw).toBeNull(); + // ops/ is ordinary content and DOES sync (#2404). const ops = await engine.getPage('ops/deploy'); - expect(ops).toBeNull(); + expect(ops).not.toBeNull(); }); test('sync stores last_commit and last_run in config', async () => { diff --git a/test/import-git-fastpath-prune.test.ts b/test/import-git-fastpath-prune.test.ts new file mode 100644 index 000000000..f591646e2 --- /dev/null +++ b/test/import-git-fastpath-prune.test.ts @@ -0,0 +1,90 @@ +/** + * #2607 — the `sync --full` git fast path applies the same prune gate as + * incremental sync. + * + * Bug class: `collectSyncableFiles` on a git work tree takes the + * `git ls-files` fast path, which historically filtered ONLY by + * strategy/extension + .gitignore — no `pruneDir`, so `sync --full` + * imported (and resurrected previously-soft-deleted) pages under dot-dirs + * and vendored trees that incremental sync's `isSyncable` excludes. The two + * enumeration modes cycled content in and out depending on which ran last. + * + * Fix: `isCollectibleForWalker` (shared by the git fast path AND the FS-walk + * emit filter) now rejects any path with a segment `pruneDir` would block — + * the same segment rule `classifySync` applies on the incremental path. + * + * No PGLite needed: `collectSyncableFiles` is pure filesystem + git. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join, relative } from 'path'; +import { collectSyncableFiles } from '../src/commands/import.ts'; +import { isSyncable } from '../src/core/sync.ts'; + +let repo: string; + +function rel(files: string[]): string[] { + return files.map((f) => relative(repo, f)); +} + +beforeAll(() => { + repo = mkdtempSync(join(tmpdir(), 'gbrain-fastpath-')); + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' }); + + // Ordinary content — must be collected. + mkdirSync(join(repo, 'notes'), { recursive: true }); + writeFileSync(join(repo, 'notes/real.md'), '---\ntitle: Real\n---\nbody\n'); + mkdirSync(join(repo, 'ops'), { recursive: true }); + writeFileSync(join(repo, 'ops/tasks.md'), '---\ntitle: Tasks\n---\nbody\n'); + + // TRACKED files under excluded trees — `git ls-files` returns these, so + // only the prune gate keeps them out (this is the #2607 divergence). + mkdirSync(join(repo, '.obsidian'), { recursive: true }); + writeFileSync(join(repo, '.obsidian/plugin-notes.md'), 'not a page\n'); + mkdirSync(join(repo, 'vendor/pkg'), { recursive: true }); + writeFileSync(join(repo, 'vendor/pkg/notes.md'), 'vendored\n'); + mkdirSync(join(repo, 'node_modules/dep'), { recursive: true }); + writeFileSync(join(repo, 'node_modules/dep/CHANGELOG.md'), 'dep changelog\n'); + mkdirSync(join(repo, 'people/pedro.raw'), { recursive: true }); + writeFileSync(join(repo, 'people/pedro.raw/source.md'), 'raw sidecar\n'); + + // Metafiles — excluded on both routes (pre-existing #345 behavior). + writeFileSync(join(repo, 'README.md'), '# repo\n'); + writeFileSync(join(repo, 'notes/index.md'), '# index\n'); + + execSync('git add -A -f && git commit -m "fixture"', { cwd: repo, stdio: 'pipe' }); +}); + +afterAll(() => { + if (repo) rmSync(repo, { recursive: true, force: true }); +}); + +describe('#2607 — git fast path excludes what incremental sync excludes', () => { + test('tracked files under pruned dirs are NOT collected', () => { + const files = rel(collectSyncableFiles(repo, { strategy: 'markdown' })); + expect(files).toContain('notes/real.md'); + expect(files).toContain('ops/tasks.md'); // ordinary content (#2404) + expect(files).not.toContain('.obsidian/plugin-notes.md'); + expect(files).not.toContain('vendor/pkg/notes.md'); + expect(files).not.toContain('node_modules/dep/CHANGELOG.md'); + expect(files).not.toContain('people/pedro.raw/source.md'); + // Metafiles stay excluded too. + expect(files).not.toContain('README.md'); + expect(files).not.toContain('notes/index.md'); + }); + + test('full-sync enumeration agrees with incremental isSyncable for every collected file', () => { + // The single-source-of-truth contract: nothing the full path collects may + // be something the incremental path would refuse to sync. + const files = rel(collectSyncableFiles(repo, { strategy: 'markdown' })); + for (const f of files) { + expect({ path: f, syncable: isSyncable(f) }).toEqual({ path: f, syncable: true }); + } + expect(files.length).toBeGreaterThan(0); + }); +}); diff --git a/test/sync-isSyncable-shape.test.ts b/test/sync-isSyncable-shape.test.ts index 44b29c2c8..7c715e590 100644 --- a/test/sync-isSyncable-shape.test.ts +++ b/test/sync-isSyncable-shape.test.ts @@ -28,7 +28,8 @@ describe('#1433 — isSyncable / unsyncableReason are duals of one classifier', { path: 'RESOLVER.md', expected: 'metafile', note: 'top-level master routing config (closes #345)' }, { path: 'brain/RESOLVER.md', expected: 'metafile', note: 'RESOLVER.md anywhere is metafile (closes #345)' }, { path: 'people/alice.txt', expected: 'strategy', note: '.txt rejected by markdown strategy' }, - { path: 'ops/scratch/note.md', expected: 'pruned-dir', note: 'ops/ is pruned' }, + { path: 'ops/scratch/note.md', expected: null, note: 'ops/ is ordinary content, not pruned (#2404)' }, + { path: 'vendor/pkg/note.md', expected: 'pruned-dir', note: 'vendor/ is pruned' }, { path: '.git/notes.md', expected: 'pruned-dir', note: 'hidden dir pruned' }, { path: 'node_modules/foo/README.md', expected: 'pruned-dir', note: 'node_modules pruned' }, ]; diff --git a/test/sync-ops-pages.serial.test.ts b/test/sync-ops-pages.serial.test.ts new file mode 100644 index 000000000..cda7bc12b --- /dev/null +++ b/test/sync-ops-pages.serial.test.ts @@ -0,0 +1,129 @@ +/** + * #2404 — `ops/` is ordinary content: sync imports `ops/*` files and never + * deletes `ops/*` pages. + * + * Bug class: `'ops'` was hardcoded in PRUNE_DIR_NAMES (a v0.2.0-era carve-out), + * so `classifySync` treated ANY path with an `ops` segment as 'pruned-dir': + * - committed `ops/*.md` files were never imported (even by `sync --full`); + * - a modified `ops/*` file fell into the unsyncableModified delete loop, + * whose #1433 guard only skipped 'metafile' — so put-created `ops/*` pages + * (e.g. the bundled daily-task-manager's canonical `ops/tasks`) were + * silently deleted on every sync. + * + * Fix: remove `'ops'` from PRUNE_DIR_NAMES, and harden the delete loop to also + * skip 'pruned-dir' classifications (a page under a genuinely-pruned dir can + * only exist via a deliberate put_page — never delete it on a file edit). + * + * Modeled on test/sync-metafile-skip.serial.test.ts (the #1433 iron rule). + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; + +let engine: PGLiteEngine; +let repoPath: string; + +function gitInit(repo: string): void { + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "test@test.com"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "Test"', { cwd: repo, stdio: 'pipe' }); +} + +describe('#2404 — ops/ pages sync like any other content', () => { + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }, 60_000); + + afterAll(async () => { + if (engine) await engine.disconnect(); + }, 60_000); + + beforeEach(async () => { + await resetPgliteState(engine); + repoPath = mkdtempSync(join(tmpdir(), 'gbrain-ops-')); + gitInit(repoPath); + mkdirSync(join(repoPath, 'topics'), { recursive: true }); + writeFileSync(join(repoPath, 'topics/foo.md'), [ + '---', 'type: concept', 'title: Foo', '---', '', 'Baseline content.', + ].join('\n')); + execSync('git add -A && git commit -m "initial"', { cwd: repoPath, stdio: 'pipe' }); + }); + + afterEach(() => { + if (repoPath) rmSync(repoPath, { recursive: true, force: true }); + }); + + test('a committed ops/*.md file is imported by sync', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + mkdirSync(join(repoPath, 'ops'), { recursive: true }); + writeFileSync(join(repoPath, 'ops/tasks.md'), [ + '---', 'type: concept', 'title: Tasks', '---', '', 'Open tasks live here.', + ].join('\n')); + execSync('git add -A && git commit -m "add ops/tasks"', { cwd: repoPath, stdio: 'pipe' }); + + const result = await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true }); + expect(['first_sync', 'synced']).toContain(result.status); + + // Pre-fix: ops/* was 'pruned-dir' → imported=0 for it, even on --full. + const page = await engine.getPage('ops/tasks'); + expect(page).not.toBeNull(); + expect(page?.compiled_truth).toContain('Open tasks'); + }, 60_000); + + test('an edited ops/*.md updates its page instead of deleting it', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + mkdirSync(join(repoPath, 'ops'), { recursive: true }); + writeFileSync(join(repoPath, 'ops/tasks.md'), [ + '---', 'type: concept', 'title: Tasks', '---', '', 'v1', + ].join('\n')); + execSync('git add -A && git commit -m "add ops/tasks"', { cwd: repoPath, stdio: 'pipe' }); + await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true }); + + writeFileSync(join(repoPath, 'ops/tasks.md'), [ + '---', 'type: concept', 'title: Tasks', '---', '', 'v2 with a new task', + ].join('\n')); + execSync('git add -A && git commit -m "edit ops/tasks"', { cwd: repoPath, stdio: 'pipe' }); + + // Pre-fix: this incremental sync hit the unsyncableModified delete loop + // ("Deleted un-syncable page: ops/tasks" — the autopilot kill-loop). + const second = await performSync(engine, { repoPath, noPull: true, noEmbed: true }); + expect(['synced', 'up_to_date', 'first_sync']).toContain(second.status); + + const page = await engine.getPage('ops/tasks'); + expect(page).not.toBeNull(); + expect(page?.compiled_truth).toContain('v2'); + }, 60_000); + + test('hardening: a put-created page under a STILL-pruned dir survives a file edit (pruned-dir delete guard)', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + // node_modules stays in PRUNE_DIR_NAMES. Commit a file there, then seed a + // same-path page via putPage — the deliberate-put precondition. + mkdirSync(join(repoPath, 'node_modules/pkg'), { recursive: true }); + writeFileSync(join(repoPath, 'node_modules/pkg/notes.md'), 'v1\n'); + execSync('git add -A -f && git commit -m "vendor file"', { cwd: repoPath, stdio: 'pipe' }); + await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true }); + + await engine.putPage('node_modules/pkg/notes', { + type: 'concept', + title: 'Deliberate put page', + compiled_truth: 'Created via put_page; must survive sync.', + timeline: '', + frontmatter: { type: 'concept' }, + }); + + writeFileSync(join(repoPath, 'node_modules/pkg/notes.md'), 'v2\n'); + execSync('git add -A -f && git commit -m "edit vendor file"', { cwd: repoPath, stdio: 'pipe' }); + await performSync(engine, { repoPath, noPull: true, noEmbed: true }); + + // Pre-fix: reason 'pruned-dir' was not guarded → page deleted. + const survivor = await engine.getPage('node_modules/pkg/notes'); + expect(survivor).not.toBeNull(); + }, 60_000); +}); diff --git a/test/sync-reconcile-db-only.serial.test.ts b/test/sync-reconcile-db-only.serial.test.ts new file mode 100644 index 000000000..a531a7dca --- /dev/null +++ b/test/sync-reconcile-db-only.serial.test.ts @@ -0,0 +1,142 @@ +/** + * #2426 (bug 3) — `sync --full` delete-reconcile preserves DB-only pages. + * + * Bug class: the full-sync reconcile soft-deleted ANY file-backed page whose + * `source_path` was absent from the working tree — including pages whose + * markdown was NEVER committed to git (write-through that never made it to + * the remote, then a fresh clone). "Absent from git" is the SYMPTOM of the + * missing write-through commit, not evidence the content is disposable; one + * production pass soft-deleted thousands of genuine pages this way. + * + * Fix: the reconcile partitions stale pages by git history — a path that ever + * appeared as an ADD was genuinely deleted (reconcile as before); a path with + * NO history is DB-only write-through: keep the page and re-export its + * markdown to the working tree so it's file-backed again. + * + * Builds on the #2828 mass-delete valve (this guard covers the below-valve + * cases the ratio check can't see). + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync, existsSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { listEverCommittedPaths } from '../src/commands/sync.ts'; + +let engine: PGLiteEngine; +let repoPath: string; + +function gitInit(repo: string): void { + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' }); +} + +describe('listEverCommittedPaths (#2426)', () => { + test('returns every path ever added, including later-deleted ones; null for non-git dirs', () => { + const repo = mkdtempSync(join(tmpdir(), 'gbrain-ecp-')); + try { + gitInit(repo); + writeFileSync(join(repo, 'kept.md'), 'kept\n'); + writeFileSync(join(repo, 'gone.md'), 'gone\n'); + execSync('git add -A && git commit -m add', { cwd: repo, stdio: 'pipe' }); + execSync('git rm -q gone.md && git commit -m rm', { cwd: repo, stdio: 'pipe' }); + + const set = listEverCommittedPaths(repo); + expect(set).not.toBeNull(); + expect(set!.has('kept.md')).toBe(true); + expect(set!.has('gone.md')).toBe(true); // deleted, but WAS committed + expect(set!.has('never-committed.md')).toBe(false); + + const plain = mkdtempSync(join(tmpdir(), 'gbrain-ecp-plain-')); + try { + expect(listEverCommittedPaths(plain)).toBeNull(); + } finally { + rmSync(plain, { recursive: true, force: true }); + } + } finally { + rmSync(repo, { recursive: true, force: true }); + } + }); +}); + +describe('#2426 — full-sync reconcile keeps never-committed (DB-only) pages', () => { + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }, 60_000); + + afterAll(async () => { + if (engine) await engine.disconnect(); + }, 60_000); + + beforeEach(async () => { + await resetPgliteState(engine); + repoPath = mkdtempSync(join(tmpdir(), 'gbrain-dbonly-')); + gitInit(repoPath); + mkdirSync(join(repoPath, 'topics'), { recursive: true }); + writeFileSync(join(repoPath, 'topics/keep.md'), [ + '---', 'type: concept', 'title: Keep', '---', '', 'still here', + ].join('\n')); + writeFileSync(join(repoPath, 'topics/gone.md'), [ + '---', 'type: concept', 'title: Gone', '---', '', 'will be git-rm-ed', + ].join('\n')); + execSync('git add -A && git commit -m initial', { cwd: repoPath, stdio: 'pipe' }); + }); + + afterEach(() => { + if (repoPath) rmSync(repoPath, { recursive: true, force: true }); + }); + + test('genuinely-deleted pages reconcile; never-committed pages are kept and re-exported', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + + // Full sync #1: both file-backed pages land. + const first = await performSync(engine, { + repoPath, full: true, sourceId: 'default', noPull: true, noEmbed: true, + }); + expect(['first_sync', 'synced']).toContain(first.status); + expect(await engine.getPage('topics/keep')).not.toBeNull(); + expect(await engine.getPage('topics/gone')).not.toBeNull(); + + // A DB-only write-through casualty: the page row exists with a + // source_path, but its file was never committed and is absent from the + // clone (e.g. write-through was never pushed, then the repo was re-cloned). + await engine.putPage('memories/lost', { + type: 'concept', + title: 'Lost write-through', + compiled_truth: 'Years of content that must not be reconciled away.', + timeline: '', + frontmatter: { type: 'concept' }, + }); + await engine.executeRaw( + `UPDATE pages SET source_path = $1 WHERE slug = $2 AND source_id = $3`, + ['memories/lost.md', 'memories/lost', 'default'], + ); + + // A genuine deletion: topics/gone.md removed via git. + execSync('git rm -q topics/gone.md && git commit -m "rm gone"', { cwd: repoPath, stdio: 'pipe' }); + await engine.setConfig('sync.repo_path', repoPath); + + // Full sync #2 runs the delete-reconcile. + const second = await performSync(engine, { + repoPath, full: true, sourceId: 'default', noPull: true, noEmbed: true, + }); + expect(['first_sync', 'synced']).toContain(second.status); + + // The genuinely-deleted page is reconciled away… + expect(await engine.getPage('topics/gone')).toBeNull(); + // …the still-present page survives… + expect(await engine.getPage('topics/keep')).not.toBeNull(); + // …and the DB-only page is PRESERVED (pre-fix: soft-deleted here)… + const lost = await engine.getPage('memories/lost'); + expect(lost).not.toBeNull(); + expect(lost?.compiled_truth).toContain('must not be reconciled'); + // …and re-exported to the working tree so it is file-backed again. + expect(existsSync(join(repoPath, 'memories/lost.md'))).toBe(true); + }, 120_000); +}); diff --git a/test/sync-strategy.test.ts b/test/sync-strategy.test.ts index fb2fc893b..37477ea16 100644 --- a/test/sync-strategy.test.ts +++ b/test/sync-strategy.test.ts @@ -56,8 +56,10 @@ describe('isSyncable with strategy', () => { expect(isSyncable('.git/config.js', { strategy: 'code' })).toBe(false); // README.md is skipped under markdown expect(isSyncable('README.md', { strategy: 'markdown' })).toBe(false); - // ops/ directory always skipped - expect(isSyncable('ops/migrate.py', { strategy: 'code' })).toBe(false); + // ops/ is ordinary content — NOT skipped (#2404) + expect(isSyncable('ops/migrate.py', { strategy: 'code' })).toBe(true); + // vendored trees always skipped + expect(isSyncable('vendor/pkg/migrate.py', { strategy: 'code' })).toBe(false); // .raw/ sidecar always skipped expect(isSyncable('dir/.raw/code.ts', { strategy: 'code' })).toBe(false); }); diff --git a/test/sync.test.ts b/test/sync.test.ts index ae7901585..204408f02 100644 --- a/test/sync.test.ts +++ b/test/sync.test.ts @@ -95,9 +95,10 @@ describe('isSyncable', () => { expect(isSyncable('people/README.md')).toBe(false); }); - test('rejects ops/ directory', () => { - expect(isSyncable('ops/deploy-log.md')).toBe(false); - expect(isSyncable('ops/config.md')).toBe(false); + test('accepts ops/ — ordinary content directory, not pruned (#2404)', () => { + expect(isSyncable('ops/deploy-log.md')).toBe(true); + expect(isSyncable('ops/config.md')).toBe(true); + expect(isSyncable('ops/tasks.md')).toBe(true); }); // ──────────────────────────────────────────────────────────────── @@ -128,8 +129,15 @@ describe('pruneDir', () => { expect(pruneDir('.vscode')).toBe(false); }); - test('blocks ops (gbrain operational dir)', () => { - expect(pruneDir('ops')).toBe(false); + test('allows ops — ordinary content dir, not a vendor tree (#2404)', () => { + expect(pruneDir('ops')).toBe(true); + }); + + test('blocks vendored / generated trees', () => { + expect(pruneDir('vendor')).toBe(false); + expect(pruneDir('dist')).toBe(false); + expect(pruneDir('build')).toBe(false); + expect(pruneDir('venv')).toBe(false); }); test('blocks *.raw sidecar dirs (gbrain convention)', () => { diff --git a/test/write-through-commit.serial.test.ts b/test/write-through-commit.serial.test.ts new file mode 100644 index 000000000..23b985639 --- /dev/null +++ b/test/write-through-commit.serial.test.ts @@ -0,0 +1,115 @@ +/** + * #2426 (bug 1) — write-through reaches git on durability-hardened repos. + * + * Bug class: `put_page` / capture / enrichment wrote `.md` into + * `sync.repo_path` but NOTHING ever committed it. The post-commit hook only + * fires after a commit — and write-through never made one — so write-through + * content accumulated uncommitted forever: never pushed, `last_sync_at` + * frozen (HEAD never moved), and silently deleted by a later `sync --full` + * delete-reconcile. + * + * Fix: `writePageThrough` best-effort commits the artifact (path-limited) + * when the repo carries the gbrain durability post-commit hook (i.e. the + * user opted in via `gbrain sources harden`); the hook then background-pushes. + * Unhardened repos keep the old write-only behavior. + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync, chmodSync } from 'fs'; +import { execSync, execFileSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { writePageThrough } from '../src/core/write-through.ts'; + +let engine: PGLiteEngine; +let repo: string; + +function git(cwd: string, ...args: string[]): string { + return execFileSync('git', ['-C', cwd, ...args], { + stdio: ['ignore', 'pipe', 'pipe'], encoding: 'utf-8', + }).trim(); +} + +/** Install a hook file carrying the gbrain durability banner (the detection + * key `isDurabilityHardened` looks for) with a no-op body so tests never + * attempt a real push. */ +function installFakeDurabilityHook(repoPath: string): void { + const hooksDir = join(repoPath, '.git', 'hooks'); + mkdirSync(hooksDir, { recursive: true }); + const hookPath = join(hooksDir, 'post-commit'); + writeFileSync(hookPath, [ + '#!/usr/bin/env bash', + '# gbrain brain-durability post-commit hook (v0.42.44+)', + 'exit 0', + '', + ].join('\n')); + chmodSync(hookPath, 0o755); +} + +async function seedPage(slug: string): Promise { + await engine.putPage(slug, { + type: 'concept', + title: 'Write-through page', + compiled_truth: 'Content that must reach git.', + timeline: '', + frontmatter: { type: 'concept' }, + }); +} + +describe('#2426 — writePageThrough auto-commit', () => { + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }, 60_000); + + afterAll(async () => { + if (engine) await engine.disconnect(); + }, 60_000); + + beforeEach(async () => { + await resetPgliteState(engine); + repo = mkdtempSync(join(tmpdir(), 'gbrain-wt-')); + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "t@t.t"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' }); + writeFileSync(join(repo, 'seed.md'), 'seed\n'); + execSync('git add -A && git commit -m init', { cwd: repo, stdio: 'pipe' }); + await engine.setConfig('sync.repo_path', repo); + }); + + afterEach(() => { + if (repo) rmSync(repo, { recursive: true, force: true }); + }); + + test('on a hardened repo, the write-through artifact is committed (path-limited)', async () => { + installFakeDurabilityHook(repo); + // Unrelated dirty edit — must NOT be swept into the write-through commit. + writeFileSync(join(repo, 'seed.md'), 'dirty unrelated edit\n'); + + await seedPage('notes/hello'); + const result = await writePageThrough(engine, 'notes/hello'); + + expect(result.written).toBe(true); + expect(result.committed).toBe(true); + // The artifact is committed… + expect(git(repo, 'log', '-1', '--format=%s')).toBe('gbrain: write-through notes/hello'); + expect(git(repo, 'log', '-1', '--name-only', '--format=')).toBe('notes/hello.md'); + expect(git(repo, 'status', '--porcelain', 'notes/hello.md')).toBe(''); + // …and the unrelated edit stays uncommitted (explicit-path discipline). + expect(git(repo, 'status', '--porcelain', 'seed.md')).not.toBe(''); + }, 60_000); + + test('on an unhardened repo, the file is written but NOT committed (no behavior change)', async () => { + await seedPage('notes/plain'); + const result = await writePageThrough(engine, 'notes/plain'); + + expect(result.written).toBe(true); + expect(result.committed).toBeUndefined(); + // Untracked, uncommitted — the pre-existing contract. + expect(git(repo, 'status', '--porcelain', 'notes/plain.md')).toContain('?? notes/plain.md'); + expect(git(repo, 'log', '-1', '--format=%s')).toBe('init'); + }, 60_000); +});