diff --git a/docs/TESTING.md b/docs/TESTING.md index 69399d7a7..9062dce92 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -189,6 +189,7 @@ Unit tests and what they cover: - `test/orphans.test.ts` — orphans command: detection, pseudo filtering, text/json/count outputs, MCP op. - `test/postgres-engine.test.ts` — `statement_timeout` scoping: `sql.begin` + `SET LOCAL` shape, source-level grep guardrail against a reintroduced bare `SET statement_timeout`. - `test/sync.test.ts` — sync logic + regression guard asserting top-level `engine.transaction` is not called. +- `test/sync-pull-failed-anchor.serial.test.ts` — #3068 regression: a failed internal `git pull` (local-path origin vs `protocol.file.allow=never`) with zero imports returns `partial`/`pull_failed` (not `up_to_date`), freezes `last_commit` + `last_sync_at`, recovers after a manual pull; fall-through import of local commits preserved. Serial: pins `GBRAIN_HOME` to a temp dir for the whole file. - `test/sync-concurrency.test.ts` — `autoConcurrency()` thresholds + PGLite-forces-serial + explicit-override clamping; `shouldRunParallel()` explicit-bypasses-floor contract; `parseWorkers()` validation rejecting `'0'`/`'-3'`/`'foo'`/`'1.5'`/trailing chars. - `test/sync-parallel.test.ts` — PGLite-routed coverage of the bookmark gate under concurrency, head-drift gate, vanished-file failure capture, PGLite-stays-serial, and the `gbrain-sync` writer-lock contract. - `test/sync-failures.test.ts` — `classifyErrorCode` regex coverage for all 12 codes against literal production message strings from `markdown.ts` and `import-file.ts`; `summarizeFailuresByCode` sort + pre-classified-honor; `recordSyncFailures` code-field persistence; `acknowledgeSyncFailures` `AcknowledgeResult` shape + backfill on legacy entries. diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 2e86cc25c..8e0c25134 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -306,7 +306,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/embedding-dim-check.ts` extension — facts.embedding dim drift surface. `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (migration v40 falls back to `vector` on pgvector < 0.7); regex ordering is halfvec-before-vector (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (NOT bare REINDEX, which doesn't rewrite the index after a column-type change). `assertFactsEmbeddingDimMatchesConfig(engine)` is the preflight — throws `FactsEmbeddingDimMismatchError` (tagged `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool MUST_ABORT semantics) when configured dim ≠ column width; cached per-engine via `WeakMap`; PGLite engines silently skip. Doctor check `facts_embedding_width_consistency` (registered after `embedding_width_consistency`) reuses the same helpers with an identical ALTER recipe. Pinned by `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). Test seam `__resetFactsEmbeddingCastCacheForTest()` clears the per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. `synthesize_concepts` writes concept pages through `importFromContent` (#2163: the same parse→chunk→embed pipeline put_page uses, with put_page's `isAvailable('embedding')` → `noEmbed` gate) so `concepts/` pages carry `content_chunks` + embeddings and are reachable by retrieval (where `source-boost.ts` weights them 1.3×). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. Below the valve, stale pages are partitioned by git history via exported `listEverCommittedPaths(repoPath)` (one `git log --all --no-renames --diff-filter=A --name-only` pass; null on non-git dirs → unchanged behavior): a stale path that EVER existed in history was genuinely deleted → reconciled; a path with NO history is DB-only write-through (never committed/pushed, e.g. lost to a fresh clone) → the page is KEPT and its markdown re-exported to the working tree via `writePageThrough`, with a stderr hint to commit it (#2426 — "absent from git" is the symptom of the missing write-through commit, not evidence the content is disposable). Pinned by `test/sync-reconcile-db-only.serial.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). Monorepo subdir sources (#753/#774): `--src-subpath ` (or a repo path that IS a subdir — auto-discovery via `discoverGitRoot`, i.e. `git rev-parse --show-toplevel`) splits the repo path into `gitContextRoot` (all git ops: pull/diff/rev-parse/cat-file) and `syncScopeRoot` (walk/import/delete/rename scope); scoped syncs use git-root-relative slugs + `source_path` (full sync threads `slugRoot` into `runImport`) so full and incremental agree; NAV-1/NAV-2 realpath containment rejects `../`-traversal and symlinked scopes resolving outside the repo BEFORE any git op, and a per-file realpath guard (`isPathSafe`) refuses symlink-escape files in the incremental drain and rename reimport (fail-closed into `failedFiles`, so the bookmark can't advance past an escape); the full-sync reconcile is scope-restricted so a scoped sync never sweeps out-of-scope pages. `--exclude ` (repeatable) filters scope-relative paths in both full and incremental paths; exclusion never deletes previously-imported pages (conservative, matching the #1433 metafile posture); an all-excluded run warns loudly (NAV-4). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. The stale-file decision routes through the pure, exported `planReconcileDeletes(rows, currentFiles, isSyncablePath)`: it normalizes path separators on both sides of the membership test (a Windows `path.relative` backslash path vs a git-derived forward-slash `source_path` would otherwise mark every page stale and wipe the source) and computes a mass-delete signal — when the reconcile would delete more than `MASS_RECONCILE_RATIO` (50%) of the file-backed pages the strategy manages, on a source holding more than `MASS_RECONCILE_MIN_PAGES` (20) of them, the delete is REFUSED with a loud stderr warning (that shape is almost always a path-comparison bug or the wrong repo path, not a genuine bulk deletion); `GBRAIN_ALLOW_MASS_RECONCILE=1` restores the unguarded delete for genuinely intended bulk removals. Pinned by `test/sync-reconcile-mass-delete.test.ts`. Below the valve, stale pages are partitioned by git history via exported `listEverCommittedPaths(repoPath)` (one `git log --all --no-renames --diff-filter=A --name-only` pass; null on non-git dirs → unchanged behavior): a stale path that EVER existed in history was genuinely deleted → reconciled; a path with NO history is DB-only write-through (never committed/pushed, e.g. lost to a fresh clone) → the page is KEPT and its markdown re-exported to the working tree via `writePageThrough`, with a stderr hint to commit it (#2426 — "absent from git" is the symptom of the missing write-through commit, not evidence the content is disposable). Pinned by `test/sync-reconcile-db-only.serial.test.ts`. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). Monorepo subdir sources (#753/#774): `--src-subpath ` (or a repo path that IS a subdir — auto-discovery via `discoverGitRoot`, i.e. `git rev-parse --show-toplevel`) splits the repo path into `gitContextRoot` (all git ops: pull/diff/rev-parse/cat-file) and `syncScopeRoot` (walk/import/delete/rename scope); scoped syncs use git-root-relative slugs + `source_path` (full sync threads `slugRoot` into `runImport`) so full and incremental agree; NAV-1/NAV-2 realpath containment rejects `../`-traversal and symlinked scopes resolving outside the repo BEFORE any git op, and a per-file realpath guard (`isPathSafe`) refuses symlink-escape files in the incremental drain and rename reimport (fail-closed into `failedFiles`, so the bookmark can't advance past an escape); the full-sync reconcile is scope-restricted so a scoped sync never sweeps out-of-scope pages. `--exclude ` (repeatable) filters scope-relative paths in both full and incremental paths; exclusion never deletes previously-imported pages (conservative, matching the #1433 metafile posture); an all-excluded run warns loudly (NAV-4). A warn-and-continue internal `git pull` failure (non-timeout class — e.g. a local-path origin rejected by `protocol.file.allow=never`) still falls through to sync the local working tree, but a ZERO-import run after a failed pull returns `partial` with `reason: 'pull_failed'` instead of `up_to_date`: `last_commit` AND the `last_sync_at` heartbeat stay frozen (so doctor `sync_freshness` / `sources status` staleness fires), the single-source CLI exits non-zero, `sync --all` exits non-zero if any source hit it (JSON envelope carries the per-source `reason`), and the autopilot cycle's sync phase maps it to `warn`. Timeout-class partials keep their pre-existing exit-0 / phase-`ok` semantics (they converge on retry; a failing pull does not). Pinned by `test/sync-pull-failed-anchor.serial.test.ts`. - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). `collectSyncableFiles`' shared emit filter `isCollectibleForWalker` applies the SAME segment-level `pruneDir` gate as incremental sync's `classifySync` — load-bearing for the `git ls-files` fast path, which enumerates tracked files under dot-dirs/vendored trees that the FS walk never descends into; without it `sync --full` imported (and resurrected soft-deleted) pages incremental sync excludes (#2607). Pinned by `test/import-git-fastpath-prune.test.ts`. `runImport` opts also carry `exclude` (glob filter over dir-relative paths, threaded by `performFullSync` for `sync --exclude`; warns when every file is excluded — NAV-4) and `slugRoot` (slug/`source_path` base for monorepo subdir syncs, #753/#774; the resume checkpoint stays dir-relative per `resumeFilter`'s contract).- `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. - `src/core/sort-newest-first.ts` — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line here instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (descending order, mixed prefixes, empty, single-element, in-place-mutation contract). - `src/core/cycle.ts` — brain maintenance cycle primitive (9 phases). `runCycle(engine: BrainEngine | null, opts: CycleOpts): Promise` composes phases in semantic order: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → embed → orphans. `synthesize` runs after sync (cross-references see a fresh brain) and before extract (auto-link materializes its writes); `patterns` runs after extract so it reads a fresh graph (subagent put_page sets `ctx.remote=true` and skips auto-link/timeline by default, so extract is the canonical materialization); `recompute_emotional_weight` sees the union of `syncPagesAffected` + `synthesizeWrittenSlugs` incrementally, or all pages when neither anchor is set (full backfill via `gbrain dream --phase recompute_emotional_weight`). `CycleReport.schema_version: "1"` is stable; `totals` is additive (`pages_emotional_weight_recomputed`, `transcripts_processed`, `synth_pages_written`, `patterns_written`). Three callers: `gbrain dream` CLI, `gbrain autopilot` daemon inline path, the Minions `autopilot-cycle` handler. Coordination via `gbrain_cycle_locks` DB table + `~/.gbrain/cycle.lock` file lock with PID-liveness for PGLite. `yieldBetweenPhases` runs between phases; `yieldDuringPhase` is in-phase keepalive (synthesize/patterns renew the cycle-lock TTL during long waits). Engine nullable; lock-skip on read-only phase selections. `CycleOpts.signal?: AbortSignal` propagates the worker's abort signal with `checkAborted()` between every phase. `runPhaseSync` returns `pagesAffected` via `SyncPhaseResult` (threaded to `runPhaseExtract` as the 4th arg) and takes `willRunExtractPhase: boolean` setting `noExtract: phases.includes('extract')` so `gbrain dream --phase sync` doesn't silently lose extraction. `resolveSourceForDir(engine, brainDir)` threads `sourceId` to `performSync()` so sync reads the per-source `sources.last_commit` anchor (not the drift-prone global `config.sync.last_commit`). `CycleOpts.brainDir` is `string | null`; when null (checkout-less postgres/Supabase brain) the 6 filesystem phases (lint/backlinks/sync/synthesize/extract/patterns) skip with `details.reason: 'no_brain_dir'` and the DB-only phases run; `resolveSourceForDir` is null-tolerant. `cycleSourceId = opts.sourceId ?? resolveSourceForDir(engine, brainDir)` is the canonical per-source scope for `extract_facts`/`extract_atoms`/calibration — and for `synthesize` (#1586: threaded as `SynthesizePhaseOpts.sourceId` so synthesized pages land in the cycle's resolved source, not `'default'`) — so `gbrain dream --source repo-a` reconciles repo-a's facts even with no checkout (instead of scoping to `'default'` while stamping repo-a fresh). `deriveStatus` counts `edges_resolved`/`edges_ambiguous` as work so an edges-only cycle reports `ok` not `clean`; the `jobs.ts` `autopilot-cycle` + phase-wrapper handlers pass `null` (not `'.'`) when no repo is configured. The cycle is SPLIT for autopilot fan-out (#2194/#2227): `PHASE_SCOPE` partitions `ALL_PHASES` into `GLOBAL_PHASES` (brain-wide: embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile/synthesize_concepts/skillopt) and `NON_GLOBAL_PHASES` (source + mixed). Per-source `autopilot-cycle` jobs run only `NON_GLOBAL_PHASES` and stamp `last_source_cycle_at`; the single `autopilot-global-maintenance` job runs `GLOBAL_PHASES` (no `sourceId`) and stamps the brain-level `autopilot.last_global_at` config key (`LAST_GLOBAL_AT_KEY`). `last_full_cycle_at` is still written alongside `last_source_cycle_at` on a per-source success for doctor/legacy readers (no longer a gate for the brain-wide phases). Pinned by `test/dream-postgres.serial.test.ts` + `test/jobs-autopilot-cycle-braindir.serial.test.ts` + `test/autopilot-global-maintenance.test.ts`. diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 2763063f4..377aaa033 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -213,7 +213,7 @@ export interface SyncResult { * cron operators can disambiguate timeout vs pull-timeout in monitoring. */ filesImported?: number; - reason?: 'timeout' | 'pull_timeout' | 'stall_timeout' | 'checkpoint_unavailable'; + reason?: 'timeout' | 'pull_timeout' | 'pull_failed' | 'stall_timeout' | 'checkpoint_unavailable'; /** * v0.42.x (#1794): cumulative file paths durably banked to the checkpoint * across THIS run + prior resumed runs. Surfaced on every partial/blocked @@ -1761,7 +1761,7 @@ function buildPartialResult(opts: { modified: number; deleted: number; renamed: number; - reason: 'timeout' | 'pull_timeout' | 'stall_timeout' | 'checkpoint_unavailable'; + reason: 'timeout' | 'pull_timeout' | 'pull_failed' | 'stall_timeout' | 'checkpoint_unavailable'; bankedFiles?: number; }): SyncResult { return { @@ -1992,6 +1992,15 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 0); if (lastCommit === headCommit && !versionMismatch && !versionNeverSet && !hasDetachedWorkingTreeChanges) { + // #3068: the pull failed and nothing local advanced — this run imported + // NOTHING and the remote may hold commits we could not fetch. Reporting + // `up_to_date` here (and bumping the heartbeat below) is exactly the + // silent-wedge from the issue: every scheduled sync exits 0 forever while + // the source is stale. Return `partial` instead (not a clean status, and + // last_sync_at stays frozen so doctor/sources-status staleness fires). + // The anchor is untouched; the next sync retries the pull from the same + // bookmark. + if (pullFailed) { + serr( + `[sync] git pull failed and no local changes imported — reporting partial ` + + `(not up_to_date); sync anchor unchanged at ${lastCommit.slice(0, 8)}.`, + ); + return buildPartialResult({ + fromCommit: lastCommit, + toCommit: lastCommit, + filesImported: 0, + pagesAffected: [], + chunksCreated: 0, + added: 0, modified: 0, deleted: 0, renamed: 0, + reason: 'pull_failed', + }); + } // v0.42.52.0 (PR #22xx): bump last_sync_at as a heartbeat on every successful // 0-changes sync. D4 invariant ("never advance last_commit on partial") is // preserved: last_sync_at is a monitoring signal (doctor sync_freshness @@ -2392,6 +2425,27 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 0) process.exit(1); + // #3068: any source wedged on a failed pull (partial/pull_failed) makes + // the whole --all run non-zero — it will not self-heal on retry, so a + // green exit would hide it from cron/monitoring. Timeout-class partials + // keep the pre-existing exit-0 behavior (they converge on retry). + const pullFailedCount = perSourceResults.filter( + (r) => r.status === 'ok' && r.result?.status === 'partial' && r.result.reason === 'pull_failed', + ).length; + if (errCount > 0 || pullFailedCount > 0) process.exit(1); return; } @@ -4725,6 +4789,16 @@ See also: process.off('SIGINT', onSingleSourceSigint); } printSyncResult(result); + // #3068: a pull_failed partial is NOT a success — unlike timeout-class + // partials (which converge on retry), a failing pull will not self-heal. + // Exit non-zero so cron/monitoring sees the wedge instead of a green run. + // Routed through the owned verdict channel (NOT bare `process.exitCode`, + // which PGLite's Emscripten runtime clobbers mid-run — see + // src/core/cli-force-exit.ts). + if (result.status === 'partial' && result.reason === 'pull_failed') { + const { setCliExitVerdict } = await import('../core/cli-force-exit.ts'); + setCliExitVerdict(1); + } // v0.42.7 (#1696, D5): extraction-lag nudge after a completed single-source // sync. Fire on every non-error completion (synced | first_sync | up_to_date) // — NOT just 'synced'; a fresh/--full import (`first_sync`) is the biggest @@ -5430,6 +5504,17 @@ function printSyncResult(result: SyncResult, sink: NodeJS.WriteStream = process. write(` Fix the files then re-run 'gbrain sync', or 'gbrain sync --skip-failed' to move on.`); break; case 'partial': + // #3068: a failed (non-timeout) pull with zero imports gets its own + // message — "imported 0 of 0" reads like success, but the local + // checkout may be behind a remote we could not fetch. + if (result.reason === 'pull_failed') { + write( + `Sync INCOMPLETE at ${result.fromCommit?.slice(0, 8) ?? ''}: ` + + `git pull failed — the local checkout may be behind its remote.`, + ); + write(` Fix the pull (see the warning above), then re-run 'gbrain sync' (last_commit unchanged; safe to retry).`); + break; + } // v0.41.13.0 (T7 / D-V3-5): --timeout fired before the bookmark write // so last_commit is UNCHANGED. The next sync re-walks the same diff // and content_hash short-circuits already-imported files at ~10ms each. diff --git a/src/core/cycle.ts b/src/core/cycle.ts index 757b4021b..0b5195795 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -935,13 +935,21 @@ async function runPhaseSync( // sync's inline extract still runs to preserve prior behavior. }); const syncedCount = result.added + result.modified; + // #3068: a pull_failed partial means the internal git pull failed and the + // run imported nothing — the source may be silently behind its remote and + // will not self-heal. Surface it as 'warn' (not 'ok') so a scheduled cycle + // doesn't report a clean run over a wedged source. Timeout-class partials + // keep the pre-existing 'ok' mapping (they converge on retry by design). + const pullFailedPartial = result.status === 'partial' && result.reason === 'pull_failed'; return { phase: 'sync', - status: result.status === 'blocked_by_failures' ? 'warn' : 'ok', + status: result.status === 'blocked_by_failures' || pullFailedPartial ? 'warn' : 'ok', duration_ms: 0, summary: dryRun ? `${syncedCount} page(s) would sync, ${result.deleted} would delete` - : `+${result.added} added, ~${result.modified} modified, -${result.deleted} deleted`, + : pullFailedPartial + ? `git pull failed, nothing imported — source may be behind its remote (sync anchor unchanged)` + : `+${result.added} added, ~${result.modified} modified, -${result.deleted} deleted`, details: { added: result.added, modified: result.modified, @@ -950,6 +958,7 @@ async function runPhaseSync( chunksCreated: result.chunksCreated, failedFiles: result.failedFiles ?? 0, syncStatus: result.status, + ...(result.reason ? { syncReason: result.reason } : {}), dryRun, }, pagesAffected: result.pagesAffected, diff --git a/test/sync-pull-failed-anchor.serial.test.ts b/test/sync-pull-failed-anchor.serial.test.ts new file mode 100644 index 000000000..90302ca85 --- /dev/null +++ b/test/sync-pull-failed-anchor.serial.test.ts @@ -0,0 +1,190 @@ +import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs'; +import { join } from 'path'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; + +/** + * #3068 regression: a failed internal `git pull` (warn-and-continue class, + * e.g. a local-filesystem-path origin rejected by the SSRF flag + * `protocol.file.allow=never`) combined with a zero-import run must NOT be + * reported as a clean `up_to_date` sync: + * + * - status must be `partial` with reason `pull_failed` (not `up_to_date`), + * - `last_commit` (the sync anchor) must not move, + * - `last_sync_at` (the freshness heartbeat doctor/sources-status read) + * must not be bumped — otherwise a permanently-failing pull keeps the + * source looking fresh forever while it silently goes stale. + * + * The fall-through-to-working-tree design is unchanged: local commits still + * import when the remote is unreachable, and once the operator repairs the + * checkout (e.g. a manual `git pull`, which does not carry the SSRF flags) + * the next sync imports the missed content from the untouched anchor. + * + * The local-path-origin topology below reproduces the pull failure + * deterministically: `pullRepo` always passes `-c protocol.file.allow=never`, + * so its internal pull fails on every cycle while plain `git pull` succeeds. + */ +describe('#3068: failed git pull + zero imports must not report up_to_date', () => { + let engine: PGLiteEngine; + const dirs: string[] = []; + // Hermetic GBRAIN_HOME: performFullSync (first sync) reads/writes the + // sync-failure ledger under the gbrain home — never touch the real one. + let isolatedHome: string; + let origGbrainHome: string | undefined; + + beforeAll(async () => { + origGbrainHome = process.env.GBRAIN_HOME; + isolatedHome = mkdtempSync(join(tmpdir(), 'gbrain-3068-home-')); + process.env.GBRAIN_HOME = isolatedHome; + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + if (origGbrainHome !== undefined) process.env.GBRAIN_HOME = origGbrainHome; + else delete process.env.GBRAIN_HOME; + rmSync(isolatedHome, { recursive: true, force: true }); + }); + + beforeEach(async () => { + await resetPgliteState(engine); + }); + + afterEach(() => { + while (dirs.length) { + const d = dirs.pop(); + if (d) rmSync(d, { recursive: true, force: true }); + } + }); + + function personMd(title: string, body: string): string { + return ['---', 'type: person', `title: ${title}`, '---', '', body].join('\n'); + } + + function mkUpstream(files: Record): string { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-3068-upstream-')); + dirs.push(dir); + execSync('git init', { cwd: dir, stdio: 'pipe' }); + execSync('git config user.email "test@test.com"', { cwd: dir, stdio: 'pipe' }); + execSync('git config user.name "Test"', { cwd: dir, stdio: 'pipe' }); + for (const [rel, content] of Object.entries(files)) { + mkdirSync(join(dir, rel, '..'), { recursive: true }); + writeFileSync(join(dir, rel), content); + } + execSync('git add -A && git commit -m "initial"', { cwd: dir, stdio: 'pipe' }); + return dir; + } + + /** Clone `upstream` to a sibling temp dir — origin is a local filesystem path. */ + function mkMirror(upstream: string): string { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-3068-mirror-')); + dirs.push(dir); + rmSync(dir, { recursive: true, force: true }); + execSync(`git clone ${JSON.stringify(upstream)} ${JSON.stringify(dir)}`, { stdio: 'pipe' }); + return dir; + } + + async function sourceRow(): Promise<{ last_commit: string | null; last_sync_at: string | null }> { + const rows = await engine.executeRaw<{ last_commit: string | null; last_sync_at: string | null }>( + `SELECT last_commit, last_sync_at FROM sources WHERE id = 'default'`, + ); + return rows[0] ?? { last_commit: null, last_sync_at: null }; + } + + // Pull ENABLED (no noPull) — the internal pull must actually run and fail. + const SYNC_OPTS = { noEmbed: true, noExtract: true, sourceId: 'default' } as const; + + test('zero-import sync after failed pull reports partial/pull_failed; anchor and heartbeat frozen; manual pull recovers', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const upstream = mkUpstream({ 'people/alice.md': personMd('Alice', 'Alice is a person.') }); + const mirror = mkMirror(upstream); + + // First sync: the internal pull already fails (local-path origin), but the + // fall-through imports the working tree — unchanged behavior. + const first = await performSync(engine, { repoPath: mirror, ...SYNC_OPTS }); + expect(first.status).toBe('first_sync'); + const afterFirst = await sourceRow(); + expect(afterFirst.last_commit).not.toBeNull(); + expect(afterFirst.last_sync_at).not.toBeNull(); + + // New content lands upstream; the mirror's internal pull can never fetch it. + writeFileSync(join(upstream, 'people/bob.md'), personMd('Bob', 'zzuniquetoken99 new content.')); + execSync('git add -A && git commit -m "new page"', { cwd: upstream, stdio: 'pipe' }); + + // Wait so a (buggy) heartbeat bump would be observable as a changed timestamp. + await new Promise((r) => setTimeout(r, 1100)); + + // Pre-fix this reported `up_to_date`, bumped last_sync_at, and exited clean + // forever — the #3068 silent wedge. + const wedged = await performSync(engine, { repoPath: mirror, ...SYNC_OPTS }); + expect(wedged.status).toBe('partial'); + expect(wedged.reason).toBe('pull_failed'); + expect(wedged.added + wedged.modified + wedged.deleted + wedged.renamed).toBe(0); + expect(wedged.fromCommit).toBe(afterFirst.last_commit); + expect(wedged.toCommit).toBe(afterFirst.last_commit ?? ''); + + const afterWedged = await sourceRow(); + expect(afterWedged.last_commit).toBe(afterFirst.last_commit); // anchor unchanged + expect(afterWedged.last_sync_at).toEqual(afterFirst.last_sync_at); // heartbeat NOT bumped + + // Recovery: a manual pull (no SSRF flags) fast-forwards the mirror; the + // next sync imports the missed content from the untouched anchor. + execSync('git pull', { cwd: mirror, stdio: 'pipe' }); + const recovered = await performSync(engine, { repoPath: mirror, ...SYNC_OPTS }); + expect(recovered.status).toBe('synced'); + expect(recovered.added).toBe(1); + expect(await engine.getPage('people/bob')).not.toBeNull(); + + const afterRecovered = await sourceRow(); + expect(afterRecovered.last_commit).not.toBe(afterFirst.last_commit); // anchor advanced with the import + }); + + test('failed pull with local syncable commits still imports them (fall-through preserved)', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const upstream = mkUpstream({ 'people/alice.md': personMd('Alice', 'Alice is a person.') }); + const mirror = mkMirror(upstream); + + const first = await performSync(engine, { repoPath: mirror, ...SYNC_OPTS }); + expect(first.status).toBe('first_sync'); + + // A local commit in the mirror itself — importable without any pull. + execSync('git config user.email "test@test.com"', { cwd: mirror, stdio: 'pipe' }); + execSync('git config user.name "Test"', { cwd: mirror, stdio: 'pipe' }); + writeFileSync(join(mirror, 'people/carol.md'), personMd('Carol', 'Carol is local.')); + execSync('git add -A && git commit -m "local carol"', { cwd: mirror, stdio: 'pipe' }); + + // The internal pull still fails, but there is real local work — the + // warn-and-continue design imports it and advances the anchor over the + // commits that were actually imported. + const synced = await performSync(engine, { repoPath: mirror, ...SYNC_OPTS }); + expect(synced.status).toBe('synced'); + expect(synced.added).toBe(1); + expect(await engine.getPage('people/carol')).not.toBeNull(); + }); + + test('local no-syncable-content commit after failed pull also reports partial without advancing the anchor', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const upstream = mkUpstream({ 'people/alice.md': personMd('Alice', 'Alice is a person.') }); + const mirror = mkMirror(upstream); + + const first = await performSync(engine, { repoPath: mirror, ...SYNC_OPTS }); + expect(first.status).toBe('first_sync'); + const anchor = (await sourceRow()).last_commit; + + // A local commit with no syncable content (non-markdown file). + execSync('git config user.email "test@test.com"', { cwd: mirror, stdio: 'pipe' }); + execSync('git config user.name "Test"', { cwd: mirror, stdio: 'pipe' }); + writeFileSync(join(mirror, 'notes.txt'), 'not syncable'); + execSync('git add -A && git commit -m "local txt"', { cwd: mirror, stdio: 'pipe' }); + + const result = await performSync(engine, { repoPath: mirror, ...SYNC_OPTS }); + expect(result.status).toBe('partial'); + expect(result.reason).toBe('pull_failed'); + expect((await sourceRow()).last_commit).toBe(anchor); // anchor unchanged + }); +});