diff --git a/CHANGELOG.md b/CHANGELOG.md index e3f1ecaa0..480091cd8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,19 @@ All notable changes to GBrain will be documented in this file. +## [0.42.35.0] - 2026-06-07 + +**A bookmark left pointing at a rewritten-away commit no longer freezes your brain in an endless full re-walk.** When a source's history is rewritten — a force-push, a `master`→`main` consolidation, a squash — the commit gbrain recorded as "last synced" can fall outside the branch's current history. The old guard treated that the same as a missing commit and fell back to re-importing the entire repository on every run. On a large brain with a cross-region database that full walk never finishes inside the sync timeout, so the bookmark never advanced and the source went quietly stale with no error surfaced. + +The fix is a smaller, exact diff. `git diff A..B` compares two trees and does not require A to be an ancestor of B, so when the recorded commit's object is still on disk (the common case right after a rewrite) gbrain now diffs directly against it and imports only the real delta — the changed files, not the whole tree. A clear `[sync] last_commit … history rewritten` line marks the recovery. Only when the commit object is genuinely gone does sync fall back to a full reconcile, and that reconcile now also purges pages whose source files were removed — so a full sync is finally authoritative for deletes, not just imports (manually authored `put_page` pages and metafiles are never swept). Sibling of the v0.42.32.0 silent-staleness fix (gbrain#1939); closes gbrain#1970. + +### Fixed +- **Sync recovers from an unreachable `last_commit` instead of full-walking forever (gbrain#1970).** A bookmark orphaned by a history rewrite is now diffed tree-to-tree directly when its object is still present, importing only the changed files; an oversized or failed diff degrades to a full reconcile instead of throwing. Only a truly-absent (gc'd) object forces a full reconcile. +- **A full sync now purges deleted files.** `performFullSync` reconciles deletions — pages whose backing file is gone are removed (gated to file-backed pages via `source_path`; manual `put_page` pages and metafiles are spared). This makes both the object-absent recovery path and every `--full` sync authoritative for deletes, not just imports. +- **Rename to an unsyncable path deletes the stale page.** A syncable file renamed to a non-syncable destination (which git reports as a rename, not a delete) now removes the old page instead of leaving it orphaned. + +### To take advantage of v0.42.35.0 +- Nothing to do. The next `gbrain sync` after upgrading self-heals a stuck bookmark automatically; watch for the one-line `[sync] last_commit … history rewritten` recovery message. If a source has been stale since a force-push or branch consolidation, this is the release that unsticks it. ## [0.42.34.0] - 2026-06-07 **Relationship questions now get relationship answers.** Ask "who invested in widget-co", "who introduced me to alice-example", or "what connects fund-a and fund-b" and gbrain resolves the named entity and walks its typed-edge graph (`invested_in`, `works_at`, `founded`, `attended`, `advises`, …) to surface the answer — even when no single page mentions both sides. Until now the graph only re-ranked results that keyword/vector search had already found; a relationship that lived purely in the edges (an investor whose page never names the company) was invisible. It now enters retrieval as a first-class candidate. diff --git a/VERSION b/VERSION index 5f3886e1f..a2e98cff7 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.34.0 +0.42.35.0 diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index e5186de7a..38e091ba1 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -288,7 +288,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/embedding-dim-check.ts` extension — facts.embedding dim drift surface. `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (migration v40 falls back to `vector` on pgvector < 0.7); regex ordering is halfvec-before-vector (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (NOT bare REINDEX, which doesn't rewrite the index after a column-type change). `assertFactsEmbeddingDimMatchesConfig(engine)` is the preflight — throws `FactsEmbeddingDimMismatchError` (tagged `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool MUST_ABORT semantics) when configured dim ≠ column width; cached per-engine via `WeakMap`; PGLite engines silently skip. Doctor check `facts_embedding_width_consistency` (registered after `embedding_width_consistency`) reuses the same helpers with an identical ALTER recipe. Pinned by `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). Test seam `__resetFactsEmbeddingCastCacheForTest()` clears the per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path; `SyncOpts.lockId?: string` is the explicit override. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths in two `op_checkpoints` rows (both keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` — an `op:'sync'` paths row + an `op:'sync-target'` target row) via `recordCompleted` every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files, and advances `last_commit`/`last_sync_at` ONLY at full import completion. A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` reject when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error, 2 cost-prompt-not-confirmed. The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The `sync --all` cost gate is mode-aware via `willEmbedSynchronously` + `shouldBlockSync` from `src/core/embedding.ts` (federated_v2 resolved once up front): the DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog `sumStaleChunkChars({signature})` priced via `estimateCostFromChars` at `currentEmbeddingPricePerMTok()` + queued-job count, NEVER exits 2); the INLINE path BLOCKS only when the new-content estimate exceeds `sync.cost_gate_min_usd` (default $0.50). The new-content estimate (`estimateInlineNewTokens`) contributes ZERO for sources provably unchanged since last sync (`isSourceUnchangedSinceSync` HEAD==last_commit + clean tree AND `chunker_version === CURRENT`) and the full-tree ceiling for changed sources; the pre-existing stale backlog is shown informationally but NEVER gated on (sync doesn't sweep it inline). Helpers `resolveCostGateFloorUsd(engine)` (fail-open $0.50, accepts 0) + `resolveBackfillCapUsd(engine)`. JSON envelopes gain `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | confirmation_required`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path; `SyncOpts.lockId?: string` is the explicit override. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths in two `op_checkpoints` rows (both keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` — an `op:'sync'` paths row + an `op:'sync-target'` target row) via `recordCompleted` every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files, and advances `last_commit`/`last_sync_at` ONLY at full import completion. A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` reject when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error, 2 cost-prompt-not-confirmed. The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The `sync --all` cost gate is mode-aware via `willEmbedSynchronously` + `shouldBlockSync` from `src/core/embedding.ts` (federated_v2 resolved once up front): the DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog `sumStaleChunkChars({signature})` priced via `estimateCostFromChars` at `currentEmbeddingPricePerMTok()` + queued-job count, NEVER exits 2); the INLINE path BLOCKS only when the new-content estimate exceeds `sync.cost_gate_min_usd` (default $0.50). The new-content estimate (`estimateInlineNewTokens`) contributes ZERO for sources provably unchanged since last sync (`isSourceUnchangedSinceSync` HEAD==last_commit + clean tree AND `chunker_version === CURRENT`) and the full-tree ceiling for changed sources; the pre-existing stale backlog is shown informationally but NEVER gated on (sync doesn't sweep it inline). Helpers `resolveCostGateFloorUsd(engine)` (fail-open $0.50, accepts 0) + `resolveBackfillCapUsd(engine)`. JSON envelopes gain `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | confirmation_required`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). - `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. - `src/core/sort-newest-first.ts` — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line here instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (descending order, mixed prefixes, empty, single-element, in-place-mutation contract). diff --git a/package.json b/package.json index 7d6f87318..b736d5cee 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.34.0" + "version": "0.42.35.0" } diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 164389db7..bada4c63a 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -1246,21 +1246,50 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise isSyncable(r.from, syncOpts) && !isSyncable(r.to, syncOpts)) + .map(r => r.from); const filtered: SyncManifest = { added: manifest.added.filter(p => isSyncable(p, syncOpts)), modified: manifest.modified.filter(p => isSyncable(p, syncOpts)), - deleted: manifest.deleted.filter(p => isSyncable(p, syncOpts)), + deleted: unique([ + ...manifest.deleted.filter(p => isSyncable(p, syncOpts)), + ...renamedToUnsyncable, + ]), renamed: manifest.renamed.filter(r => isSyncable(r.to, syncOpts)), }; @@ -2394,6 +2450,68 @@ async function performFullSync( ); } + // #1970 (F-A): runImport is import-only — it never purges pages whose backing + // file was deleted since the last sync. A full re-import is authoritative for + // the whole tree, so reconcile deletes here too (this is what makes the + // object-absent fallback at performSyncInner correct for deletes, not just + // imports). Runs only on an advancing full sync (we're past the + // !fullGate.advanced early-return). + // + // SAFETY — must NOT re-introduce the #1433 stale-page data loss. A page is + // deleted ONLY when ALL three hold: + // 1. source_path != null → file-backed pages only; put_page/manual + // pages (null source_path) are never swept. + // 2. isSyncable(source_path) → excludes metafiles (README/log.md, the + // #1433 class) AND the wrong strategy (a markdown sync can't delete a + // code page, and vice versa). + // 3. source_path ∉ current → the backing file is genuinely gone from the + // working tree (collectSyncableFiles == the same enumeration runImport + // used, so paths are in the identical relative form as source_path). + // Skipped on the legacy no-sourceId path (the batch delete primitives require + // a sourceId; matches every other source-scoped feature). + let reconciledDeletes = 0; + if (opts.sourceId) { + const sid = opts.sourceId; + const reconcileSyncOpts = opts.strategy ? { strategy: opts.strategy } : undefined; + // collectSyncableFiles returns ABSOLUTE paths; source_path is stored + // repo-relative (importFile uses `relative(dir, filePath)`), so relativize + // to the same form before membership-testing — otherwise every page looks + // stale and the reconcile would wrongly delete live pages. + const current = new Set( + collectSyncableFiles(repoPath, { strategy: opts.strategy ?? 'markdown' }) + .map(abs => relative(repoPath, abs)), + ); + const rows = await engine.executeRaw<{ slug: string; source_path: string | null }>( + `SELECT slug, source_path FROM pages WHERE source_id = $1 AND source_path IS NOT NULL AND deleted_at IS NULL`, + [sid], + ); + const staleSlugs = rows + .filter(r => r.source_path != null + && isSyncable(r.source_path, reconcileSyncOpts) + && !current.has(r.source_path)) + .map(r => r.slug); + if (staleSlugs.length > 0) { + const deleteScopedOpts = { sourceId: sid }; + for (let i = 0; i < staleSlugs.length; i += DELETE_BATCH_SIZE) { + const batch = staleSlugs.slice(i, i + DELETE_BATCH_SIZE); + try { + const deleted = await engine.deletePages(batch, deleteScopedOpts); + reconciledDeletes += deleted.length; + } catch { + // Per-slug fallback on a batch blip (mirrors the incremental delete + // loop). A stale page that won't delete is best-effort, not fatal. + for (const slug of batch) { + try { await engine.deletePage(slug, deleteScopedOpts); reconciledDeletes++; } + catch { /* best-effort */ } + } + } + } + if (reconciledDeletes > 0) { + slog(` Reconciled ${reconciledDeletes} stale page(s) whose source file was removed.`); + } + } + } + // Full sync doesn't track pagesAffected, so fall back to embed --stale. // v0.37 fix wave (Lane D.3 + CDX2-8): switched to runEmbedCore for the // same reason as the incremental path — surface dim-mismatch via hint @@ -2421,7 +2539,7 @@ async function performFullSync( toCommit: headCommit, added: result.imported, modified: 0, - deleted: 0, + deleted: reconciledDeletes, renamed: 0, chunksCreated: result.chunksCreated, embedded, diff --git a/test/sync.test.ts b/test/sync.test.ts index 5a6bf9596..acac5e348 100644 --- a/test/sync.test.ts +++ b/test/sync.test.ts @@ -670,3 +670,244 @@ describe('sync auto-embed arguments', () => { expect(buildAutoEmbedArgs(['people/alice'])).toEqual(['--slugs', 'people/alice']); }); }); + +// #1970: sync silently full-walks forever when last_commit is unreachable. +// The bookmark can point at a commit orphaned by a history rewrite (force-push, +// master→main consolidation, squash). The old guard sent BOTH "object missing" +// AND "not an ancestor" to a blind full re-walk that never advanced the bookmark. +// The fix: only a truly-absent object forces a full reconcile; a present-but- +// non-ancestor bookmark is diffed tree-to-tree directly (`git diff A..B` needs +// no ancestry). Plus F-A (full-sync delete reconcile), F-B (oversized-diff +// fallback), F-C (rename-to-unsyncable deletes the old page). +describe('#1970: unreachable last_commit bookmark recovery', () => { + let engine: PGLiteEngine; + const repos: string[] = []; + + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + beforeEach(async () => { + await resetPgliteState(engine); + }); + + afterEach(() => { + while (repos.length) { + const d = repos.pop(); + if (d) rmSync(d, { recursive: true, force: true }); + } + }); + + function personMd(title: string, body: string): string { + return ['---', 'type: person', `title: ${title}`, '---', '', body].join('\n'); + } + + /** Create a temp git repo seeded with the given files + an initial commit. */ + function mkRepo(files: Record): string { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-1970-')); + repos.push(dir); + execSync('git init', { cwd: dir, stdio: 'pipe' }); + execSync('git config user.email "test@test.com"', { cwd: dir, stdio: 'pipe' }); + execSync('git config user.name "Test"', { cwd: dir, stdio: 'pipe' }); + for (const [rel, content] of Object.entries(files)) { + mkdirSync(join(dir, rel, '..'), { recursive: true }); + writeFileSync(join(dir, rel), content); + } + execSync('git add -A && git commit -m "initial"', { cwd: dir, stdio: 'pipe' }); + return dir; + } + + const SYNC_OPTS = { noPull: true, noEmbed: true, noExtract: true, sourceId: 'default' } as const; + + async function bookmark(): Promise { + const rows = await engine.executeRaw<{ last_commit: string | null }>( + `SELECT last_commit FROM sources WHERE id = 'default'`, + ); + return rows[0]?.last_commit ?? null; + } + + async function captureLog(fn: () => Promise): Promise<{ result: T; out: string }> { + const lines: string[] = []; + const origLog = console.log; + console.log = (...args: unknown[]) => { lines.push(args.map(String).join(' ')); }; + try { + const result = await fn(); + return { result, out: lines.join('\n') }; + } finally { + console.log = origLog; + } + } + + test('orphan-present (not an ancestor): diffs tree-to-tree, imports only the delta, advances bookmark', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const repo = mkRepo({ + 'people/alice.md': personMd('Alice', 'Alice is a person.'), + 'people/bob.md': personMd('Bob', 'Bob is a person.'), + }); + + const first = await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + expect(first.status).toBe('first_sync'); + const orphan = await bookmark(); + expect(orphan).not.toBeNull(); + + // Rewrite history: amend the only commit (adds delta.md). The previous tip + // is now orphaned but still on disk — cat-file succeeds, is-ancestor fails. + writeFileSync(join(repo, 'people/carol.md'), personMd('Carol', 'Carol joins.')); + execSync('git add -A && git commit --amend -m "amended with carol"', { cwd: repo, stdio: 'pipe' }); + + // Sanity: the stored bookmark is present but no longer an ancestor of HEAD. + expect(execSync(`git cat-file -t ${orphan}`, { cwd: repo }).toString().trim()).toBe('commit'); + let isAncestor = true; + try { execSync(`git merge-base --is-ancestor ${orphan} HEAD`, { cwd: repo, stdio: 'pipe' }); } + catch { isAncestor = false; } + expect(isAncestor).toBe(false); + + const { result, out } = await captureLog(() => performSync(engine, { repoPath: repo, ...SYNC_OPTS })); + + // Incremental diff path (status 'synced'), NOT a full re-walk ('first_sync'). + expect(result.status).toBe('synced'); + expect(result.added).toBe(1); + expect(out).toContain('not an ancestor of HEAD'); + expect(await engine.getPage('people/carol')).not.toBeNull(); + // Bookmark advanced off the orphan onto the rewritten HEAD. + const advanced = await bookmark(); + expect(advanced).not.toBe(orphan); + expect(advanced).toBe(execSync('git rev-parse HEAD', { cwd: repo }).toString().trim()); + }); + + test('orphan-absent (object gc\'d): falls back to a full reconcile', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const repo = mkRepo({ 'people/alice.md': personMd('Alice', 'Alice is a person.') }); + + await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + // Simulate an orphaned-AND-pruned bookmark: a valid-shaped SHA with no object. + await engine.executeRaw( + `UPDATE sources SET last_commit = $1 WHERE id = 'default'`, + ['deadbeefdeadbeefdeadbeefdeadbeefdeadbeef'], + ); + writeFileSync(join(repo, 'people/bob.md'), personMd('Bob', 'Bob is a person.')); + execSync('git add -A && git commit -m "add bob"', { cwd: repo, stdio: 'pipe' }); + + const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + // Object absent → authoritative full reconcile. + expect(result.status).toBe('first_sync'); + expect(await engine.getPage('people/bob')).not.toBeNull(); + expect(await engine.getPage('people/alice')).not.toBeNull(); + }); + + test('divergence: a file present in the orphan tree but dropped from HEAD is deleted', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const repo = mkRepo({ + 'people/alice.md': personMd('Alice', 'Alice is a person.'), + 'people/bob.md': personMd('Bob', 'Bob is a person.'), + }); + await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + expect(await engine.getPage('people/bob')).not.toBeNull(); + + // Rewrite the tip: drop bob, edit alice. Orphans the prior tip (still on disk). + execSync('git rm people/bob.md', { cwd: repo, stdio: 'pipe' }); + writeFileSync(join(repo, 'people/alice.md'), personMd('Alice', 'Alice was corrected.')); + execSync('git add -A && git commit --amend -m "drop bob, edit alice"', { cwd: repo, stdio: 'pipe' }); + + const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + expect(result.status).toBe('synced'); + expect(await engine.getPage('people/bob')).toBeNull(); // deleted + const alice = await engine.getPage('people/alice'); + expect(alice!.compiled_truth).toContain('corrected'); // updated + }); + + test('F-C: a rename whose destination is unsyncable deletes the old page', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const repo = mkRepo({ 'people/carol.md': personMd('Carol', 'Carol is a person.') }); + await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + expect(await engine.getPage('people/carol')).not.toBeNull(); + + // git mv keeps content identical → classified as a 100% rename (R100). + // The destination .txt is unsyncable, so without the F-C fix the old page + // would linger (the rename drops out of both `renamed` and `deleted`). + execSync('git mv people/carol.md people/carol.txt', { cwd: repo, stdio: 'pipe' }); + execSync('git commit -m "rename carol to txt"', { cwd: repo, stdio: 'pipe' }); + + const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + expect(result.status).toBe('synced'); + expect(await engine.getPage('people/carol')).toBeNull(); + }); + + test('F-A: full reconcile purges stale file-backed pages but spares manual + metafile pages', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const repo = mkRepo({ + 'people/alice.md': personMd('Alice', 'Alice is a person.'), + 'people/bob.md': personMd('Bob', 'Bob is a person.'), + }); + await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + + // A manually-curated page (put_page) — source_path stays NULL. + await engine.putPage('manual/note', { + type: 'note', title: 'Manual Note', compiled_truth: 'Hand-authored, not from a file.', + }, { sourceId: 'default' }); + // A metafile-backed page (e.g. an older import or direct put_page of log.md). + // Its source_path is unsyncable, so the reconcile must NOT delete it (#1433). + await engine.putPage('people/log', { + type: 'note', title: 'Log', compiled_truth: 'metafile page', source_path: 'people/log.md', + }, { sourceId: 'default' }); + + // Delete bob's backing file, then force a full reconcile. + execSync('git rm people/bob.md', { cwd: repo, stdio: 'pipe' }); + execSync('git commit -m "remove bob"', { cwd: repo, stdio: 'pipe' }); + + const result = await performSync(engine, { repoPath: repo, full: true, ...SYNC_OPTS }); + expect(result.status).toBe('first_sync'); + expect(result.deleted).toBeGreaterThanOrEqual(1); + + expect(await engine.getPage('people/bob')).toBeNull(); // stale file-backed → purged + expect(await engine.getPage('people/alice')).not.toBeNull(); // still present → kept + expect(await engine.getPage('manual/note')).not.toBeNull(); // null source_path → spared + expect(await engine.getPage('people/log')).not.toBeNull(); // metafile source_path → spared + }); + + test('F-B: an undiffable-but-present bookmark falls back to a full reconcile instead of throwing', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const repo = mkRepo({ 'people/alice.md': personMd('Alice', 'Alice is a person.') }); + await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + + // A blob SHA: cat-file -t succeeds ("blob", so objectPresent=true), but + // `git diff ..HEAD` errors — the same failure shape as an oversized + // post-rewrite diff hitting git()'s timeout/buffer limits. Must fall back, + // not throw. + const blob = execSync('git rev-parse HEAD:people/alice.md', { cwd: repo }).toString().trim(); + await engine.executeRaw(`UPDATE sources SET last_commit = $1 WHERE id = 'default'`, [blob]); + + const result = await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + expect(result.status).toBe('first_sync'); // fell back cleanly + expect(await engine.getPage('people/alice')).not.toBeNull(); + }); + + test('convergence: after orphan recovery, a later commit syncs incrementally to up_to_date', async () => { + const { performSync } = await import('../src/commands/sync.ts'); + const repo = mkRepo({ 'people/alice.md': personMd('Alice', 'Alice is a person.') }); + await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + + // Orphan + recover. + writeFileSync(join(repo, 'people/bob.md'), personMd('Bob', 'Bob is a person.')); + execSync('git add -A && git commit --amend -m "amended with bob"', { cwd: repo, stdio: 'pipe' }); + const recovered = await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + expect(recovered.status).toBe('synced'); + + // A subsequent ordinary commit now syncs incrementally (bookmark is sane). + writeFileSync(join(repo, 'people/carol.md'), personMd('Carol', 'Carol joins.')); + execSync('git add -A && git commit -m "add carol"', { cwd: repo, stdio: 'pipe' }); + const next = await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + expect(next.status).toBe('synced'); + expect(next.added).toBe(1); + + // No further changes → up_to_date (converged). + const settled = await performSync(engine, { repoPath: repo, ...SYNC_OPTS }); + expect(settled.status).toBe('up_to_date'); + }); +});