mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.42.17.0 fix(sync): resumable incremental sync — killed mid-import no longer loses progress (#1794) (#1808)
* feat(sync): checkpoint primitives for resumable sync (#1794) - op-checkpoint.ts: syncFingerprint({sourceId, lastCommit}) keyed on the anchor (never HEAD) so the checkpoint survives a growing backlog. - source-health.ts: commitTimeMs(localPath, sha) for stamping newest_content_at against a pinned (non-HEAD) commit. - sync-concurrency.ts: resolveMaxConnections + clampWorkersForConnectionBudget for the opt-in GBRAIN_MAX_CONNECTIONS single-sync footprint clamp. * feat(sync): resumable incremental sync via pinned-target checkpoint (#1794) performSyncInner now drains a fixed lastCommit..pin range, banking completed file paths to op_checkpoints and advancing last_commit (+ last_sync_at) ONLY at full import completion. A killed/aborted/blocked run leaves the anchor untouched and resumes from the banked set next run — the convergence fix. - Pinned target: completion advances to the pin, not live HEAD, so commits landing after the pin are a clean next-sync diff (kills the staleness window). History rewrite (pin not an ancestor of HEAD) discards the checkpoint + re-pins. - Forward-progress head gate: merge-base --is-ancestor pin HEAD replaces the strict "HEAD == captured" gate that blocked on every concurrent enrich commit. - Vanished-on-disk added file -> skip + checkpoint, not a failedFiles block. - Large syncs defer extract/embed to the resumable --stale sweeps (convergence == import convergence); small syncs keep inline extract/facts/embed. - GBRAIN_MAX_CONNECTIONS clamp on the worker fan-out (opt-in). - Typed SyncLockBusyError; the Minion sync handler (jobs.ts) marks the job SKIPPED (not failed) on a held lock so cron/autopilot defers cleanly. * feat(doctor): pool_budget check for GBRAIN_MAX_CONNECTIONS (#1794) computePoolBudgetCheck + checkPoolBudget warn when the parent pool leaves no room for a parallel sync worker under GBRAIN_MAX_CONNECTIONS, pointing at GBRAIN_POOL_SIZE=2. Registered in the ops category set. * test(sync): resumable-sync regression suite + vanished-file contract (#1794) - sync-resumable-import.serial.test.ts (13 cases): convergence regression, resume-skips-checkpointed, pinned-target/forward-drift, history-rewrite re-pin, last_sync_at-not-bumped-on-block + good-file banking, vanished-file skip, dry-run/empty-diff, + pure fingerprint/clamp/pool-budget helpers. - sync-parallel.test.ts: vanished-mid-sync added file now asserts the new skip contract (supersedes the v0.22.13 CODEX-3 failedFiles behavior). * chore: bump version and changelog (v0.42.17.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
3fe449361c
commit
fd2fde9d26
@@ -2,6 +2,46 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.17.0] - 2026-06-03
|
||||
|
||||
**A huge `gbrain sync` can no longer get stuck forever losing all its progress when it's killed partway through.** If your brain suddenly grows by tens of thousands of pages (say a background process is enriching one page per commit, all night long), the next sync has a giant backlog to import. If that sync gets killed before it finishes — a session timeout, a laptop sleep, anything — it used to throw away **everything** it had done and start over from zero. The next hour the backlog was even bigger, so it got killed again, and again. It could never catch up. This release makes sync **resumable**: a killed sync banks what it imported, and the next run picks up where it left off. It converges.
|
||||
|
||||
Two related things were quietly making it worse, both fixed here:
|
||||
|
||||
- **Sync used to give up the moment a new commit landed during the run.** If something else was committing to the same repo while sync worked (exactly the "enriching all night" case), sync saw the moving target and aborted with "blocked." Now sync locks onto a fixed target snapshot, drains to it, and lets the new commits land in the next run. Normal forward progress no longer blocks anything.
|
||||
- **A big sync would hammer your database connection pool.** On a small pooler (Supabase's 20-client default) a single sync could exhaust the slots and starve its own retries. New opt-in `GBRAIN_MAX_CONNECTIONS` caps a sync's footprint, and `gbrain doctor` nudges you to lower `GBRAIN_POOL_SIZE` when the math doesn't fit.
|
||||
|
||||
You don't have to do anything — `gbrain sync` is resumable by default. Two knobs if you want them:
|
||||
|
||||
```
|
||||
# How often progress is banked (files between checkpoint flushes; default 1000)
|
||||
export GBRAIN_SYNC_CHECKPOINT_EVERY=1000
|
||||
|
||||
# Cap a single sync's DB connections on a low-cap pooler (opt-in; off by default)
|
||||
export GBRAIN_MAX_CONNECTIONS=16
|
||||
```
|
||||
|
||||
What you'd see on a 44,000-file backlog, killed at 16% three times:
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| Progress kept after a kill | 0% (full re-walk) | banked up to the last checkpoint |
|
||||
| New commits during the sync | blocks the whole run | ignored this run, picked up next run |
|
||||
| Does it ever finish? | no — backlog outran every attempt | yes — each run banks real progress |
|
||||
|
||||
Things to know about: the sync bookmark (`last_commit`) still only advances when the import fully completes, so a killed sync correctly looks "stale" and gets retried. For large syncs, link/timeline/embedding extraction is deferred to the resumable `gbrain extract --stale` / `gbrain embed --stale` sweeps (and the autopilot cycle) instead of running inline — that keeps a 44K-page extraction pass from re-blocking the import. Small syncs are unchanged: they still extract and embed inline.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- **Resumable incremental sync (`src/commands/sync.ts`).** `performSyncInner` now drives the import loop against a **pinned target commit** held in a DB checkpoint (`op_checkpoints`, two rows keyed by `syncFingerprint(sourceId, lastCommit)`). Each batch of imported files is flushed to the checkpoint; a killed/aborted/blocked run banks the completed set and leaves `last_commit` unchanged. The next run resume-filters the diff against the banked set and continues. `last_commit` (and `last_sync_at`) advance only at full import completion, then both checkpoint rows clear.
|
||||
- **Pinned target eliminates the staleness window.** The checkpoint pins the target commit at the first run and drains `lastCommit..pin`; completion advances to the pin (not live HEAD), so commits landing after the pin are a clean next-sync diff and never get skipped. A history rewrite (pin no longer an ancestor of HEAD) discards the checkpoint and re-pins.
|
||||
- **Forward-progress head gate.** The pre-existing strict "HEAD == captured" head-drift gate (which blocked on any concurrent commit) is replaced by a pin-reachability check: forward progress is safe; only a real rewrite blocks.
|
||||
- **`commitTimeMs(localPath, sha)`** added to `src/core/source-health.ts` — stamps `newest_content_at` against the pinned commit.
|
||||
- **`syncFingerprint({ sourceId, lastCommit })`** added to `src/core/op-checkpoint.ts` — keyed on the anchor (never HEAD) so the checkpoint survives a growing backlog.
|
||||
- **Connection-budget clamp (`src/core/sync-concurrency.ts`).** New `resolveMaxConnections()` + `clampWorkersForConnectionBudget()`; opt-in via `GBRAIN_MAX_CONNECTIONS`, no-op when unset (existing brains unchanged). `gbrain doctor` gains a `pool_budget` check that warns when the parent pool leaves no room for a worker.
|
||||
- **Vanished-on-disk added files are skipped, not failed.** A file added in the diff but deleted from disk by a commit after the pin (normal forward delete) is skipped and checkpointed instead of blocking the run.
|
||||
- **Cleaner single-flight backpressure.** `performSync` throws a typed `SyncLockBusyError`; the Minion `sync` handler catches it and marks the job *skipped* (not failed), so a cron/autopilot tick that hits a held lock defers to the holder without polluting the failed-job/crash metrics.
|
||||
- **Tests.** `test/sync-resumable-import.serial.test.ts` (13 cases): convergence regression, resume-skips-checkpointed, pinned-target/forward-drift, history-rewrite re-pin, `last_sync_at` not bumped on a blocked run + good-file banking, vanished-file skip, dry-run/empty-diff, plus pure-helper coverage for the fingerprint, clamp, and pool-budget math.
|
||||
## [0.42.16.0] - 2026-06-02
|
||||
|
||||
**`gbrain doctor` now tells you the brain is OOM-looping in one line, ranks every
|
||||
|
||||
File diff suppressed because one or more lines are too long
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.16.0"
|
||||
"version": "0.42.17.0"
|
||||
}
|
||||
|
||||
@@ -691,6 +691,9 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
// v0.41.19.0 (Issue 5): sync --all consolidation nudge for multi-source brains.
|
||||
checks.push(await checkSyncConsolidation(engine));
|
||||
|
||||
// v0.42.x (#1794, 4A): pool-budget nudge when GBRAIN_MAX_CONNECTIONS is set.
|
||||
checks.push(await checkPoolBudget(engine));
|
||||
|
||||
// v0.42.7 (#1696): link-extraction lag. Strictly SQL (single indexed COUNT),
|
||||
// safe on the thin-client/remote path — remote operators on checkout-less
|
||||
// Postgres brains are exactly who can't otherwise see the extraction backlog.
|
||||
@@ -3444,6 +3447,65 @@ export async function checkSyncConsolidation(engine: BrainEngine): Promise<Check
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.x (#1794, 4A) — pure pool-budget check. When `GBRAIN_MAX_CONNECTIONS`
|
||||
* is set (the operator opted into the single-source connection clamp), verify
|
||||
* the parent pool leaves room for at least one parallel worker. If even the
|
||||
* parent pool alone is at/over the budget, sync clamps to serial AND every
|
||||
* other gbrain process competes for the same cap — the operator should lower
|
||||
* `GBRAIN_POOL_SIZE`. Pure so it's unit-testable without env/engine.
|
||||
*/
|
||||
export function computePoolBudgetCheck(
|
||||
maxConnections: number | undefined,
|
||||
parentPool: number,
|
||||
perWorkerPool: number,
|
||||
): Check {
|
||||
if (maxConnections === undefined) {
|
||||
return {
|
||||
name: 'pool_budget',
|
||||
status: 'ok',
|
||||
message: 'GBRAIN_MAX_CONNECTIONS not set — connection budget clamp disabled (default behavior).',
|
||||
};
|
||||
}
|
||||
if (parentPool + perWorkerPool > maxConnections) {
|
||||
return {
|
||||
name: 'pool_budget',
|
||||
status: 'warn',
|
||||
message:
|
||||
`GBRAIN_MAX_CONNECTIONS=${maxConnections} leaves no room for a parallel sync worker ` +
|
||||
`(parent pool ${parentPool} + ${perWorkerPool} per-worker > ${maxConnections}). ` +
|
||||
`Sync will run serial. If you hit EMAXCONNSESSION, lower the parent pool: ` +
|
||||
'`gbrain config` / set GBRAIN_POOL_SIZE=2 (recommended for low-cap poolers like Supabase Supavisor).',
|
||||
};
|
||||
}
|
||||
const maxWorkers = Math.floor((maxConnections - parentPool) / perWorkerPool);
|
||||
return {
|
||||
name: 'pool_budget',
|
||||
status: 'ok',
|
||||
message:
|
||||
`GBRAIN_MAX_CONNECTIONS=${maxConnections}: room for up to ${maxWorkers} parallel sync ` +
|
||||
`worker(s) (parent pool ${parentPool} + ${perWorkerPool} per-worker).`,
|
||||
};
|
||||
}
|
||||
|
||||
/** Thin env/engine wrapper over `computePoolBudgetCheck`. */
|
||||
export async function checkPoolBudget(_engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
const { resolveMaxConnections } = await import('../core/sync-concurrency.ts');
|
||||
const { resolvePoolSize } = await import('../core/db.ts');
|
||||
const maxConnections = resolveMaxConnections();
|
||||
const parentPool = resolvePoolSize();
|
||||
const perWorkerPool = Math.min(2, resolvePoolSize(2));
|
||||
return computePoolBudgetCheck(maxConnections, parentPool, perWorkerPool);
|
||||
} catch (err) {
|
||||
return {
|
||||
name: 'pool_budget',
|
||||
status: 'ok',
|
||||
message: `Skipped (${err instanceof Error ? err.message : String(err)})`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.38 — per-source `last_full_cycle_at` freshness check.
|
||||
*
|
||||
|
||||
+23
-4
@@ -1172,10 +1172,29 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
// standalone handler dropped it. Callers that want inline extract can
|
||||
// pass { noExtract: false } in job params explicitly.
|
||||
const noExtract = job.data.noExtract !== false;
|
||||
const result = await performSync(engine, {
|
||||
repoPath, sourceId, noPull, noEmbed, noExtract,
|
||||
concurrency: concurrencyOverride,
|
||||
});
|
||||
let result;
|
||||
try {
|
||||
result = await performSync(engine, {
|
||||
repoPath, sourceId, noPull, noEmbed, noExtract,
|
||||
concurrency: concurrencyOverride,
|
||||
});
|
||||
} catch (err) {
|
||||
// v0.42.x (#1794, Part B): single-flight backpressure. A concurrent
|
||||
// sync (manual run, sibling autopilot tick) holds the per-source lock.
|
||||
// SKIP cleanly — mark the job done, NOT failed — so the holder finishes
|
||||
// without this tick polluting the failed-jobs count + supervisor crash
|
||||
// metrics. The next scheduled tick resumes against the (by then
|
||||
// advanced) anchor.
|
||||
const { SyncLockBusyError } = await import('./sync.ts');
|
||||
if (err instanceof SyncLockBusyError) {
|
||||
console.error(
|
||||
`[sync] skipped: sync already in progress for ${sourceId ?? 'default'} ` +
|
||||
`(lock ${err.lockKey} held).`,
|
||||
);
|
||||
return { skipped: true, reason: 'sync_in_progress', source_id: sourceId ?? 'default' };
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
|
||||
// v0.40 D22: auto_embed_backfill defaults TRUE when sourceId is set AND
|
||||
// the feature flag is enabled. Submits a child embed-backfill job
|
||||
|
||||
+315
-56
@@ -39,6 +39,8 @@ import {
|
||||
parseWorkers,
|
||||
parseDurationSeconds,
|
||||
DEFAULT_PARALLEL_SOURCES,
|
||||
resolveMaxConnections,
|
||||
clampWorkersForConnectionBudget,
|
||||
} from '../core/sync-concurrency.ts';
|
||||
import {
|
||||
tryAcquireDbLock,
|
||||
@@ -58,8 +60,58 @@ import { getDefaultSourcePath } from '../core/source-resolver.ts';
|
||||
// remote staleness path reads a column instead of shelling out to git.
|
||||
// lagFromContentMs is the remote/column comparator (buildSyncStatusReport
|
||||
// backs the get_status_snapshot MCP op — must NOT shell out to git).
|
||||
import { newestCommitMs, lagFromContentMs } from '../core/source-health.ts';
|
||||
import { newestCommitMs, commitTimeMs, lagFromContentMs } from '../core/source-health.ts';
|
||||
import { sortNewestFirst } from '../core/sort-newest-first.ts';
|
||||
import {
|
||||
loadOpCheckpoint,
|
||||
recordCompleted,
|
||||
clearOpCheckpoint,
|
||||
resumeFilter,
|
||||
syncFingerprint,
|
||||
type OpCheckpointKey,
|
||||
} from '../core/op-checkpoint.ts';
|
||||
|
||||
/**
|
||||
* v0.42.x (#1794) -- resumable incremental sync checkpoint.
|
||||
*
|
||||
* Two op-checkpoint rows back one resumable incremental sync, both keyed by
|
||||
* the same syncFingerprint(sourceId, lastCommit):
|
||||
* - op 'sync' -> completed_keys = repo-relative file paths drained
|
||||
* - op 'sync-target' -> completed_keys = [pinnedTargetCommit]
|
||||
*
|
||||
* Splitting the pinned target into its own row keeps the path set free of any
|
||||
* sentinel that could collide with a real filename, and both rows clear
|
||||
* together on full completion. The fingerprint encodes ONLY (sourceId,
|
||||
* lastCommit) -- never the target or live HEAD -- so the checkpoint survives
|
||||
* every killed-and-resumed run while lastCommit..HEAD grows underneath it.
|
||||
*/
|
||||
const SYNC_CKPT_OP = 'sync';
|
||||
const SYNC_TARGET_OP = 'sync-target';
|
||||
|
||||
function syncCheckpointKeys(
|
||||
sourceId: string | undefined,
|
||||
lastCommit: string,
|
||||
): { paths: OpCheckpointKey; target: OpCheckpointKey } {
|
||||
const fp = syncFingerprint({ sourceId, lastCommit });
|
||||
return {
|
||||
paths: { op: SYNC_CKPT_OP, fingerprint: fp },
|
||||
target: { op: SYNC_TARGET_OP, fingerprint: fp },
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Files flushed to the checkpoint per batch. Larger = fewer JSONB rewrites
|
||||
* (less write amplification on huge backlogs) at the cost of more re-done
|
||||
* import work on a kill (cheap -- content_hash short-circuits the re-import).
|
||||
*/
|
||||
const SYNC_CHECKPOINT_EVERY_DEFAULT = 1000;
|
||||
|
||||
function resolveSyncCheckpointEvery(): number {
|
||||
const raw = process.env.GBRAIN_SYNC_CHECKPOINT_EVERY;
|
||||
if (!raw) return SYNC_CHECKPOINT_EVERY_DEFAULT;
|
||||
const n = Number(raw);
|
||||
return Number.isInteger(n) && n > 0 ? n : SYNC_CHECKPOINT_EVERY_DEFAULT;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.7 (#1696, D5): which terminal sync statuses warrant the end-of-sync
|
||||
@@ -641,6 +693,23 @@ See also:
|
||||
console.log(`job_id=${job.id}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.x (#1794, Part B): typed lock-busy error so callers can distinguish a
|
||||
* benign "another sync holds the lock, skip cleanly" from a real failure.
|
||||
* Subclasses Error so existing CLI handlers that print `err.message` keep the
|
||||
* rich `formatLockBusyMessage` text verbatim. The Minion `sync` handler catches
|
||||
* this to mark the job skipped (not failed) — single-flight backpressure: the
|
||||
* cron/autopilot sync defers to the holder instead of erroring + retrying noisily.
|
||||
*/
|
||||
export class SyncLockBusyError extends Error {
|
||||
readonly lockKey: string;
|
||||
constructor(message: string, lockKey: string) {
|
||||
super(message);
|
||||
this.name = 'SyncLockBusyError';
|
||||
this.lockKey = lockKey;
|
||||
}
|
||||
}
|
||||
|
||||
export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<SyncResult> {
|
||||
// v0.22.13 CODEX-2: cross-process writer lock prevents two concurrent
|
||||
// syncs from racing on the same last_commit anchor (last writer wins,
|
||||
@@ -675,7 +744,7 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
return await withRefreshingLock(engine, lockKey, () => performSyncInner(engine, opts));
|
||||
} catch (err) {
|
||||
if (err instanceof LockUnavailableError) {
|
||||
throw new Error(await formatLockBusyMessage(engine, lockKey));
|
||||
throw new SyncLockBusyError(await formatLockBusyMessage(engine, lockKey), lockKey);
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
@@ -684,7 +753,7 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise<
|
||||
// Legacy global-lock path (single-default-source brains).
|
||||
const lockHandle = await tryAcquireDbLock(engine, lockKey);
|
||||
if (!lockHandle) {
|
||||
throw new Error(await formatLockBusyMessage(engine, lockKey));
|
||||
throw new SyncLockBusyError(await formatLockBusyMessage(engine, lockKey), lockKey);
|
||||
}
|
||||
try {
|
||||
return await performSyncInner(engine, opts);
|
||||
@@ -1181,6 +1250,48 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
return performFullSync(engine, repoPath, headCommit, opts);
|
||||
}
|
||||
|
||||
// v0.42.x (#1794): resumable incremental sync — resolve the PINNED target.
|
||||
// last_commit advances only at FULL import completion, so a killed run keeps
|
||||
// lastCommit fixed and the checkpoint key stable across every resume even as
|
||||
// the enrich process races HEAD forward underneath us. We drain
|
||||
// `lastCommit..pin`; commits past the pin are a clean next-sync diff (this is
|
||||
// what kills the staleness window — see plan).
|
||||
// - valid in-flight checkpoint (pin still reachable from HEAD) → resume it.
|
||||
// - rewrite / force-push (pin no longer an ancestor) → discard, re-pin to HEAD.
|
||||
// - no checkpoint → pin = HEAD (the normal single-shot case).
|
||||
const ckpt = syncCheckpointKeys(opts.sourceId, lastCommit);
|
||||
const checkpointEvery = resolveSyncCheckpointEvery();
|
||||
let pin = headCommit;
|
||||
let completedPaths: string[] = [];
|
||||
{
|
||||
const storedTargetArr = await loadOpCheckpoint(engine, ckpt.target);
|
||||
const storedTarget = storedTargetArr[0] ?? null;
|
||||
if (storedTarget) {
|
||||
let pinReachable = false;
|
||||
try {
|
||||
git(repoPath, ['merge-base', '--is-ancestor', storedTarget, headCommit]);
|
||||
pinReachable = true;
|
||||
} catch {
|
||||
pinReachable = false;
|
||||
}
|
||||
if (pinReachable) {
|
||||
pin = storedTarget;
|
||||
completedPaths = await loadOpCheckpoint(engine, ckpt.paths);
|
||||
slog(
|
||||
`[sync] resuming checkpoint: ${completedPaths.length} file(s) already done; ` +
|
||||
`draining ${lastCommit.slice(0, 8)}..${pin.slice(0, 8)} (pinned target).`,
|
||||
);
|
||||
} else {
|
||||
slog(
|
||||
`[sync] checkpoint target ${storedTarget.slice(0, 8)} no longer reachable ` +
|
||||
`(history rewritten); restarting against HEAD.`,
|
||||
);
|
||||
await clearOpCheckpoint(engine, ckpt.paths);
|
||||
await clearOpCheckpoint(engine, ckpt.target);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// v0.20.0 Cathedral II Layer 12 (codex SP-1 fix): before returning
|
||||
// 'up_to_date' on git-HEAD equality, check the chunker version gate.
|
||||
// If sources.chunker_version mismatches CURRENT_CHUNKER_VERSION, force
|
||||
@@ -1221,8 +1332,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
return result;
|
||||
}
|
||||
|
||||
// Diff using git diff (net result, not per-commit)
|
||||
const diffOutput = git(repoPath, ['diff', '--name-status', '-M', `${lastCommit}..${headCommit}`]);
|
||||
// Diff using git diff (net result, not per-commit). v0.42.x (#1794): diff
|
||||
// against the PINNED target, not live HEAD. With a fixed (lastCommit, pin)
|
||||
// both endpoints are stable across every resume, so the manifest is
|
||||
// deterministic and resumeFilter maps cleanly onto completed paths.
|
||||
const diffOutput = git(repoPath, ['diff', '--name-status', '-M', `${lastCommit}..${pin}`]);
|
||||
const manifest = buildSyncManifest(diffOutput);
|
||||
if (detachedWorkingTreeManifest) {
|
||||
manifest.added = unique([...manifest.added, ...detachedWorkingTreeManifest.added]);
|
||||
@@ -1307,14 +1421,19 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
|
||||
if (totalChanges === 0) {
|
||||
// Update sync state even with no syncable changes (git advanced)
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit, newestCommitMs(repoPath));
|
||||
// Update sync state even with no syncable changes (git advanced). v0.42.x
|
||||
// (#1794): advance to the PINNED target, and clear any checkpoint (a resume
|
||||
// whose remaining range turned out to have no syncable changes still
|
||||
// completes cleanly here).
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(repoPath, pin));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
await clearOpCheckpoint(engine, ckpt.paths);
|
||||
await clearOpCheckpoint(engine, ckpt.target);
|
||||
return {
|
||||
status: 'up_to_date',
|
||||
fromCommit: lastCommit,
|
||||
toCommit: headCommit,
|
||||
toCommit: pin,
|
||||
added: 0, modified: 0, deleted: 0, renamed: 0,
|
||||
chunksCreated: 0,
|
||||
embedded: 0,
|
||||
@@ -1327,6 +1446,31 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
slog(`Large sync (${totalChanges} files). Importing text, deferring embeddings.`);
|
||||
}
|
||||
|
||||
// v0.42.x (#1794): we have real work — persist the PIN now so a crash before
|
||||
// the first path-flush still resumes to THIS target (not re-pin to a newer
|
||||
// HEAD). Idempotent on a resume (the row already holds pin). Never on dry-run
|
||||
// (returned above).
|
||||
await recordCompleted(engine, ckpt.target, [pin]);
|
||||
|
||||
// v0.42.x (#1794): the cross-run completed-path set. Seeded from the loaded
|
||||
// checkpoint so a resume skips already-drained files. `markCompleted` flushes
|
||||
// every `checkpointEvery` additions; on a kill the worst case lost is one
|
||||
// batch of import work (cheap to redo — content_hash short-circuits).
|
||||
const completed = new Set<string>(completedPaths);
|
||||
let sinceFlush = 0;
|
||||
const flushCheckpoint = async (): Promise<void> => {
|
||||
// `[...completed]` is a synchronous snapshot (atomic under concurrent
|
||||
// worker `completed.add`), so a parallel flush can't observe a torn set.
|
||||
await recordCompleted(engine, ckpt.paths, [...completed]);
|
||||
};
|
||||
const markCompleted = async (path: string): Promise<void> => {
|
||||
completed.add(path);
|
||||
if (++sinceFlush >= checkpointEvery) {
|
||||
sinceFlush = 0;
|
||||
await flushCheckpoint();
|
||||
}
|
||||
};
|
||||
|
||||
const pagesAffected: string[] = [];
|
||||
let chunksCreated = 0;
|
||||
// v0.41.13.0 (T2): tracks add+modify files actually persisted so far.
|
||||
@@ -1342,10 +1486,14 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// D-V3-1 invariant: callable ONLY in pre-bookmark phases (pull, delete,
|
||||
// rename, import). After the bookmark write at writeSyncAnchor('last_commit'),
|
||||
// partial is impossible because extract + embed run to completion.
|
||||
// v0.42.x (#1794): toCommit reports the PINNED target (what this run drains
|
||||
// to), not live HEAD. The checkpoint is flushed periodically inside the loops
|
||||
// (markCompleted), so on abort the banked completed set survives — last_commit
|
||||
// stays at lastCommit (unchanged) and the next run resumes from the checkpoint.
|
||||
const partial = (reason: 'timeout' | 'pull_timeout'): SyncResult =>
|
||||
buildPartialResult({
|
||||
fromCommit: lastCommit,
|
||||
toCommit: headCommit,
|
||||
toCommit: pin,
|
||||
filesImported,
|
||||
pagesAffected: [...pagesAffected],
|
||||
chunksCreated,
|
||||
@@ -1415,17 +1563,21 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// OLD per-path loop. The batch engine surface requires sourceId per D5
|
||||
// (multi-source-bug-class defense at the type level). Production callers
|
||||
// that thread sourceId via resolveSourceWithTier get the new fast path.
|
||||
if (filtered.deleted.length > 0) {
|
||||
progress.start('sync.deletes', filtered.deleted.length);
|
||||
// v0.42.x (#1794): resume-filter the delete set so a resumed run skips paths
|
||||
// already drained in a prior run (deletes are idempotent, but skipping avoids
|
||||
// re-resolving + re-deleting tens of thousands of already-gone pages).
|
||||
const deletesToDo = resumeFilter(filtered.deleted, [...completed]);
|
||||
if (deletesToDo.length > 0) {
|
||||
progress.start('sync.deletes', deletesToDo.length);
|
||||
if (opts.sourceId) {
|
||||
const sid = opts.sourceId;
|
||||
const deleteScopedOpts = { sourceId: sid };
|
||||
for (let i = 0; i < filtered.deleted.length; i += DELETE_BATCH_SIZE) {
|
||||
for (let i = 0; i < deletesToDo.length; i += DELETE_BATCH_SIZE) {
|
||||
if (opts.signal?.aborted) {
|
||||
progress.finish();
|
||||
return partial('timeout');
|
||||
}
|
||||
const batch = filtered.deleted.slice(i, i + DELETE_BATCH_SIZE);
|
||||
const batch = deletesToDo.slice(i, i + DELETE_BATCH_SIZE);
|
||||
|
||||
// Phase A: batch slug resolution (1 round-trip per batch).
|
||||
let pathSlugMap: Map<string, string>;
|
||||
@@ -1447,6 +1599,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// slugs (paths in filtered.deleted but with no DB row) so
|
||||
// downstream extract/embed don't waste lookups.
|
||||
pagesAffected.push(...deleted);
|
||||
// v0.42.x (#1794): the whole batch is handled (deleted or already
|
||||
// gone); checkpoint every path so a resume skips it.
|
||||
for (const p of batch) await markCompleted(p);
|
||||
} catch (err) {
|
||||
// D7 decompose: a transient blip on this batch shouldn't lose all
|
||||
// 500 deletes. Fall back to per-slug deletePage for THIS batch
|
||||
@@ -1456,6 +1611,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
try {
|
||||
await engine.deletePage(slugs[j], deleteScopedOpts);
|
||||
pagesAffected.push(slugs[j]);
|
||||
await markCompleted(batch[j]);
|
||||
} catch (perSlugErr) {
|
||||
failedFiles.push({
|
||||
path: batch[j],
|
||||
@@ -1464,7 +1620,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
}
|
||||
}
|
||||
progress.tick(batch.length, `deletes ${Math.min(i + DELETE_BATCH_SIZE, filtered.deleted.length)}/${filtered.deleted.length}`);
|
||||
progress.tick(batch.length, `deletes ${Math.min(i + DELETE_BATCH_SIZE, deletesToDo.length)}/${deletesToDo.length}`);
|
||||
}
|
||||
} else {
|
||||
// Legacy no-sourceId path. The engine batch methods require sourceId
|
||||
@@ -1472,7 +1628,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// sourceId is unset, fall back to the original per-path loop. Slow
|
||||
// but correct; production callers all thread sourceId so this branch
|
||||
// is functionally dead post-v0.34.1.
|
||||
for (const path of filtered.deleted) {
|
||||
for (const path of deletesToDo) {
|
||||
if (opts.signal?.aborted) {
|
||||
progress.finish();
|
||||
return partial('timeout');
|
||||
@@ -1481,6 +1637,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
try {
|
||||
await engine.deletePage(slug, deleteOpts);
|
||||
pagesAffected.push(slug);
|
||||
await markCompleted(path);
|
||||
} catch (err) {
|
||||
failedFiles.push({
|
||||
path,
|
||||
@@ -1503,8 +1660,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// + embedding), so the per-iteration updateSlug + importFile loop stays;
|
||||
// only the upfront slug-resolve N+1 gets batched. The try/catch around
|
||||
// updateSlug for slug-doesn't-exist preserves verbatim.
|
||||
if (filtered.renamed.length > 0) {
|
||||
progress.start('sync.renames', filtered.renamed.length);
|
||||
// v0.42.x (#1794): resume-filter renames on the destination path.
|
||||
const renamesToDo = filtered.renamed.filter(r => !completed.has(r.to));
|
||||
if (renamesToDo.length > 0) {
|
||||
progress.start('sync.renames', renamesToDo.length);
|
||||
// v0.18.0+ multi-source: scope updateSlug so the rename only touches the
|
||||
// source-A row, not every same-slug row across sources (which would
|
||||
// either sweep them all OR violate (source_id, slug) UNIQUE).
|
||||
@@ -1518,7 +1677,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
const fromSlugByPath = new Map<string, string>();
|
||||
if (opts.sourceId) {
|
||||
const sid = opts.sourceId;
|
||||
const fromPaths = filtered.renamed.map(r => r.from);
|
||||
const fromPaths = renamesToDo.map(r => r.from);
|
||||
for (let i = 0; i < fromPaths.length; i += DELETE_BATCH_SIZE) {
|
||||
if (opts.signal?.aborted) {
|
||||
progress.finish();
|
||||
@@ -1537,7 +1696,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
}
|
||||
|
||||
for (const { from, to } of filtered.renamed) {
|
||||
for (const { from, to } of renamesToDo) {
|
||||
// v0.41.13.0 (T2 / D-V4-2): per-iteration abort check. Renames call
|
||||
// importFile() at line 1173-style sites which can be slow on big files;
|
||||
// refactor commits with 200+ renames must respect --timeout.
|
||||
@@ -1562,6 +1721,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
if (result.status === 'imported') chunksCreated += result.chunks;
|
||||
}
|
||||
pagesAffected.push(newSlug);
|
||||
await markCompleted(to);
|
||||
progress.tick(1, newSlug);
|
||||
}
|
||||
progress.finish();
|
||||
@@ -1591,15 +1751,47 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// ones. See src/core/sort-newest-first.ts for the policy.
|
||||
sortNewestFirst(addsAndMods);
|
||||
|
||||
// v0.42.x (#1794): resume-filter the import set so a resumed run only
|
||||
// processes files it hasn't already drained. This is the convergence win —
|
||||
// a killed run banks `completed`, the next run skips it (no per-file disk
|
||||
// read or content_hash DB lookup for done files).
|
||||
const importsToDo = resumeFilter(addsAndMods, [...completed]);
|
||||
|
||||
// v0.22.13 (PR #490 Q5): one source of truth for the concurrency decision.
|
||||
// engine.kind === 'pglite' → forced 1; explicit opts.concurrency wins;
|
||||
// auto path returns DEFAULT_PARALLEL_WORKERS only when fileCount > 100.
|
||||
const explicitConcurrency = opts.concurrency !== undefined;
|
||||
const effectiveConcurrency = autoConcurrency(engine, addsAndMods.length, opts.concurrency);
|
||||
const runParallel = shouldRunParallel(effectiveConcurrency, addsAndMods.length, explicitConcurrency);
|
||||
let effectiveConcurrency = autoConcurrency(engine, importsToDo.length, opts.concurrency);
|
||||
// v0.42.x (#1794, 4A): clamp the worker fan-out under GBRAIN_MAX_CONNECTIONS
|
||||
// (opt-in; no-op when unset). The parent engine holds ~resolvePoolSize()
|
||||
// connections; each parallel worker opens its own pool of
|
||||
// min(2, resolvePoolSize(2)). When the budget can't fit even one extra
|
||||
// worker, the clamp returns 1 and we fall through to the serial path
|
||||
// (parent pool only). The doctor `pool_budget` nudge covers the case where
|
||||
// the parent pool alone already exceeds the budget.
|
||||
const maxConnections = resolveMaxConnections();
|
||||
if (maxConnections !== undefined && engine.kind !== 'pglite') {
|
||||
const { resolvePoolSize } = await import('../core/db.ts');
|
||||
const parentPool = resolvePoolSize();
|
||||
const perWorkerPool = Math.min(2, resolvePoolSize(2));
|
||||
const clampResult = clampWorkersForConnectionBudget(effectiveConcurrency, {
|
||||
maxConnections,
|
||||
parentPool,
|
||||
perWorkerPool,
|
||||
});
|
||||
if (clampResult.clamped) {
|
||||
serr(
|
||||
` [sync] GBRAIN_MAX_CONNECTIONS=${maxConnections}: clamped workers ` +
|
||||
`${effectiveConcurrency} -> ${clampResult.workers} ` +
|
||||
`(parent ${parentPool} + ${clampResult.workers}x${perWorkerPool} per-worker).`,
|
||||
);
|
||||
}
|
||||
effectiveConcurrency = clampResult.workers;
|
||||
}
|
||||
const runParallel = shouldRunParallel(effectiveConcurrency, importsToDo.length, explicitConcurrency);
|
||||
|
||||
if (addsAndMods.length > 0) {
|
||||
progress.start('sync.imports', addsAndMods.length);
|
||||
if (importsToDo.length > 0) {
|
||||
progress.start('sync.imports', importsToDo.length);
|
||||
|
||||
// Core import logic shared by serial and parallel paths.
|
||||
// repoPath is validated non-null at the top of performSyncInner; narrow for TS.
|
||||
@@ -1607,15 +1799,18 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
async function importOnePath(eng: BrainEngine, path: string): Promise<void> {
|
||||
const filePath = join(syncRepoPath, path);
|
||||
if (!existsSync(filePath)) {
|
||||
// CODEX-3 (v0.22.13): a file the diff said exists at headCommit but
|
||||
// is gone from disk means the working tree has drifted (someone ran
|
||||
// `git checkout` / `git reset` mid-sync, or the file was deleted
|
||||
// post-diff). Record as a failure so last_commit does NOT advance —
|
||||
// the silent-skip-then-advance pathology was the bug.
|
||||
failedFiles.push({
|
||||
path,
|
||||
error: 'file vanished mid-sync (working tree drifted from headCommit)',
|
||||
});
|
||||
// v0.42.x (#1794, Codex #3): the diff is against the PINNED target, but
|
||||
// importFile reads the live working tree. A file added in lastCommit..pin
|
||||
// that's gone from disk was deleted by a commit AFTER the pin (normal
|
||||
// forward progress from the enrich process). It genuinely doesn't exist
|
||||
// at live HEAD, so there's nothing to import — SKIP and mark it
|
||||
// completed rather than failing the run. The post-loop pin-reachability
|
||||
// gate catches a real history REWRITE (the dangerous drift); a benign
|
||||
// forward delete is handled by the next sync's pin..HEAD diff (which
|
||||
// will show this path deleted). The pre-v0.42 "record as failure" was
|
||||
// correct only when the gate compared HEAD == captured; under pinning a
|
||||
// forward delete must not block.
|
||||
await markCompleted(path);
|
||||
progress.tick(1, `skip:${path}`);
|
||||
return;
|
||||
}
|
||||
@@ -1640,8 +1835,15 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// persist. partial() reports this so cron operators see how
|
||||
// much actually landed before --timeout fired.
|
||||
filesImported++;
|
||||
// v0.42.x (#1794): checkpoint this path so a kill banks it.
|
||||
await markCompleted(path);
|
||||
} else if (result.status === 'skipped' && (result as any).error) {
|
||||
failedFiles.push({ path, error: String((result as any).error) });
|
||||
} else {
|
||||
// status 'skipped' with no error == content_hash short-circuit
|
||||
// (already imported, unchanged). It IS done for checkpoint purposes,
|
||||
// so mark it completed (matches import-checkpoint's posture).
|
||||
await markCompleted(path);
|
||||
}
|
||||
} catch (e: unknown) {
|
||||
const msg = e instanceof Error ? e.message : String(e);
|
||||
@@ -1658,7 +1860,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// assertion if config is missing.
|
||||
const config = loadConfig();
|
||||
if (engine.kind === 'pglite' || !config?.database_url) {
|
||||
for (const path of addsAndMods) {
|
||||
for (const path of importsToDo) {
|
||||
// v0.41.13.0 (T2 / D-V3-2): per-iteration abort check. PGLite
|
||||
// serial fallback inside the parallel branch (database_url unset).
|
||||
if (opts.signal?.aborted) {
|
||||
@@ -1671,11 +1873,11 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
const { PostgresEngine } = await import('../core/postgres-engine.ts');
|
||||
const { resolvePoolSize } = await import('../core/db.ts');
|
||||
const workerPoolSize = Math.min(2, resolvePoolSize(2));
|
||||
const workerCount = Math.min(effectiveConcurrency, addsAndMods.length);
|
||||
const workerCount = Math.min(effectiveConcurrency, importsToDo.length);
|
||||
const databaseUrl = config.database_url;
|
||||
|
||||
// Q4 (v0.22.13): banner on stderr so stdout stays clean for --json.
|
||||
serr(` Parallel sync: ${workerCount} workers for ${addsAndMods.length} files`);
|
||||
serr(` Parallel sync: ${workerCount} workers for ${importsToDo.length} files`);
|
||||
|
||||
const workerEngines: InstanceType<typeof PostgresEngine>[] = [];
|
||||
try {
|
||||
@@ -1701,8 +1903,8 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// naturally (no mid-transaction kill).
|
||||
if (opts.signal?.aborted) break;
|
||||
const idx = queueIndex++;
|
||||
if (idx >= addsAndMods.length) break;
|
||||
await importOnePath(eng, addsAndMods[idx]);
|
||||
if (idx >= importsToDo.length) break;
|
||||
await importOnePath(eng, importsToDo[idx]);
|
||||
}
|
||||
}),
|
||||
);
|
||||
@@ -1722,7 +1924,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
} else {
|
||||
// Serial path (small auto diffs or explicit --workers 1).
|
||||
for (const path of addsAndMods) {
|
||||
for (const path of importsToDo) {
|
||||
// v0.41.13.0 (T2 / D-V3-2): per-iteration abort check at the
|
||||
// primary serial site.
|
||||
if (opts.signal?.aborted) {
|
||||
@@ -1746,20 +1948,40 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
}
|
||||
|
||||
// CODEX-3 (v0.22.13): head-drift gate. If git HEAD moved during the import
|
||||
// window (someone ran `git checkout` or `git pull` in another terminal /
|
||||
// sibling Conductor workspace), the chunks we just imported reflect a
|
||||
// different tree than `headCommit` claims. Refuse to advance last_commit
|
||||
// so the next sync re-walks against the new HEAD. The lock from CODEX-2
|
||||
// prevents *this* gbrain process from stepping on itself; this gate
|
||||
// catches drift caused by external `git` commands the lock cannot see.
|
||||
// v0.42.x (#1794): bank the final completed set before the gate so a block /
|
||||
// rewrite still persists everything drained this run (the next run resumes).
|
||||
await flushCheckpoint();
|
||||
|
||||
// v0.42.x (#1794, T3): pin-reachability gate, replacing the pre-v0.42 strict
|
||||
// "HEAD == captured" head-drift gate. CODEX-3 originally blocked on ANY HEAD
|
||||
// movement to catch external `git checkout`/`reset` that would make the
|
||||
// imported chunks reflect a different tree. But the #1794 repro has an enrich
|
||||
// process committing to the SAME repo every ~2 min, so the strict gate
|
||||
// blocked every run — a co-equal cause of non-convergence. Under pinning we
|
||||
// drain a FIXED lastCommit..pin range, so:
|
||||
// - HEAD == pin → nothing moved; advance.
|
||||
// - HEAD is descendant of pin (forward progress) → SAFE. The new commits
|
||||
// are outside this run's range and get picked up by the next sync's
|
||||
// pin..HEAD diff. Advance to pin.
|
||||
// - pin NOT an ancestor of HEAD (history REWRITE / reset / force-push) →
|
||||
// the tree we imported against is gone. Block; do not advance.
|
||||
try {
|
||||
const currentHead = git(repoPath, ['rev-parse', 'HEAD']);
|
||||
if (currentHead !== headCommit) {
|
||||
failedFiles.push({
|
||||
path: '<head>',
|
||||
error: `git HEAD drifted during sync: captured ${headCommit.slice(0, 8)}, now ${currentHead.slice(0, 8)}`,
|
||||
});
|
||||
if (currentHead !== pin) {
|
||||
let pinStillReachable = false;
|
||||
try {
|
||||
git(repoPath, ['merge-base', '--is-ancestor', pin, currentHead]);
|
||||
pinStillReachable = true;
|
||||
} catch {
|
||||
pinStillReachable = false;
|
||||
}
|
||||
if (!pinStillReachable) {
|
||||
failedFiles.push({
|
||||
path: '<head>',
|
||||
error: `git history rewritten during sync: pinned target ${pin.slice(0, 8)} is no longer an ancestor of HEAD ${currentHead.slice(0, 8)}`,
|
||||
});
|
||||
}
|
||||
// else: forward progress (enrich committed on top) — safe, advance to pin.
|
||||
}
|
||||
} catch (e) {
|
||||
// rev-parse failure is itself a drift signal (worktree disappeared).
|
||||
@@ -1777,7 +1999,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
// the failed files. Escape hatches: --skip-failed acknowledges the
|
||||
// current set, --retry-failed re-parses before running the normal sync.
|
||||
if (failedFiles.length > 0) {
|
||||
recordSyncFailures(failedFiles, headCommit);
|
||||
recordSyncFailures(failedFiles, pin);
|
||||
// Emit structured summary grouped by error code so the operator
|
||||
// can see *why* files failed, not just how many.
|
||||
const codeBreakdown = formatCodeBreakdown(failedFiles);
|
||||
@@ -1789,12 +2011,15 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
`'gbrain sync --skip-failed' to acknowledge and move on.`,
|
||||
);
|
||||
// Update last_run + repo_path (progress on infra) but NOT last_commit.
|
||||
// v0.42.x (#1794): the checkpoint is INTENTIONALLY left in place — the
|
||||
// banked `completed` set (flushed above) lets the next run skip the files
|
||||
// already drained and re-attempt only the failures.
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
return {
|
||||
status: 'blocked_by_failures',
|
||||
fromCommit: lastCommit,
|
||||
toCommit: headCommit,
|
||||
toCommit: pin,
|
||||
added: filtered.added.length,
|
||||
modified: filtered.modified.length,
|
||||
deleted: filtered.deleted.length,
|
||||
@@ -1816,14 +2041,31 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
}
|
||||
|
||||
// Update sync state AFTER all changes succeed (source-scoped when
|
||||
// opts.sourceId is set, global config otherwise).
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', headCommit, newestCommitMs(repoPath));
|
||||
// opts.sourceId is set, global config otherwise). v0.42.x (#1794): advance to
|
||||
// the PINNED target (not live HEAD) — commits past the pin are the next
|
||||
// sync's pin..HEAD diff. `commitTimeMs(pin)` stamps newest_content_at against
|
||||
// the commit we actually drained to. `last_sync_at` is bumped HERE and ONLY
|
||||
// here (inside writeSyncAnchor) — never on a killed partial — so the
|
||||
// autopilot scheduler never sees a stuck source as "fresh".
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'last_commit', pin, commitTimeMs(repoPath, pin));
|
||||
await engine.setConfig('sync.last_run', new Date().toISOString());
|
||||
await writeSyncAnchor(engine, opts.sourceId, 'repo_path', repoPath);
|
||||
// v0.20.0 Cathedral II Layer 12: persist the chunker version we just
|
||||
// finished with so the next sync's up_to_date gate respects it. Only
|
||||
// source-scoped syncs track this (see readChunkerVersion for rationale).
|
||||
await writeChunkerVersion(engine, opts.sourceId, String(CHUNKER_VERSION));
|
||||
// v0.42.x (#1794): import drained the full lastCommit..pin range — the anchor
|
||||
// is advanced and both checkpoint rows clear here. CONVERGENCE CONTRACT: sync
|
||||
// convergence == IMPORT convergence. Downstream (extract/facts/embed below)
|
||||
// is DELIBERATELY decoupled from the anchor: it is size-gated (inline only for
|
||||
// small syncs) and otherwise handled by its own resumable sweeps
|
||||
// (extract --stale watermark, embed --stale / embed-backfill, the extract_facts
|
||||
// + conversation_facts_backfill cycle phases). Coupling a 44k-page facts/embed
|
||||
// pass into the anchor gate would re-introduce the exact non-convergence this
|
||||
// fix exists to kill. A kill mid-downstream just means the banked pages get
|
||||
// their links/embeddings/facts from the next stale sweep.
|
||||
await clearOpCheckpoint(engine, ckpt.paths);
|
||||
await clearOpCheckpoint(engine, ckpt.target);
|
||||
|
||||
// Log ingest
|
||||
await engine.logIngest({
|
||||
@@ -1833,13 +2075,30 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
|
||||
summary: `Sync: +${filtered.added.length} ~${filtered.modified.length} -${filtered.deleted.length} R${filtered.renamed.length}, ${chunksCreated} chunks, ${elapsed}ms`,
|
||||
});
|
||||
|
||||
// Auto-extract links + timeline (always, extraction is cheap CPU).
|
||||
// Auto-extract links + timeline (cheap CPU, but skip-inline for LARGE syncs).
|
||||
// Thread opts.sourceId so the extract phase reconciles edges + timeline
|
||||
// entries against the right source — pre-fix (Data R1 HIGH 1) this phase
|
||||
// bypassed sourceId entirely and the bare-slug subquery in addTimelineEntry
|
||||
// (Data R1 HIGH 2) crashed with 21000 in multi-source brains.
|
||||
//
|
||||
// v0.42.x (#1794, T4): size-gate inline extract on `totalChanges <= 100`
|
||||
// (same threshold as noEmbed). A large sync (the #1794 case) would otherwise
|
||||
// run links+timeline extraction over tens of thousands of pages inline,
|
||||
// re-coupling a slow pass into the just-decoupled convergence path. Instead we
|
||||
// leave `links_extracted_at` UNSTAMPED so the resumable `extract --stale`
|
||||
// watermark sweep (run by the autopilot cycle / on demand) picks the pages
|
||||
// up. For resumed large syncs, pagesAffected holds only THIS run's slugs, but
|
||||
// the stale sweep scans the whole source, so banked-across-runs pages are
|
||||
// covered regardless.
|
||||
const extractOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined;
|
||||
if (!opts.noExtract && pagesAffected.length > 0) {
|
||||
if (!opts.noExtract && totalChanges > 100 && pagesAffected.length > 0) {
|
||||
slog(
|
||||
` Large sync: deferring link/timeline extraction. ` +
|
||||
`Run 'gbrain extract --stale${opts.sourceId ? ` --source-id ${opts.sourceId}` : ''}' ` +
|
||||
`(or let the autopilot cycle's extract phase sweep it).`,
|
||||
);
|
||||
}
|
||||
if (!opts.noExtract && totalChanges <= 100 && pagesAffected.length > 0) {
|
||||
try {
|
||||
const { extractLinksForSlugs, extractTimelineForSlugs, stampExtracted } = await import('./extract.ts');
|
||||
const linksCreated = await extractLinksForSlugs(engine, repoPath, pagesAffected, extractOpts);
|
||||
|
||||
@@ -137,6 +137,7 @@ export const OPS_CHECK_NAMES: ReadonlySet<string> = new Set([
|
||||
'orphan_clones',
|
||||
'pgbouncer_prepare',
|
||||
'pgvector',
|
||||
'pool_budget',
|
||||
'progressive_batch_audit_health',
|
||||
'queue_health',
|
||||
'reranker_health',
|
||||
|
||||
@@ -320,6 +320,27 @@ export function mentionsFingerprint(p: {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.42.x (#1794) — Fingerprint for resumable incremental `sync`. Completed
|
||||
* keys are repo-relative file PATHS (add/modify/delete/rename-to) drained so
|
||||
* far; a reserved sentinel entry also carries the pinned target commit (see
|
||||
* sync.ts).
|
||||
*
|
||||
* The key deliberately encodes ONLY `(sourceId, lastCommit)` — NOT the target
|
||||
* or live HEAD. `lastCommit` is the anchor, which the resumable sync advances
|
||||
* only at full completion, so the fingerprint is stable across every killed-
|
||||
* and-resumed run while the backlog (lastCommit..HEAD) grows underneath it.
|
||||
* Keying on HEAD would mint a fresh checkpoint each hour as the enrich process
|
||||
* commits, defeating resume — the exact bug this fix exists to kill.
|
||||
*/
|
||||
export function syncFingerprint(p: { sourceId?: string; lastCommit: string }): string {
|
||||
return fingerprint({
|
||||
mode: 'sync',
|
||||
source: p.sourceId ?? 'default',
|
||||
lastCommit: p.lastCommit,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Cycle's purge phase calls this to drop stale checkpoints. 7-day TTL is
|
||||
* deliberately generous — any reasonable long-running op finishes inside
|
||||
|
||||
@@ -133,6 +133,38 @@ export function newestCommitMs(localPath: string | null): number | null {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Committer time of a SPECIFIC commit SHA, in epoch ms, or `null` when not
|
||||
* determinable (non-git path, unknown/garbage sha, git unavailable, timeout).
|
||||
* Sibling of `newestCommitMs`, but pinned to an arbitrary commit instead of
|
||||
* HEAD — used by the resumable incremental sync (#1794) to stamp
|
||||
* `sources.newest_content_at` against the PINNED target commit the run drained
|
||||
* to, not whatever HEAD happens to be (HEAD may have moved past the pin via a
|
||||
* concurrent committer). For the pin == HEAD case this equals `newestCommitMs`.
|
||||
*
|
||||
* Fail-open: every error path returns `null`. Shell-injection-safe
|
||||
* (execFileSync array args), matching the `newestCommitMs` / git-head.ts posture.
|
||||
*/
|
||||
export function commitTimeMs(localPath: string | null, sha: string | null): number | null {
|
||||
if (!localPath || !sha) return null;
|
||||
try {
|
||||
const out = execFileSync('git', ['-C', localPath, 'show', '-s', '--format=%ct', sha], {
|
||||
encoding: 'utf8',
|
||||
timeout: 10_000,
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
}).trim();
|
||||
if (!out) return null;
|
||||
// `git show -s --format=%ct <sha>` prints only the committer epoch for the
|
||||
// resolved commit; take the first line in case the sha resolves to an
|
||||
// annotated-tag-ish ref that prints extra lines.
|
||||
const first = out.split('\n')[0]?.trim();
|
||||
const ms = Number(first) * 1000;
|
||||
return Number.isFinite(ms) ? ms : null;
|
||||
} catch {
|
||||
return null; // not a git repo / unknown sha / git unavailable / timeout
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Commit-relative lag in seconds from a STORED content timestamp (the
|
||||
* `newest_content_at` column), for REMOTE consumers that cannot shell out:
|
||||
|
||||
@@ -162,6 +162,52 @@ export function parseDurationSeconds(s: string | undefined, flagName: string): n
|
||||
throw new Error(`${flagName}: unsupported unit "${unit}"`);
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// v0.42.x (#1794, 4A) — connection-budget clamp.
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Resolve `GBRAIN_MAX_CONNECTIONS` (a positive integer) or undefined when
|
||||
* unset/invalid. OPT-IN: when unset, the budget clamp is a no-op and the
|
||||
* well-tuned defaults are unchanged. Operators behind a low-cap pooler
|
||||
* (Supabase Supavisor's 20-client transaction pooler is the #1794 trigger)
|
||||
* set this so a single source's worker fan-out stays under the cap instead of
|
||||
* starving retries with EMAXCONNSESSION.
|
||||
*/
|
||||
export function resolveMaxConnections(): number | undefined {
|
||||
const raw = process.env.GBRAIN_MAX_CONNECTIONS;
|
||||
if (!raw) return undefined;
|
||||
const n = Number(raw);
|
||||
return Number.isInteger(n) && n > 0 ? n : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Clamp a single sync's worker count so its connection footprint
|
||||
* (`parentPool + workers * perWorkerPool`) stays at or under `maxConnections`.
|
||||
*
|
||||
* Pure. No-op (returns `workers` unchanged, `clamped:false`) when
|
||||
* `maxConnections` is undefined — preserving every existing brain's behavior.
|
||||
* When even one worker won't fit (`parentPool + perWorkerPool > budget`),
|
||||
* returns 1: the caller's serial path uses only the parent pool, and the
|
||||
* doctor nudge tells the operator to lower `GBRAIN_POOL_SIZE`.
|
||||
*
|
||||
* Deliberately clamps only the worker COUNT (not per-worker poolSize) — the
|
||||
* dominant lever — to keep the clamp narrow per the 4A scope.
|
||||
*/
|
||||
export function clampWorkersForConnectionBudget(
|
||||
workers: number,
|
||||
opts: { maxConnections?: number; parentPool: number; perWorkerPool: number },
|
||||
): { workers: number; clamped: boolean } {
|
||||
const { maxConnections, parentPool, perWorkerPool } = opts;
|
||||
if (maxConnections === undefined || perWorkerPool <= 0) {
|
||||
return { workers, clamped: false };
|
||||
}
|
||||
const maxWorkers = Math.floor((maxConnections - parentPool) / perWorkerPool);
|
||||
if (maxWorkers < 1) return { workers: 1, clamped: workers > 1 };
|
||||
if (workers > maxWorkers) return { workers: maxWorkers, clamped: true };
|
||||
return { workers, clamped: false };
|
||||
}
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// v0.41.16.0 — D9 wrapper for bulk-command --workers.
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -185,15 +185,14 @@ describe('sync-parallel: head-drift gate (CODEX-3)', () => {
|
||||
expect(await engine.getConfig('sync.last_commit')).toBe(head);
|
||||
});
|
||||
|
||||
test('vanished-mid-sync file produces a failedFiles entry', async () => {
|
||||
test('vanished-mid-sync added file is skipped, not a failure (v0.42.x #1794)', async () => {
|
||||
// First sync: clean state for incremental.
|
||||
seedRepoWithMarkdown(repoPath, 3);
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
|
||||
// Add a file, commit, then delete the file from disk WITHOUT amending the
|
||||
// commit — diff says it exists at HEAD, but the file is gone. This is the
|
||||
// "checkout/race deleted my file mid-sync" simulation.
|
||||
// commit — diff says it exists at HEAD, but the file is gone.
|
||||
writeFileSync(join(repoPath, 'people/will-vanish.md'), [
|
||||
'---', 'type: person', 'title: Vanish', '---', '', 'body',
|
||||
].join('\n'));
|
||||
@@ -203,10 +202,17 @@ describe('sync-parallel: head-drift gate (CODEX-3)', () => {
|
||||
const result = await performSync(engine, {
|
||||
repoPath, noPull: true, noEmbed: true,
|
||||
});
|
||||
// Per CODEX-3 (v0.22.13): vanished files now go into failedFiles
|
||||
// (prior behavior was a benign skip, which let last_commit advance).
|
||||
expect(result.status).toBe('blocked_by_failures');
|
||||
expect(result.failedFiles ?? 0).toBeGreaterThan(0);
|
||||
// v0.42.x (#1794, Codex #3) SUPERSEDES the v0.22.13 CODEX-3 behavior: under
|
||||
// the pinned-target resumable sync, importFile reads the live working tree,
|
||||
// so a file added in lastCommit..pin but deleted from disk by a commit AFTER
|
||||
// the pin is normal forward progress from a concurrent committer — it
|
||||
// genuinely doesn't exist at HEAD, so SKIP it (don't block the run). A real
|
||||
// history REWRITE is caught by the separate pin-reachability gate. The
|
||||
// vanished file is never created; the next sync's pin..HEAD diff shows it
|
||||
// deleted.
|
||||
expect(result.status).toBe('synced');
|
||||
expect(result.failedFiles ?? 0).toBe(0);
|
||||
expect(await engine.getPage('people/will-vanish')).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,402 @@
|
||||
/**
|
||||
* v0.42.x (#1794) — resumable incremental sync: pinned-target path-set
|
||||
* checkpoint, last_commit advances only at full import completion.
|
||||
*
|
||||
* The defect: a large incremental sync killed mid-import never advanced
|
||||
* `last_commit`, so every retry re-walked the whole (growing) diff and the
|
||||
* backlog outran every attempt. Two co-conspirators: the strict head-drift
|
||||
* gate aborted on normal forward progress (the enrich process commits to the
|
||||
* same repo mid-sync), and per-run anchor advance would have stranded
|
||||
* modifications past the pin.
|
||||
*
|
||||
* These are the IRON-RULE regression tests. Marked `.serial.test.ts` because
|
||||
* they spawn git subprocesses, mutate `process.env.GBRAIN_SYNC_CHECKPOINT_EVERY`,
|
||||
* and share one PGLite engine across tests (PGLite forces serial workers, so
|
||||
* the batch loop is exercised without the parallel worker-pool branch).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, unlinkSync } from 'fs';
|
||||
import { execSync } from 'child_process';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import {
|
||||
loadOpCheckpoint,
|
||||
recordCompleted,
|
||||
syncFingerprint,
|
||||
} from '../src/core/op-checkpoint.ts';
|
||||
import { commitTimeMs } from '../src/core/source-health.ts';
|
||||
import {
|
||||
clampWorkersForConnectionBudget,
|
||||
resolveMaxConnections,
|
||||
} from '../src/core/sync-concurrency.ts';
|
||||
import { computePoolBudgetCheck } from '../src/commands/doctor.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let repoPath: string;
|
||||
|
||||
function gitInit(repo: string): void {
|
||||
execSync('git init', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.email "test@test.com"', { cwd: repo, stdio: 'pipe' });
|
||||
execSync('git config user.name "Test"', { cwd: repo, stdio: 'pipe' });
|
||||
}
|
||||
|
||||
function head(repo: string): string {
|
||||
return execSync('git rev-parse HEAD', { cwd: repo, stdio: 'pipe' }).toString().trim();
|
||||
}
|
||||
|
||||
function pageMd(title: string): string {
|
||||
return `---\ntype: concept\ntitle: ${title}\n---\n\nBody for ${title}.\n`;
|
||||
}
|
||||
|
||||
/** Write a markdown page under notes/, stage + commit, return new HEAD. */
|
||||
function commitPages(repo: string, files: Record<string, string>, msg: string): string {
|
||||
mkdirSync(join(repo, 'notes'), { recursive: true });
|
||||
for (const [name, body] of Object.entries(files)) {
|
||||
writeFileSync(join(repo, 'notes', name), body);
|
||||
}
|
||||
execSync('git add -A', { cwd: repo, stdio: 'pipe' });
|
||||
execSync(`git commit -m ${JSON.stringify(msg)}`, { cwd: repo, stdio: 'pipe' });
|
||||
return head(repo);
|
||||
}
|
||||
|
||||
/** Read the no-sourceId config anchor. */
|
||||
async function lastCommitConfig(): Promise<string | null> {
|
||||
return engine.getConfig('sync.last_commit');
|
||||
}
|
||||
|
||||
async function ckptPaths(lastCommit: string): Promise<string[]> {
|
||||
const fp = syncFingerprint({ lastCommit });
|
||||
return loadOpCheckpoint(engine, { op: 'sync', fingerprint: fp });
|
||||
}
|
||||
async function ckptTarget(lastCommit: string): Promise<string[]> {
|
||||
const fp = syncFingerprint({ lastCommit });
|
||||
return loadOpCheckpoint(engine, { op: 'sync-target', fingerprint: fp });
|
||||
}
|
||||
async function seedCheckpoint(lastCommit: string, target: string, paths: string[]): Promise<void> {
|
||||
const fp = syncFingerprint({ lastCommit });
|
||||
await recordCompleted(engine, { op: 'sync-target', fingerprint: fp }, [target]);
|
||||
await recordCompleted(engine, { op: 'sync', fingerprint: fp }, paths);
|
||||
}
|
||||
|
||||
describe('#1794 — resumable incremental sync (pinned target)', () => {
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
}, 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
}, 60_000);
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
repoPath = mkdtempSync(join(tmpdir(), 'gbrain-1794-'));
|
||||
gitInit(repoPath);
|
||||
// Baseline commit so the first sync has a real anchor.
|
||||
commitPages(repoPath, { 'base.md': pageMd('Base') }, 'initial');
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
delete process.env.GBRAIN_SYNC_CHECKPOINT_EVERY;
|
||||
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// ── A. CRITICAL: resume skips checkpointed paths (the mechanism) ──────────
|
||||
test('[CRITICAL] resume skips paths already in the checkpoint', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
const full = await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true });
|
||||
expect(['first_sync', 'synced']).toContain(full.status);
|
||||
const c0 = (await lastCommitConfig())!;
|
||||
expect(c0).toBeTruthy();
|
||||
|
||||
// Six new pages at C1.
|
||||
const c1 = commitPages(repoPath, {
|
||||
'a.md': pageMd('A'), 'b.md': pageMd('B'), 'c.md': pageMd('C'),
|
||||
'd.md': pageMd('D'), 'e.md': pageMd('E'), 'f.md': pageMd('F'),
|
||||
}, 'six pages');
|
||||
|
||||
// Simulate a prior killed run that drained a,b,c (pinned at C1) but DID
|
||||
// NOT import them — so if resume honors the checkpoint, a,b,c stay absent.
|
||||
await seedCheckpoint(c0, c1, ['notes/a.md', 'notes/b.md', 'notes/c.md']);
|
||||
|
||||
const res = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(res.status).toBe('synced');
|
||||
|
||||
// d,e,f imported; a,b,c skipped (proof resumeFilter honored the checkpoint).
|
||||
expect(await engine.getPage('notes/d')).not.toBeNull();
|
||||
expect(await engine.getPage('notes/e')).not.toBeNull();
|
||||
expect(await engine.getPage('notes/f')).not.toBeNull();
|
||||
expect(await engine.getPage('notes/a')).toBeNull();
|
||||
expect(await engine.getPage('notes/b')).toBeNull();
|
||||
expect(await engine.getPage('notes/c')).toBeNull();
|
||||
|
||||
// Anchor advanced to the pinned target; checkpoint cleared.
|
||||
expect(await lastCommitConfig()).toBe(c1);
|
||||
expect(await ckptPaths(c0)).toEqual([]);
|
||||
expect(await ckptTarget(c0)).toEqual([]);
|
||||
}, 60_000);
|
||||
|
||||
// ── B. CRITICAL: real abort banks progress, second run converges ──────────
|
||||
test('[CRITICAL] killed mid-import banks progress; next run converges', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
process.env.GBRAIN_SYNC_CHECKPOINT_EVERY = '1'; // flush after every file
|
||||
|
||||
await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true });
|
||||
const c0 = (await lastCommitConfig())!;
|
||||
|
||||
// Many files so the import window is wide enough for the abort to land mid-loop.
|
||||
const files: Record<string, string> = {};
|
||||
for (let i = 0; i < 30; i++) files[`p${i}.md`] = pageMd(`P${i}`);
|
||||
const c1 = commitPages(repoPath, files, 'thirty pages');
|
||||
|
||||
// Abort once the checkpoint shows at least one banked path.
|
||||
const ac = new AbortController();
|
||||
const fp = syncFingerprint({ lastCommit: c0 });
|
||||
const poll = setInterval(() => {
|
||||
loadOpCheckpoint(engine, { op: 'sync', fingerprint: fp })
|
||||
.then((paths) => { if (paths.length >= 1 && !ac.signal.aborted) ac.abort(); })
|
||||
.catch(() => {});
|
||||
}, 1);
|
||||
const aborted = await performSync(engine, { repoPath, noPull: true, noEmbed: true, signal: ac.signal });
|
||||
clearInterval(poll);
|
||||
|
||||
if (aborted.status === 'partial') {
|
||||
// Anchor MUST NOT advance on a partial.
|
||||
expect(await lastCommitConfig()).toBe(c0);
|
||||
// Banked at least one, not all (proof the kill banked progress).
|
||||
const banked = await ckptPaths(c0);
|
||||
expect(banked.length).toBeGreaterThanOrEqual(1);
|
||||
expect(banked.length).toBeLessThan(30);
|
||||
}
|
||||
|
||||
// Second run resumes and converges regardless of whether the abort landed.
|
||||
// (If the abort never fired, the first run already completed → up_to_date.)
|
||||
const second = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(['synced', 'up_to_date']).toContain(second.status);
|
||||
expect(await lastCommitConfig()).toBe(c1);
|
||||
for (let i = 0; i < 30; i++) {
|
||||
expect(await engine.getPage(`notes/p${i}`)).not.toBeNull();
|
||||
}
|
||||
// Checkpoint cleared on completion.
|
||||
expect(await ckptPaths(c0)).toEqual([]);
|
||||
}, 90_000);
|
||||
|
||||
// ── C. CRITICAL: pinned target — forward commits don't block, land next sync
|
||||
test('[CRITICAL] advances to the pinned target, not live HEAD', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true });
|
||||
const c0 = (await lastCommitConfig())!;
|
||||
|
||||
const c1 = commitPages(repoPath, { 'x.md': pageMd('X') }, 'add x');
|
||||
// Simulate a started-but-unfinished run pinned at C1 (nothing drained yet).
|
||||
await seedCheckpoint(c0, c1, []);
|
||||
// The enrich process commits FORWARD past the pin while we were "down".
|
||||
const c2 = commitPages(repoPath, { 'y.md': pageMd('Y') }, 'add y (forward drift)');
|
||||
expect(c2).not.toBe(c1);
|
||||
|
||||
// Run 1: drains C0..C1 only, advances to the PIN (C1), not live HEAD (C2).
|
||||
const r1 = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(r1.status).toBe('synced');
|
||||
expect(await engine.getPage('notes/x')).not.toBeNull();
|
||||
expect(await engine.getPage('notes/y')).toBeNull(); // past the pin, not yet
|
||||
expect(await lastCommitConfig()).toBe(c1);
|
||||
|
||||
// Run 2: now anchored at C1, diff C1..C2 picks up y.
|
||||
const r2 = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(r2.status).toBe('synced');
|
||||
expect(await engine.getPage('notes/y')).not.toBeNull();
|
||||
expect(await lastCommitConfig()).toBe(c2);
|
||||
}, 60_000);
|
||||
|
||||
// ── D. Rewrite of the pin → discard checkpoint, re-pin to HEAD ─────────────
|
||||
test('history rewrite (pin not ancestor of HEAD) discards checkpoint + re-pins', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true });
|
||||
const c0 = (await lastCommitConfig())!;
|
||||
|
||||
const c1 = commitPages(repoPath, { 'x.md': pageMd('X') }, 'add x');
|
||||
// Stale checkpoint pinned at C1 claiming a ghost path drained.
|
||||
await seedCheckpoint(c0, c1, ['notes/ghost.md']);
|
||||
|
||||
// Rewrite: reset hard back to C0, then commit a DIFFERENT file. Now C1 is
|
||||
// dangling (not an ancestor of the new HEAD).
|
||||
execSync(`git reset --hard ${c0}`, { cwd: repoPath, stdio: 'pipe' });
|
||||
const c1b = commitPages(repoPath, { 'z.md': pageMd('Z') }, 'rewritten branch');
|
||||
expect(c1b).not.toBe(c1);
|
||||
|
||||
const res = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(res.status).toBe('synced');
|
||||
// Re-pinned to the live HEAD (C1b), drained z; x and ghost never existed here.
|
||||
expect(await engine.getPage('notes/z')).not.toBeNull();
|
||||
expect(await engine.getPage('notes/x')).toBeNull();
|
||||
expect(await engine.getPage('notes/ghost')).toBeNull();
|
||||
expect(await lastCommitConfig()).toBe(c1b);
|
||||
expect(await ckptPaths(c0)).toEqual([]);
|
||||
}, 60_000);
|
||||
|
||||
// ── E + G. blocked_by_failures: last_sync_at NOT bumped, good file banked ──
|
||||
test('[Codex #2/#5] blocked sync leaves last_sync_at + last_commit unchanged; good file banked', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
const sid = 'srcE';
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path) VALUES ($1, $2, $3)`,
|
||||
[sid, sid, repoPath],
|
||||
);
|
||||
|
||||
await performSync(engine, { repoPath, sourceId: sid, full: true, noPull: true, noEmbed: true });
|
||||
const beforeRows = await engine.executeRaw<{ last_commit: string | null; last_sync_at: string | null }>(
|
||||
`SELECT last_commit, last_sync_at FROM sources WHERE id = $1`, [sid],
|
||||
);
|
||||
const c0 = beforeRows[0].last_commit!;
|
||||
const t0 = beforeRows[0].last_sync_at;
|
||||
expect(c0).toBeTruthy();
|
||||
expect(t0).toBeTruthy();
|
||||
|
||||
// One good page + one with a frontmatter slug that doesn't match the
|
||||
// path-derived slug → SLUG_MISMATCH import failure (the same reliable
|
||||
// failure the e2e sync test uses).
|
||||
mkdirSync(join(repoPath, 'notes'), { recursive: true });
|
||||
writeFileSync(join(repoPath, 'notes/good.md'), pageMd('Good'));
|
||||
writeFileSync(
|
||||
join(repoPath, 'notes/bad.md'),
|
||||
['---', 'type: concept', 'title: Bad', 'slug: wrong-slug', '---', '', 'Body.'].join('\n'),
|
||||
);
|
||||
execSync('git add -A && git commit -m "good + bad"', { cwd: repoPath, stdio: 'pipe' });
|
||||
|
||||
const res = await performSync(engine, { repoPath, sourceId: sid, noPull: true, noEmbed: true });
|
||||
expect(res.status).toBe('blocked_by_failures');
|
||||
|
||||
// Codex #2: blocked path writes neither last_commit NOR last_sync_at.
|
||||
const afterRows = await engine.executeRaw<{ last_commit: string | null; last_sync_at: string | null }>(
|
||||
`SELECT last_commit, last_sync_at FROM sources WHERE id = $1`, [sid],
|
||||
);
|
||||
expect(afterRows[0].last_commit).toBe(c0);
|
||||
// last_sync_at is a Date instance; compare by value (unchanged == #2 fix).
|
||||
expect(String(afterRows[0].last_sync_at)).toBe(String(t0));
|
||||
|
||||
// Codex #5 / banking: the good file imported + is banked; the bad one isn't.
|
||||
expect(await engine.getPage('notes/good', { sourceId: sid })).not.toBeNull();
|
||||
const banked = await (async () => {
|
||||
const fp = syncFingerprint({ sourceId: sid, lastCommit: c0 });
|
||||
return loadOpCheckpoint(engine, { op: 'sync', fingerprint: fp });
|
||||
})();
|
||||
expect(banked).toContain('notes/good.md');
|
||||
expect(banked).not.toContain('notes/bad.md');
|
||||
}, 60_000);
|
||||
|
||||
// ── F. Codex #3: file added in range but deleted from disk → skip, not block
|
||||
test('[Codex #3] vanished-on-disk added file is skipped, not a failure', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true });
|
||||
const c1 = commitPages(repoPath, { 'keep.md': pageMd('Keep'), 'gone.md': pageMd('Gone') }, 'two pages');
|
||||
|
||||
// Delete gone.md from disk WITHOUT committing — it's still 'added' in the
|
||||
// C0..C1 diff, but importFile won't find it (forward-delete simulation).
|
||||
unlinkSync(join(repoPath, 'notes/gone.md'));
|
||||
|
||||
const res = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(res.status).toBe('synced'); // NOT blocked_by_failures
|
||||
expect(await engine.getPage('notes/keep')).not.toBeNull();
|
||||
expect(await engine.getPage('notes/gone')).toBeNull();
|
||||
expect(await lastCommitConfig()).toBe(c1);
|
||||
}, 60_000);
|
||||
|
||||
// ── H. Dry-run shape unchanged; totalChanges==0 advances to pin ───────────
|
||||
test('--dry-run reports counts without writing; non-syncable-only diff advances anchor', async () => {
|
||||
const { performSync } = await import('../src/commands/sync.ts');
|
||||
|
||||
await performSync(engine, { repoPath, full: true, noPull: true, noEmbed: true });
|
||||
const c0 = (await lastCommitConfig())!;
|
||||
|
||||
commitPages(repoPath, { 'dry.md': pageMd('Dry') }, 'add dry');
|
||||
const dry = await performSync(engine, { repoPath, noPull: true, noEmbed: true, dryRun: true });
|
||||
expect(dry.status).toBe('dry_run');
|
||||
expect(dry.added).toBeGreaterThanOrEqual(1);
|
||||
expect(await lastCommitConfig()).toBe(c0); // dry-run wrote nothing
|
||||
expect(await engine.getPage('notes/dry')).toBeNull();
|
||||
|
||||
// First, sync the dry.md commit so the anchor is past it.
|
||||
await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(await engine.getPage('notes/dry')).not.toBeNull();
|
||||
|
||||
// Commit only a NON-syncable file (.txt) → filtered totalChanges == 0 →
|
||||
// advance to pin + clear checkpoint, status up_to_date.
|
||||
mkdirSync(join(repoPath, 'notes'), { recursive: true });
|
||||
writeFileSync(join(repoPath, 'notes/readme.txt'), 'not syncable');
|
||||
execSync('git add -A && git commit -m "txt only"', { cwd: repoPath, stdio: 'pipe' });
|
||||
const cTxt = head(repoPath);
|
||||
|
||||
const res = await performSync(engine, { repoPath, noPull: true, noEmbed: true });
|
||||
expect(['up_to_date', 'synced']).toContain(res.status);
|
||||
expect(await lastCommitConfig()).toBe(cTxt);
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
// ───────────────────────────────────────────────────────────────────────────
|
||||
// Pure helpers — no PGLite, no git (fail-open cases) needed.
|
||||
// ───────────────────────────────────────────────────────────────────────────
|
||||
describe('#1794 pure helpers', () => {
|
||||
test('syncFingerprint is stable on (sourceId, lastCommit) and varies otherwise', () => {
|
||||
const a = syncFingerprint({ sourceId: 's', lastCommit: 'abc' });
|
||||
expect(syncFingerprint({ sourceId: 's', lastCommit: 'abc' })).toBe(a);
|
||||
expect(syncFingerprint({ sourceId: 's', lastCommit: 'def' })).not.toBe(a);
|
||||
expect(syncFingerprint({ sourceId: 't', lastCommit: 'abc' })).not.toBe(a);
|
||||
// sourceId undefined defaults to 'default' but stays stable.
|
||||
const u = syncFingerprint({ lastCommit: 'abc' });
|
||||
expect(syncFingerprint({ lastCommit: 'abc' })).toBe(u);
|
||||
});
|
||||
|
||||
test('commitTimeMs fails open to null on bad input', () => {
|
||||
expect(commitTimeMs(null, 'abc')).toBeNull();
|
||||
expect(commitTimeMs('/definitely/not/a/repo', 'abc')).toBeNull();
|
||||
});
|
||||
|
||||
test('clampWorkersForConnectionBudget — no-op when budget unset', () => {
|
||||
expect(clampWorkersForConnectionBudget(4, { parentPool: 10, perWorkerPool: 2 }))
|
||||
.toEqual({ workers: 4, clamped: false });
|
||||
});
|
||||
|
||||
test('clampWorkersForConnectionBudget — clamps to fit budget', () => {
|
||||
// budget 16, parent 10, perWorker 2 → room for (16-10)/2 = 3 workers.
|
||||
expect(clampWorkersForConnectionBudget(4, { maxConnections: 16, parentPool: 10, perWorkerPool: 2 }))
|
||||
.toEqual({ workers: 3, clamped: true });
|
||||
// already fits → unchanged.
|
||||
expect(clampWorkersForConnectionBudget(2, { maxConnections: 16, parentPool: 10, perWorkerPool: 2 }))
|
||||
.toEqual({ workers: 2, clamped: false });
|
||||
// parent pool eats the whole budget → fall back to serial (1).
|
||||
expect(clampWorkersForConnectionBudget(4, { maxConnections: 10, parentPool: 10, perWorkerPool: 2 }))
|
||||
.toEqual({ workers: 1, clamped: true });
|
||||
});
|
||||
|
||||
test('resolveMaxConnections parses env, ignores garbage', () => {
|
||||
const prev = process.env.GBRAIN_MAX_CONNECTIONS;
|
||||
try {
|
||||
delete process.env.GBRAIN_MAX_CONNECTIONS;
|
||||
expect(resolveMaxConnections()).toBeUndefined();
|
||||
process.env.GBRAIN_MAX_CONNECTIONS = '20';
|
||||
expect(resolveMaxConnections()).toBe(20);
|
||||
process.env.GBRAIN_MAX_CONNECTIONS = 'nope';
|
||||
expect(resolveMaxConnections()).toBeUndefined();
|
||||
process.env.GBRAIN_MAX_CONNECTIONS = '0';
|
||||
expect(resolveMaxConnections()).toBeUndefined();
|
||||
} finally {
|
||||
if (prev === undefined) delete process.env.GBRAIN_MAX_CONNECTIONS;
|
||||
else process.env.GBRAIN_MAX_CONNECTIONS = prev;
|
||||
}
|
||||
});
|
||||
|
||||
test('computePoolBudgetCheck — ok when unset, warn when parent pool eats budget', () => {
|
||||
expect(computePoolBudgetCheck(undefined, 10, 2).status).toBe('ok');
|
||||
expect(computePoolBudgetCheck(20, 10, 2).status).toBe('ok');
|
||||
expect(computePoolBudgetCheck(10, 10, 2).status).toBe('warn'); // 10+2 > 10
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user