mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(sync): migration v98 last_refreshed_at + deleteLockRowIfStale helper Schema foundation for v0.41.15.0's `gbrain sync --break-lock --max-age <s>` flag. Adds `gbrain_cycle_locks.last_refreshed_at TIMESTAMPTZ` as the heartbeat signal that distinguishes wedged-but-alive lock holders from healthy long-running syncs that are actively refreshing. Why last_refreshed_at not acquired_at: `withRefreshingLock` already bumps `ttl_expires_at` every ~5 min while work runs, but leaves `acquired_at` at the original timestamp. A 35-min media-corpus sync that's healthy has `acquired_at` 35 min ago but `last_refreshed_at` 30 seconds ago. Using acquired_at for --max-age would steal healthy locks; last_refreshed_at correctly identifies only holders whose JS interval has stopped firing. D-V4-1 rollout safety: migration v98 backfills `last_refreshed_at = NOW()` (NOT `= acquired_at`) so pre-upgrade holders running the old binary get a 30-min protection window. After that window all pre-upgrade syncs are either complete (lock released) OR genuinely wedged (--max-age does the right thing). Documented as a known caveat in CHANGELOG. D-V4-mech-4 SQL cast: deleteLockRowIfStale uses `$N * INTERVAL '1 second'` not `$N::interval` (Postgres does not cast integer to interval the latter way). Atomic DELETE keyed on (id, holder_pid, last_refreshed_at < NOW() - $N * INTERVAL '1 second') RETURNING id, last_refreshed_at — no TOCTOU between inspect + delete. D-V4-mech-3 schema-snapshot parity: column added to all 3 snapshots so fresh init paths (pglite-schema.ts, schema.sql) initialize correctly without depending on the migration runner. schema-embedded.ts regenerated via `bun run build:schema`. Pinned by 13 PGLite cases in test/sync-break-lock-all.test.ts: tryAcquireDbLock writes on INSERT, withRefreshingLock refresh bumps both columns, inspectLock surfaces the new field, deleteLockRowIfStale refuses fresh / breaks stale / safe on holder_pid mismatch / refuses NULL (pre-v98). R1 + R6 regression invariants from the v4 plan. Closes #1472 (RFC from @garrytan-agents) — schema foundation only; performSync abort threading + CLI flags + consumer threading land in follow-up commits in this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(sync): --timeout + --max-age + partial status + per-source AbortController The CLI surface for v0.41.15.0. Wires `gbrain sync --timeout <s>` (graceful self-termination) and `gbrain sync --break-lock --all --max-age <s>` (cron-self-heal) end-to-end through `performSync`, `runOne`, `runBreakLock`, and all `SyncResult.status` consumers. Surface 1: `gbrain sync --timeout <s>` - New `SyncOpts.signal?: AbortSignal` threads through `performSync` → `withRefreshingLock` work callback → `performSyncInner`. - D-V3-1 honest scope: abort checks fire ONLY in pre-bookmark phases (pull, delete, rename, import). Extract + embed run to completion if reached. The `last_commit` bookmark write at sync.ts:1261 is the invariant boundary — partial CANNOT advance the bookmark because the abort checkpoints sit strictly before that write. - D-V3-2 per-iteration: abort check at top of every loop iteration (delete, rename, serial import, each parallel worker's while loop) matches the per-file granularity the existing loops already have. - D-V3-3 per-source AbortController: `--timeout --all` creates ONE controller inside runOne per source so each gets its own budget; NOT a shared global controller (which would starve later sources). try/finally + timer.unref() guarantees cleanup on throw. - D-V4-mech-7 pull error.cause: pullRepo wraps execFileSync errors in GitOperationError. The catch inspects e.cause.code === 'ETIMEDOUT' and e.cause.signal === 'SIGTERM' (NOT the top-level error) to distinguish timeout (partial reason='pull_timeout') from ordinary pull failure (existing warn-and-continue, R2 invariant preserved). Surface 2: `gbrain sync --break-lock [--all] [--max-age <s>]` - Drops the --all refusal at sync.ts:1610. When combined with --all, runBreakLock iterates every active source and prints per-source verdict. - --max-age routes through the new deleteLockRowIfStale helper from db-lock.ts (atomic age-gated DELETE; no TOCTOU). Healthy refreshing holders survive by construction; only wedged-but-alive holders trip. D-V3-5 partial-status consumer threading (conservative posture matching blocked_by_failures): - printSyncResult: new `case 'partial':` arm reports filesImported + reason; tells operator to re-run to continue. - manageGitignore (both single-source and parallel runOne sites, plus watch mode): excludes partial from the gate. A partial sync's db_only path set isn't fully reconciled. - Auto-embed-backfill enqueue inside runOne: excludes partial. The next clean sync will re-walk and re-decide. CLI flag parsing (T16): - parseDurationSeconds in sync-concurrency.ts: accepts 60s/10m/1h/bare int; rejects 0/negatives/decimals/garbage. Names the failing flag in the error message. - --timeout requires --source OR --all (validation rejects bare `gbrain sync --timeout`). - --max-age requires --break-lock; mutually exclusive with --force-break-lock. Coverage: - 15 unit cases (test/sync-timeout.test.ts) pin parseDurationSeconds + SyncResult union additivity. - 2 E2E cases (test/e2e/sync-parallel.test.ts) pin the abort-mid-import contract against real Postgres: status='partial', last_commit unchanged, filesImported bounded. Closes #1472 (RFC from @garrytan-agents) — CLI surface; schema foundation landed in the previous commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(heavy): sync_timeout_rescue.sh reproducer for the cron-cascade 10K-page seed × 4 sources × deliberately tight --timeout × 3 sequential cron emulations. Asserts every source reaches `last_commit === HEAD` within 3 waves. Proves the v0.41.15.0 fix breaks the cascade the PR #1472 RFC documented. Workload (tests/heavy/_sync_timeout_rescue_workload.ts) is PGLite-only because the PGLite engine forces serial sync internally (parallelEligible excludes it). The parallel-fan-out + per-source AbortController case lives in test/e2e/sync-parallel.test.ts against real Postgres. This heavy test pins the contract that matters for cron: aborts → partial returns → next wave content_hash-short-circuits + makes new progress. Smoke-tested locally at PAGES=50 WAVES=2 TIMEOUT_SECONDS=2: every source converges within 2 waves. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(v0.41.15.0): CHANGELOG + README + TODOS + version bump Bumps VERSION + package.json to 0.41.15.0 (next slot after master's v0.41.14.0). CHANGELOG entry leads ELI10 per gstack voice rules and documents the 3 intentional honest gaps: 1. --timeout covers pull + delete + rename + import only; extract + embed run to completion (D-V3-1 honest scope). 2. First 30 min after migration v98, --max-age cannot identify wedged pre-upgrade holders (D-V4-1 rollout trade-off). 3. Full-sync triggers (first sync, --full, chunker-version rewalk) don't respect --timeout yet (deferred to v0.42+). README troubleshooting section: paste-ready cron pattern with shell timeout(1) for OS-level process isolation + gbrain's --timeout for graceful self-termination half-a-minute earlier. TODOS.md: v0.42+ entries for subprocess fan-out (revisit if shell timeout(1) proves insufficient), full-sync --timeout coverage via AbortSignal in runImport, and runFactsBackstop microtask-queue process-alive caveat. llms-full.txt regenerated via `bun run build:llms`. Closes #1472 (RFC from @garrytan-agents). Credit to @garrytan-agents in the CHANGELOG for surfacing the production cron-failure data that motivated the work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test-isolation): rewrite JSDoc to not match mock.module() lint regex scripts/check-test-isolation.sh greps for the literal string `mock.module(` to flag top-level module mocks (R2 rule — top-level mocks leak across files in the shard process). The regex doesn't know about comments, so my two new test files tripped the lint with JSDoc lines literally describing the rule: test/sync-timeout.test.ts:11 "* `mock.module()` (R2). Engine ..." test/sync-break-lock-all.test.ts:15 "* mock.module(), no process.env ..." Both files had ZERO actual mock.module() calls — only the comment text matched. Rewrote both JSDocs to refer to "top-level module mocks" instead of the literal token. Same meaning; doesn't trip the regex. `bun run check:test-isolation` now passes (714 non-serial unit files scanned). `bun run verify` clean (22/22 checks pass). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
249 lines
10 KiB
TypeScript
249 lines
10 KiB
TypeScript
/**
|
|
* v0.41.13.0 — PGLite tests for the sync break-lock + max-age + abort
|
|
* threading wave.
|
|
*
|
|
* Coverage diagram targets:
|
|
* - tryAcquireDbLock writes last_refreshed_at = NOW() on INSERT (R4 baseline).
|
|
* - withRefreshingLock-style refresh bumps both ttl_expires_at AND
|
|
* last_refreshed_at (R5 + new column).
|
|
* - inspectLock surfaces last_refreshed_at + ms_since_last_refresh.
|
|
* - deleteLockRowIfStale: refuses fresh, breaks stale, holder_pid mismatch
|
|
* refuses, NULL last_refreshed_at refuses (pre-v98-style row).
|
|
* - migration v98 backfills last_refreshed_at = NOW() (R6).
|
|
*
|
|
* Test isolation: canonical PGLite block per CLAUDE.md R3 + R4. No
|
|
* top-level module mocks (R2 — `mock.module` calls leak across files in
|
|
* the shard process); no process.env mutations.
|
|
*/
|
|
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
|
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
|
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
|
import {
|
|
tryAcquireDbLock,
|
|
inspectLock,
|
|
deleteLockRow,
|
|
deleteLockRowIfStale,
|
|
} from '../src/core/db-lock.ts';
|
|
|
|
let engine: PGLiteEngine;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({});
|
|
await engine.initSchema();
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
await resetPgliteState(engine);
|
|
// Also clear gbrain_cycle_locks since resetPgliteState focuses on user data
|
|
// and the lock table is per-test state we want fresh.
|
|
await engine.executeRaw('DELETE FROM gbrain_cycle_locks', []);
|
|
});
|
|
|
|
// Helper: read raw row for assertions against the new column shape.
|
|
async function readLockRow(lockId: string) {
|
|
const rows = await engine.executeRaw<{
|
|
id: string;
|
|
holder_pid: number;
|
|
acquired_at: string;
|
|
ttl_expires_at: string;
|
|
last_refreshed_at: string | null;
|
|
}>(
|
|
`SELECT id, holder_pid, acquired_at, ttl_expires_at, last_refreshed_at
|
|
FROM gbrain_cycle_locks WHERE id = $1`,
|
|
[lockId],
|
|
);
|
|
return rows[0] ?? null;
|
|
}
|
|
|
|
describe('tryAcquireDbLock writes last_refreshed_at (v0.41.13.0 T5)', () => {
|
|
test('fresh INSERT sets last_refreshed_at to a non-null timestamp', async () => {
|
|
const handle = await tryAcquireDbLock(engine, 'test:fresh-acquire', 30);
|
|
expect(handle).not.toBeNull();
|
|
const row = await readLockRow('test:fresh-acquire');
|
|
expect(row).not.toBeNull();
|
|
expect(row!.last_refreshed_at).not.toBeNull();
|
|
// The acquired_at and last_refreshed_at are set in the same INSERT
|
|
// (both NOW()) so they should be within a few ms of each other.
|
|
const acq = new Date(row!.acquired_at).getTime();
|
|
const ref = new Date(row!.last_refreshed_at!).getTime();
|
|
expect(Math.abs(acq - ref)).toBeLessThan(1000);
|
|
await handle!.release();
|
|
});
|
|
|
|
test('takeover (TTL-expired) refreshes last_refreshed_at too', async () => {
|
|
// Insert a stale lock row directly with TTL already expired AND an OLD
|
|
// last_refreshed_at so we can verify the takeover bumps it.
|
|
const oldTs = new Date(Date.now() - 60 * 60 * 1000).toISOString(); // 1h ago
|
|
await engine.executeRaw(
|
|
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at)
|
|
VALUES ($1, $2, $3, $4, $5, $4)`,
|
|
['test:takeover', 99999, 'fake-host', oldTs, oldTs],
|
|
);
|
|
const before = await readLockRow('test:takeover');
|
|
// PGLite returns timestamps as Date objects; normalize via .toISOString().
|
|
expect(new Date(before!.last_refreshed_at!).toISOString()).toBe(oldTs);
|
|
|
|
const handle = await tryAcquireDbLock(engine, 'test:takeover', 30);
|
|
expect(handle).not.toBeNull();
|
|
|
|
const after = await readLockRow('test:takeover');
|
|
expect(new Date(after!.last_refreshed_at!).toISOString()).not.toBe(oldTs);
|
|
// After takeover, last_refreshed_at should be recent.
|
|
const refMs = new Date(after!.last_refreshed_at!).getTime();
|
|
expect(Date.now() - refMs).toBeLessThan(5000);
|
|
await handle!.release();
|
|
});
|
|
|
|
test('refresh() bumps both ttl_expires_at AND last_refreshed_at', async () => {
|
|
const handle = await tryAcquireDbLock(engine, 'test:refresh', 30);
|
|
expect(handle).not.toBeNull();
|
|
const before = await readLockRow('test:refresh');
|
|
|
|
// Sleep just a hair so the timestamp changes are observable.
|
|
await new Promise(r => setTimeout(r, 50));
|
|
await handle!.refresh();
|
|
|
|
const after = await readLockRow('test:refresh');
|
|
expect(new Date(after!.ttl_expires_at).getTime()).toBeGreaterThan(
|
|
new Date(before!.ttl_expires_at).getTime(),
|
|
);
|
|
expect(new Date(after!.last_refreshed_at!).getTime()).toBeGreaterThan(
|
|
new Date(before!.last_refreshed_at!).getTime(),
|
|
);
|
|
await handle!.release();
|
|
});
|
|
});
|
|
|
|
describe('inspectLock surfaces last_refreshed_at (v0.41.13.0 T5)', () => {
|
|
test('returns last_refreshed_at + ms_since_last_refresh on a live lock', async () => {
|
|
const handle = await tryAcquireDbLock(engine, 'test:inspect', 30);
|
|
expect(handle).not.toBeNull();
|
|
const snap = await inspectLock(engine, 'test:inspect');
|
|
expect(snap).not.toBeNull();
|
|
expect(snap!.last_refreshed_at).toBeInstanceOf(Date);
|
|
expect(snap!.ms_since_last_refresh).not.toBeNull();
|
|
// Fresh acquire → ms_since_last_refresh should be tiny.
|
|
expect(snap!.ms_since_last_refresh!).toBeLessThan(5000);
|
|
await handle!.release();
|
|
});
|
|
|
|
test('returns null for last_refreshed_at when the row has NULL (pre-v98 fallback)', async () => {
|
|
await engine.executeRaw(
|
|
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at)
|
|
VALUES ('test:null-ref', 12345, 'h', NOW(), NOW() + INTERVAL '30 minutes', NULL)`,
|
|
[],
|
|
);
|
|
const snap = await inspectLock(engine, 'test:null-ref');
|
|
expect(snap).not.toBeNull();
|
|
expect(snap!.last_refreshed_at).toBeNull();
|
|
expect(snap!.ms_since_last_refresh).toBeNull();
|
|
});
|
|
|
|
test('returns null for absent lock', async () => {
|
|
const snap = await inspectLock(engine, 'test:does-not-exist');
|
|
expect(snap).toBeNull();
|
|
});
|
|
});
|
|
|
|
describe('deleteLockRowIfStale (v0.41.13.0 T4 + D-V4-mech-4/5)', () => {
|
|
test('refuses to break a fresh lock (no rows deleted)', async () => {
|
|
const handle = await tryAcquireDbLock(engine, 'test:fresh', 30);
|
|
expect(handle).not.toBeNull();
|
|
const snap = await inspectLock(engine, 'test:fresh');
|
|
|
|
// max-age 1800s (30 min); lock is fresh → refuse.
|
|
const result = await deleteLockRowIfStale(engine, 'test:fresh', snap!.holder_pid, 1800);
|
|
expect(result.deleted).toBe(false);
|
|
expect(result.lastRefreshedAt).toBeNull();
|
|
|
|
// Row still present after the no-op delete.
|
|
const after = await readLockRow('test:fresh');
|
|
expect(after).not.toBeNull();
|
|
await handle!.release();
|
|
});
|
|
|
|
test('breaks a stale lock (last_refreshed_at older than max-age)', async () => {
|
|
// Insert a row with last_refreshed_at 1 hour ago.
|
|
const oldTs = new Date(Date.now() - 60 * 60 * 1000).toISOString();
|
|
await engine.executeRaw(
|
|
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at)
|
|
VALUES ('test:stale', 54321, 'h', NOW(), NOW() + INTERVAL '30 minutes', $1)`,
|
|
[oldTs],
|
|
);
|
|
// max-age 1800s (30 min); lock has not refreshed in 1h → break.
|
|
const result = await deleteLockRowIfStale(engine, 'test:stale', 54321, 1800);
|
|
expect(result.deleted).toBe(true);
|
|
expect(result.lastRefreshedAt).toBeInstanceOf(Date);
|
|
expect(Math.abs(result.lastRefreshedAt!.getTime() - new Date(oldTs).getTime()))
|
|
.toBeLessThan(1000);
|
|
// Row gone.
|
|
expect(await readLockRow('test:stale')).toBeNull();
|
|
});
|
|
|
|
test('refuses on holder_pid mismatch (PID-safe)', async () => {
|
|
const oldTs = new Date(Date.now() - 60 * 60 * 1000).toISOString();
|
|
await engine.executeRaw(
|
|
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at)
|
|
VALUES ('test:wrong-pid', 11111, 'h', NOW(), NOW() + INTERVAL '30 minutes', $1)`,
|
|
[oldTs],
|
|
);
|
|
// Even though the lock IS stale, mismatched pid → refuse.
|
|
const result = await deleteLockRowIfStale(engine, 'test:wrong-pid', 22222, 1800);
|
|
expect(result.deleted).toBe(false);
|
|
// Row still present.
|
|
expect(await readLockRow('test:wrong-pid')).not.toBeNull();
|
|
});
|
|
|
|
test('refuses when last_refreshed_at IS NULL (pre-v98 row)', async () => {
|
|
// A row with NULL last_refreshed_at is conservatively kept alive — the
|
|
// operator should run apply-migrations or use --force-break-lock.
|
|
await engine.executeRaw(
|
|
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at)
|
|
VALUES ('test:null-ref-stale', 33333, 'h', NOW() - INTERVAL '2 hours', NOW() + INTERVAL '30 minutes', NULL)`,
|
|
[],
|
|
);
|
|
const result = await deleteLockRowIfStale(engine, 'test:null-ref-stale', 33333, 1800);
|
|
expect(result.deleted).toBe(false);
|
|
expect(await readLockRow('test:null-ref-stale')).not.toBeNull();
|
|
});
|
|
|
|
test('refuses on absent row', async () => {
|
|
const result = await deleteLockRowIfStale(engine, 'test:nonexistent', 12345, 1800);
|
|
expect(result.deleted).toBe(false);
|
|
expect(result.lastRefreshedAt).toBeNull();
|
|
});
|
|
});
|
|
|
|
describe('R1 regression: existing deleteLockRow byte-stable', () => {
|
|
test('safe deleteLockRow still works with the new column present', async () => {
|
|
const handle = await tryAcquireDbLock(engine, 'test:r1', 30);
|
|
expect(handle).not.toBeNull();
|
|
const snap = await inspectLock(engine, 'test:r1');
|
|
// Pre-v98 deleteLockRow shape (no maxAge, just id + pid).
|
|
const result = await deleteLockRow(engine, 'test:r1', snap!.holder_pid);
|
|
expect(result.deleted).toBe(true);
|
|
expect(await readLockRow('test:r1')).toBeNull();
|
|
});
|
|
});
|
|
|
|
describe('R6 regression: schema bootstrap includes last_refreshed_at column', () => {
|
|
test('CREATE TABLE shape (from pglite-schema.ts snapshot) has the column', async () => {
|
|
// information_schema.columns is the canonical introspection. If the
|
|
// column is missing, the SELECT returns 0 rows.
|
|
const rows = await engine.executeRaw<{ column_name: string; data_type: string; is_nullable: string }>(
|
|
`SELECT column_name, data_type, is_nullable
|
|
FROM information_schema.columns
|
|
WHERE table_name = 'gbrain_cycle_locks' AND column_name = 'last_refreshed_at'`,
|
|
[],
|
|
);
|
|
expect(rows).toHaveLength(1);
|
|
expect(rows[0].data_type).toMatch(/timestamp/i);
|
|
expect(rows[0].is_nullable).toBe('YES');
|
|
});
|
|
});
|