v0.32.4 feat: add sync_freshness check to gbrain doctor (#872)

* feat: add sync freshness check to gbrain doctor

- Add checkSyncFreshness function to detect stale sources
- Check all sources with local_path for sync staleness
- Warn if > 24 hours, fail if > 72 hours since last sync
- Include page count drift detection (best-effort)
- Add check to both remote and local doctor flows
- Provides actionable error messages with gbrain sync commands

* chore: bump version and changelog (v0.32.4)

sync_freshness check ships in v0.32.4 — adds detection for stale federated
sources (warn at 24h, fail at 72h) plus best-effort filesystem-vs-DB drift
detection. Surfaces in both runDoctor (local) and doctorReportRemote
(thin-client).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* feat: rewrite sync_freshness as staleness-only + env overrides + 12 tests

Strip the inline FS-walk drift detector from checkSyncFreshness. Codex
outside-voice review during plan-eng-review caught that doctorReportRemote
runs in the HTTP MCP server (src/commands/serve-http.ts), so walking
DB-supplied sources.local_path values from a remotely-callable endpoint
crosses a trust boundary — an OAuth write-scoped client could mutate
local_path and probe arbitrary server filesystem paths via timing/count
signal. Drift detection belongs in the existing multi_source_drift check
which already has GBRAIN_DRIFT_LIMIT + GBRAIN_DRIFT_TIMEOUT_MS guards.

Functional fixes folded in:
- Future-last_sync_at now warns ("clock skew or corrupted timestamp")
  instead of silently falling through as ok. Negative ageMs previously
  skipped both threshold tests.
- GBRAIN_SYNC_FRESHNESS_WARN_HOURS / GBRAIN_SYNC_FRESHNESS_FAIL_HOURS
  env vars override the 24h / 72h defaults. Invalid values (NaN, <=0)
  fall back to defaults with a once-per-process stderr warn.
- Failure messages embed source.id so `gbrain sync --source <id>` matches
  the user's copy-paste (was source.name, which doesn't match the CLI flag).

checkSyncFreshness is now exported so tests can target it directly,
mirroring the takesWeightGridCheck pattern at doctor.ts:89.

12 unit tests in test/doctor.test.ts cover every branch:
empty sources, never-synced, >72h fail, 72h boundary, 24-72h warn,
24h boundary, <24h ok, future timestamp, mixed sources (highest severity
wins), executeRaw throws -> outer-catch warn, env override fires at 7h,
source.id regression.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: refresh v0.32.4 CHANGELOG + CLAUDE.md to match staleness-only scope

Drop the filesystem-vs-DB drift detector description from the CHANGELOG
entry. Document the env-var overrides (GBRAIN_SYNC_FRESHNESS_WARN_HOURS /
GBRAIN_SYNC_FRESHNESS_FAIL_HOURS), the future-timestamp warn behavior,
the source.id-in-message fix, and the codex-surfaced trust-boundary
rationale for stripping drift out of scope.

CLAUDE.md doctor.ts annotation updated to reflect the simpler surface
plus the 12 pinning tests.

llms-full.txt regenerated to track the CLAUDE.md edit (mandatory per
CLAUDE.md rule).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
garrytan-agents
2026-05-11 21:18:17 -07:00
committed by GitHub
co-authored by garrytan-agents Garry Tan Claude Opus 4.7
parent 7be17261bc
commit 59d077f1f2
8 changed files with 390 additions and 5 deletions
+57
View File
@@ -2,6 +2,62 @@
All notable changes to GBrain will be documented in this file.
## [0.32.4] - 2026-05-11
**`gbrain doctor` now catches the silent failure mode where sync hasn't run in days.**
**One new check, configurable thresholds, no surprises when clocks lie.**
Brain search becoming stale because `gbrain sync` quietly stopped running is one of the most common "agent is missing recent pages" failure modes. The cron job dies. The watcher unloads. The autopilot daemon wedges. The user finds out a week later when an agent can't find a meeting they had three days ago. v0.32.4 adds a `sync_freshness` check to both the local `gbrain doctor` and the remote-MCP `doctorReportRemote` so the staleness surfaces the next time anyone runs doctor.
The check queries `sources.last_sync_at` for every source with a `local_path` (the federated sources that actually sync from disk). Sources synced less than 24h ago are fine. Between 24h and 72h, the check warns. Past 72h — or never synced at all — it fails. Future timestamps from clock skew or corrupted DB writes also warn (instead of silently passing). The failure message embeds the source id so `gbrain sync --source <id>` is a clean copy-paste.
Defaults aren't always right. Weekly-sync teams want a 7d fail threshold. Hourly CI brains want 6h. Two env vars override:
```bash
GBRAIN_SYNC_FRESHNESS_WARN_HOURS=24 # default
GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=72 # default
```
### What this means for operators
Your next `gbrain doctor` either says `[OK] sync_freshness: All N federated source(s) synced recently` (you're fine) or names the stale source by id with a copy-pasteable fix command. Doctor goes from "everything is fine!" (while the agent silently misses three days of meetings) to "Source 'gstack' last synced 4d ago — brain search is stale!". The check runs in both the local doctor and the remote-MCP doctor, so thin-client deployments inherit the same surfacing without extra plumbing.
The check is pure staleness — no filesystem access, no expensive walks. `doctorReportRemote` runs inside the HTTP MCP server, and walking arbitrary server filesystem paths from a remotely-callable endpoint would cross a trust boundary. Filesystem-vs-DB page drift detection is intentionally out of scope here; that work belongs in the existing `multi_source_drift` check which already has `GBRAIN_DRIFT_LIMIT` and `GBRAIN_DRIFT_TIMEOUT_MS` guards. A future PR resurrects drift detection with proper guards, slug normalization tests, and a meta-file allow-list.
### To take advantage of v0.32.4
`gbrain upgrade` ships the binary. No migration, no config:
1. **Run doctor:**
```bash
gbrain doctor
```
Look for the new `sync_freshness` row. If it warns or fails, run `gbrain sync --source <id>` per the message.
2. **Thin-client users:** `gbrain remote doctor` includes the same check end-to-end through MCP.
3. **Customize thresholds** if the defaults don't fit your workflow:
```bash
# Weekly-sync project: fail only past 7 days
GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=168 gbrain doctor
# CI brain: fail at 4 hours
GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=4 gbrain doctor
```
4. **No breaking changes.** The existing doctor surface is unchanged; this is one additive row.
### Itemized changes
#### Added
- `sync_freshness` check in both `runDoctor` (local) and `doctorReportRemote` (thin-client) at `src/commands/doctor.ts:checkSyncFreshness`. Warn at 24h, fail at 72h. Names the source by id so the printed fix command (`gbrain sync --source <id>`) matches the user's copy-paste.
- Future-`last_sync_at` is now a `warn` ("clock skew or corrupted timestamp") instead of silently falling through as `ok`. Negative ageMs from a future timestamp used to skip both threshold tests.
- `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` and `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` env vars override the 24h / 72h defaults. Invalid values (NaN, ≤0) fall back to defaults with a once-per-process stderr warn.
- 12 unit tests in `test/doctor.test.ts` covering every branch: empty sources, never-synced, >72h fail with day-rounded message, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future timestamp, mixed sources (highest severity wins), executeRaw throws, env-var override, source.id-in-message regression.
- `checkSyncFreshness` exported from `doctor.ts` so tests can target it directly (mirrors the pattern used by `takesWeightGridCheck`).
#### Out of scope (deferred)
- Filesystem-vs-DB page drift detection inside `sync_freshness`. Codex outside-voice review caught that walking DB-supplied `local_path` from `doctorReportRemote` crosses a trust boundary (the HTTP MCP server can receive remote-mutated `sources.local_path` values). Drift detection will land separately, integrated with `multi_source_drift`'s existing `GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS` guards, with slug normalization tests and a meta-file allow-list, LOCAL-DOCTOR-ONLY.
## [0.32.3.0] - 2026-05-11
**Compress a 25KB AGENTS.md down to 13KB without losing routing accuracy.**
@@ -165,6 +221,7 @@ If you're on a thin-client install (no `sources.local_path`), facts still write
#### For contributors
The CI gate's function-scoped allow-list is the structural mechanism that prevents future regressions from re-introducing the DB-only failure mode. New code that needs to call a derived-table writer must either route through the existing extract / reconcile / migration layer OR carry an explicit allow-list comment with a justification.
## [0.32.0] - 2026-05-10
**5 new embedding providers + the discoverability fix that closes the 17-PR dupe cluster.**
+1 -1
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -1 +1 @@
0.32.3.0
0.32.4
+28 -1
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.32.3.0",
"version": "0.32.4",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
+1 -1
View File
@@ -303,7 +303,7 @@ This skill guarantees:
- Routing matches the canonical triggers in the frontmatter.
- Compression is only performed when the preconditions in Step 1 pass (file ≥12KB AND clean working tree, or `--force`).
- The mandatory verification gate in Step 6 fires on the user's edited file, not on sample variants. The user runs `gbrain routing-eval --json` AND the gbrain-repo harness (`node harness.mjs --variants-dir <tmp> --variants my-edit`) before committing the compressed file.
- Privacy contract preserved: no fork-specific filesystem path literals (`/data/brain/`, `/data/.openclaw/`) leak into the compressed output.
- Privacy contract preserved: no fork-specific filesystem path literals (server-side brain home, OpenClaw fork home) leak into the compressed output.
The full behavior contract is documented in the body sections above; this section exists for the conformance test.
+148
View File
@@ -344,6 +344,9 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
// surfacing layer.)
checks.push(await checkSubagentProvider(engine));
// 6. Sync freshness check
checks.push(await checkSyncFreshness(engine));
return computeDoctorReport(checks);
}
@@ -393,6 +396,145 @@ async function checkSubagentProvider(engine: BrainEngine): Promise<Check> {
}
}
// Module-scoped flag so the NaN-fallback warning fires once per process.
let _syncFreshnessEnvWarned = false;
function _resolveSyncFreshnessHours(varName: string, fallback: number): number {
const raw = process.env[varName];
if (raw === undefined || raw === '') return fallback;
const n = Number(raw);
if (!Number.isFinite(n) || n <= 0) {
if (!_syncFreshnessEnvWarned) {
_syncFreshnessEnvWarned = true;
console.warn(
`[gbrain doctor] Ignoring invalid ${varName}=${raw}; using default ${fallback}h.`,
);
}
return fallback;
}
return n;
}
/**
* Sync freshness check (v0.32.4) — verify that sources with local_path have
* been synced recently. Detects the silent failure mode where `gbrain sync`
* stopped running and brain search now misses recent pages.
*
* Pure staleness check. Reads `sources.last_sync_at` only — no filesystem
* access. Filesystem-vs-DB drift detection is intentionally out of scope:
* - doctorReportRemote runs in the HTTP MCP server (src/commands/serve-http.ts);
* walking arbitrary DB-supplied paths from a remote-callable endpoint
* crosses a trust boundary (OAuth write scope could mutate local_path).
* - Drift detection belongs in `multi_source_drift` which already has
* GBRAIN_DRIFT_LIMIT + GBRAIN_DRIFT_TIMEOUT_MS guards.
*
* Thresholds (env-overridable, default = 24h warn / 72h fail):
* - GBRAIN_SYNC_FRESHNESS_WARN_HOURS
* - GBRAIN_SYNC_FRESHNESS_FAIL_HOURS
* Invalid values (NaN, ≤0) fall back to defaults with a once-per-process warn.
*
* Edge cases handled:
* - last_sync_at IS NULL → fail "never synced"
* - last_sync_at > now() (clock skew / corrupted timestamp) → warn
* - mixed sources → highest-severity drives the overall status
* - executeRaw throws → outer-catch warn so doctor keeps running
*
* Failure messages embed `source.id` so the fix command
* `gbrain sync --source <id>` matches what the user copy-pastes.
*/
export async function checkSyncFreshness(engine: BrainEngine): Promise<Check> {
try {
const sources = await engine.executeRaw<{
id: string;
name: string;
local_path: string | null;
last_sync_at: Date | null;
}>(
`SELECT id, name, local_path, last_sync_at FROM sources WHERE local_path IS NOT NULL`,
);
if (sources.length === 0) {
return {
name: 'sync_freshness',
status: 'ok',
message: 'No federated sources to sync',
};
}
const warnHours = _resolveSyncFreshnessHours('GBRAIN_SYNC_FRESHNESS_WARN_HOURS', 24);
const failHours = _resolveSyncFreshnessHours('GBRAIN_SYNC_FRESHNESS_FAIL_HOURS', 72);
const warnMs = warnHours * 60 * 60 * 1000;
const failMs = failHours * 60 * 60 * 1000;
const now = Date.now();
const issues: string[] = [];
let hasWarnings = false;
let hasFailures = false;
for (const source of sources) {
// Embed source.id in user-visible messages so `gbrain sync --source <id>`
// matches what the user copy-pastes. Show display name in parens when set.
const display = source.name && source.name !== source.id
? `'${source.id}' (${source.name})`
: `'${source.id}'`;
if (!source.last_sync_at) {
issues.push(`Source ${display} has never been synced`);
hasFailures = true;
continue;
}
const lastSync = new Date(source.last_sync_at).getTime();
const ageMs = now - lastSync;
if (ageMs < 0) {
issues.push(
`Source ${display} has future last_sync_at — clock skew or corrupted timestamp`,
);
hasWarnings = true;
continue;
}
const ageHours = Math.floor(ageMs / (1000 * 60 * 60));
const ageDays = Math.floor(ageHours / 24);
if (ageMs > failMs) {
issues.push(`Source ${display} last synced ${ageDays}d ago — brain search is stale!`);
hasFailures = true;
} else if (ageMs > warnMs) {
issues.push(`Source ${display} last synced ${ageHours}h ago`);
hasWarnings = true;
}
}
if (hasFailures) {
return {
name: 'sync_freshness',
status: 'fail',
message: `${issues.join('; ')}. Run \`gbrain sync --source <id>\` for each stale source`,
};
}
if (hasWarnings) {
return {
name: 'sync_freshness',
status: 'warn',
message: `${issues.join('; ')}. Run \`gbrain sync --source <id>\` to refresh`,
};
}
return {
name: 'sync_freshness',
status: 'ok',
message: `All ${sources.length} federated source(s) synced recently`,
};
} catch (e) {
return {
name: 'sync_freshness',
status: 'warn',
message: `Could not check sync freshness: ${e instanceof Error ? e.message : String(e)}`,
};
}
}
/**
* Run doctor with filesystem-first, DB-second architecture.
* Filesystem checks (resolver, conformance) run without engine.
@@ -2008,6 +2150,12 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
} catch { /* config table missing on a very old brain — skip */ }
}
// Sync freshness check (v0.32 — Check that sources are synced recently)
if (engine !== null) {
progress.heartbeat('sync_freshness');
checks.push(await checkSyncFreshness(engine));
}
progress.finish();
const hasFail = outputResults(checks, jsonOutput);
+153
View File
@@ -425,3 +425,156 @@ describe('v0.31.8 — wedge migration force-retry hint (D19)', () => {
expect(remoteBlock).toMatch(/WEDGED MIGRATION\(s\) on brain host/);
});
});
// ============================================================================
// v0.32.4 — sync_freshness check
// ============================================================================
// Pure staleness probe: reads sources.last_sync_at, no filesystem access.
// Drift detection was stripped in v0.32.4 — the doctorReportRemote path runs
// in the HTTP MCP server and walking DB-supplied local_path values from there
// crosses a trust boundary. Drift belongs in multi_source_drift's existing
// guard infrastructure (GBRAIN_DRIFT_LIMIT / GBRAIN_DRIFT_TIMEOUT_MS).
// ============================================================================
describe('v0.32.4 — sync_freshness check', () => {
// Stub engine: only checkSyncFreshness's executeRaw matters. Per-case rows
// shape is `{id, name, local_path, last_sync_at}`.
function makeStubEngine(rows: any[]): any {
return { executeRaw: async () => rows };
}
function agoMs(ms: number): Date {
return new Date(Date.now() - ms);
}
test('empty sources → ok with no-federated-sources message', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
const result = await checkSyncFreshness(makeStubEngine([]));
expect(result.name).toBe('sync_freshness');
expect(result.status).toBe('ok');
expect(result.message).toBe('No federated sources to sync');
});
test('last_sync_at IS NULL → fail with "never been synced"', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: null },
]));
expect(result.status).toBe('fail');
expect(result.message).toContain('never been synced');
expect(result.message).toContain(`'wiki'`); // source.id embedded
expect(result.message).toContain('gbrain sync --source <id>');
});
test('last_sync_at > 72h ago → fail with day-rounded "Nd ago"', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(4 * 24 * 60 * 60 * 1000) },
]));
expect(result.status).toBe('fail');
expect(result.message).toMatch(/4d ago/);
expect(result.message).toContain('brain search is stale');
});
test('exact 72h boundary → warn (>72h strict; 72h source NOT yet fail)', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
// Exactly 72h. Strict `>` on fail threshold means 72h-stale is still in
// the warn window. (Tested boundary semantics.)
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(72 * 60 * 60 * 1000) },
]));
expect(result.status).toBe('warn');
expect(result.message).toContain('72h ago');
});
test('24h < last_sync_at < 72h → warn with hour-rounded "Nh ago"', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(30 * 60 * 60 * 1000) },
]));
expect(result.status).toBe('warn');
expect(result.message).toMatch(/30h ago/);
});
test('exact 24h boundary → ok (>24h strict)', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
// Exactly 24h. Strict `>` on warn threshold means 24h-stale is still ok.
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(24 * 60 * 60 * 1000) },
]));
expect(result.status).toBe('ok');
expect(result.message).toContain('synced recently');
});
test('last_sync_at <= 24h → ok with "synced recently"', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(2 * 60 * 60 * 1000) },
{ id: 'gstack', name: '', local_path: '/tmp/gstack', last_sync_at: agoMs(60 * 1000) },
]));
expect(result.status).toBe('ok');
expect(result.message).toContain('2 federated source(s)');
});
test('future last_sync_at → warn (clock skew / corrupted timestamp)', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
// 10 min in the future. Negative ageMs must NOT fall through as ok.
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: new Date(Date.now() + 10 * 60 * 1000) },
]));
expect(result.status).toBe('warn');
expect(result.message).toMatch(/future last_sync_at/);
expect(result.message).toMatch(/clock skew|corrupted timestamp/);
});
test('mixed sources (one fail + one warn) → fail with both issues listed', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(5 * 24 * 60 * 60 * 1000) },
{ id: 'gstack', name: '', local_path: '/tmp/gstack', last_sync_at: agoMs(30 * 60 * 60 * 1000) },
]));
expect(result.status).toBe('fail');
expect(result.message).toContain(`'wiki'`);
expect(result.message).toContain(`'gstack'`);
expect(result.message).toMatch(/5d ago/);
expect(result.message).toMatch(/30h ago/);
});
test('executeRaw throws → outer-catch returns warn (doctor keeps running)', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
const engine: any = {
executeRaw: async () => { throw new Error('connection refused'); },
};
const result = await checkSyncFreshness(engine);
expect(result.status).toBe('warn');
expect(result.message).toContain('Could not check sync freshness');
expect(result.message).toContain('connection refused');
});
test('env-var override: GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6 → 7h-stale fails', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
const prev = process.env.GBRAIN_SYNC_FRESHNESS_FAIL_HOURS;
process.env.GBRAIN_SYNC_FRESHNESS_FAIL_HOURS = '6';
try {
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(7 * 60 * 60 * 1000) },
]));
expect(result.status).toBe('fail');
expect(result.message).toContain('brain search is stale');
} finally {
if (prev === undefined) delete process.env.GBRAIN_SYNC_FRESHNESS_FAIL_HOURS;
else process.env.GBRAIN_SYNC_FRESHNESS_FAIL_HOURS = prev;
}
});
test('source.id embedded in messages even when source.name is set', async () => {
const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
const result = await checkSyncFreshness(makeStubEngine([
{ id: 'wiki-id', name: 'My Wiki', local_path: '/tmp/wiki', last_sync_at: null },
]));
expect(result.status).toBe('fail');
// User copy-pastes `gbrain sync --source wiki-id` (NOT "My Wiki"). Message
// must include the id so the CLI command actually works.
expect(result.message).toContain(`'wiki-id'`);
});
});