From 59d077f1f2b51b9b83ca4453fb6e123f10ec94b0 Mon Sep 17 00:00:00 2001 From: garrytan-agents Date: Mon, 11 May 2026 21:18:17 -0700 Subject: [PATCH] v0.32.4 feat: add sync_freshness check to gbrain doctor (#872) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: add sync freshness check to gbrain doctor - Add checkSyncFreshness function to detect stale sources - Check all sources with local_path for sync staleness - Warn if > 24 hours, fail if > 72 hours since last sync - Include page count drift detection (best-effort) - Add check to both remote and local doctor flows - Provides actionable error messages with gbrain sync commands * chore: bump version and changelog (v0.32.4) sync_freshness check ships in v0.32.4 — adds detection for stale federated sources (warn at 24h, fail at 72h) plus best-effort filesystem-vs-DB drift detection. Surfaces in both runDoctor (local) and doctorReportRemote (thin-client). Co-Authored-By: Claude Opus 4.7 * feat: rewrite sync_freshness as staleness-only + env overrides + 12 tests Strip the inline FS-walk drift detector from checkSyncFreshness. Codex outside-voice review during plan-eng-review caught that doctorReportRemote runs in the HTTP MCP server (src/commands/serve-http.ts), so walking DB-supplied sources.local_path values from a remotely-callable endpoint crosses a trust boundary — an OAuth write-scoped client could mutate local_path and probe arbitrary server filesystem paths via timing/count signal. Drift detection belongs in the existing multi_source_drift check which already has GBRAIN_DRIFT_LIMIT + GBRAIN_DRIFT_TIMEOUT_MS guards. Functional fixes folded in: - Future-last_sync_at now warns ("clock skew or corrupted timestamp") instead of silently falling through as ok. Negative ageMs previously skipped both threshold tests. - GBRAIN_SYNC_FRESHNESS_WARN_HOURS / GBRAIN_SYNC_FRESHNESS_FAIL_HOURS env vars override the 24h / 72h defaults. Invalid values (NaN, <=0) fall back to defaults with a once-per-process stderr warn. - Failure messages embed source.id so `gbrain sync --source ` matches the user's copy-paste (was source.name, which doesn't match the CLI flag). checkSyncFreshness is now exported so tests can target it directly, mirroring the takesWeightGridCheck pattern at doctor.ts:89. 12 unit tests in test/doctor.test.ts cover every branch: empty sources, never-synced, >72h fail, 72h boundary, 24-72h warn, 24h boundary, <24h ok, future timestamp, mixed sources (highest severity wins), executeRaw throws -> outer-catch warn, env override fires at 7h, source.id regression. Co-Authored-By: Claude Opus 4.7 * docs: refresh v0.32.4 CHANGELOG + CLAUDE.md to match staleness-only scope Drop the filesystem-vs-DB drift detector description from the CHANGELOG entry. Document the env-var overrides (GBRAIN_SYNC_FRESHNESS_WARN_HOURS / GBRAIN_SYNC_FRESHNESS_FAIL_HOURS), the future-timestamp warn behavior, the source.id-in-message fix, and the codex-surfaced trust-boundary rationale for stripping drift out of scope. CLAUDE.md doctor.ts annotation updated to reflect the simpler surface plus the 12 pinning tests. llms-full.txt regenerated to track the CLAUDE.md edit (mandatory per CLAUDE.md rule). Co-Authored-By: Claude Opus 4.7 --------- Co-authored-by: garrytan-agents Co-authored-by: Garry Tan Co-authored-by: Claude Opus 4.7 --- CHANGELOG.md | 57 +++++++++ CLAUDE.md | 2 +- VERSION | 2 +- llms-full.txt | 29 ++++- package.json | 2 +- skills/functional-area-resolver/SKILL.md | 2 +- src/commands/doctor.ts | 148 ++++++++++++++++++++++ test/doctor.test.ts | 153 +++++++++++++++++++++++ 8 files changed, 390 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 53287af0c..f9e2ba720 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,62 @@ All notable changes to GBrain will be documented in this file. +## [0.32.4] - 2026-05-11 + +**`gbrain doctor` now catches the silent failure mode where sync hasn't run in days.** +**One new check, configurable thresholds, no surprises when clocks lie.** + +Brain search becoming stale because `gbrain sync` quietly stopped running is one of the most common "agent is missing recent pages" failure modes. The cron job dies. The watcher unloads. The autopilot daemon wedges. The user finds out a week later when an agent can't find a meeting they had three days ago. v0.32.4 adds a `sync_freshness` check to both the local `gbrain doctor` and the remote-MCP `doctorReportRemote` so the staleness surfaces the next time anyone runs doctor. + +The check queries `sources.last_sync_at` for every source with a `local_path` (the federated sources that actually sync from disk). Sources synced less than 24h ago are fine. Between 24h and 72h, the check warns. Past 72h — or never synced at all — it fails. Future timestamps from clock skew or corrupted DB writes also warn (instead of silently passing). The failure message embeds the source id so `gbrain sync --source ` is a clean copy-paste. + +Defaults aren't always right. Weekly-sync teams want a 7d fail threshold. Hourly CI brains want 6h. Two env vars override: + +```bash +GBRAIN_SYNC_FRESHNESS_WARN_HOURS=24 # default +GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=72 # default +``` + +### What this means for operators + +Your next `gbrain doctor` either says `[OK] sync_freshness: All N federated source(s) synced recently` (you're fine) or names the stale source by id with a copy-pasteable fix command. Doctor goes from "everything is fine!" (while the agent silently misses three days of meetings) to "Source 'gstack' last synced 4d ago — brain search is stale!". The check runs in both the local doctor and the remote-MCP doctor, so thin-client deployments inherit the same surfacing without extra plumbing. + +The check is pure staleness — no filesystem access, no expensive walks. `doctorReportRemote` runs inside the HTTP MCP server, and walking arbitrary server filesystem paths from a remotely-callable endpoint would cross a trust boundary. Filesystem-vs-DB page drift detection is intentionally out of scope here; that work belongs in the existing `multi_source_drift` check which already has `GBRAIN_DRIFT_LIMIT` and `GBRAIN_DRIFT_TIMEOUT_MS` guards. A future PR resurrects drift detection with proper guards, slug normalization tests, and a meta-file allow-list. + +### To take advantage of v0.32.4 + +`gbrain upgrade` ships the binary. No migration, no config: + +1. **Run doctor:** + ```bash + gbrain doctor + ``` + Look for the new `sync_freshness` row. If it warns or fails, run `gbrain sync --source ` per the message. + +2. **Thin-client users:** `gbrain remote doctor` includes the same check end-to-end through MCP. + +3. **Customize thresholds** if the defaults don't fit your workflow: + ```bash + # Weekly-sync project: fail only past 7 days + GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=168 gbrain doctor + # CI brain: fail at 4 hours + GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=4 gbrain doctor + ``` + +4. **No breaking changes.** The existing doctor surface is unchanged; this is one additive row. + +### Itemized changes + +#### Added +- `sync_freshness` check in both `runDoctor` (local) and `doctorReportRemote` (thin-client) at `src/commands/doctor.ts:checkSyncFreshness`. Warn at 24h, fail at 72h. Names the source by id so the printed fix command (`gbrain sync --source `) matches the user's copy-paste. +- Future-`last_sync_at` is now a `warn` ("clock skew or corrupted timestamp") instead of silently falling through as `ok`. Negative ageMs from a future timestamp used to skip both threshold tests. +- `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` and `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` env vars override the 24h / 72h defaults. Invalid values (NaN, ≤0) fall back to defaults with a once-per-process stderr warn. +- 12 unit tests in `test/doctor.test.ts` covering every branch: empty sources, never-synced, >72h fail with day-rounded message, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future timestamp, mixed sources (highest severity wins), executeRaw throws, env-var override, source.id-in-message regression. +- `checkSyncFreshness` exported from `doctor.ts` so tests can target it directly (mirrors the pattern used by `takesWeightGridCheck`). + +#### Out of scope (deferred) +- Filesystem-vs-DB page drift detection inside `sync_freshness`. Codex outside-voice review caught that walking DB-supplied `local_path` from `doctorReportRemote` crosses a trust boundary (the HTTP MCP server can receive remote-mutated `sources.local_path` values). Drift detection will land separately, integrated with `multi_source_drift`'s existing `GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS` guards, with slug normalization tests and a meta-file allow-list, LOCAL-DOCTOR-ONLY. + ## [0.32.3.0] - 2026-05-11 **Compress a 25KB AGENTS.md down to 13KB without losing routing accuracy.** @@ -165,6 +221,7 @@ If you're on a thin-client install (no `sources.local_path`), facts still write #### For contributors The CI gate's function-scoped allow-list is the structural mechanism that prevents future regressions from re-introducing the DB-only failure mode. New code that needs to call a derived-table writer must either route through the existing extract / reconcile / migration layer OR carry an explicit allow-list comment with a justification. + ## [0.32.0] - 2026-05-10 **5 new embedding providers + the discoverability fix that closes the 17-PR dupe cluster.** diff --git a/CLAUDE.md b/CLAUDE.md index e69e58053..5502ba8e2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -162,7 +162,7 @@ strict behavior when unset. - `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. - `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. diff --git a/VERSION b/VERSION index 181d71fe3..7dd8dc580 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.32.3.0 \ No newline at end of file +0.32.4 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index c8846d249..f341baa82 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -262,7 +262,7 @@ strict behavior when unset. - `src/commands/anomalies.ts` (v0.29) — `gbrain anomalies [--since YYYY-MM-DD] [--lookback-days N] [--sigma N] [--json]`: cohort-level activity outliers. Calls `engine.findAnomalies(opts)`. Two cohort kinds in v1: tag, type. Year cohort deferred to v0.30. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses `SELECT DISTINCT ON (slug)` in a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s) while preserving `engine.getAllSlugs()`'s `Set` semantics on multi-source brains. Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. - `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -811,6 +811,31 @@ routing is narrowed to what the skill actually covers. **Skillify loop (v0.19):** skillify (the markdown orchestration), skillpack-check (agent-readable health report). +**Routing-table compression (v0.32.3.0):** `skills/functional-area-resolver/` — +two-layer dispatch pattern for shrinking large AGENTS.md / RESOLVER.md files +(>=12KB) without losing routing accuracy. Replaces one row per skill with one +entry per functional area, where each area declares its sub-skills in a +`(dispatcher for: ...)` clause. The static-prompt analog of hierarchical agent +routing (AnyTool [arXiv:2402.04253](https://arxiv.org/abs/2402.04253), RAG-MCP +[arXiv:2505.03275](https://arxiv.org/html/2505.03275v1), Anthropic Agent Skills +progressive disclosure). Empirically validated across Opus 4.7 / Sonnet 4.6 / +Haiku 4.5: +13 to +17pp over the verbose baseline at 48% the size (25KB → 13KB +on a real fork). The `(dispatcher for: ...)` clause is the load-bearing signal +— strip it and lenient accuracy collapses to 41.7% on Sonnet (the +`resolver-of-resolvers` ablation case). A/B eval surface lives at +`evals/functional-area-resolver/` (outside `skills/` deliberately so the +skillpack bundler doesn't ship eval infrastructure to downstream installs): +gateway-routed TypeScript harness, 20 training + 5 held-out fixtures, strict + +lenient scoring, three committed cross-model receipts in `baseline-runs/`. +Receipt header binds (model, prompt_template_hash, fixtures_hash, harness_sha, +ts) so future contributors can verify reproduction. Companion `rescore.mjs` +re-scores existing JSONL with lenient tolerance for zero API cost. Reproduce +with `cd evals/functional-area-resolver && node harness.mjs --model +{opus|sonnet|haiku}` (~$0.30–1.70 per model). Nine v0.33.x follow-up TODOs +filed for held-out corpus growth, cross-vendor verification, hierarchical +area-of-areas, embedding-based pre-router, and the run-1 vs run-2 +prompt-design ablation methodology. + **Operational health (v0.19.1):** smoke-test (8 post-restart health checks with auto-fix for Bun, CLI, DB, worker, Zod CJS, gateway, API key, brain repo; user-extensible via `~/.gbrain/smoke-tests.d/*.sh`). @@ -1839,6 +1864,8 @@ GBrain is those patterns, generalized. 34 skills. Install in 30 minutes. Your ag > **Embedding providers:** OpenAI is the default, but gbrain ships with **14 recipes** covering Voyage, Google Gemini, Azure OpenAI, MiniMax, Alibaba DashScope, Zhipu, Ollama (local), llama.cpp llama-server (local), LiteLLM proxy (universal), and 5 more. Run `gbrain providers list` to see them, or read [`docs/integrations/embedding-providers.md`](docs/integrations/embedding-providers.md) for setup, pricing, and a decision tree. `gbrain doctor` will surface alternative providers whose env vars you already have set. +> **New in v0.32.3.0 — compress your AGENTS.md without losing accuracy:** if your downstream agent fork has grown a 25KB+ `AGENTS.md` / `RESOLVER.md`, the new [`functional-area-resolver`](skills/functional-area-resolver/SKILL.md) skill ships a two-layer dispatch pattern that compresses 25KB → 13KB (48% the size) while **beating** the verbose baseline by +13 to +17pp across Opus 4.7, Sonnet 4.6, and Haiku 4.5. A/B eval harness, cross-model receipts, and reproduction instructions live at [`evals/functional-area-resolver/`](evals/functional-area-resolver/). The static-prompt analog of AnyTool / RAG-MCP / Anthropic Agent Skills progressive disclosure — single-LLM-pass dispatch, no second routing call. + ## Install ### On an agent platform (recommended) diff --git a/package.json b/package.json index 5017ff7da..46fe7aa49 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.32.3.0", + "version": "0.32.4", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/skills/functional-area-resolver/SKILL.md b/skills/functional-area-resolver/SKILL.md index 0ec336946..4715a74a6 100644 --- a/skills/functional-area-resolver/SKILL.md +++ b/skills/functional-area-resolver/SKILL.md @@ -303,7 +303,7 @@ This skill guarantees: - Routing matches the canonical triggers in the frontmatter. - Compression is only performed when the preconditions in Step 1 pass (file ≥12KB AND clean working tree, or `--force`). - The mandatory verification gate in Step 6 fires on the user's edited file, not on sample variants. The user runs `gbrain routing-eval --json` AND the gbrain-repo harness (`node harness.mjs --variants-dir --variants my-edit`) before committing the compressed file. -- Privacy contract preserved: no fork-specific filesystem path literals (`/data/brain/`, `/data/.openclaw/`) leak into the compressed output. +- Privacy contract preserved: no fork-specific filesystem path literals (server-side brain home, OpenClaw fork home) leak into the compressed output. The full behavior contract is documented in the body sections above; this section exists for the conformance test. diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index d021b35b7..8df3a9e4d 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -344,6 +344,9 @@ export async function doctorReportRemote(engine: BrainEngine): Promise { } } +// Module-scoped flag so the NaN-fallback warning fires once per process. +let _syncFreshnessEnvWarned = false; + +function _resolveSyncFreshnessHours(varName: string, fallback: number): number { + const raw = process.env[varName]; + if (raw === undefined || raw === '') return fallback; + const n = Number(raw); + if (!Number.isFinite(n) || n <= 0) { + if (!_syncFreshnessEnvWarned) { + _syncFreshnessEnvWarned = true; + console.warn( + `[gbrain doctor] Ignoring invalid ${varName}=${raw}; using default ${fallback}h.`, + ); + } + return fallback; + } + return n; +} + +/** + * Sync freshness check (v0.32.4) — verify that sources with local_path have + * been synced recently. Detects the silent failure mode where `gbrain sync` + * stopped running and brain search now misses recent pages. + * + * Pure staleness check. Reads `sources.last_sync_at` only — no filesystem + * access. Filesystem-vs-DB drift detection is intentionally out of scope: + * - doctorReportRemote runs in the HTTP MCP server (src/commands/serve-http.ts); + * walking arbitrary DB-supplied paths from a remote-callable endpoint + * crosses a trust boundary (OAuth write scope could mutate local_path). + * - Drift detection belongs in `multi_source_drift` which already has + * GBRAIN_DRIFT_LIMIT + GBRAIN_DRIFT_TIMEOUT_MS guards. + * + * Thresholds (env-overridable, default = 24h warn / 72h fail): + * - GBRAIN_SYNC_FRESHNESS_WARN_HOURS + * - GBRAIN_SYNC_FRESHNESS_FAIL_HOURS + * Invalid values (NaN, ≤0) fall back to defaults with a once-per-process warn. + * + * Edge cases handled: + * - last_sync_at IS NULL → fail "never synced" + * - last_sync_at > now() (clock skew / corrupted timestamp) → warn + * - mixed sources → highest-severity drives the overall status + * - executeRaw throws → outer-catch warn so doctor keeps running + * + * Failure messages embed `source.id` so the fix command + * `gbrain sync --source ` matches what the user copy-pastes. + */ +export async function checkSyncFreshness(engine: BrainEngine): Promise { + try { + const sources = await engine.executeRaw<{ + id: string; + name: string; + local_path: string | null; + last_sync_at: Date | null; + }>( + `SELECT id, name, local_path, last_sync_at FROM sources WHERE local_path IS NOT NULL`, + ); + + if (sources.length === 0) { + return { + name: 'sync_freshness', + status: 'ok', + message: 'No federated sources to sync', + }; + } + + const warnHours = _resolveSyncFreshnessHours('GBRAIN_SYNC_FRESHNESS_WARN_HOURS', 24); + const failHours = _resolveSyncFreshnessHours('GBRAIN_SYNC_FRESHNESS_FAIL_HOURS', 72); + const warnMs = warnHours * 60 * 60 * 1000; + const failMs = failHours * 60 * 60 * 1000; + + const now = Date.now(); + const issues: string[] = []; + let hasWarnings = false; + let hasFailures = false; + + for (const source of sources) { + // Embed source.id in user-visible messages so `gbrain sync --source ` + // matches what the user copy-pastes. Show display name in parens when set. + const display = source.name && source.name !== source.id + ? `'${source.id}' (${source.name})` + : `'${source.id}'`; + + if (!source.last_sync_at) { + issues.push(`Source ${display} has never been synced`); + hasFailures = true; + continue; + } + + const lastSync = new Date(source.last_sync_at).getTime(); + const ageMs = now - lastSync; + + if (ageMs < 0) { + issues.push( + `Source ${display} has future last_sync_at — clock skew or corrupted timestamp`, + ); + hasWarnings = true; + continue; + } + + const ageHours = Math.floor(ageMs / (1000 * 60 * 60)); + const ageDays = Math.floor(ageHours / 24); + + if (ageMs > failMs) { + issues.push(`Source ${display} last synced ${ageDays}d ago — brain search is stale!`); + hasFailures = true; + } else if (ageMs > warnMs) { + issues.push(`Source ${display} last synced ${ageHours}h ago`); + hasWarnings = true; + } + } + + if (hasFailures) { + return { + name: 'sync_freshness', + status: 'fail', + message: `${issues.join('; ')}. Run \`gbrain sync --source \` for each stale source`, + }; + } + if (hasWarnings) { + return { + name: 'sync_freshness', + status: 'warn', + message: `${issues.join('; ')}. Run \`gbrain sync --source \` to refresh`, + }; + } + return { + name: 'sync_freshness', + status: 'ok', + message: `All ${sources.length} federated source(s) synced recently`, + }; + } catch (e) { + return { + name: 'sync_freshness', + status: 'warn', + message: `Could not check sync freshness: ${e instanceof Error ? e.message : String(e)}`, + }; + } +} + /** * Run doctor with filesystem-first, DB-second architecture. * Filesystem checks (resolver, conformance) run without engine. @@ -2008,6 +2150,12 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo } catch { /* config table missing on a very old brain — skip */ } } + // Sync freshness check (v0.32 — Check that sources are synced recently) + if (engine !== null) { + progress.heartbeat('sync_freshness'); + checks.push(await checkSyncFreshness(engine)); + } + progress.finish(); const hasFail = outputResults(checks, jsonOutput); diff --git a/test/doctor.test.ts b/test/doctor.test.ts index fb9769b4f..a821d0342 100644 --- a/test/doctor.test.ts +++ b/test/doctor.test.ts @@ -425,3 +425,156 @@ describe('v0.31.8 — wedge migration force-retry hint (D19)', () => { expect(remoteBlock).toMatch(/WEDGED MIGRATION\(s\) on brain host/); }); }); + +// ============================================================================ +// v0.32.4 — sync_freshness check +// ============================================================================ +// Pure staleness probe: reads sources.last_sync_at, no filesystem access. +// Drift detection was stripped in v0.32.4 — the doctorReportRemote path runs +// in the HTTP MCP server and walking DB-supplied local_path values from there +// crosses a trust boundary. Drift belongs in multi_source_drift's existing +// guard infrastructure (GBRAIN_DRIFT_LIMIT / GBRAIN_DRIFT_TIMEOUT_MS). +// ============================================================================ + +describe('v0.32.4 — sync_freshness check', () => { + // Stub engine: only checkSyncFreshness's executeRaw matters. Per-case rows + // shape is `{id, name, local_path, last_sync_at}`. + function makeStubEngine(rows: any[]): any { + return { executeRaw: async () => rows }; + } + + function agoMs(ms: number): Date { + return new Date(Date.now() - ms); + } + + test('empty sources → ok with no-federated-sources message', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const result = await checkSyncFreshness(makeStubEngine([])); + expect(result.name).toBe('sync_freshness'); + expect(result.status).toBe('ok'); + expect(result.message).toBe('No federated sources to sync'); + }); + + test('last_sync_at IS NULL → fail with "never been synced"', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: null }, + ])); + expect(result.status).toBe('fail'); + expect(result.message).toContain('never been synced'); + expect(result.message).toContain(`'wiki'`); // source.id embedded + expect(result.message).toContain('gbrain sync --source '); + }); + + test('last_sync_at > 72h ago → fail with day-rounded "Nd ago"', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(4 * 24 * 60 * 60 * 1000) }, + ])); + expect(result.status).toBe('fail'); + expect(result.message).toMatch(/4d ago/); + expect(result.message).toContain('brain search is stale'); + }); + + test('exact 72h boundary → warn (>72h strict; 72h source NOT yet fail)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + // Exactly 72h. Strict `>` on fail threshold means 72h-stale is still in + // the warn window. (Tested boundary semantics.) + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(72 * 60 * 60 * 1000) }, + ])); + expect(result.status).toBe('warn'); + expect(result.message).toContain('72h ago'); + }); + + test('24h < last_sync_at < 72h → warn with hour-rounded "Nh ago"', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(30 * 60 * 60 * 1000) }, + ])); + expect(result.status).toBe('warn'); + expect(result.message).toMatch(/30h ago/); + }); + + test('exact 24h boundary → ok (>24h strict)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + // Exactly 24h. Strict `>` on warn threshold means 24h-stale is still ok. + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(24 * 60 * 60 * 1000) }, + ])); + expect(result.status).toBe('ok'); + expect(result.message).toContain('synced recently'); + }); + + test('last_sync_at <= 24h → ok with "synced recently"', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(2 * 60 * 60 * 1000) }, + { id: 'gstack', name: '', local_path: '/tmp/gstack', last_sync_at: agoMs(60 * 1000) }, + ])); + expect(result.status).toBe('ok'); + expect(result.message).toContain('2 federated source(s)'); + }); + + test('future last_sync_at → warn (clock skew / corrupted timestamp)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + // 10 min in the future. Negative ageMs must NOT fall through as ok. + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: new Date(Date.now() + 10 * 60 * 1000) }, + ])); + expect(result.status).toBe('warn'); + expect(result.message).toMatch(/future last_sync_at/); + expect(result.message).toMatch(/clock skew|corrupted timestamp/); + }); + + test('mixed sources (one fail + one warn) → fail with both issues listed', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(5 * 24 * 60 * 60 * 1000) }, + { id: 'gstack', name: '', local_path: '/tmp/gstack', last_sync_at: agoMs(30 * 60 * 60 * 1000) }, + ])); + expect(result.status).toBe('fail'); + expect(result.message).toContain(`'wiki'`); + expect(result.message).toContain(`'gstack'`); + expect(result.message).toMatch(/5d ago/); + expect(result.message).toMatch(/30h ago/); + }); + + test('executeRaw throws → outer-catch returns warn (doctor keeps running)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const engine: any = { + executeRaw: async () => { throw new Error('connection refused'); }, + }; + const result = await checkSyncFreshness(engine); + expect(result.status).toBe('warn'); + expect(result.message).toContain('Could not check sync freshness'); + expect(result.message).toContain('connection refused'); + }); + + test('env-var override: GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6 → 7h-stale fails', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const prev = process.env.GBRAIN_SYNC_FRESHNESS_FAIL_HOURS; + process.env.GBRAIN_SYNC_FRESHNESS_FAIL_HOURS = '6'; + try { + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki', name: '', local_path: '/tmp/wiki', last_sync_at: agoMs(7 * 60 * 60 * 1000) }, + ])); + expect(result.status).toBe('fail'); + expect(result.message).toContain('brain search is stale'); + } finally { + if (prev === undefined) delete process.env.GBRAIN_SYNC_FRESHNESS_FAIL_HOURS; + else process.env.GBRAIN_SYNC_FRESHNESS_FAIL_HOURS = prev; + } + }); + + test('source.id embedded in messages even when source.name is set', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + const result = await checkSyncFreshness(makeStubEngine([ + { id: 'wiki-id', name: 'My Wiki', local_path: '/tmp/wiki', last_sync_at: null }, + ])); + expect(result.status).toBe('fail'); + // User copy-pastes `gbrain sync --source wiki-id` (NOT "My Wiki"). Message + // must include the id so the CLI command actually works. + expect(result.message).toContain(`'wiki-id'`); + }); +});