2 Commits

Include renames

Author	SHA1	Message	Date
4 people	f79c1306a2	v0.41.32.0 fix(staleness): commit-relative sync staleness (supersedes #1623) (#1656) ... * fix(staleness): commit-relative sync staleness (HEAD-hash local, durable column remote) Quiet, fully-caught-up repos no longer false-alarm as SEVERELY STALE in gbrain doctor / sources status. Staleness now means "is there committed content the sync hasn't ingested?" not raw wall-clock since the last sync. - git-head.ts: requireCleanWorkingTree gains 'ignore-untracked' mode (git status --porcelain --untracked-files=no). Untracked dirs no longer defeat the freshness short-circuit — sync's incremental path keys off the commit diff and never imports untracked files, so doctor agrees with sync. - source-health.ts: newestCommitMs (HEAD committer time) + pure lagFromContentMs comparator; computeAllSourceMetrics {probeContent} routes local→live commit-hash, remote→stored column. Dead isSourceStale removed. - migration v108 sources.newest_content_at + fresh-schema blobs. - sync.ts: writeSyncAnchor stamps newest_content_at atomically with last_commit/last_sync_at; buildSyncStatusReport (remote get_status_snapshot) reads the column — no git subprocess (v0.41.27.0 trust boundary intact). - doctor.ts: checkSyncFreshness short-circuit ignores untracked; remote path reads the column; clock-skew check stays on raw wall-clock. Local consumers probe live git (catch HEAD moving to an old-dated commit, which a timestamp compare would miss); remote consumers read the durable column so a remote-callable endpoint never shells out to a DB-supplied local_path. Supersedes #1623 (re-implemented in base repo with the trust boundary preserved). Co-Authored-By: t <t@t> * chore(ci): offload tests to on-demand cloud runners from a local CLI scripts/ship-remote-tests.sh pushes the branch, dispatches the test workflow, and blocks on `gh run watch --exit-status` — a local caller (human or agent) awaits the GitHub run exactly like a local `bun run test`, with a real pass/fail exit code. Frees a load-saturated local machine (many Conductor agents running their own bun-test suites at once → load avg 120 on 16 cores → PGLite OOM/crawl). test.yml gains workflow_dispatch so the suite can be triggered from any branch. * chore: bump version and changelog (v0.41.32.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: t <t@t> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-05-30 10:56:46 -07:00
4 people	677142a680	v0.40.6.0 feat(sync): parallel sync --all + per-source lock invariant + sources status dashboard (productionized from PR #1314) (#1324) ... * v0.40.4.0 feat(sync): parallel sync --all + per-source lock invariant + sources status dashboard (productionized from PR #1314) Lands the community-authored PR #1314 with the structural fixes Codex's outside-voice review caught: the original PR's lock-id change only fired inside the --all parallel path, which would have introduced a worse race than the global-lock contention it fixed (sync --all on per-source lock racing against sync --source foo on the still-global lock). The landed version makes the per-source lock the invariant for every source-scoped sync, paired with withRefreshingLock for sources that exceed 30 minutes. What's new - gbrain sync --all parallel fan-out via continuous worker pool (D2); --parallel N flag, default min(sourceCount, --workers, 4); per-source [<source-id>] line prefix via AsyncLocalStorage (D6 + D12 + D13); stable --json envelope {schema_version:1, ...} on stdout with banners on stderr (D4 + D14); --skip-failed/--retry-failed reject under --parallel > 1 (D15 — sync-failures.jsonl is brain-global today; source-scoping filed as v0.40.4 TODO). - gbrain sources status [--json] read-only dashboard (D3 — sibling to sources list/add/remove/archive, not a sync flag, so reads + writes don't share a verb). Counts pages + chunks + embedding coverage per source. Active embedding column resolved via the registry (D16) so Voyage / multimodal brains see the right column. Archived sources excluded by caller filter. - Connection-budget stderr warning when parallel × workers × 2 > 16 with the formula in the message text (D1 + D10 — Codex P0 #3: each per-file worker opens its own PostgresEngine with poolSize=2, so the multiplication factor is 2, not 1). The load-bearing structural fix - performSync defaults to per-source lock id (gbrain-sync:<sourceId>) whenever opts.sourceId is set + wraps in withRefreshingLock. Legacy single-default-source brains keep the bare tryAcquireDbLock(SYNC_LOCK_ID) path for back-compat. - Dashboard SQL is the canonical content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL shape — the original PR shipped chunks ch JOIN ON page_slug, which would have crashed on PGLite parse and silently zeroed on Postgres via a swallow-catch. Errors from the dashboard SQL propagate (no silent zero-counts on real DB errors). Tests - New test/console-prefix.test.ts — 8 cases pinning ALS propagation, nested wraps, embedded-newline prefixing, back-compat fast path. - New test/sync-all-parallel.test.ts (replaces PR's stubbed tests) — 16 cases covering resolveParallelism, per-source lock format, buildSyncStatusReport SQL math + error propagation + envelope shape, connection-budget math, per-source prefix routing. - New test/e2e/sync-status-pglite.test.ts — IRON RULE regression: real PGLite seeds 2 sources × pages × chunks (mixed embedded/unembedded, 1 soft-deleted, 1 archived source). Validates SQL excludes both AND the active embedding column is the one used. This is the case that would have caught the PR's original broken SQL. Compatibility - No schema changes. No new dependencies. - Single-source / non-`--all` paths: bit-for-bit identical to v0.40.2. - PGLite users get serial behavior (single-connection engine). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com> * v0.40.6.0 — version bump for ship (skipping 0.40.4 + 0.40.5 for in-flight work) Reserves v0.40.4 + v0.40.5 slots for parallel waves (salem's graph-signals work and any other in-flight branches) and lands this PR's parallel-sync work at v0.40.6.0. No code change beyond the version triple and the TODOS / CLAUDE.md / CHANGELOG cross-references which were updated from "v0.40.4" to "v0.41+" to match the new follow-up version. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>	2026-05-23 10:57:32 -07:00