mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-30 03:12:32 +00:00
* v0.40.4.0 feat(sync): parallel sync --all + per-source lock invariant + sources status dashboard (productionized from PR #1314) Lands the community-authored PR #1314 with the structural fixes Codex's outside-voice review caught: the original PR's lock-id change only fired inside the --all parallel path, which would have introduced a worse race than the global-lock contention it fixed (sync --all on per-source lock racing against sync --source foo on the still-global lock). The landed version makes the per-source lock the invariant for every source-scoped sync, paired with withRefreshingLock for sources that exceed 30 minutes. What's new - gbrain sync --all parallel fan-out via continuous worker pool (D2); --parallel N flag, default min(sourceCount, --workers, 4); per-source [<source-id>] line prefix via AsyncLocalStorage (D6 + D12 + D13); stable --json envelope {schema_version:1, ...} on stdout with banners on stderr (D4 + D14); --skip-failed/--retry-failed reject under --parallel > 1 (D15 — sync-failures.jsonl is brain-global today; source-scoping filed as v0.40.4 TODO). - gbrain sources status [--json] read-only dashboard (D3 — sibling to sources list/add/remove/archive, not a sync flag, so reads + writes don't share a verb). Counts pages + chunks + embedding coverage per source. Active embedding column resolved via the registry (D16) so Voyage / multimodal brains see the right column. Archived sources excluded by caller filter. - Connection-budget stderr warning when parallel × workers × 2 > 16 with the formula in the message text (D1 + D10 — Codex P0 #3: each per-file worker opens its own PostgresEngine with poolSize=2, so the multiplication factor is 2, not 1). The load-bearing structural fix - performSync defaults to per-source lock id (gbrain-sync:<sourceId>) whenever opts.sourceId is set + wraps in withRefreshingLock. Legacy single-default-source brains keep the bare tryAcquireDbLock(SYNC_LOCK_ID) path for back-compat. - Dashboard SQL is the canonical content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL shape — the original PR shipped chunks ch JOIN ON page_slug, which would have crashed on PGLite parse and silently zeroed on Postgres via a swallow-catch. Errors from the dashboard SQL propagate (no silent zero-counts on real DB errors). Tests - New test/console-prefix.test.ts — 8 cases pinning ALS propagation, nested wraps, embedded-newline prefixing, back-compat fast path. - New test/sync-all-parallel.test.ts (replaces PR's stubbed tests) — 16 cases covering resolveParallelism, per-source lock format, buildSyncStatusReport SQL math + error propagation + envelope shape, connection-budget math, per-source prefix routing. - New test/e2e/sync-status-pglite.test.ts — IRON RULE regression: real PGLite seeds 2 sources × pages × chunks (mixed embedded/unembedded, 1 soft-deleted, 1 archived source). Validates SQL excludes both AND the active embedding column is the one used. This is the case that would have caught the PR's original broken SQL. Compatibility - No schema changes. No new dependencies. - Single-source / non-`--all` paths: bit-for-bit identical to v0.40.2. - PGLite users get serial behavior (single-connection engine). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com> * v0.40.6.0 — version bump for ship (skipping 0.40.4 + 0.40.5 for in-flight work) Reserves v0.40.4 + v0.40.5 slots for parallel waves (salem's graph-signals work and any other in-flight branches) and lands this PR's parallel-sync work at v0.40.6.0. No code change beyond the version triple and the TODOS / CLAUDE.md / CHANGELOG cross-references which were updated from "v0.40.4" to "v0.41+" to match the new follow-up version. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
146 lines
5.7 KiB
TypeScript
146 lines
5.7 KiB
TypeScript
/**
|
|
* v0.40.3.0 — per-source console line-prefixing via AsyncLocalStorage.
|
|
*
|
|
* Under `gbrain sync --all --parallel > 1`, multiple per-source syncs run
|
|
* concurrently. Without prefixing, every `console.log` from `performSync`
|
|
* (git pull lines, embed progress, drift gates, ~30+ call sites total)
|
|
* interleaves on the terminal — operators can't tell which source emitted
|
|
* which line. kubectl `--prefix` and docker-compose solve the same problem
|
|
* with `[<id>] ` prefixes that ride along with each line.
|
|
*
|
|
* Usage:
|
|
* await withSourcePrefix(src.id, () => performSync(engine, opts));
|
|
* // Inside performSync (and its callees), call slog() / serr() instead
|
|
* // of console.log() / console.error(). When invoked under a wrap, the
|
|
* // prefix is automatically prepended to each emitted line. Outside the
|
|
* // wrap (single-source sync, doctor, anywhere else), slog/serr are
|
|
* // identical to console.log/console.error — back-compat preserved.
|
|
*
|
|
* Multi-line strings: each newline-separated segment gets its own prefix,
|
|
* so a `serr('phase started\\n details: x')` under prefix `[foo]` emits:
|
|
* [foo] phase started
|
|
* [foo] details: x
|
|
*
|
|
* Why source.id and not source.name:
|
|
* `sources add --name` accepts arbitrary text — operators (or attackers)
|
|
* could embed newlines or control characters that break grep filtering
|
|
* and impersonate other sources' lines. `source.id` is slug-validated by
|
|
* the existing sources schema (lowercase alphanumeric + dash) and safe to
|
|
* prefix raw. The per-source banner (one-shot at start of each source)
|
|
* can still display `source.name` for human readability.
|
|
*
|
|
* Why AsyncLocalStorage:
|
|
* Propagates through every `await` boundary without manual threading.
|
|
* `withSourcePrefix(id, () => performSync(...))` covers performSync AND
|
|
* every async function it calls (runImport, runEmbedCore, parallel-worker
|
|
* subphases) as long as those functions use slog/serr. Nested wraps
|
|
* restore the outer prefix on exit.
|
|
*
|
|
* Coverage in v0.40.3.0 (see CLAUDE.md for the canonical list):
|
|
* - src/commands/sync.ts (performSync + in-file callees)
|
|
* - src/commands/embed.ts (runEmbedCore + helpers)
|
|
* - src/core/progress.ts (heartbeat / progress writer)
|
|
*
|
|
* Anything outside those modules that writes directly to stdout/stderr will
|
|
* NOT get the prefix. If you find a delegate-module line that escapes the
|
|
* prefix under parallel sync, it's a missed migration target — file an
|
|
* issue (per D12 → A full-lake; honest about scope).
|
|
*/
|
|
|
|
import { AsyncLocalStorage } from 'node:async_hooks';
|
|
|
|
const __prefixStore = new AsyncLocalStorage<string>();
|
|
|
|
/**
|
|
* Run `fn` with an active per-source prefix `id`. Within the closure,
|
|
* `slog` and `serr` will prepend `[id] ` to every line they emit. The
|
|
* prefix is automatically propagated through `await` boundaries.
|
|
*
|
|
* Nested wraps replace the active prefix for the inner closure and
|
|
* restore the outer prefix when the inner returns/throws.
|
|
*
|
|
* `id` should be a slug-validated source identifier (NOT a free-form
|
|
* display name) — see module docstring for the security rationale.
|
|
*/
|
|
export function withSourcePrefix<T>(id: string, fn: () => Promise<T>): Promise<T> {
|
|
return __prefixStore.run(id, fn);
|
|
}
|
|
|
|
/**
|
|
* Read the currently-active per-source prefix. Returns null when called
|
|
* outside a `withSourcePrefix` scope. Test seam; production code should
|
|
* use `slog` / `serr` instead of reading the prefix directly.
|
|
*/
|
|
export function getSourcePrefix(): string | null {
|
|
return __prefixStore.getStore() ?? null;
|
|
}
|
|
|
|
/**
|
|
* Prefix-aware replacement for `console.log`. When called inside a
|
|
* `withSourcePrefix(id, ...)` scope, prepends `[id] ` to every line of
|
|
* the formatted output. Outside the scope, behaves exactly like
|
|
* `console.log` (writes to stdout).
|
|
*/
|
|
export function slog(...args: unknown[]): void {
|
|
const prefix = getSourcePrefix();
|
|
if (prefix === null) {
|
|
// Back-compat fast path: bare console.log semantics.
|
|
// eslint-disable-next-line no-console
|
|
console.log(...args);
|
|
return;
|
|
}
|
|
process.stdout.write(prefixLines(formatArgs(args), prefix) + '\n');
|
|
}
|
|
|
|
/**
|
|
* Prefix-aware replacement for `console.error`. Same prefix semantics as
|
|
* `slog`, but writes to stderr. Use for warnings, errors, and any output
|
|
* that should NOT pollute `--json` stdout.
|
|
*/
|
|
export function serr(...args: unknown[]): void {
|
|
const prefix = getSourcePrefix();
|
|
if (prefix === null) {
|
|
// eslint-disable-next-line no-console
|
|
console.error(...args);
|
|
return;
|
|
}
|
|
process.stderr.write(prefixLines(formatArgs(args), prefix) + '\n');
|
|
}
|
|
|
|
/**
|
|
* Format an args list the way `console.log` would, but as a single string.
|
|
* Strings stay raw; everything else goes through JSON.stringify with a
|
|
* fallback to String() for non-serializable values. Multi-arg invocations
|
|
* join with spaces (matches console.log's default delimiter).
|
|
*/
|
|
function formatArgs(args: unknown[]): string {
|
|
return args
|
|
.map((a) => {
|
|
if (typeof a === 'string') return a;
|
|
if (a instanceof Error) return a.stack ?? a.message;
|
|
try {
|
|
return JSON.stringify(a);
|
|
} catch {
|
|
return String(a);
|
|
}
|
|
})
|
|
.join(' ');
|
|
}
|
|
|
|
/**
|
|
* Prepend `[prefix] ` to every line of `text`. Preserves empty trailing
|
|
* lines (so a trailing newline in input stays a trailing newline in
|
|
* output — the caller's `+ '\\n'` adds the final separator). Embedded
|
|
* newlines inside the string each get their own prefix.
|
|
*
|
|
* Examples (prefix = 'foo'):
|
|
* '' → '[foo] '
|
|
* 'a' → '[foo] a'
|
|
* 'a\\nb' → '[foo] a\\n[foo] b'
|
|
* 'a\\nb\\n' → '[foo] a\\n[foo] b\\n[foo] '
|
|
*/
|
|
function prefixLines(text: string, prefix: string): string {
|
|
const tag = `[${prefix}] `;
|
|
return text.split('\n').map((line) => tag + line).join('\n');
|
|
}
|