Files
gbrain/src/core/console-prefix.ts
T
677142a680 v0.40.6.0 feat(sync): parallel sync --all + per-source lock invariant + sources status dashboard (productionized from PR #1314) (#1324)
* v0.40.4.0 feat(sync): parallel sync --all + per-source lock invariant + sources status dashboard (productionized from PR #1314)

Lands the community-authored PR #1314 with the structural fixes Codex's
outside-voice review caught: the original PR's lock-id change only fired
inside the --all parallel path, which would have introduced a worse race
than the global-lock contention it fixed (sync --all on per-source lock
racing against sync --source foo on the still-global lock). The landed
version makes the per-source lock the invariant for every source-scoped
sync, paired with withRefreshingLock for sources that exceed 30 minutes.

What's new
- gbrain sync --all parallel fan-out via continuous worker pool (D2);
  --parallel N flag, default min(sourceCount, --workers, 4); per-source
  [<source-id>] line prefix via AsyncLocalStorage (D6 + D12 + D13);
  stable --json envelope {schema_version:1, ...} on stdout with banners
  on stderr (D4 + D14); --skip-failed/--retry-failed reject under
  --parallel > 1 (D15 — sync-failures.jsonl is brain-global today;
  source-scoping filed as v0.40.4 TODO).
- gbrain sources status [--json] read-only dashboard (D3 — sibling to
  sources list/add/remove/archive, not a sync flag, so reads + writes
  don't share a verb). Counts pages + chunks + embedding coverage per
  source. Active embedding column resolved via the registry (D16) so
  Voyage / multimodal brains see the right column. Archived sources
  excluded by caller filter.
- Connection-budget stderr warning when parallel × workers × 2 > 16 with
  the formula in the message text (D1 + D10 — Codex P0 #3: each per-file
  worker opens its own PostgresEngine with poolSize=2, so the
  multiplication factor is 2, not 1).

The load-bearing structural fix
- performSync defaults to per-source lock id (gbrain-sync:<sourceId>)
  whenever opts.sourceId is set + wraps in withRefreshingLock. Legacy
  single-default-source brains keep the bare tryAcquireDbLock(SYNC_LOCK_ID)
  path for back-compat.
- Dashboard SQL is the canonical content_chunks ch JOIN pages pg ON
  pg.id = ch.page_id WHERE pg.deleted_at IS NULL shape — the original PR
  shipped chunks ch JOIN ON page_slug, which would have crashed on PGLite
  parse and silently zeroed on Postgres via a swallow-catch. Errors from
  the dashboard SQL propagate (no silent zero-counts on real DB errors).

Tests
- New test/console-prefix.test.ts — 8 cases pinning ALS propagation,
  nested wraps, embedded-newline prefixing, back-compat fast path.
- New test/sync-all-parallel.test.ts (replaces PR's stubbed tests) —
  16 cases covering resolveParallelism, per-source lock format,
  buildSyncStatusReport SQL math + error propagation + envelope shape,
  connection-budget math, per-source prefix routing.
- New test/e2e/sync-status-pglite.test.ts — IRON RULE regression: real
  PGLite seeds 2 sources × pages × chunks (mixed embedded/unembedded,
  1 soft-deleted, 1 archived source). Validates SQL excludes both AND
  the active embedding column is the one used. This is the case that
  would have caught the PR's original broken SQL.

Compatibility
- No schema changes. No new dependencies.
- Single-source / non-`--all` paths: bit-for-bit identical to v0.40.2.
- PGLite users get serial behavior (single-connection engine).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com>

* v0.40.6.0 — version bump for ship (skipping 0.40.4 + 0.40.5 for in-flight work)

Reserves v0.40.4 + v0.40.5 slots for parallel waves (salem's graph-signals
work and any other in-flight branches) and lands this PR's parallel-sync
work at v0.40.6.0. No code change beyond the version triple and the
TODOS / CLAUDE.md / CHANGELOG cross-references which were updated from
"v0.40.4" to "v0.41+" to match the new follow-up version.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
2026-05-23 10:57:32 -07:00

146 lines
5.7 KiB
TypeScript

/**
* v0.40.3.0 — per-source console line-prefixing via AsyncLocalStorage.
*
* Under `gbrain sync --all --parallel > 1`, multiple per-source syncs run
* concurrently. Without prefixing, every `console.log` from `performSync`
* (git pull lines, embed progress, drift gates, ~30+ call sites total)
* interleaves on the terminal — operators can't tell which source emitted
* which line. kubectl `--prefix` and docker-compose solve the same problem
* with `[<id>] ` prefixes that ride along with each line.
*
* Usage:
* await withSourcePrefix(src.id, () => performSync(engine, opts));
* // Inside performSync (and its callees), call slog() / serr() instead
* // of console.log() / console.error(). When invoked under a wrap, the
* // prefix is automatically prepended to each emitted line. Outside the
* // wrap (single-source sync, doctor, anywhere else), slog/serr are
* // identical to console.log/console.error — back-compat preserved.
*
* Multi-line strings: each newline-separated segment gets its own prefix,
* so a `serr('phase started\\n details: x')` under prefix `[foo]` emits:
* [foo] phase started
* [foo] details: x
*
* Why source.id and not source.name:
* `sources add --name` accepts arbitrary text — operators (or attackers)
* could embed newlines or control characters that break grep filtering
* and impersonate other sources' lines. `source.id` is slug-validated by
* the existing sources schema (lowercase alphanumeric + dash) and safe to
* prefix raw. The per-source banner (one-shot at start of each source)
* can still display `source.name` for human readability.
*
* Why AsyncLocalStorage:
* Propagates through every `await` boundary without manual threading.
* `withSourcePrefix(id, () => performSync(...))` covers performSync AND
* every async function it calls (runImport, runEmbedCore, parallel-worker
* subphases) as long as those functions use slog/serr. Nested wraps
* restore the outer prefix on exit.
*
* Coverage in v0.40.3.0 (see CLAUDE.md for the canonical list):
* - src/commands/sync.ts (performSync + in-file callees)
* - src/commands/embed.ts (runEmbedCore + helpers)
* - src/core/progress.ts (heartbeat / progress writer)
*
* Anything outside those modules that writes directly to stdout/stderr will
* NOT get the prefix. If you find a delegate-module line that escapes the
* prefix under parallel sync, it's a missed migration target — file an
* issue (per D12 → A full-lake; honest about scope).
*/
import { AsyncLocalStorage } from 'node:async_hooks';
const __prefixStore = new AsyncLocalStorage<string>();
/**
* Run `fn` with an active per-source prefix `id`. Within the closure,
* `slog` and `serr` will prepend `[id] ` to every line they emit. The
* prefix is automatically propagated through `await` boundaries.
*
* Nested wraps replace the active prefix for the inner closure and
* restore the outer prefix when the inner returns/throws.
*
* `id` should be a slug-validated source identifier (NOT a free-form
* display name) — see module docstring for the security rationale.
*/
export function withSourcePrefix<T>(id: string, fn: () => Promise<T>): Promise<T> {
return __prefixStore.run(id, fn);
}
/**
* Read the currently-active per-source prefix. Returns null when called
* outside a `withSourcePrefix` scope. Test seam; production code should
* use `slog` / `serr` instead of reading the prefix directly.
*/
export function getSourcePrefix(): string | null {
return __prefixStore.getStore() ?? null;
}
/**
* Prefix-aware replacement for `console.log`. When called inside a
* `withSourcePrefix(id, ...)` scope, prepends `[id] ` to every line of
* the formatted output. Outside the scope, behaves exactly like
* `console.log` (writes to stdout).
*/
export function slog(...args: unknown[]): void {
const prefix = getSourcePrefix();
if (prefix === null) {
// Back-compat fast path: bare console.log semantics.
// eslint-disable-next-line no-console
console.log(...args);
return;
}
process.stdout.write(prefixLines(formatArgs(args), prefix) + '\n');
}
/**
* Prefix-aware replacement for `console.error`. Same prefix semantics as
* `slog`, but writes to stderr. Use for warnings, errors, and any output
* that should NOT pollute `--json` stdout.
*/
export function serr(...args: unknown[]): void {
const prefix = getSourcePrefix();
if (prefix === null) {
// eslint-disable-next-line no-console
console.error(...args);
return;
}
process.stderr.write(prefixLines(formatArgs(args), prefix) + '\n');
}
/**
* Format an args list the way `console.log` would, but as a single string.
* Strings stay raw; everything else goes through JSON.stringify with a
* fallback to String() for non-serializable values. Multi-arg invocations
* join with spaces (matches console.log's default delimiter).
*/
function formatArgs(args: unknown[]): string {
return args
.map((a) => {
if (typeof a === 'string') return a;
if (a instanceof Error) return a.stack ?? a.message;
try {
return JSON.stringify(a);
} catch {
return String(a);
}
})
.join(' ');
}
/**
* Prepend `[prefix] ` to every line of `text`. Preserves empty trailing
* lines (so a trailing newline in input stays a trailing newline in
* output — the caller's `+ '\\n'` adds the final separator). Embedded
* newlines inside the string each get their own prefix.
*
* Examples (prefix = 'foo'):
* '' → '[foo] '
* 'a' → '[foo] a'
* 'a\\nb' → '[foo] a\\n[foo] b'
* 'a\\nb\\n' → '[foo] a\\n[foo] b\\n[foo] '
*/
function prefixLines(text: string, prefix: string): string {
const tag = `[${prefix}] `;
return text.split('\n').map((line) => tag + line).join('\n');
}