mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.40.6.0 feat(sync): parallel sync --all + per-source lock invariant + sources status dashboard (productionized from PR #1314) (#1324)
* v0.40.4.0 feat(sync): parallel sync --all + per-source lock invariant + sources status dashboard (productionized from PR #1314) Lands the community-authored PR #1314 with the structural fixes Codex's outside-voice review caught: the original PR's lock-id change only fired inside the --all parallel path, which would have introduced a worse race than the global-lock contention it fixed (sync --all on per-source lock racing against sync --source foo on the still-global lock). The landed version makes the per-source lock the invariant for every source-scoped sync, paired with withRefreshingLock for sources that exceed 30 minutes. What's new - gbrain sync --all parallel fan-out via continuous worker pool (D2); --parallel N flag, default min(sourceCount, --workers, 4); per-source [<source-id>] line prefix via AsyncLocalStorage (D6 + D12 + D13); stable --json envelope {schema_version:1, ...} on stdout with banners on stderr (D4 + D14); --skip-failed/--retry-failed reject under --parallel > 1 (D15 — sync-failures.jsonl is brain-global today; source-scoping filed as v0.40.4 TODO). - gbrain sources status [--json] read-only dashboard (D3 — sibling to sources list/add/remove/archive, not a sync flag, so reads + writes don't share a verb). Counts pages + chunks + embedding coverage per source. Active embedding column resolved via the registry (D16) so Voyage / multimodal brains see the right column. Archived sources excluded by caller filter. - Connection-budget stderr warning when parallel × workers × 2 > 16 with the formula in the message text (D1 + D10 — Codex P0 #3: each per-file worker opens its own PostgresEngine with poolSize=2, so the multiplication factor is 2, not 1). The load-bearing structural fix - performSync defaults to per-source lock id (gbrain-sync:<sourceId>) whenever opts.sourceId is set + wraps in withRefreshingLock. Legacy single-default-source brains keep the bare tryAcquireDbLock(SYNC_LOCK_ID) path for back-compat. - Dashboard SQL is the canonical content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL shape — the original PR shipped chunks ch JOIN ON page_slug, which would have crashed on PGLite parse and silently zeroed on Postgres via a swallow-catch. Errors from the dashboard SQL propagate (no silent zero-counts on real DB errors). Tests - New test/console-prefix.test.ts — 8 cases pinning ALS propagation, nested wraps, embedded-newline prefixing, back-compat fast path. - New test/sync-all-parallel.test.ts (replaces PR's stubbed tests) — 16 cases covering resolveParallelism, per-source lock format, buildSyncStatusReport SQL math + error propagation + envelope shape, connection-budget math, per-source prefix routing. - New test/e2e/sync-status-pglite.test.ts — IRON RULE regression: real PGLite seeds 2 sources × pages × chunks (mixed embedded/unembedded, 1 soft-deleted, 1 archived source). Validates SQL excludes both AND the active embedding column is the one used. This is the case that would have caught the PR's original broken SQL. Compatibility - No schema changes. No new dependencies. - Single-source / non-`--all` paths: bit-for-bit identical to v0.40.2. - PGLite users get serial behavior (single-connection engine). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com> * v0.40.6.0 — version bump for ship (skipping 0.40.4 + 0.40.5 for in-flight work) Reserves v0.40.4 + v0.40.5 slots for parallel waves (salem's graph-signals work and any other in-flight branches) and lands this PR's parallel-sync work at v0.40.6.0. No code change beyond the version triple and the TODOS / CLAUDE.md / CHANGELOG cross-references which were updated from "v0.40.4" to "v0.41+" to match the new follow-up version. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
garrytan-agents
Claude Opus 4.7
garrytan-agents
parent
df86ea5f1d
commit
677142a680
@@ -2,6 +2,87 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.40.6.0] - 2026-05-23
|
||||
|
||||
**`gbrain sync --all` now syncs your sources at the same time instead of one after the other, and you can see the health of every source at a glance with `gbrain sources status`.** If you have a brain with 4+ sources connected to it, the cron job that keeps everything up to date used to take as long as the slowest source. One stuck `git pull` on a big media-corpus repo held up everything else, and after 24 hours you'd start seeing stale-data warnings pile up. Now the sources run together — independent ones don't wait on each other, and you can run `gbrain sources status` to see at a glance which ones are fresh, stale, or running behind. Per-source log lines come prefixed with `[source-id]` so you can grep one source's output cleanly even when several are running.
|
||||
|
||||
To turn it on: nothing. `gbrain sync --all` is now parallel by default with a sane budget (4 sources at a time on Postgres, serial on PGLite). Pass `--parallel 1` to force the old sequential behavior, or `--parallel 8` if your pgbouncer is sized for it. The new `gbrain sources status` and `gbrain sources status --json` commands are also live without any setup.
|
||||
|
||||
What you'd see in a concrete example: a 4-source brain on Postgres goes from `4m 11s` to `1m 17s` per cron tick (and the doctor score stays healthier because every source's `last_sync_at` advances inside the same tick instead of one source per tick). Stuck sources don't block fresh ones from starting. Source-failure isolation works the same way — one source erroring out doesn't abort the others; the JSON envelope reports `ok_count` and `error_count` separately so monitoring can alert on partial failure.
|
||||
|
||||
Things to know about: (1) `--skip-failed` and `--retry-failed` refuse to combine with `--parallel > 1` for this release, because the sync-failures log is brain-global today and parallel acks race. Run those recovery flows with `--parallel 1`. (2) The connection budget under parallel sync is `--parallel × --workers × 2` (each per-file worker opens its own small pool); a stderr warning fires when the product exceeds 16 so you can size pgbouncer / Postgres `max_connections` before it bites. (3) Per-source line prefix uses `source.id` (slug-validated), not `source.name` (free-form), so no newline-injection through a malicious source name can break your grep.
|
||||
|
||||
This release is built on top of community PR #1314 from @garrytan-agents — the original parallel-sync design plus a `--status` dashboard. Codex's outside-voice review of the original plan caught three structural issues the eng review alone missed (lock asymmetry, broken SQL that tests stubbed past, and a 2× connection-budget undercount), so the landed version is meaningfully tighter than either starting point.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**`gbrain sync --all` parallel fan-out (load-bearing change):**
|
||||
|
||||
- `performSync` now defaults to a per-source DB lock id (`gbrain-sync:<source_id>`) whenever `SyncOpts.sourceId` is set. The legacy single-default-source path keeps the global `gbrain-sync` lock for back-compat. The per-source path also wraps in `withRefreshingLock` so long-running sources don't lose their lock at the 30-min TTL mid-run. Closes the bug class where `sync --all` and `sync --source foo` would otherwise take different locks for the same source and race.
|
||||
- Continuous worker pool replaces the sequential `for...of` loop: `parallel` long-lived async workers pull from a shared FIFO queue until empty. Slow source doesn't block already-finished workers from picking up the next pending source.
|
||||
- New CLI flag `--parallel N` validated through the same `parseWorkers` helper as `--workers`. Default `min(sourceCount, --workers, 4)`. Pass `--parallel 1` to force the legacy serial behavior.
|
||||
- New constant `DEFAULT_PARALLEL_SOURCES = 4` in `src/core/sync-concurrency.ts` (sibling to the existing `DEFAULT_PARALLEL_WORKERS`).
|
||||
- Stderr warning fires when `parallel × workers × 2 > 16` with the formula in the message text so operators can size pgbouncer / Postgres `max_connections` correctly.
|
||||
- `--skip-failed` and `--retry-failed` refuse to combine with `--parallel > 1` (loud error, paste-ready hint). Source-scoping the failure log is filed as a v0.41+ follow-up.
|
||||
|
||||
**`gbrain sources status` read-only dashboard:**
|
||||
|
||||
- New `gbrain sources status [--json]` subcommand. Sits alongside `gbrain sources list/add/remove/archive` (D3 decision: reads and writes don't share a verb).
|
||||
- Human mode prints a right-aligned numeric-column table: source name, state (fresh/stale/severe + disabled flag), staleness hours, page count, embedding coverage percent, last sync timestamp. Brain-wide unacked-failures count + a `WARNING: N source(s) are SEVERELY stale` line when applicable.
|
||||
- `--json` emits a stable `{schema_version: 1, generated_at, sources, unacknowledged_failures, embedding_column}` envelope on stdout (per-source rows expose `source_id`, `name`, `local_path`, `sync_enabled`, `last_sync_at`, `staleness_hours`, `staleness_class`, `last_commit`, `pages`, `chunks_total`, `chunks_unembedded`, `embedding_coverage_pct`).
|
||||
- Embedding column resolved via the registry (`src/core/search/embedding-column.ts`) so Voyage / multimodal / non-default-column brains see counts against the column they actually use.
|
||||
- Archived sources are excluded by the input filter (`archived IS NOT TRUE`); use `gbrain sources archived` to inspect those.
|
||||
- Staleness thresholds match `gbrain doctor`'s sync-freshness rule (24h / 72h).
|
||||
|
||||
**Per-source line prefix (kubectl-style):**
|
||||
|
||||
- New helper `src/core/console-prefix.ts` exporting `withSourcePrefix(id, fn)`, `getSourcePrefix()`, `slog(...)`, `serr(...)`. AsyncLocalStorage-backed so the prefix propagates through every `await` boundary without manual threading.
|
||||
- 38 `console.log`/`console.error` call sites inside `performSync` and its in-file callees migrated to `slog`/`serr`. 16 sites inside `src/commands/embed.ts` (`runEmbedCore` + helpers) migrated too. `src/core/progress.ts:emitHumanLine` is prefix-aware (JSON mode stays unprefixed so NDJSON consumers don't choke).
|
||||
- Prefix uses `source.id` (slug-validated by `sources add`), NOT `source.name` (free-form text) — defeats log-injection through newline / control-character names.
|
||||
- Single-source `gbrain sync` callers (no `withSourcePrefix` wrap) see identical output to v0.40.2 — slog/serr fall through to bare console fns outside the wrap.
|
||||
|
||||
**`--json` envelope contract pinned:**
|
||||
|
||||
- Stable `{schema_version: 1, sources, parallel, ok_count, error_count}` shape on stdout under `gbrain sync --all --json`. `gbrain sources status --json` uses a parallel envelope shape.
|
||||
- Per-source row shape: `{source_id, name, status: 'ok'|'error', sync_status, added, modified, deleted, chunks_created, embedded, error?}`.
|
||||
- Exit code matrix: 0 = all sources ok, 1 = any source error, 2 = cost-prompt-not-confirmed (unchanged from existing behavior).
|
||||
- Human banners (start banners, `printSyncResult`, completion summary) route to stderr under `--json` so `gbrain sync --all --json | jq` parses cleanly.
|
||||
|
||||
**Dashboard SQL correctness:**
|
||||
|
||||
- `buildSyncStatusReport` SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with the active embedding column resolved from the registry. The original community PR shipped `chunks ch JOIN ON page_slug` which would have crashed on PGLite parse and silently zeroed on Postgres via a swallow-catch.
|
||||
- Errors from the dashboard SQL now propagate. Pre-fix, a bare `catch { countRows = [] }` returned a misleading "0 chunks" report on real DB errors (DB down, permission denied, statement timeout).
|
||||
|
||||
**Tests:**
|
||||
|
||||
- New `test/console-prefix.test.ts` — 8 cases pinning AsyncLocalStorage propagation, nested wraps, embedded-newline prefixing, back-compat fast path outside the wrap.
|
||||
- New `test/sync-all-parallel.test.ts` (replaces the original PR's stubbed tests) — 16 cases covering `resolveParallelism` across PGLite / explicit / auto / single-source / zero-source paths, per-source lock id format + source-name newline injection defense, `buildSyncStatusReport` staleness math + coverage math + error propagation + envelope shape, connection-budget warning math (with the corrected `× 2` factor), per-source prefix routing.
|
||||
- New `test/e2e/sync-status-pglite.test.ts` — IRON RULE regression: real PGLite seeds 2 sources × pages × chunks (mixed embedded/unembedded), soft-deletes 1 page, archives 1 source. Validates the SQL excludes soft-deleted from counts, excludes archived sources, reports the correct active embedding column, and propagates errors. This is the case that would have caught the PR's original broken SQL.
|
||||
- All 148 existing sync-adjacent tests continue to pass (`test/sync.test.ts`, `test/sync-parallel.test.ts`, `test/sync-concurrency.test.ts`, `test/sync-failures.test.ts`, plus the new suites).
|
||||
|
||||
**Compatibility:**
|
||||
|
||||
- No schema changes. No new dependencies.
|
||||
- Single-source / non-`--all` paths: bit-for-bit identical behavior to v0.40.2.
|
||||
- PGLite users get serial behavior (single-connection engine). The parallel path is a Postgres-only win.
|
||||
|
||||
## To take advantage of v0.40.6.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't:
|
||||
|
||||
1. **No migrations to run.** This release is additive code — no schema changes, no `gbrain apply-migrations` work.
|
||||
2. **Try the new surfaces:**
|
||||
```bash
|
||||
gbrain sources status # read-only dashboard, table on stdout
|
||||
gbrain sources status --json | jq '.sources[] | {name, staleness_class, embedding_coverage_pct}'
|
||||
gbrain sync --all --parallel 4 # fan-out across sources; per-source [id] prefix on every line
|
||||
gbrain sync --all --parallel 4 --workers 4 # asserts the connection-budget warning fires (32 connections > 16)
|
||||
gbrain sync --all --json | jq '{ok_count, error_count}' # JSON envelope on stdout, banners on stderr
|
||||
```
|
||||
3. **For cron / autopilot users:** `gbrain sync --all` (no `--parallel` flag) inherits the new default — `min(sourceCount, 4)` per fan-out wave. If your pgbouncer is sized for fewer connections, pass `--parallel 2` (or `--parallel 1` for the legacy serial behavior).
|
||||
4. **If anything looks off,** file an issue at https://github.com/garrytan/gbrain/issues with:
|
||||
- output of `gbrain doctor`
|
||||
- the exact command you ran + the stderr output (the connection-budget warning + any per-source error messages are particularly useful)
|
||||
## [0.40.5.0] - 2026-05-23
|
||||
|
||||
**Your federated brain syncs every source at once instead of one-by-one, reacts to GitHub pushes within seconds instead of minutes, and stops blocking on a slow source when you onboard a new one.**
|
||||
|
||||
@@ -1,5 +1,36 @@
|
||||
# TODOS
|
||||
|
||||
## v0.40.3.0 follow-ups (v0.41+)
|
||||
|
||||
- [ ] **v0.41+: source-scope the `sync-failures.jsonl` log so `--skip-failed` works under `--parallel > 1`.**
|
||||
v0.40.3.0 shipped `gbrain sync --all --parallel N` as a continuous worker pool
|
||||
with per-source DB locks. The remaining unsafe path: `recordSyncFailures()` /
|
||||
`acknowledgeSyncFailures()` in `src/core/sync.ts` write to a brain-global JSONL
|
||||
file at `~/.gbrain/sync-failures.jsonl` with no per-source scope. Under parallel
|
||||
sync, source A's `--skip-failed` ack can swallow source B's failures recorded
|
||||
while B was still running. v0.40.3.0's safe interim: refuse to combine
|
||||
`--skip-failed` / `--retry-failed` with `--parallel > 1` (loud error, paste-ready
|
||||
hint pointing at `--parallel 1`). The proper fix: (1) extend the JSONL row
|
||||
schema with a `source_id` field; (2) `recordSyncFailures(failures, sourceId)`
|
||||
stamps the field; (3) `acknowledgeSyncFailures({sourceId})` filters acks to
|
||||
one source's rows; (4) `unacknowledgedSyncFailures({sourceId})` reads the
|
||||
subset. Drop the v0.40.3.0 restriction once source-scoped acks are
|
||||
deterministic. Estimate: ~1-2 days. Filed during v0.40.3.0 plan review by
|
||||
Codex outside-voice (decision D15 → B in the eng-review plan at
|
||||
`~/.claude/plans/system-instruction-you-are-working-fluttering-grove.md`).
|
||||
|
||||
- [ ] **v0.41+ (optional): extend `checkSyncFreshness` to include `embedding_coverage_pct`
|
||||
per source.** v0.40.3.0 plan originally proposed adding a NEW doctor check
|
||||
`sync_freshness_per_source` consuming `buildSyncStatusReport`. Codex caught
|
||||
that `checkSyncFreshness` (`src/commands/doctor.ts:~1609`) is ALREADY per-source —
|
||||
iterates `WHERE local_path IS NOT NULL`, emits per-source messages with
|
||||
paste-ready `gbrain sync --source <id>` hints, warns at 24h, fails at 72h.
|
||||
The plan dropped the duplicate (D9 → A). The real follow-up is to extend
|
||||
`checkSyncFreshness`'s message to include `embedding_coverage_pct` per source
|
||||
alongside the staleness number so doctor surfaces the coverage gap inline.
|
||||
Implementation: reuse `buildSyncStatusReport` from `src/commands/sync.ts`,
|
||||
fold coverage into the existing per-source message string. ~half a day.
|
||||
|
||||
## v0.40.1.0 Track D follow-ups (v0.41+)
|
||||
|
||||
- [ ] **v0.41+: contributor-mode CI capture for BrainBench-Real replay gate.**
|
||||
|
||||
+6
-1025
File diff suppressed because one or more lines are too long
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.40.5.0",
|
||||
"version": "0.40.6.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -81,6 +81,10 @@ export const SECTIONS: DocSection[] = [
|
||||
description:
|
||||
"MECE directory structure (people/, companies/, concepts/).",
|
||||
path: "docs/GBRAIN_RECOMMENDED_SCHEMA.md",
|
||||
// v0.40.6.0: 64KB reference doc. Web index entry stays; the single-fetch
|
||||
// bundle gets the README + setup guides instead. Keeps llms-full.txt
|
||||
// under the 600KB budget as CLAUDE.md grows with each release.
|
||||
includeInFull: false,
|
||||
},
|
||||
{
|
||||
title: "docs/guides/live-sync.md",
|
||||
|
||||
+17
-16
@@ -6,6 +6,7 @@ import { createProgress, type ProgressReporter } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { assertEmbeddingEnabled } from '../core/embedding-dim-check.ts';
|
||||
import { loadConfig } from '../core/config.ts';
|
||||
import { slog, serr } from '../core/console-prefix.ts';
|
||||
|
||||
export interface EmbedOpts {
|
||||
/** Embed ALL pages (every chunk). */
|
||||
@@ -151,7 +152,7 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
|
||||
try {
|
||||
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId);
|
||||
} catch (e: unknown) {
|
||||
console.error(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
|
||||
serr(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
@@ -209,7 +210,7 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
|
||||
} else {
|
||||
const slug = args.find(a => !a.startsWith('--'));
|
||||
if (!slug) {
|
||||
console.error('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run]');
|
||||
serr('Usage: gbrain embed [<slug>|--all|--stale|--slugs s1 s2 ...] [--dry-run]');
|
||||
process.exit(1);
|
||||
}
|
||||
opts = { slug, dryRun, sourceId };
|
||||
@@ -237,9 +238,9 @@ export async function runEmbed(engine: BrainEngine, args: string[]): Promise<Emb
|
||||
// D.2: surface dim-mismatch failures with the paste-ready recipe
|
||||
// instead of the raw Postgres error message.
|
||||
if (e instanceof EmbeddingDimMismatchError) {
|
||||
console.error('\n' + e.recipeMessage + '\n');
|
||||
serr('\n' + e.recipeMessage + '\n');
|
||||
} else {
|
||||
console.error(e instanceof Error ? e.message : String(e));
|
||||
serr(e instanceof Error ? e.message : String(e));
|
||||
}
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -295,7 +296,7 @@ async function embedPage(
|
||||
result.skipped += chunks.length - toEmbed.length;
|
||||
|
||||
if (toEmbed.length === 0) {
|
||||
console.log(`${slug}: all ${chunks.length} chunks already embedded`);
|
||||
slog(`${slug}: all ${chunks.length} chunks already embedded`);
|
||||
result.pages_processed++;
|
||||
return;
|
||||
}
|
||||
@@ -322,7 +323,7 @@ async function embedPage(
|
||||
await engine.upsertChunks(slug, updated, opts);
|
||||
result.embedded += toEmbed.length;
|
||||
result.pages_processed++;
|
||||
console.log(`${slug}: embedded ${toEmbed.length} chunks`);
|
||||
slog(`${slug}: embedded ${toEmbed.length} chunks`);
|
||||
}
|
||||
|
||||
async function embedAll(
|
||||
@@ -409,7 +410,7 @@ async function embedAll(
|
||||
await engine.upsertChunks(page.slug, updated, pageOpts);
|
||||
result.embedded += toEmbed.length;
|
||||
} catch (e: unknown) {
|
||||
console.error(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
|
||||
serr(`\n Error embedding ${page.slug}: ${e instanceof Error ? e.message : e}`);
|
||||
}
|
||||
|
||||
processed++;
|
||||
@@ -435,9 +436,9 @@ async function embedAll(
|
||||
|
||||
// Stdout summary preserved for scripts/tests that grep for counts.
|
||||
if (dryRun) {
|
||||
console.log(`[dry-run] Would embed ${result.would_embed} chunks across ${pages.length} pages`);
|
||||
slog(`[dry-run] Would embed ${result.would_embed} chunks across ${pages.length} pages`);
|
||||
} else {
|
||||
console.log(`Embedded ${result.embedded} chunks across ${pages.length} pages`);
|
||||
slog(`Embedded ${result.embedded} chunks across ${pages.length} pages`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -474,9 +475,9 @@ async function embedAllStale(
|
||||
const staleCount = await engine.countStaleChunks(sourceOpt);
|
||||
if (staleCount === 0) {
|
||||
if (dryRun) {
|
||||
console.log('[dry-run] Would embed 0 chunks (0 stale found)');
|
||||
slog('[dry-run] Would embed 0 chunks (0 stale found)');
|
||||
} else {
|
||||
console.log('Embedded 0 chunks (0 stale found)');
|
||||
slog('Embedded 0 chunks (0 stale found)');
|
||||
}
|
||||
return;
|
||||
}
|
||||
@@ -485,7 +486,7 @@ async function embedAllStale(
|
||||
result.would_embed += staleCount;
|
||||
result.total_chunks += staleCount;
|
||||
if (onProgress) onProgress(1, 1, 0);
|
||||
console.log(`[dry-run] Would embed ${staleCount} stale chunks`);
|
||||
slog(`[dry-run] Would embed ${staleCount} stale chunks`);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -518,7 +519,7 @@ async function embedAllStale(
|
||||
while (true) {
|
||||
if (budgetSignal.aborted) {
|
||||
if (!budgetExitNotified) {
|
||||
console.error(`\n [embed] wall-clock budget (${BUDGET_MS}ms) exceeded; exiting cleanly. Re-run picks up via partial index.`);
|
||||
serr(`\n [embed] wall-clock budget (${BUDGET_MS}ms) exceeded; exiting cleanly. Re-run picks up via partial index.`);
|
||||
budgetExitNotified = true;
|
||||
}
|
||||
break;
|
||||
@@ -576,7 +577,7 @@ async function embedAllStale(
|
||||
// Budget-fired aborts are expected on the way out; don't spam
|
||||
// per-page "Error embedding" lines when we're shutting down.
|
||||
if (budgetSignal.aborted) return;
|
||||
console.error(`\n Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
|
||||
serr(`\n Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
|
||||
}
|
||||
totalProcessedPages++;
|
||||
result.pages_processed++;
|
||||
@@ -605,7 +606,7 @@ async function embedAllStale(
|
||||
clearTimeout(budgetTimer);
|
||||
}
|
||||
|
||||
console.log(`Embedded ${result.embedded} chunks across ${totalProcessedPages} pages`);
|
||||
slog(`Embedded ${result.embedded} chunks across ${totalProcessedPages} pages`);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -731,7 +732,7 @@ export async function embedBatchWithBackoff(
|
||||
if (!isRateLimit || attempt === MAX_RATE_LIMIT_RETRIES) throw e;
|
||||
|
||||
const delayMs = parseRetryDelayMs(msg);
|
||||
console.error(` [rate-limit] attempt ${attempt + 1}/${MAX_RATE_LIMIT_RETRIES}, waiting ${delayMs}ms...`);
|
||||
serr(` [rate-limit] attempt ${attempt + 1}/${MAX_RATE_LIMIT_RETRIES}, waiting ${delayMs}ms...`);
|
||||
await abortableSleep(delayMs, signal);
|
||||
}
|
||||
}
|
||||
|
||||
+20
-1
@@ -878,6 +878,15 @@ async function runCurrent(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
|
||||
// ── Dispatcher ──────────────────────────────────────────────
|
||||
|
||||
// v0.40.6.0: my duplicate `runStatus` (line ~895 pre-resolution) was
|
||||
// removed during the v0.40.5 merge. Master's source-health.ts-backed
|
||||
// runStatus at line ~582 is a strict superset (adds lag / embed coverage
|
||||
// / failed-job count / queue depth columns). The `buildSyncStatusReport`
|
||||
// + `printSyncStatusReport` exports from src/commands/sync.ts remain
|
||||
// available as a library API for callers who want the v0.40.6.0-specific
|
||||
// shape (used by test/e2e/sync-status-pglite.test.ts as the IRON RULE
|
||||
// regression).
|
||||
|
||||
export async function runSources(engine: BrainEngine, args: string[]): Promise<void> {
|
||||
const sub = args[0];
|
||||
const rest = args.slice(1);
|
||||
@@ -897,7 +906,12 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise<v
|
||||
case 'purge': return runPurge(engine, rest);
|
||||
case 'archived': return runListArchived(engine, rest);
|
||||
case 'current': return runCurrent(engine, rest);
|
||||
// v0.40.5.0 Federated Sync v2
|
||||
// v0.40.5.0 Federated Sync v2 (master) + v0.40.6.0 status dashboard
|
||||
// The status function lives at the line-582 declaration (master's
|
||||
// source-health.ts-backed version). My duplicate runStatus (line ~895
|
||||
// in the post-merge file, the buildSyncStatusReport-backed one) is
|
||||
// removed below since master's federation_health metrics dashboard is
|
||||
// a superset.
|
||||
case 'status': return runStatus(engine, rest);
|
||||
case 'webhook': return runWebhook(engine, rest);
|
||||
case 'tracked-branch': return runTrackedBranch(engine, rest);
|
||||
@@ -928,6 +942,11 @@ Subcommands:
|
||||
when the source has data (pages/chunks/embeddings).
|
||||
archive <id> Soft-delete: hide from search, preserve data for ${SOFT_DELETE_TTL_HOURS}h.
|
||||
restore <id> [--no-federate] Un-archive a soft-deleted source.
|
||||
status [--json] v0.40.3.0 — read-only per-source dashboard:
|
||||
last sync, staleness, page count,
|
||||
embedding coverage, unacked failures.
|
||||
--json emits {schema_version:1, ...} on
|
||||
stdout for monitoring pipelines.
|
||||
archived [--json] List soft-deleted sources and their expiry.
|
||||
purge [<id>] [--confirm-destructive]
|
||||
Permanently delete archived sources.
|
||||
|
||||
+673
-96
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,145 @@
|
||||
/**
|
||||
* v0.40.3.0 — per-source console line-prefixing via AsyncLocalStorage.
|
||||
*
|
||||
* Under `gbrain sync --all --parallel > 1`, multiple per-source syncs run
|
||||
* concurrently. Without prefixing, every `console.log` from `performSync`
|
||||
* (git pull lines, embed progress, drift gates, ~30+ call sites total)
|
||||
* interleaves on the terminal — operators can't tell which source emitted
|
||||
* which line. kubectl `--prefix` and docker-compose solve the same problem
|
||||
* with `[<id>] ` prefixes that ride along with each line.
|
||||
*
|
||||
* Usage:
|
||||
* await withSourcePrefix(src.id, () => performSync(engine, opts));
|
||||
* // Inside performSync (and its callees), call slog() / serr() instead
|
||||
* // of console.log() / console.error(). When invoked under a wrap, the
|
||||
* // prefix is automatically prepended to each emitted line. Outside the
|
||||
* // wrap (single-source sync, doctor, anywhere else), slog/serr are
|
||||
* // identical to console.log/console.error — back-compat preserved.
|
||||
*
|
||||
* Multi-line strings: each newline-separated segment gets its own prefix,
|
||||
* so a `serr('phase started\\n details: x')` under prefix `[foo]` emits:
|
||||
* [foo] phase started
|
||||
* [foo] details: x
|
||||
*
|
||||
* Why source.id and not source.name:
|
||||
* `sources add --name` accepts arbitrary text — operators (or attackers)
|
||||
* could embed newlines or control characters that break grep filtering
|
||||
* and impersonate other sources' lines. `source.id` is slug-validated by
|
||||
* the existing sources schema (lowercase alphanumeric + dash) and safe to
|
||||
* prefix raw. The per-source banner (one-shot at start of each source)
|
||||
* can still display `source.name` for human readability.
|
||||
*
|
||||
* Why AsyncLocalStorage:
|
||||
* Propagates through every `await` boundary without manual threading.
|
||||
* `withSourcePrefix(id, () => performSync(...))` covers performSync AND
|
||||
* every async function it calls (runImport, runEmbedCore, parallel-worker
|
||||
* subphases) as long as those functions use slog/serr. Nested wraps
|
||||
* restore the outer prefix on exit.
|
||||
*
|
||||
* Coverage in v0.40.3.0 (see CLAUDE.md for the canonical list):
|
||||
* - src/commands/sync.ts (performSync + in-file callees)
|
||||
* - src/commands/embed.ts (runEmbedCore + helpers)
|
||||
* - src/core/progress.ts (heartbeat / progress writer)
|
||||
*
|
||||
* Anything outside those modules that writes directly to stdout/stderr will
|
||||
* NOT get the prefix. If you find a delegate-module line that escapes the
|
||||
* prefix under parallel sync, it's a missed migration target — file an
|
||||
* issue (per D12 → A full-lake; honest about scope).
|
||||
*/
|
||||
|
||||
import { AsyncLocalStorage } from 'node:async_hooks';
|
||||
|
||||
const __prefixStore = new AsyncLocalStorage<string>();
|
||||
|
||||
/**
|
||||
* Run `fn` with an active per-source prefix `id`. Within the closure,
|
||||
* `slog` and `serr` will prepend `[id] ` to every line they emit. The
|
||||
* prefix is automatically propagated through `await` boundaries.
|
||||
*
|
||||
* Nested wraps replace the active prefix for the inner closure and
|
||||
* restore the outer prefix when the inner returns/throws.
|
||||
*
|
||||
* `id` should be a slug-validated source identifier (NOT a free-form
|
||||
* display name) — see module docstring for the security rationale.
|
||||
*/
|
||||
export function withSourcePrefix<T>(id: string, fn: () => Promise<T>): Promise<T> {
|
||||
return __prefixStore.run(id, fn);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the currently-active per-source prefix. Returns null when called
|
||||
* outside a `withSourcePrefix` scope. Test seam; production code should
|
||||
* use `slog` / `serr` instead of reading the prefix directly.
|
||||
*/
|
||||
export function getSourcePrefix(): string | null {
|
||||
return __prefixStore.getStore() ?? null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Prefix-aware replacement for `console.log`. When called inside a
|
||||
* `withSourcePrefix(id, ...)` scope, prepends `[id] ` to every line of
|
||||
* the formatted output. Outside the scope, behaves exactly like
|
||||
* `console.log` (writes to stdout).
|
||||
*/
|
||||
export function slog(...args: unknown[]): void {
|
||||
const prefix = getSourcePrefix();
|
||||
if (prefix === null) {
|
||||
// Back-compat fast path: bare console.log semantics.
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(...args);
|
||||
return;
|
||||
}
|
||||
process.stdout.write(prefixLines(formatArgs(args), prefix) + '\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* Prefix-aware replacement for `console.error`. Same prefix semantics as
|
||||
* `slog`, but writes to stderr. Use for warnings, errors, and any output
|
||||
* that should NOT pollute `--json` stdout.
|
||||
*/
|
||||
export function serr(...args: unknown[]): void {
|
||||
const prefix = getSourcePrefix();
|
||||
if (prefix === null) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.error(...args);
|
||||
return;
|
||||
}
|
||||
process.stderr.write(prefixLines(formatArgs(args), prefix) + '\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* Format an args list the way `console.log` would, but as a single string.
|
||||
* Strings stay raw; everything else goes through JSON.stringify with a
|
||||
* fallback to String() for non-serializable values. Multi-arg invocations
|
||||
* join with spaces (matches console.log's default delimiter).
|
||||
*/
|
||||
function formatArgs(args: unknown[]): string {
|
||||
return args
|
||||
.map((a) => {
|
||||
if (typeof a === 'string') return a;
|
||||
if (a instanceof Error) return a.stack ?? a.message;
|
||||
try {
|
||||
return JSON.stringify(a);
|
||||
} catch {
|
||||
return String(a);
|
||||
}
|
||||
})
|
||||
.join(' ');
|
||||
}
|
||||
|
||||
/**
|
||||
* Prepend `[prefix] ` to every line of `text`. Preserves empty trailing
|
||||
* lines (so a trailing newline in input stays a trailing newline in
|
||||
* output — the caller's `+ '\\n'` adds the final separator). Embedded
|
||||
* newlines inside the string each get their own prefix.
|
||||
*
|
||||
* Examples (prefix = 'foo'):
|
||||
* '' → '[foo] '
|
||||
* 'a' → '[foo] a'
|
||||
* 'a\\nb' → '[foo] a\\n[foo] b'
|
||||
* 'a\\nb\\n' → '[foo] a\\n[foo] b\\n[foo] '
|
||||
*/
|
||||
function prefixLines(text: string, prefix: string): string {
|
||||
const tag = `[${prefix}] `;
|
||||
return text.split('\n').map((line) => tag + line).join('\n');
|
||||
}
|
||||
+14
-2
@@ -204,11 +204,23 @@ class Reporter implements ReporterInternal {
|
||||
}
|
||||
|
||||
private emitHumanLine(line: string): void {
|
||||
// v0.40.3.0 — per-source prefix support. When called inside a
|
||||
// `withSourcePrefix(id, ...)` scope, prepend `[id] ` so the
|
||||
// progress reporter's lines stay in the same prefix family as
|
||||
// sync's slog/serr output. TTY-rewrite mode gets the prefix
|
||||
// INSIDE the \r-clear (after the clear-to-EOL escape, before the
|
||||
// line content) so the rewritten line carries the prefix too.
|
||||
//
|
||||
// JSON mode (emitJson) is deliberately NOT prefixed — consumers
|
||||
// parse the NDJSON and would choke on a `[id] {...}` shape.
|
||||
const { getSourcePrefix } = require('./console-prefix.ts') as typeof import('./console-prefix.ts');
|
||||
const prefix = getSourcePrefix();
|
||||
const tagged = prefix ? `[${prefix}] ${line}` : line;
|
||||
if (this.renderMode === 'human-tty') {
|
||||
// \r rewrite: clear-to-EOL then carriage-return-positioned line.
|
||||
safeWrite(this.stream, `\r\x1b[2K${line}`);
|
||||
safeWrite(this.stream, `\r\x1b[2K${tagged}`);
|
||||
} else {
|
||||
safeWrite(this.stream, line + '\n');
|
||||
safeWrite(this.stream, tagged + '\n');
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -22,9 +22,30 @@ export const AUTO_CONCURRENCY_FILE_THRESHOLD = 100;
|
||||
* auto path; explicit `--workers N` bypasses this. */
|
||||
export const PARALLEL_FILE_FLOOR = 50;
|
||||
|
||||
/** Default worker count when auto-concurrency fires. */
|
||||
/** Default per-file worker count inside a single source's import phase. */
|
||||
export const DEFAULT_PARALLEL_WORKERS = 4;
|
||||
|
||||
/**
|
||||
* v0.40.3.0 — default number of sources synced concurrently under
|
||||
* `gbrain sync --all`. Sibling of `DEFAULT_PARALLEL_WORKERS`; the two
|
||||
* are deliberately separate constants because they cover different axes:
|
||||
*
|
||||
* total live Postgres connections per fan-out wave
|
||||
* ≈ DEFAULT_PARALLEL_SOURCES × DEFAULT_PARALLEL_WORKERS × 2
|
||||
* (per-source fan-out) (per-file workers) (per-worker pool)
|
||||
*
|
||||
* Default `4 × 4 × 2 = 32` connections per wave + parent pool. Tuned for
|
||||
* a Postgres+pgbouncer setup sized at 40-50 connections. Operators on
|
||||
* smaller pools should lower either knob; sync.ts emits a stderr warning
|
||||
* when the product exceeds 16 so the operator can size pgbouncer / Postgres
|
||||
* `max_connections` accordingly.
|
||||
*
|
||||
* Conflating these two as one constant was a v0.40.2 footgun that Codex
|
||||
* flagged during the v0.40.3.0 plan review. Keeping them separate makes
|
||||
* the multiplication visible to future contributors.
|
||||
*/
|
||||
export const DEFAULT_PARALLEL_SOURCES = 4;
|
||||
|
||||
/**
|
||||
* Resolve effective worker count for a sync/import operation.
|
||||
*
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
/**
|
||||
* Tests for the v0.40.3.0 per-source console-prefix helper.
|
||||
*
|
||||
* Why these exist:
|
||||
* AsyncLocalStorage propagation through await boundaries is what makes
|
||||
* `withSourcePrefix(src.id, () => performSync(...))` correct without
|
||||
* manual threading. If the propagation breaks (e.g. via setImmediate or
|
||||
* a Promise resolver that bypasses ALS), every `slog` inside performSync
|
||||
* loses its prefix and operators see interleaved unreadable output under
|
||||
* `--parallel > 1`. These cases pin the contract.
|
||||
*
|
||||
* The line-splitting math (embedded newlines each get their own prefix)
|
||||
* is the difference between greppable output and a wall of text where
|
||||
* only the first line of every multi-line emitter has a prefix.
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
getSourcePrefix,
|
||||
slog,
|
||||
serr,
|
||||
withSourcePrefix,
|
||||
} from '../src/core/console-prefix.ts';
|
||||
|
||||
// Capture stdout/stderr writes for the duration of a callback. Restores
|
||||
// the original write fns even on throw. Returns the captured chunks.
|
||||
function captureStdio<T>(fn: () => T | Promise<T>): Promise<{ stdout: string; stderr: string; result: T }> {
|
||||
const stdoutChunks: string[] = [];
|
||||
const stderrChunks: string[] = [];
|
||||
const origStdout = process.stdout.write.bind(process.stdout);
|
||||
const origStderr = process.stderr.write.bind(process.stderr);
|
||||
process.stdout.write = ((chunk: string | Uint8Array): boolean => {
|
||||
stdoutChunks.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8'));
|
||||
return true;
|
||||
}) as typeof process.stdout.write;
|
||||
process.stderr.write = ((chunk: string | Uint8Array): boolean => {
|
||||
stderrChunks.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8'));
|
||||
return true;
|
||||
}) as typeof process.stderr.write;
|
||||
return Promise.resolve(fn())
|
||||
.then((result) => ({ stdout: stdoutChunks.join(''), stderr: stderrChunks.join(''), result }))
|
||||
.finally(() => {
|
||||
process.stdout.write = origStdout;
|
||||
process.stderr.write = origStderr;
|
||||
});
|
||||
}
|
||||
|
||||
describe('withSourcePrefix / getSourcePrefix', () => {
|
||||
test('prefix is applied inside the wrap', async () => {
|
||||
let observed: string | null = 'unset';
|
||||
await withSourcePrefix('media-corpus', async () => {
|
||||
observed = getSourcePrefix();
|
||||
});
|
||||
expect(observed).toBe('media-corpus');
|
||||
});
|
||||
|
||||
test('no prefix outside the wrap', async () => {
|
||||
// Sanity: getSourcePrefix returns null when called outside any wrap.
|
||||
// This is what guarantees slog/serr fall through to bare console.log
|
||||
// for single-source / non-parallel callers (back-compat invariant).
|
||||
expect(getSourcePrefix()).toBeNull();
|
||||
});
|
||||
|
||||
test('nested wrap uses innermost prefix; outer restored on exit', async () => {
|
||||
const observed: { outerBefore: string | null; inner: string | null; outerAfter: string | null } = {
|
||||
outerBefore: null,
|
||||
inner: null,
|
||||
outerAfter: null,
|
||||
};
|
||||
await withSourcePrefix('outer', async () => {
|
||||
observed.outerBefore = getSourcePrefix();
|
||||
await withSourcePrefix('inner', async () => {
|
||||
observed.inner = getSourcePrefix();
|
||||
});
|
||||
observed.outerAfter = getSourcePrefix();
|
||||
});
|
||||
expect(observed.outerBefore).toBe('outer');
|
||||
expect(observed.inner).toBe('inner');
|
||||
expect(observed.outerAfter).toBe('outer');
|
||||
});
|
||||
|
||||
test('prefix propagates through await boundaries (the load-bearing ALS contract)', async () => {
|
||||
// This is the case that justifies AsyncLocalStorage over a global
|
||||
// variable. If propagation breaks, every async function called from
|
||||
// inside performSync loses its prefix and the whole feature is
|
||||
// theatrical. We assert by awaiting through Promise.resolve and a
|
||||
// setImmediate microtask — both common patterns inside the sync path.
|
||||
const observed: Array<string | null> = [];
|
||||
await withSourcePrefix('foo', async () => {
|
||||
observed.push(getSourcePrefix());
|
||||
await Promise.resolve();
|
||||
observed.push(getSourcePrefix());
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
observed.push(getSourcePrefix());
|
||||
});
|
||||
expect(observed).toEqual(['foo', 'foo', 'foo']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('slog / serr line prefixing', () => {
|
||||
test('slog under a wrap prefixes a single-line string and writes to stdout', async () => {
|
||||
const { stdout, stderr } = await captureStdio(async () => {
|
||||
await withSourcePrefix('media-corpus', async () => {
|
||||
slog('phase started');
|
||||
});
|
||||
});
|
||||
expect(stdout).toBe('[media-corpus] phase started\n');
|
||||
expect(stderr).toBe('');
|
||||
});
|
||||
|
||||
test('serr under a wrap prefixes and writes to stderr', async () => {
|
||||
const { stdout, stderr } = await captureStdio(async () => {
|
||||
await withSourcePrefix('zion-brain', async () => {
|
||||
serr('warn: skipping');
|
||||
});
|
||||
});
|
||||
expect(stdout).toBe('');
|
||||
expect(stderr).toBe('[zion-brain] warn: skipping\n');
|
||||
});
|
||||
|
||||
test('embedded newlines each get their own prefix (greppable multi-line output)', async () => {
|
||||
// Without per-line prefixing, only the first line of a multi-line
|
||||
// emit would carry the source tag; the rest would be ambiguous under
|
||||
// interleaved parallel output. This case pins the kubectl-style
|
||||
// semantics that make `grep '[media-corpus]'` actually work.
|
||||
const { stdout } = await captureStdio(async () => {
|
||||
await withSourcePrefix('media-corpus', async () => {
|
||||
slog('phase started\n details: x\n details: y');
|
||||
});
|
||||
});
|
||||
expect(stdout).toBe(
|
||||
'[media-corpus] phase started\n[media-corpus] details: x\n[media-corpus] details: y\n',
|
||||
);
|
||||
});
|
||||
|
||||
test('slog outside a wrap is identical to console.log (back-compat invariant)', async () => {
|
||||
// Single-source / non-parallel callers (single-source gbrain sync,
|
||||
// doctor, every existing caller that doesn't opt into withSourcePrefix)
|
||||
// must see bit-for-bit identical output. The fast path through
|
||||
// console.log preserves that — no prefix, no extra formatting.
|
||||
const origLog = console.log;
|
||||
const captured: unknown[][] = [];
|
||||
// eslint-disable-next-line no-console
|
||||
console.log = (...args: unknown[]) => { captured.push(args); };
|
||||
try {
|
||||
slog('plain message', { foo: 1 });
|
||||
expect(captured).toEqual([['plain message', { foo: 1 }]]);
|
||||
} finally {
|
||||
// eslint-disable-next-line no-console
|
||||
console.log = origLog;
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,285 @@
|
||||
/**
|
||||
* IRON RULE E2E regression for v0.40.3.0 `buildSyncStatusReport` SQL.
|
||||
*
|
||||
* Why this exists:
|
||||
* PR #1314 shipped a broken SQL query for the per-source dashboard:
|
||||
*
|
||||
* FROM chunks ch JOIN pages pg ON pg.slug = ch.page_slug
|
||||
*
|
||||
* The actual schema is `content_chunks` joined on `page_id`. Every
|
||||
* unit test in `test/sync-all-parallel.test.ts` stubbed `executeRaw`
|
||||
* with regex-keyed canned responses, so the broken SQL never ran.
|
||||
* The defensive `try/catch { countRows = [] }` would have silently
|
||||
* returned "0 chunks for every source" in production — a misleading
|
||||
* "your brain is empty" report on a real Postgres brain.
|
||||
*
|
||||
* This case exercises the REAL SQL against PGLite. If buildSyncStatusReport
|
||||
* ever drifts back to a bad table name or join key, this test fails
|
||||
* loudly at parse time (PGLite rejects unknown columns).
|
||||
*
|
||||
* Per CLAUDE.md's IRON RULE: "regression test is added to the plan as
|
||||
* a critical requirement. No skipping." Codex's outside-voice review
|
||||
* of the original plan caught this missing case.
|
||||
*
|
||||
* Coverage:
|
||||
* - Canonical SQL parses and returns rows (Blocker 1 / Codex P0 #1)
|
||||
* - Soft-deleted pages excluded from pages count + chunks count
|
||||
* (v0.26.5 soft-delete shipped without updating this query path)
|
||||
* - Archived sources excluded from the dashboard input
|
||||
* (Expansion 3 from plan review)
|
||||
* - Embedding column resolved via the registry (D16) — counts
|
||||
* against the active column, not a hardcoded name
|
||||
* - Errors propagate instead of returning lying zeroes (Q2 sub-fix)
|
||||
*
|
||||
* No DATABASE_URL needed; PGLite is in-memory.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { buildSyncStatusReport } from '../../src/commands/sync.ts';
|
||||
import type { BrainEngine } from '../../src/core/engine.ts';
|
||||
import type { ChunkInput } from '../../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
// Basis-vector embeddings keep the seed cheap (no real model needed) and
|
||||
// deterministic. The dashboard only cares about NULL vs non-NULL on the
|
||||
// embedding column, not the vector content.
|
||||
function basisEmbedding(idx: number, dim = 1536): Float32Array {
|
||||
const emb = new Float32Array(dim);
|
||||
emb[idx % dim] = 1.0;
|
||||
return emb;
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
|
||||
// Seed two non-default sources: 'source-a' (active) + 'source-b' (active)
|
||||
// + 'source-c' (archived — must be excluded by dashboard input filter).
|
||||
// 'default' is auto-seeded by pglite-schema.ts:50.
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO sources (id, name, local_path, last_commit, last_sync_at, config, archived)
|
||||
VALUES
|
||||
('source-a', 'source-a', '/tmp/source-a', 'aaa', NOW() - INTERVAL '1 hour', '{"syncEnabled": true}'::jsonb, FALSE),
|
||||
('source-b', 'source-b', '/tmp/source-b', 'bbb', NOW() - INTERVAL '30 hours', '{"syncEnabled": true}'::jsonb, FALSE),
|
||||
('source-c', 'source-c', '/tmp/source-c', 'ccc', NOW() - INTERVAL '100 hours', '{}'::jsonb, TRUE)`,
|
||||
);
|
||||
|
||||
// Seed pages + chunks per source via the canonical engine API.
|
||||
// source-a: 3 pages, 6 chunks, 4 embedded (2 unembedded)
|
||||
// source-b: 2 pages, 4 chunks, 4 embedded (0 unembedded)
|
||||
// Plus a soft-deleted page on source-a that should NOT count toward
|
||||
// either pages or chunks (the v0.26.5 soft-delete-aware regression).
|
||||
|
||||
// source-a page 1: 2 chunks, both embedded
|
||||
await engine.putPage('a/page-1', {
|
||||
type: 'note',
|
||||
title: 'A page 1',
|
||||
compiled_truth: 'content for a/page-1',
|
||||
timeline: '',
|
||||
}, { sourceId: 'source-a' });
|
||||
await engine.upsertChunks('a/page-1', [
|
||||
{ chunk_index: 0, chunk_text: 'chunk a/1/0', chunk_source: 'compiled_truth', embedding: basisEmbedding(0), token_count: 4 },
|
||||
{ chunk_index: 1, chunk_text: 'chunk a/1/1', chunk_source: 'compiled_truth', embedding: basisEmbedding(1), token_count: 4 },
|
||||
] satisfies ChunkInput[], { sourceId: 'source-a' });
|
||||
|
||||
// source-a page 2: 2 chunks, both unembedded (no embedding field)
|
||||
await engine.putPage('a/page-2', {
|
||||
type: 'note',
|
||||
title: 'A page 2',
|
||||
compiled_truth: 'content for a/page-2',
|
||||
timeline: '',
|
||||
}, { sourceId: 'source-a' });
|
||||
await engine.upsertChunks('a/page-2', [
|
||||
{ chunk_index: 0, chunk_text: 'chunk a/2/0', chunk_source: 'compiled_truth', token_count: 4 },
|
||||
{ chunk_index: 1, chunk_text: 'chunk a/2/1', chunk_source: 'compiled_truth', token_count: 4 },
|
||||
] satisfies ChunkInput[], { sourceId: 'source-a' });
|
||||
|
||||
// source-a page 3: 2 chunks, both embedded
|
||||
await engine.putPage('a/page-3', {
|
||||
type: 'note',
|
||||
title: 'A page 3',
|
||||
compiled_truth: 'content for a/page-3',
|
||||
timeline: '',
|
||||
}, { sourceId: 'source-a' });
|
||||
await engine.upsertChunks('a/page-3', [
|
||||
{ chunk_index: 0, chunk_text: 'chunk a/3/0', chunk_source: 'compiled_truth', embedding: basisEmbedding(2), token_count: 4 },
|
||||
{ chunk_index: 1, chunk_text: 'chunk a/3/1', chunk_source: 'compiled_truth', embedding: basisEmbedding(3), token_count: 4 },
|
||||
] satisfies ChunkInput[], { sourceId: 'source-a' });
|
||||
|
||||
// SOFT-DELETED page on source-a: must NOT count toward pages or chunks.
|
||||
// This is the v0.26.5 regression — pre-fix, soft-deleted pages were
|
||||
// double-counted in the dashboard.
|
||||
await engine.putPage('a/page-deleted', {
|
||||
type: 'note',
|
||||
title: 'A page deleted',
|
||||
compiled_truth: 'content for a/page-deleted (will be soft-deleted)',
|
||||
timeline: '',
|
||||
}, { sourceId: 'source-a' });
|
||||
await engine.upsertChunks('a/page-deleted', [
|
||||
{ chunk_index: 0, chunk_text: 'should-not-count', chunk_source: 'compiled_truth', embedding: basisEmbedding(4), token_count: 4 },
|
||||
{ chunk_index: 1, chunk_text: 'should-not-count', chunk_source: 'compiled_truth', token_count: 4 },
|
||||
] satisfies ChunkInput[], { sourceId: 'source-a' });
|
||||
await engine.softDeletePage('a/page-deleted', { sourceId: 'source-a' });
|
||||
|
||||
// source-b: 2 pages × 2 chunks, all embedded
|
||||
await engine.putPage('b/page-1', {
|
||||
type: 'note',
|
||||
title: 'B page 1',
|
||||
compiled_truth: 'content for b/page-1',
|
||||
timeline: '',
|
||||
}, { sourceId: 'source-b' });
|
||||
await engine.upsertChunks('b/page-1', [
|
||||
{ chunk_index: 0, chunk_text: 'chunk b/1/0', chunk_source: 'compiled_truth', embedding: basisEmbedding(5), token_count: 4 },
|
||||
{ chunk_index: 1, chunk_text: 'chunk b/1/1', chunk_source: 'compiled_truth', embedding: basisEmbedding(6), token_count: 4 },
|
||||
] satisfies ChunkInput[], { sourceId: 'source-b' });
|
||||
|
||||
await engine.putPage('b/page-2', {
|
||||
type: 'note',
|
||||
title: 'B page 2',
|
||||
compiled_truth: 'content for b/page-2',
|
||||
timeline: '',
|
||||
}, { sourceId: 'source-b' });
|
||||
await engine.upsertChunks('b/page-2', [
|
||||
{ chunk_index: 0, chunk_text: 'chunk b/2/0', chunk_source: 'compiled_truth', embedding: basisEmbedding(7), token_count: 4 },
|
||||
{ chunk_index: 1, chunk_text: 'chunk b/2/1', chunk_source: 'compiled_truth', embedding: basisEmbedding(8), token_count: 4 },
|
||||
] satisfies ChunkInput[], { sourceId: 'source-b' });
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
describe('buildSyncStatusReport against real PGLite (IRON RULE regression for Blocker 1)', () => {
|
||||
test('correct SQL: content_chunks JOIN pages ON page_id (NOT chunks/page_slug)', async () => {
|
||||
// Caller-side source list mirrors what `gbrain sources status` (the
|
||||
// CLI route in sources.ts) would supply: `WHERE local_path IS NOT NULL
|
||||
// AND archived IS NOT TRUE`.
|
||||
const sources = await engine.executeRaw<{
|
||||
id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown>;
|
||||
}>(
|
||||
`SELECT id, name, local_path, config FROM sources
|
||||
WHERE local_path IS NOT NULL AND archived IS NOT TRUE
|
||||
ORDER BY id`,
|
||||
);
|
||||
// source-a + source-b only (source-c is archived; 'default' has no local_path).
|
||||
expect(sources.map((s) => s.id).sort()).toEqual(['source-a', 'source-b']);
|
||||
|
||||
// The SQL must not throw. Pre-fix, the broken SQL would have thrown
|
||||
// `relation "chunks" does not exist` on PGLite (and Postgres). Post-
|
||||
// fix, the canonical `content_chunks JOIN pages ON page_id` shape parses.
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
|
||||
expect(report.schema_version).toBe(1);
|
||||
expect(report.sources).toHaveLength(2);
|
||||
|
||||
const byId = new Map(report.sources.map((s) => [s.source_id, s]));
|
||||
|
||||
// source-a: 3 active pages (4th is soft-deleted and excluded).
|
||||
// 6 active chunks (2 from the soft-deleted page are excluded).
|
||||
// 4 chunks embedded, 2 unembedded.
|
||||
const a = byId.get('source-a')!;
|
||||
expect(a.pages).toBe(3);
|
||||
expect(a.chunks_total).toBe(6);
|
||||
expect(a.chunks_unembedded).toBe(2);
|
||||
expect(a.embedding_coverage_pct).toBeCloseTo(66.7, 1);
|
||||
|
||||
// source-b: 2 pages × 2 chunks = 4 chunks, all embedded.
|
||||
const b = byId.get('source-b')!;
|
||||
expect(b.pages).toBe(2);
|
||||
expect(b.chunks_total).toBe(4);
|
||||
expect(b.chunks_unembedded).toBe(0);
|
||||
expect(b.embedding_coverage_pct).toBe(100);
|
||||
});
|
||||
|
||||
test('soft-deleted pages excluded from pages count (v0.26.5 regression)', async () => {
|
||||
// Verifies the `WHERE pg.deleted_at IS NULL` clause in BOTH subqueries
|
||||
// of the dashboard SQL. Pre-fix the original PR query would have
|
||||
// counted the soft-deleted page as part of source-a's totals.
|
||||
const sources = await engine.executeRaw<{
|
||||
id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown>;
|
||||
}>(
|
||||
`SELECT id, name, local_path, config FROM sources WHERE id = 'source-a'`,
|
||||
);
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
const a = report.sources.find((s) => s.source_id === 'source-a')!;
|
||||
expect(a.pages).toBe(3); // 4 raw rows, 1 soft-deleted → 3 active
|
||||
expect(a.chunks_total).toBe(6); // 8 raw chunks, 2 on soft-deleted page → 6 active
|
||||
});
|
||||
|
||||
test('archived sources are excluded by the caller-side filter (Expansion 3)', async () => {
|
||||
// The dashboard input filter is `archived IS NOT TRUE`. source-c was
|
||||
// seeded with archived=true and must not appear in the report.
|
||||
const sources = await engine.executeRaw<{
|
||||
id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown>;
|
||||
}>(
|
||||
`SELECT id, name, local_path, config FROM sources
|
||||
WHERE local_path IS NOT NULL AND archived IS NOT TRUE`,
|
||||
);
|
||||
const ids = sources.map((s) => s.id);
|
||||
expect(ids).not.toContain('source-c');
|
||||
});
|
||||
|
||||
test('embedding column reported in envelope is what the SQL counted against (D16)', async () => {
|
||||
// D16 → A: dashboard counts unembedded chunks against the ACTIVE
|
||||
// embedding column (resolved via the registry). The envelope's
|
||||
// `embedding_column` field exposes which column the SQL used so
|
||||
// operators can verify Voyage / multimodal setups are reported
|
||||
// correctly. Default brains (no embedding_model config) resolve to
|
||||
// 'embedding'. Non-default brains would see their override here.
|
||||
const sources = await engine.executeRaw<{
|
||||
id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown>;
|
||||
}>(
|
||||
`SELECT id, name, local_path, config FROM sources
|
||||
WHERE local_path IS NOT NULL AND archived IS NOT TRUE`,
|
||||
);
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
expect(typeof report.embedding_column).toBe('string');
|
||||
expect(report.embedding_column.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('errors propagate (Q2 sub-fix — no silent swallowing of real DB errors)', async () => {
|
||||
// Wrap a wrapped engine that throws on the count query. Pre-fix the
|
||||
// PR's bare `catch { countRows = [] }` would have masked this and
|
||||
// returned "0 chunks for every source" — exactly the misleading
|
||||
// report the operator should NEVER see when their DB is actually
|
||||
// broken.
|
||||
const sources = await engine.executeRaw<{
|
||||
id: string;
|
||||
name: string;
|
||||
local_path: string | null;
|
||||
config: Record<string, unknown>;
|
||||
}>(
|
||||
`SELECT id, name, local_path, config FROM sources
|
||||
WHERE local_path IS NOT NULL AND archived IS NOT TRUE`,
|
||||
);
|
||||
|
||||
const proxyEngine: BrainEngine = {
|
||||
kind: engine.kind,
|
||||
executeRaw: async (sql: string, params?: unknown[]) => {
|
||||
if (/WITH s AS \(\s*SELECT unnest/.test(sql)) {
|
||||
throw new Error('synthetic test error: count query failed');
|
||||
}
|
||||
return (engine.executeRaw as (sql: string, params?: unknown[]) => Promise<unknown[]>)(sql, params);
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
await expect(buildSyncStatusReport(proxyEngine, sources)).rejects.toThrow(
|
||||
/count query failed/,
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,387 @@
|
||||
/**
|
||||
* Tests for the v0.40.3.0 `sync --all` parallel fan-out + read-only
|
||||
* `gbrain sources status` dashboard.
|
||||
*
|
||||
* Why this exists:
|
||||
* The CLI `sync --all` path used to walk sources SEQUENTIALLY via a
|
||||
* `for...of` loop. On a 4-source brain, one stalled source held up
|
||||
* every other source's sync, causing staleness penalties to pile up
|
||||
* between cron ticks. Operators reported manual workarounds (8 ad-hoc
|
||||
* parallel workers wrapping `sync --source <id>`) and the cycle's
|
||||
* autopilot-fanout path already proves source dispatch is safe to
|
||||
* parallelize when each source has its own DB lock.
|
||||
*
|
||||
* PR #1314 (community) proposed the parallel fan-out. Codex's
|
||||
* outside-voice review caught three P0s the PR (and the initial plan)
|
||||
* missed: lock asymmetry, hardcoded chunks table name, and 2x
|
||||
* connection-budget understatement. v0.40.3.0 ships the corrected
|
||||
* design. These tests pin every contract:
|
||||
*
|
||||
* 1. resolveParallelism() picks the right concurrency budget across
|
||||
* all the inputs (PGLite, explicit --parallel, --workers ceiling,
|
||||
* source-count floor, zero-source guard).
|
||||
* 2. The lock-identity invariant: any sync with sourceId set takes
|
||||
* the per-source lock `gbrain-sync:<source_id>`, NOT the global
|
||||
* `gbrain-sync` lock. Closes the bug class where `sync --all`
|
||||
* and `sync --source foo` would otherwise race.
|
||||
* 3. buildSyncStatusReport() returns a stable structured shape
|
||||
* readable by both --json output and the human-facing table.
|
||||
* SQL is the canonical `content_chunks JOIN pages ON page_id`
|
||||
* shape with deleted_at + archived filters — the regression
|
||||
* guard for Codex's P0 #1 SQL bug.
|
||||
* 4. Continuous worker pool (D2): slow source doesn't block other
|
||||
* workers; one source throwing doesn't abort the others;
|
||||
* completion order is independent of source order.
|
||||
* 5. Connection-budget warning fires at parallel × workers × 2 > 16
|
||||
* with the formula in the message text (D1 + D10).
|
||||
* 6. --json envelope shape is {schema_version: 1, sources, parallel,
|
||||
* ok_count, error_count} (D14).
|
||||
* 7. --skip-failed / --retry-failed reject when --parallel > 1
|
||||
* with a loud paste-ready hint (D15).
|
||||
* 8. Per-source prefix uses source.id (NOT source.name) so
|
||||
* operators can grep cleanly and no log-injection vector exists
|
||||
* through arbitrary source names (D13).
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
resolveParallelism,
|
||||
buildSyncStatusReport,
|
||||
} from '../src/commands/sync.ts';
|
||||
import { SYNC_LOCK_ID, syncLockId } from '../src/core/db-lock.ts';
|
||||
import { withSourcePrefix, slog } from '../src/core/console-prefix.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
// ── resolveParallelism ──────────────────────────────────────────────
|
||||
|
||||
describe('resolveParallelism', () => {
|
||||
test('PGLite always serial regardless of source count or flags', () => {
|
||||
expect(resolveParallelism({ sourceCount: 10, engineKind: 'pglite' })).toBe(1);
|
||||
expect(resolveParallelism({ sourceCount: 10, engineKind: 'pglite', explicitParallel: 8 })).toBe(1);
|
||||
expect(resolveParallelism({ sourceCount: 10, engineKind: 'pglite', workers: 8 })).toBe(1);
|
||||
});
|
||||
|
||||
test('explicit --parallel wins and is clamped to sourceCount', () => {
|
||||
expect(resolveParallelism({ sourceCount: 4, engineKind: 'postgres', explicitParallel: 2 })).toBe(2);
|
||||
// Capped by source count (no point dispatching more workers than work).
|
||||
expect(resolveParallelism({ sourceCount: 2, engineKind: 'postgres', explicitParallel: 8 })).toBe(2);
|
||||
// Floor of 1.
|
||||
expect(resolveParallelism({ sourceCount: 4, engineKind: 'postgres', explicitParallel: 1 })).toBe(1);
|
||||
});
|
||||
|
||||
test('auto path: min(sourceCount, workers || DEFAULT_PARALLEL_SOURCES)', () => {
|
||||
// sourceCount < default ceiling → bounded by sourceCount.
|
||||
expect(resolveParallelism({ sourceCount: 2, engineKind: 'postgres' })).toBe(2);
|
||||
// sourceCount > default ceiling → bounded by the 4-worker ceiling.
|
||||
expect(resolveParallelism({ sourceCount: 12, engineKind: 'postgres' })).toBe(4);
|
||||
// --workers tightens the ceiling.
|
||||
expect(resolveParallelism({ sourceCount: 12, engineKind: 'postgres', workers: 2 })).toBe(2);
|
||||
// --workers above the safety ceiling is itself clamped to 4.
|
||||
expect(resolveParallelism({ sourceCount: 12, engineKind: 'postgres', workers: 32 })).toBe(4);
|
||||
});
|
||||
|
||||
test('single-source --all short-circuits to serial (no fan-out value)', () => {
|
||||
expect(resolveParallelism({ sourceCount: 1, engineKind: 'postgres' })).toBe(1);
|
||||
expect(resolveParallelism({ sourceCount: 1, engineKind: 'postgres', explicitParallel: 8 })).toBe(1);
|
||||
});
|
||||
|
||||
test('zero-source edge case returns 1 (no division by zero, no negative worker count)', () => {
|
||||
expect(resolveParallelism({ sourceCount: 0, engineKind: 'postgres' })).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
// ── lock-identity invariant (D8) ─────────────────────────────────────
|
||||
|
||||
describe('per-source lock id (D8 — Codex P0 #1 fix)', () => {
|
||||
test('per-source lock id format is namespaced via syncLockId helper', () => {
|
||||
// v0.40.5.0 (master) introduced syncLockId(sourceId) as the canonical
|
||||
// helper. v0.40.6.0 (this branch) builds on it. Two sources -> two
|
||||
// distinct lock ids. SYNC_LOCK_ID = syncLockId('default') is a
|
||||
// back-compat alias for the legacy single-source path.
|
||||
const idA = syncLockId('source-a');
|
||||
const idB = syncLockId('source-b');
|
||||
expect(idA).not.toBe(idB);
|
||||
// Distinct from the default lock so the cycle's default-source acquire
|
||||
// doesn't block a per-source `sync --all` worker.
|
||||
expect(idA).not.toBe(SYNC_LOCK_ID);
|
||||
expect(idA).not.toBe(syncLockId('default'));
|
||||
});
|
||||
|
||||
test('source.id only — newline injection through source.name is impossible', () => {
|
||||
// D13 fix: lock id derives from source.id (slug-validated by `sources
|
||||
// add`), NOT source.name (free-form text accepted by --name). This
|
||||
// pins the contract that adversarial names can't smuggle in newlines
|
||||
// or control characters into the lock id.
|
||||
const sourceWithEvilName = {
|
||||
id: 'media-corpus',
|
||||
name: 'evil\nname: hijacked\n[other-source]',
|
||||
};
|
||||
const lockId = syncLockId(sourceWithEvilName.id);
|
||||
expect(lockId).not.toContain('\n');
|
||||
expect(lockId).not.toContain('evil');
|
||||
expect(lockId).toContain('media-corpus');
|
||||
});
|
||||
});
|
||||
|
||||
// ── buildSyncStatusReport ────────────────────────────────────────────
|
||||
|
||||
describe('buildSyncStatusReport', () => {
|
||||
// Minimal engine stub: implements executeRaw with a script of canned
|
||||
// responses keyed by the first SQL keyword. The real engine uses
|
||||
// postgres-js with tagged templates; tests use raw executeRaw so we
|
||||
// can pin the dashboard query without booting Postgres.
|
||||
//
|
||||
// SQL regex changed from PR's original `chunks/page_slug` (broken;
|
||||
// would have shipped silently) to the canonical `content_chunks` +
|
||||
// `page_id` shape. The IRON RULE regression case lives in
|
||||
// test/e2e/sync-status-pglite.test.ts and exercises real SQL.
|
||||
function makeEngine(scripts: {
|
||||
sourceRows?: Array<{ id: string; last_commit: string | null; last_sync_at: string | null }>;
|
||||
countRows?: Array<{ source_id: string; pages: number; chunks_total: number; chunks_unembedded: number }>;
|
||||
}): BrainEngine {
|
||||
return {
|
||||
kind: 'postgres',
|
||||
executeRaw: async (sql: string) => {
|
||||
if (/FROM sources WHERE id = ANY/i.test(sql)) {
|
||||
return scripts.sourceRows ?? [];
|
||||
}
|
||||
// Codex P0 #1 regression: must match the CORRECT SQL shape
|
||||
// (content_chunks + page_id), not the broken PR shape.
|
||||
if (/content_chunks cc[\s\S]*JOIN pages pg ON pg\.id = cc\.page_id/.test(sql)) {
|
||||
return scripts.countRows ?? [];
|
||||
}
|
||||
return [];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
}
|
||||
|
||||
test('returns staleness_class fresh/stale/severe based on last_sync_at age', async () => {
|
||||
const now = Date.now();
|
||||
const freshIso = new Date(now - 1 * 60 * 60 * 1000).toISOString(); // 1h ago
|
||||
const staleIso = new Date(now - 30 * 60 * 60 * 1000).toISOString(); // 30h ago
|
||||
const severeIso = new Date(now - 100 * 60 * 60 * 1000).toISOString(); // 100h ago
|
||||
|
||||
const sources = [
|
||||
{ id: 'fresh', name: 'fresh', local_path: '/tmp/a', config: { syncEnabled: true } },
|
||||
{ id: 'stale', name: 'stale', local_path: '/tmp/b', config: { syncEnabled: true } },
|
||||
{ id: 'severe', name: 'severe', local_path: '/tmp/c', config: { syncEnabled: true } },
|
||||
{ id: 'never', name: 'never', local_path: '/tmp/d', config: { syncEnabled: true } },
|
||||
];
|
||||
const engine = makeEngine({
|
||||
sourceRows: [
|
||||
{ id: 'fresh', last_commit: 'a'.repeat(40), last_sync_at: freshIso },
|
||||
{ id: 'stale', last_commit: 'b'.repeat(40), last_sync_at: staleIso },
|
||||
{ id: 'severe', last_commit: 'c'.repeat(40), last_sync_at: severeIso },
|
||||
{ id: 'never', last_commit: null, last_sync_at: null },
|
||||
],
|
||||
countRows: [
|
||||
{ source_id: 'fresh', pages: 100, chunks_total: 200, chunks_unembedded: 0 },
|
||||
{ source_id: 'stale', pages: 50, chunks_total: 100, chunks_unembedded: 25 },
|
||||
{ source_id: 'severe', pages: 10, chunks_total: 20, chunks_unembedded: 20 },
|
||||
// 'never' source: no count rows → defaults to 0 pages, 0 chunks.
|
||||
],
|
||||
});
|
||||
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
expect(report.sources).toHaveLength(4);
|
||||
|
||||
const byId = new Map(report.sources.map((s) => [s.source_id, s]));
|
||||
expect(byId.get('fresh')!.staleness_class).toBe('fresh');
|
||||
expect(byId.get('stale')!.staleness_class).toBe('stale');
|
||||
expect(byId.get('severe')!.staleness_class).toBe('severe');
|
||||
expect(byId.get('never')!.staleness_class).toBe('unknown');
|
||||
expect(byId.get('never')!.staleness_hours).toBeNull();
|
||||
});
|
||||
|
||||
test('embedding_coverage_pct computed from chunks_total vs chunks_unembedded', async () => {
|
||||
const sources = [
|
||||
{ id: 'a', name: 'a', local_path: '/tmp/a', config: {} },
|
||||
{ id: 'b', name: 'b', local_path: '/tmp/b', config: {} },
|
||||
{ id: 'c', name: 'c', local_path: '/tmp/c', config: {} },
|
||||
];
|
||||
const engine = makeEngine({
|
||||
sourceRows: [
|
||||
{ id: 'a', last_commit: null, last_sync_at: null },
|
||||
{ id: 'b', last_commit: null, last_sync_at: null },
|
||||
{ id: 'c', last_commit: null, last_sync_at: null },
|
||||
],
|
||||
countRows: [
|
||||
{ source_id: 'a', pages: 10, chunks_total: 100, chunks_unembedded: 0 },
|
||||
{ source_id: 'b', pages: 10, chunks_total: 100, chunks_unembedded: 50 },
|
||||
// c: zero chunks → coverage reported as 100% (vacuously complete; no
|
||||
// divide-by-zero blowup).
|
||||
],
|
||||
});
|
||||
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
const byId = new Map(report.sources.map((s) => [s.source_id, s]));
|
||||
expect(byId.get('a')!.embedding_coverage_pct).toBe(100);
|
||||
expect(byId.get('b')!.embedding_coverage_pct).toBe(50);
|
||||
expect(byId.get('c')!.embedding_coverage_pct).toBe(100);
|
||||
});
|
||||
|
||||
test('disabled source is reflected in sync_enabled flag', async () => {
|
||||
const sources = [
|
||||
{ id: 'on', name: 'on', local_path: '/tmp/on', config: { syncEnabled: true } },
|
||||
{ id: 'off', name: 'off', local_path: '/tmp/off', config: { syncEnabled: false } },
|
||||
{ id: 'default', name: 'default', local_path: '/tmp/default', config: {} },
|
||||
];
|
||||
const engine = makeEngine({
|
||||
sourceRows: sources.map((s) => ({ id: s.id, last_commit: null, last_sync_at: null })),
|
||||
countRows: [],
|
||||
});
|
||||
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
const byId = new Map(report.sources.map((s) => [s.source_id, s]));
|
||||
// syncEnabled omitted defaults to true (matches the loop's `!== false` check).
|
||||
expect(byId.get('on')!.sync_enabled).toBe(true);
|
||||
expect(byId.get('off')!.sync_enabled).toBe(false);
|
||||
expect(byId.get('default')!.sync_enabled).toBe(true);
|
||||
});
|
||||
|
||||
test('count-query failure propagates (Q2 sub-fix — no silent zero-counts)', async () => {
|
||||
// v0.40.3.0 (Codex Q2): the PR's bare `catch { countRows = [] }`
|
||||
// would return a misleading "0 chunks" report on real DB errors.
|
||||
// Now the thrown error propagates so the operator sees the real
|
||||
// problem instead of a lying dashboard.
|
||||
const sources = [
|
||||
{ id: 'a', name: 'a', local_path: '/tmp/a', config: {} },
|
||||
];
|
||||
const engine = {
|
||||
kind: 'postgres',
|
||||
executeRaw: async (sql: string) => {
|
||||
if (/FROM sources WHERE id = ANY/i.test(sql)) {
|
||||
return [{ id: 'a', last_commit: null, last_sync_at: null }];
|
||||
}
|
||||
throw new Error('canceling statement due to statement timeout');
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
// The function MUST throw, not swallow.
|
||||
await expect(buildSyncStatusReport(engine, sources)).rejects.toThrow(
|
||||
/statement timeout/,
|
||||
);
|
||||
});
|
||||
|
||||
test('empty source list returns empty array with schema_version: 1, not crash', async () => {
|
||||
const engine = makeEngine({ sourceRows: [], countRows: [] });
|
||||
const report = await buildSyncStatusReport(engine, []);
|
||||
expect(report.sources).toEqual([]);
|
||||
expect(report.schema_version).toBe(1);
|
||||
expect(typeof report.generated_at).toBe('string');
|
||||
expect(typeof report.embedding_column).toBe('string');
|
||||
});
|
||||
|
||||
test('stable schema_version: 1 envelope shape (D14 JSON contract)', async () => {
|
||||
const sources = [
|
||||
{ id: 'a', name: 'a', local_path: '/tmp/a', config: {} },
|
||||
];
|
||||
const engine = makeEngine({
|
||||
sourceRows: [{ id: 'a', last_commit: 'abc', last_sync_at: new Date().toISOString() }],
|
||||
countRows: [{ source_id: 'a', pages: 5, chunks_total: 10, chunks_unembedded: 0 }],
|
||||
});
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
// Pin every top-level key — the envelope is a public surface; monitoring
|
||||
// pipelines bind to it. New fields are additive; existing names + types
|
||||
// must not change without bumping schema_version.
|
||||
expect(report.schema_version).toBe(1);
|
||||
expect(typeof report.generated_at).toBe('string');
|
||||
expect(Array.isArray(report.sources)).toBe(true);
|
||||
expect(typeof report.unacknowledged_failures).toBe('number');
|
||||
expect(typeof report.embedding_column).toBe('string');
|
||||
expect(report.sources).toHaveLength(1);
|
||||
const s = report.sources[0];
|
||||
expect(s.source_id).toBe('a');
|
||||
expect(s.name).toBe('a');
|
||||
expect(s.sync_enabled).toBe(true);
|
||||
expect(s.embedding_coverage_pct).toBe(100);
|
||||
});
|
||||
|
||||
test('embedding column is reported in envelope so operators can verify the registry resolved correctly', async () => {
|
||||
// D16 → A: dashboard counts unembedded chunks against the resolved
|
||||
// active embedding column (not hardcoded `embedding`). The column
|
||||
// name is surfaced in the envelope so operators inspecting their
|
||||
// Voyage / multimodal setup can confirm the right column was used.
|
||||
const sources = [
|
||||
{ id: 'a', name: 'a', local_path: '/tmp/a', config: {} },
|
||||
];
|
||||
const engine = makeEngine({
|
||||
sourceRows: [{ id: 'a', last_commit: null, last_sync_at: null }],
|
||||
countRows: [],
|
||||
});
|
||||
const report = await buildSyncStatusReport(engine, sources);
|
||||
// The default registry resolves 'embedding' for OpenAI-default setups.
|
||||
// Voyage / multimodal brains would see a different name here. The
|
||||
// contract is "the value is present and non-empty"; the specific
|
||||
// string varies by configured embedding model.
|
||||
expect(report.embedding_column.length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ── per-source console prefix (D6 + D13) ─────────────────────────────
|
||||
|
||||
describe('per-source line prefix under withSourcePrefix', () => {
|
||||
test('slog under wrap emits [source-id] prefix; outside wrap emits bare output', async () => {
|
||||
// Captures both paths in one case. The wrap propagation is what
|
||||
// makes the entire "per-source greppable parallel output" feature
|
||||
// work; without it, parallel sync interleaves illegibly.
|
||||
//
|
||||
// Outside-wrap path routes through `console.log` (not
|
||||
// `process.stdout.write` directly) so we patch both sinks.
|
||||
// Inside-wrap path routes through `process.stdout.write` because
|
||||
// slog needs raw stream control to emit prefixed lines.
|
||||
const stdoutOrig = process.stdout.write.bind(process.stdout);
|
||||
const consoleLogOrig = console.log;
|
||||
const stdoutChunks: string[] = [];
|
||||
const consoleLogChunks: unknown[][] = [];
|
||||
process.stdout.write = ((chunk: string | Uint8Array): boolean => {
|
||||
stdoutChunks.push(typeof chunk === 'string' ? chunk : Buffer.from(chunk).toString('utf-8'));
|
||||
return true;
|
||||
}) as typeof process.stdout.write;
|
||||
// eslint-disable-next-line no-console
|
||||
console.log = (...args: unknown[]) => { consoleLogChunks.push(args); };
|
||||
try {
|
||||
// Outside wrap: bare console.log fast path.
|
||||
slog('outside-wrap');
|
||||
// Inside wrap: prefixed line via process.stdout.write.
|
||||
await withSourcePrefix('media-corpus', async () => {
|
||||
slog('inside-wrap');
|
||||
});
|
||||
// Outside-wrap landed on console.log (no prefix, no [tag]).
|
||||
const flatConsole = consoleLogChunks.flat().map(String).join(' ');
|
||||
expect(flatConsole).toContain('outside-wrap');
|
||||
expect(flatConsole).not.toMatch(/\[.*\]/);
|
||||
// Inside-wrap landed on process.stdout.write WITH the source.id prefix.
|
||||
const stdoutText = stdoutChunks.join('');
|
||||
expect(stdoutText).toContain('[media-corpus] inside-wrap');
|
||||
} finally {
|
||||
process.stdout.write = stdoutOrig;
|
||||
// eslint-disable-next-line no-console
|
||||
console.log = consoleLogOrig;
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── connection-budget warning (D1 + D10) ─────────────────────────────
|
||||
|
||||
describe('connection-budget warning math (D10 — Codex P0 #3 fix)', () => {
|
||||
// The warning formula is `parallel × workers × 2 > 16`. Tests pin the
|
||||
// math rather than invoking runSync (which would require a real engine
|
||||
// and a real --all run). The thresholds are documented in
|
||||
// sync.ts:resolveParallelism docstring + DEFAULT_PARALLEL_SOURCES.
|
||||
test('triggers at parallel × workers × 2 > 16 (default 4×4×2=32 fires)', () => {
|
||||
const cases: Array<{ parallel: number; workers: number; expected: boolean }> = [
|
||||
{ parallel: 4, workers: 4, expected: true }, // 32 — default fires
|
||||
{ parallel: 2, workers: 4, expected: false }, // 16 — exact boundary, silent
|
||||
{ parallel: 2, workers: 2, expected: false }, // 8 — silent
|
||||
{ parallel: 1, workers: 4, expected: false }, // 8 — silent
|
||||
{ parallel: 8, workers: 1, expected: false }, // 16 — exact boundary, silent
|
||||
{ parallel: 8, workers: 2, expected: true }, // 32 — fires
|
||||
{ parallel: 4, workers: 3, expected: true }, // 24 — fires
|
||||
];
|
||||
for (const c of cases) {
|
||||
const budget = c.parallel * c.workers * 2;
|
||||
const triggered = budget > 16;
|
||||
expect(triggered).toBe(c.expected);
|
||||
}
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user