Files
gbrain/test/pglite-workers-clamp.test.ts
T
8ab733471b v0.41.17.0 feat: --workers N on every bulk command + facts dim doctor parity (#1519)
* feat(worker-pool): shared sliding pool + bounded semaphore + PGLite-clamp wrapper

T1 + T2 of the v0.41.16.0 workers cathedral. New src/core/worker-pool.ts is
the canonical primitive every --workers N bulk command in this wave (and
future bulk commands) builds on. Atomic-claim invariant enforced by
scripts/check-worker-pool-atomicity.sh (wired into bun run verify).
BudgetExhausted bypass + AbortSignal composition baked into the helper so
budget caps are a structural ceiling under concurrency, not a per-caller
convention.

The new resolveWorkersWithClamp wrapper composes existing autoConcurrency
with PGLite-clamp + per-(command, requested) stderr dedup. Deliberately
NOT a modification to shared autoConcurrency (silent today, used by sync
+ import); embed.ts keeps GBRAIN_EMBED_CONCURRENCY || 20 default per
codex #13.

23 + 12 + 9 = 44 hermetic tests pin every contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: structural + dim-check regression suites for v0.41.16.0 wave

- test/embed-helper-migration.test.ts (T3): asserts embed.ts's two
  sliding-pool sites are migrated to runSlidingPool, pre-migration
  shapes (let nextIdx = 0, Promise.all(Array.from(...))) are gone,
  GBRAIN_EMBED_CONCURRENCY || 20 default preserved, failureLabel
  threads page.slug. Per codex #16/#17 these are invariant assertions,
  not byte-equality on progress event ORDERING.
- test/embedding-dim-check-facts.test.ts (T6): readFactsEmbeddingDim
  covers vector(N) + halfvec(N), halfvec-before-vector regex ordering
  pinned (codex #19), buildFactsAlterRecipe emits DROP INDEX + ALTER
  USING + CREATE INDEX (codex #18, not bare REINDEX),
  FactsEmbeddingDimMismatchError tagged class shape,
  assertFactsEmbeddingDimMatchesConfig PGLite skip + Postgres absent-
  column skip, doctor check + insert-cast wiring assertions.
- test/extract-conversation-facts-workers.test.ts (T5): helper
  exports (extractConversationFactsLockId, PER_PAGE_LOCK_TTL_MINUTES),
  structural wiring (runSlidingPool, resolveWorkersWithClamp,
  withRefreshingLock, LockUnavailableError, delete-orphans-first
  before segment loop, preflight before pool, exit 3 when lock_skipped
  > 0), Minion handler round-trip.
- test/extract-workers.test.ts (T7): --workers wiring on all 3 inner
  fs-walk loops (extractForSlugs, extractLinksFromDir,
  extractTimelineFromDir) + CLI parse + opts threading through
  runExtractCore.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump v0.41.16.0 → v0.41.17.0 (queue collision with PR #1510)

PR #1510 (garrytan/dynamic-regex-conversation-formats) claimed v0.41.16.0
on master in parallel. Advancing this wave to v0.41.17.0 so both can land
cleanly. Pure mechanical version bump:

- VERSION + package.json → 0.41.17.0
- CHANGELOG.md header + "To take advantage of v0.41.17.0" block
- TODOS.md section header + v0.41.18+ forward references
- CLAUDE.md inline version tags
- Regenerated llms-full.txt / llms.txt

No code changes. The actual workers cathedral feature set is unchanged
from the two prior commits in this branch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): search-image-column probes column dim at runtime

CI shard 5 failed on `searchVector column routing (v0.27.1)` with:
  error: expected 1280 dimensions, not 1536

The test had a hardcoded `fakeText1536` helper that seeded chunks at
1536-d vectors. Master's default embedding model switched from OpenAI
text-embedding-3-large (1536) to ZeroEntropy zembed-1 (1280) so a fresh
PGLite brain on CI now sizes content_chunks.embedding at 1280; the
test's 1536-d INSERT trips pgvector's CheckExpectedDim.

Fix: probe `content_chunks.embedding` width via
`readContentChunksEmbeddingDim(engine)` in `beforeAll`, store in
`TEXT_DIM`, and build `fakeTextDefault(seed)` at that width. The test
now passes regardless of which default ships (the model has flipped
twice and may flip again). Local dev (1536 from older config) and CI
fresh-install (1280 from new default) both pass.

Image-side vectors stay at 1024 (matches Voyage multimodal-3 + the
column's fixed width on the image side).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): bump PGLite hook timeout for shard-4 deep-process files

facts-anti-loop.test.ts and ingest-capture.test.ts were timing out in CI
shard 4 with "beforeEach/afterEach hook timed out" after the v0.41.16.0
master merge brought migration count to 99. When these files run deep in
a shard process that has already created ~20 PGLite engines, the WASM
cold-start + 95-migration replay legitimately exceeds bun's 5s default
hook timeout (observed 5.6s and 7.3s locally when reproducing).

Bun's --timeout=60000 from scripts/test-shard.sh covers TEST timeouts
but NOT hook timeouts; those default to 5s and must be set per-hook via
the optional 2nd arg to beforeAll/afterAll.

Reproduced locally by running the first 21 shard-4 files via
  head -21 /tmp/shard4-list.txt | xargs bun test
  → 179 pass, 2 fail (both with hook-timeout error)

After fix:
  → 198 pass, 0 fail (the 4 anti-loop + 15 ingest-capture tests recover)

Full shard 4 with fix:  955 pass, 0 fail.
Full shard 5 with fix:  1261 pass, 0 fail.

Also added a defensive diagnostic to the two put_page tests: if
facts_backstop is missing in the response payload, throw with the full
payload + isError so future failures surface the actual handler error
instead of a bare "expected {...} got undefined" assertion. No-op when
the test passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:29:03 -07:00

143 lines
5.5 KiB
TypeScript

/**
* Tests for resolveWorkersWithClamp (v0.41.15.0, D9).
*
* Pins the PGLite-clamp + stderr-warn contract that every bulk command
* routes through. No real engine needed — the wrapper branches on
* `engine.kind` only, so synthetic stubs satisfy the contract.
*
* R1+R2 compliant: no process.env mutation, no mock.module. Lives in
* the parallel fast loop.
*/
import { describe, test, expect, beforeEach } from 'bun:test';
import {
resolveWorkersWithClamp,
_resetWorkersClampWarningsForTest,
DEFAULT_PARALLEL_WORKERS,
} from '../src/core/sync-concurrency.ts';
import type { BrainEngine } from '../src/core/engine.ts';
const PGLITE: Pick<BrainEngine, 'kind'> = { kind: 'pglite' };
const POSTGRES: Pick<BrainEngine, 'kind'> = { kind: 'postgres' };
/**
* Capture console.error output for the duration of `fn`. Returns the
* collected lines. Restores the original spy on exit, including on
* throw. Avoids env mutation per the test-isolation lint.
*/
async function captureStderr<T>(fn: () => Promise<T> | T): Promise<{ result: T; lines: string[] }> {
const lines: string[] = [];
const orig = console.error;
console.error = (...args: unknown[]) => {
lines.push(args.map((a) => (typeof a === 'string' ? a : JSON.stringify(a))).join(' '));
};
try {
const result = await fn();
return { result, lines };
} finally {
console.error = orig;
}
}
beforeEach(() => {
_resetWorkersClampWarningsForTest();
});
describe('resolveWorkersWithClamp — PGLite branch', () => {
test('PGLite + override=5 → clamp to 1 + stderr warn + reason=pglite_clamp', async () => {
const { result, lines } = await captureStderr(() =>
resolveWorkersWithClamp(PGLITE as BrainEngine, 5, 'extract-conversation-facts', 0),
);
expect(result.workers).toBe(1);
expect(result.wasClamped).toBe(true);
expect(result.requested).toBe(5);
expect(result.reason).toBe('pglite_clamp');
expect(lines.length).toBe(1);
expect(lines[0]).toContain('[extract-conversation-facts]');
expect(lines[0]).toContain('workers=5 requested');
expect(lines[0]).toContain('clamped to 1');
expect(lines[0]).toContain('PGLite');
});
test('PGLite + override=1 → no clamp, no warn, reason=override', async () => {
const { result, lines } = await captureStderr(() =>
resolveWorkersWithClamp(PGLITE as BrainEngine, 1, 'extract-conversation-facts', 0),
);
expect(result.workers).toBe(1);
expect(result.wasClamped).toBe(false);
expect(result.requested).toBe(1);
expect(result.reason).toBe('override');
expect(lines.length).toBe(0);
});
test('PGLite + no override → silent default to 1', async () => {
const { result, lines } = await captureStderr(() =>
resolveWorkersWithClamp(PGLITE as BrainEngine, undefined, 'extract', 0),
);
expect(result.workers).toBe(1);
expect(result.wasClamped).toBe(false);
expect(result.requested).toBeUndefined();
expect(result.reason).toBe('default');
expect(lines.length).toBe(0);
});
test('PGLite warning emits ONCE per (command, requested) pair within a process', async () => {
const { lines } = await captureStderr(async () => {
resolveWorkersWithClamp(PGLITE as BrainEngine, 20, 'extract-conversation-facts', 0);
resolveWorkersWithClamp(PGLITE as BrainEngine, 20, 'extract-conversation-facts', 0);
resolveWorkersWithClamp(PGLITE as BrainEngine, 20, 'extract-conversation-facts', 0);
});
expect(lines.length).toBe(1);
});
test('different (command, requested) tuples each warn once', async () => {
const { lines } = await captureStderr(async () => {
resolveWorkersWithClamp(PGLITE as BrainEngine, 5, 'extract', 0);
resolveWorkersWithClamp(PGLITE as BrainEngine, 20, 'extract', 0);
resolveWorkersWithClamp(PGLITE as BrainEngine, 5, 'reindex-code', 0);
});
expect(lines.length).toBe(3);
});
});
describe('resolveWorkersWithClamp — Postgres branch', () => {
test('Postgres + override=5 → passthrough, no warn, reason=override', async () => {
const { result, lines } = await captureStderr(() =>
resolveWorkersWithClamp(POSTGRES as BrainEngine, 5, 'extract-conversation-facts', 0),
);
expect(result.workers).toBe(5);
expect(result.wasClamped).toBe(false);
expect(result.requested).toBe(5);
expect(result.reason).toBe('override');
expect(lines.length).toBe(0);
});
test('Postgres + override=0 normalized to 1 via Math.max in autoConcurrency', async () => {
const r = resolveWorkersWithClamp(POSTGRES as BrainEngine, 0, 'extract', 0);
expect(r.workers).toBe(1);
expect(r.requested).toBe(0);
});
test('Postgres + no override + large fileCount → DEFAULT_PARALLEL_WORKERS, reason=auto', async () => {
const r = resolveWorkersWithClamp(POSTGRES as BrainEngine, undefined, 'extract', 1000);
expect(r.workers).toBe(DEFAULT_PARALLEL_WORKERS);
expect(r.reason).toBe('auto');
});
test('Postgres + no override + small fileCount → 1, reason=default', async () => {
const r = resolveWorkersWithClamp(POSTGRES as BrainEngine, undefined, 'extract', 10);
expect(r.workers).toBe(1);
expect(r.reason).toBe('default');
});
});
describe('resolveWorkersWithClamp — stderr message shape regression', () => {
test('warning includes paste-ready engine-switch hint', async () => {
const { lines } = await captureStderr(() =>
resolveWorkersWithClamp(PGLITE as BrainEngine, 20, 'reindex-code', 0),
);
expect(lines[0]).toContain('use Postgres for parallel writes');
expect(lines[0]).toContain('single-writer');
});
});