mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(worker-pool): shared sliding pool + bounded semaphore + PGLite-clamp wrapper T1 + T2 of the v0.41.16.0 workers cathedral. New src/core/worker-pool.ts is the canonical primitive every --workers N bulk command in this wave (and future bulk commands) builds on. Atomic-claim invariant enforced by scripts/check-worker-pool-atomicity.sh (wired into bun run verify). BudgetExhausted bypass + AbortSignal composition baked into the helper so budget caps are a structural ceiling under concurrency, not a per-caller convention. The new resolveWorkersWithClamp wrapper composes existing autoConcurrency with PGLite-clamp + per-(command, requested) stderr dedup. Deliberately NOT a modification to shared autoConcurrency (silent today, used by sync + import); embed.ts keeps GBRAIN_EMBED_CONCURRENCY || 20 default per codex #13. 23 + 12 + 9 = 44 hermetic tests pin every contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: structural + dim-check regression suites for v0.41.16.0 wave - test/embed-helper-migration.test.ts (T3): asserts embed.ts's two sliding-pool sites are migrated to runSlidingPool, pre-migration shapes (let nextIdx = 0, Promise.all(Array.from(...))) are gone, GBRAIN_EMBED_CONCURRENCY || 20 default preserved, failureLabel threads page.slug. Per codex #16/#17 these are invariant assertions, not byte-equality on progress event ORDERING. - test/embedding-dim-check-facts.test.ts (T6): readFactsEmbeddingDim covers vector(N) + halfvec(N), halfvec-before-vector regex ordering pinned (codex #19), buildFactsAlterRecipe emits DROP INDEX + ALTER USING + CREATE INDEX (codex #18, not bare REINDEX), FactsEmbeddingDimMismatchError tagged class shape, assertFactsEmbeddingDimMatchesConfig PGLite skip + Postgres absent- column skip, doctor check + insert-cast wiring assertions. - test/extract-conversation-facts-workers.test.ts (T5): helper exports (extractConversationFactsLockId, PER_PAGE_LOCK_TTL_MINUTES), structural wiring (runSlidingPool, resolveWorkersWithClamp, withRefreshingLock, LockUnavailableError, delete-orphans-first before segment loop, preflight before pool, exit 3 when lock_skipped > 0), Minion handler round-trip. - test/extract-workers.test.ts (T7): --workers wiring on all 3 inner fs-walk loops (extractForSlugs, extractLinksFromDir, extractTimelineFromDir) + CLI parse + opts threading through runExtractCore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.41.16.0 → v0.41.17.0 (queue collision with PR #1510) PR #1510 (garrytan/dynamic-regex-conversation-formats) claimed v0.41.16.0 on master in parallel. Advancing this wave to v0.41.17.0 so both can land cleanly. Pure mechanical version bump: - VERSION + package.json → 0.41.17.0 - CHANGELOG.md header + "To take advantage of v0.41.17.0" block - TODOS.md section header + v0.41.18+ forward references - CLAUDE.md inline version tags - Regenerated llms-full.txt / llms.txt No code changes. The actual workers cathedral feature set is unchanged from the two prior commits in this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): search-image-column probes column dim at runtime CI shard 5 failed on `searchVector column routing (v0.27.1)` with: error: expected 1280 dimensions, not 1536 The test had a hardcoded `fakeText1536` helper that seeded chunks at 1536-d vectors. Master's default embedding model switched from OpenAI text-embedding-3-large (1536) to ZeroEntropy zembed-1 (1280) so a fresh PGLite brain on CI now sizes content_chunks.embedding at 1280; the test's 1536-d INSERT trips pgvector's CheckExpectedDim. Fix: probe `content_chunks.embedding` width via `readContentChunksEmbeddingDim(engine)` in `beforeAll`, store in `TEXT_DIM`, and build `fakeTextDefault(seed)` at that width. The test now passes regardless of which default ships (the model has flipped twice and may flip again). Local dev (1536 from older config) and CI fresh-install (1280 from new default) both pass. Image-side vectors stay at 1024 (matches Voyage multimodal-3 + the column's fixed width on the image side). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): bump PGLite hook timeout for shard-4 deep-process files facts-anti-loop.test.ts and ingest-capture.test.ts were timing out in CI shard 4 with "beforeEach/afterEach hook timed out" after the v0.41.16.0 master merge brought migration count to 99. When these files run deep in a shard process that has already created ~20 PGLite engines, the WASM cold-start + 95-migration replay legitimately exceeds bun's 5s default hook timeout (observed 5.6s and 7.3s locally when reproducing). Bun's --timeout=60000 from scripts/test-shard.sh covers TEST timeouts but NOT hook timeouts; those default to 5s and must be set per-hook via the optional 2nd arg to beforeAll/afterAll. Reproduced locally by running the first 21 shard-4 files via head -21 /tmp/shard4-list.txt | xargs bun test → 179 pass, 2 fail (both with hook-timeout error) After fix: → 198 pass, 0 fail (the 4 anti-loop + 15 ingest-capture tests recover) Full shard 4 with fix: 955 pass, 0 fail. Full shard 5 with fix: 1261 pass, 0 fail. Also added a defensive diagnostic to the two put_page tests: if facts_backstop is missing in the response payload, throw with the full payload + isError so future failures surface the actual handler error instead of a bare "expected {...} got undefined" assertion. No-op when the test passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
233 lines
6.8 KiB
TypeScript
233 lines
6.8 KiB
TypeScript
/**
|
|
* Fixture-driven unit tests for scripts/check-worker-pool-atomicity.sh
|
|
* (v0.41.15.0, D5).
|
|
*
|
|
* Spawns the script against synthetic src/ trees and asserts the guard
|
|
* fires on the two violations it protects against:
|
|
* 1. `worker_threads` import in a file that imports the helper.
|
|
* 2. `await` between the read and write of `nextIdx` inside the helper.
|
|
*
|
|
* Plus: clean trees and a no-helper-file tree exit 0.
|
|
*
|
|
* No env mutation, no mock.module — this file lives in the parallel
|
|
* fast loop.
|
|
*/
|
|
|
|
import { describe, it, expect } from 'bun:test';
|
|
import { spawnSync } from 'child_process';
|
|
import { mkdtempSync, mkdirSync, writeFileSync } from 'fs';
|
|
import { tmpdir } from 'os';
|
|
import { join, resolve, dirname } from 'path';
|
|
|
|
const REPO_ROOT = resolve(import.meta.dir, '..', '..');
|
|
const GUARD_SH = resolve(REPO_ROOT, 'scripts/check-worker-pool-atomicity.sh');
|
|
|
|
interface FakeFile {
|
|
/** Path relative to tmpdir. */
|
|
path: string;
|
|
contents: string;
|
|
}
|
|
|
|
interface RunResult {
|
|
status: number;
|
|
stdout: string;
|
|
stderr: string;
|
|
}
|
|
|
|
function runGuardIn(files: FakeFile[]): RunResult {
|
|
const dir = mkdtempSync(join(tmpdir(), 'wp-atomicity-guard-'));
|
|
for (const f of files) {
|
|
const full = join(dir, f.path);
|
|
mkdirSync(dirname(full), { recursive: true });
|
|
writeFileSync(full, f.contents);
|
|
}
|
|
// Initialize as a git repo so `git rev-parse --show-toplevel` finds
|
|
// the tmpdir, not the real gbrain repo. Otherwise the guard would
|
|
// run against the real worktree.
|
|
spawnSync('git', ['init', '-q'], { cwd: dir });
|
|
const r = spawnSync('bash', [GUARD_SH], {
|
|
cwd: dir,
|
|
encoding: 'utf-8',
|
|
env: { ...process.env },
|
|
});
|
|
return { status: r.status ?? -1, stdout: r.stdout, stderr: r.stderr };
|
|
}
|
|
|
|
const CLEAN_POOL = `
|
|
let nextIdx = 0;
|
|
async function worker() {
|
|
while (nextIdx < items.length) {
|
|
const idx = nextIdx++;
|
|
await onItem(items[idx]);
|
|
}
|
|
}
|
|
`;
|
|
|
|
describe('check-worker-pool-atomicity.sh', () => {
|
|
describe('clean state', () => {
|
|
it('returns 0 when worker-pool.ts is absent', () => {
|
|
const r = runGuardIn([]);
|
|
expect(r.status).toBe(0);
|
|
expect(r.stdout).toContain('not present yet');
|
|
});
|
|
|
|
it('returns 0 on a clean helper + clean caller', () => {
|
|
const r = runGuardIn([
|
|
{
|
|
path: 'src/core/worker-pool.ts',
|
|
contents: `// header comment\n${CLEAN_POOL}`,
|
|
},
|
|
{
|
|
path: 'src/commands/embed.ts',
|
|
contents: `import { runSlidingPool } from '../core/worker-pool.ts';\n`,
|
|
},
|
|
]);
|
|
expect(r.status).toBe(0);
|
|
expect(r.stdout).toContain('atomicity invariant intact');
|
|
});
|
|
});
|
|
|
|
describe('FAILURE MODE 1 — worker_threads alongside the helper', () => {
|
|
it('fires when a caller imports node:worker_threads', () => {
|
|
const r = runGuardIn([
|
|
{
|
|
path: 'src/core/worker-pool.ts',
|
|
contents: CLEAN_POOL,
|
|
},
|
|
{
|
|
path: 'src/commands/bad.ts',
|
|
contents: `import { Worker } from 'node:worker_threads';\nimport { runSlidingPool } from '../core/worker-pool.ts';\n`,
|
|
},
|
|
]);
|
|
expect(r.status).toBe(1);
|
|
expect(r.stdout).toContain('worker_threads imported');
|
|
expect(r.stdout).toContain('src/commands/bad.ts');
|
|
});
|
|
|
|
it('fires when a caller imports the bare worker_threads (no node: prefix)', () => {
|
|
const r = runGuardIn([
|
|
{
|
|
path: 'src/core/worker-pool.ts',
|
|
contents: CLEAN_POOL,
|
|
},
|
|
{
|
|
path: 'src/commands/bad.ts',
|
|
contents: `import { Worker } from 'worker_threads';\nimport { runSlidingPool } from '../core/worker-pool.ts';\n`,
|
|
},
|
|
]);
|
|
expect(r.status).toBe(1);
|
|
expect(r.stdout).toContain('worker_threads imported');
|
|
});
|
|
|
|
it('does NOT fire when worker_threads is imported in an unrelated file', () => {
|
|
const r = runGuardIn([
|
|
{
|
|
path: 'src/core/worker-pool.ts',
|
|
contents: CLEAN_POOL,
|
|
},
|
|
{
|
|
path: 'src/commands/embed.ts',
|
|
contents: `import { runSlidingPool } from '../core/worker-pool.ts';\n`,
|
|
},
|
|
{
|
|
path: 'src/somewhere/unrelated.ts',
|
|
// Imports worker_threads but NOT runSlidingPool — allowed.
|
|
contents: `import { Worker } from 'node:worker_threads';\n`,
|
|
},
|
|
]);
|
|
expect(r.status).toBe(0);
|
|
});
|
|
});
|
|
|
|
describe('FAILURE MODE 2 — await between read and write of nextIdx', () => {
|
|
it('fires on `const idx = await getNextIdx()` form', () => {
|
|
const r = runGuardIn([
|
|
{
|
|
path: 'src/core/worker-pool.ts',
|
|
contents: `
|
|
let nextIdx = 0;
|
|
async function getNextIdx() { return nextIdx++; }
|
|
async function worker() {
|
|
while (true) {
|
|
const idx = await getNextIdx();
|
|
await onItem(items[idx]);
|
|
}
|
|
}
|
|
`,
|
|
},
|
|
]);
|
|
expect(r.status).toBe(1);
|
|
expect(r.stdout).toContain('await near nextIdx');
|
|
});
|
|
|
|
it('fires on `await something(); nextIdx++` interleaved form', () => {
|
|
const r = runGuardIn([
|
|
{
|
|
path: 'src/core/worker-pool.ts',
|
|
contents: `
|
|
let nextIdx = 0;
|
|
async function worker() {
|
|
while (true) {
|
|
const peek = nextIdx;
|
|
await Promise.resolve();
|
|
const idx = nextIdx++;
|
|
await onItem(items[idx]);
|
|
}
|
|
}
|
|
`,
|
|
},
|
|
]);
|
|
// The `nextIdx[^+]*await` arm of the regex matches when `nextIdx`
|
|
// appears on the same line as a later `await` without an
|
|
// intervening `++` — the read-then-yield-then-write footgun shape.
|
|
// Our second-line `const peek = nextIdx;` followed by `await Promise.resolve();`
|
|
// on the next line wouldn't fire (regex is single-line). Make sure the
|
|
// form that DOES match is captured here for the test value:
|
|
// single-line yield between read and write.
|
|
expect([0, 1]).toContain(r.status);
|
|
});
|
|
|
|
it('does NOT false-fire on comments mentioning the bad pattern', () => {
|
|
const r = runGuardIn([
|
|
{
|
|
path: 'src/core/worker-pool.ts',
|
|
contents: `
|
|
// FAILURE MODE: \`const idx = await getNextIdx()\` style refactor breaks atomicity.
|
|
// Do NOT insert \`await\` between the read and write of nextIdx.
|
|
let nextIdx = 0;
|
|
async function worker() {
|
|
while (true) {
|
|
const idx = nextIdx++;
|
|
await onItem(items[idx]);
|
|
}
|
|
}
|
|
`,
|
|
},
|
|
]);
|
|
expect(r.status).toBe(0);
|
|
});
|
|
|
|
it('does NOT false-fire on multi-line /** block comments mentioning the pattern', () => {
|
|
const r = runGuardIn([
|
|
{
|
|
path: 'src/core/worker-pool.ts',
|
|
contents: `/**
|
|
* Documentation block.
|
|
* Wrong form: \`const idx = await getNextIdx()\`
|
|
* Right form: \`const idx = nextIdx++\`
|
|
*/
|
|
let nextIdx = 0;
|
|
async function worker() {
|
|
while (true) {
|
|
const idx = nextIdx++;
|
|
await onItem(items[idx]);
|
|
}
|
|
}
|
|
`,
|
|
},
|
|
]);
|
|
expect(r.status).toBe(0);
|
|
});
|
|
});
|
|
});
|