Files
gbrain/test/parallel.test.ts
T
df86ea5f1d v0.40.5.0 Federated Sync v2 — parallel source sync + push triggers + per-source health (#1322)
* wip: federated sync v2 pre-merge snapshot

* v0.40.5.0 Federated Sync v2 — parallel source sync + push triggers + per-source health

Bump VERSION + package.json + CHANGELOG header + migration walkthrough filename
to v0.40.5.0 (claiming the next free slot in the v0.40.x patch series after
master's v0.40.1.0).

What ships (6 components, all behind sync.federated_v2 feature flag default-on):
1. Per-source sync lock — syncLockId(sourceId), phantom-redirect parity
2. Parallel sync --all — pMapAllSettled fan-out, --max-sources N cap
3. embed-backfill minion handler — D2 per-source lock + D6 $10/job budget + D15.1
   fire-and-forget submission + D19 source-level cooldown + 24h $25 rolling cap
4. sync trigger CLI + POST /webhooks/github — HMAC-verified (60 req/min/IP),
   X-GitHub-Event=push + ref filter against tracked_branch
5. sources status + federation_health doctor — batched GROUP BY pipeline
   (4 queries instead of 6×N per-source roundtrips)
6. sources federate/unfederate hook — auto-submit embed-backfill on flip

Correctness fixes (unconditional):
- D21: sync.ts:959 facts backstop now passes sourceId to engine.getPage
- D15.4: redactSourceConfig + CI guard prevent webhook_secret leak
- D15.5: safeHexEqual extracted to src/core/timing-safe.ts

Schema:
- Migration v89 (sources_github_repo_index): partial expression index on
  config->>'github_repo' for fast webhook source-lookup

Tests:
- 14 new test files, 112 cases. 4 IRON-RULE regressions pinned (SYNC_LOCK_ID
  back-compat, phantom per-source lock, embed-backfill kill+resume,
  webhook HMAC prefix-strip). All 9449 unit tests pass.

Caught at test-write time: the webhook handler had a Buffer.from('sha256=...',
'hex') truncation bug — without the prefix-strip, every signature would have
"matched" empty buffers. Pinned by a test/sources-webhook.test.ts IRON-RULE.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(check-source-config-leak): tighten regex to source-row patterns only

The v0.40.5.0 wave added scripts/check-source-config-leak.sh with a
too-broad pattern (JSON\.stringify\(.*config) that flagged any variable
named 'config' — catching the GLOBAL gbrain config.json serializers in
src/commands/init.ts (status envelopes) and src/core/config.ts (the
config-file write site). On the CI runner without rg installed, the
grep -rE fallback fired correctly and produced 4 false positives that
broke the `verify` script.

Tightened the patterns to specifically match `(source|src|row|s).config`
property access — the actual risk shape (a sources-table row being
serialized whole). The global gbrain config has a different shape and
threat model (file-mode 0o600 at the write site), so it's safe to
exempt at the regex level rather than per-file whitelist.

Also fixed a latent bug: the rg branch used `--include='*.ts'` (grep's
flag, not rg's). rg silently rejected it and CANDIDATES came back empty,
so the local-dev runs (which have rg) would never have caught a real
leak. Now branches on tool availability: `-g '*.ts'` for rg, `--include`
for grep -rE. Both branches verified against a synthetic leak fixture.

Also added init.ts + config.ts to the whitelist as a belt-and-suspenders
since they handle gbrain-global config (not source rows) and could
otherwise reflect-back via regex iteration.

CI: `bun run verify` exit 0 locally with both the original false-positive
fixture (clean repo) and a synthetic leak fixture (correctly caught,
exit 1).

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 10:21:59 -07:00

84 lines
3.1 KiB
TypeScript

/**
* Tests for src/core/parallel.ts (v0.40 Federated Sync v2).
*
* Pins: result order matches input order, one rejection doesn't kill others,
* concurrency cap is a hard ceiling, empty input returns empty, invalid
* concurrency throws TypeError.
*/
import { describe, test, expect } from 'bun:test';
import { pMapAllSettled } from '../src/core/parallel.ts';
describe('pMapAllSettled', () => {
test('returns results in input order', async () => {
const items = [1, 2, 3, 4, 5];
const results = await pMapAllSettled(items, 2, async (n) => n * 10);
expect(results).toHaveLength(5);
expect(results.map((r) => (r.status === 'fulfilled' ? r.value : null))).toEqual([10, 20, 30, 40, 50]);
});
test('one rejection does not kill other items', async () => {
const items = ['a', 'b', 'c'];
const results = await pMapAllSettled(items, 2, async (s) => {
if (s === 'b') throw new Error('intentional');
return s.toUpperCase();
});
expect(results[0]).toEqual({ status: 'fulfilled', value: 'A' });
expect(results[1].status).toBe('rejected');
expect(results[1].status === 'rejected' && (results[1].reason as Error).message).toBe('intentional');
expect(results[2]).toEqual({ status: 'fulfilled', value: 'C' });
});
test('hard ceiling: never exceeds concurrency in flight', async () => {
let inFlight = 0;
let peak = 0;
const items = Array.from({ length: 20 }, (_, i) => i);
await pMapAllSettled(items, 3, async () => {
inFlight++;
peak = Math.max(peak, inFlight);
await new Promise((r) => setTimeout(r, 5));
inFlight--;
return 'ok';
});
expect(peak).toBeLessThanOrEqual(3);
expect(peak).toBeGreaterThan(0);
});
test('empty input returns empty array', async () => {
const results = await pMapAllSettled([], 5, async () => 'x');
expect(results).toEqual([]);
});
test('concurrency > items.length runs all in parallel (no semaphore stall)', async () => {
const start = Date.now();
const items = [1, 2, 3];
await pMapAllSettled(items, 100, async () => {
await new Promise((r) => setTimeout(r, 50));
return 'ok';
});
const elapsed = Date.now() - start;
// Sequential would be ~150ms; concurrent ~50ms. Allow generous slack.
expect(elapsed).toBeLessThan(140);
});
test('throws TypeError on concurrency < 1', async () => {
await expect(pMapAllSettled([1], 0, async (x) => x)).rejects.toThrow(TypeError);
await expect(pMapAllSettled([1], -3, async (x) => x)).rejects.toThrow(TypeError);
});
test('throws TypeError on non-integer concurrency', async () => {
await expect(pMapAllSettled([1], 1.5, async (x) => x)).rejects.toThrow(TypeError);
await expect(pMapAllSettled([1], NaN, async (x) => x)).rejects.toThrow(TypeError);
});
test('passes index to fn', async () => {
const seen: Array<[string, number]> = [];
await pMapAllSettled(['a', 'b', 'c'], 2, async (item, i) => {
seen.push([item, i]);
return item;
});
// Sort by item since execution order isn't guaranteed.
seen.sort((a, b) => a[0].localeCompare(b[0]));
expect(seen).toEqual([['a', 0], ['b', 1], ['c', 2]]);
});
});