mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-28 14:59:47 +00:00
v0.41.11.1 ci: cut CI wallclock from 9min to 4.5min (#1457)
* feat(eval-longmemeval): RunOpts.engine seam for shared benchmark brain Adds optional `engine?: PGLiteEngine` field to RunOpts. When set, runEvalLongMemEval uses the caller-provided engine and skips the withBenchmarkBrain wrapper (no fresh PGLite create, no disconnect on exit). When unset, the production CLI path is unchanged: withBenchmarkBrain creates and disposes a fresh engine per invocation. Designed for the test seam that's about to land: one beforeAll-created brain shared across all 13 runEvalLongMemEval calls in test/eval-longmemeval-e2e.slow.test.ts, amortizing the ~1-3s PGLite cold-create cost. runOneQuestion already calls resetTables() as its first line so per-test isolation is preserved across the shared engine. Pure additive seam — every existing caller (CLI, current tests that already create engines via withBenchmarkBrain implicitly) keeps its current behavior because opts.engine defaults to undefined. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(test): split eval-longmemeval slow tests + share engine across e2e half The 884-line test/eval-longmemeval.slow.test.ts was the heaviest single file in CI at ~359s on the matrix. Split by runEvalLongMemEval usage: - test/eval-longmemeval.slow.test.ts (trimmed): 8 pure describes, 15 tests. Harness lifecycle, resetTables, schema-migration robustness, warm-create speed gate, adapter haystackToPages, source-boost guard, loadResumeSet, buildByTypeSummary. Local wall: 1.985s, projected CI ~42s. - test/eval-longmemeval-e2e.slow.test.ts (NEW): 8 e2e describes, 11 tests. Every describe that calls runEvalLongMemEval — 13 call sites total. Threads a single beforeAll-created PGLite via the v0.41.10 RunOpts.engine seam. Local wall: 9.33s (was 15.09s without sharing); projected CI ~196s (was ~317s). - test/helpers/longmemeval-stub.ts (NEW): shared makeStubClient + StubCall. Matches the existing test/helpers/ convention (with-env.ts, reset-pglite.ts). Single source of truth across the two split files. - scripts/test-weights.json: replaced 359087ms entry with TWO entries (42000ms pure, 196000ms e2e). Projected linearly from local wall-clock × 21 CI scaling factor. First post-merge CI run will refine via scripts/mine-shard-weights.ts. Test count is preserved: 15 pure + 11 e2e = 26, matches original file. No production code changes in this commit — only test reorganization + opt-in to the RunOpts.engine seam from the previous commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ci(test): bump matrix 6→10, dedicate two slow files, cache bun-install CI matrix wallclock: ~9 min → ~4.5 min. Three coordinated changes. 1. .github/workflows/test.yml matrix bumped from 6 → 10 shards. Per-shard total drops from 532s → 272s. Honest concurrency-budget call: total gated jobs go 13 → 18, so 2 concurrent PRs ≈ 36 queued, past the GH free-tier ~20 ceiling — single-PR runs unaffected, multi-PR days see queue pressure. Worth it for the 4-min CI saving. 2. Two slow files pulled out of the matrix and into their own dedicated jobs (sibling to verify, serial-tests): - slow-eval-longmemeval runs test/eval-longmemeval-e2e.slow.test.ts (~196s after the engine-sharing seam from the previous two commits). - slow-entity-resolve-perf runs test/entity-resolve-perf.slow.test.ts (~159s, single non-subdivisible perf test). The 60s default bun timeout is too tight for this file — bumped to 300000ms. scripts/test-shard.sh excludes both via -not -name clauses so the matrix sweep doesn't double-run them. Both new jobs wire into cache-write.needs and test-status.needs so CI gates on them. 3. actions/cache for ~/.bun/install/cache added to every job that runs bun install (test matrix, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf). Keyed on bun.lock hash. Saves ~15s per job on cache hit; first-PR push pays full cost, subsequent runs hit cache. Total CI wallclock now bounded by max(matrix ~4.5min, slow-eval ~3.3min, slow-entity-resolve-perf ~2.6min) = ~4.5 min. The matrix is back to being the floor; no single test file dominates a shard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: v0.41.10.0 — CI wallclock 9min → 4.5min VERSION + package.json + CHANGELOG entry for the three preceding commits: feat(eval-longmemeval): RunOpts.engine seam for shared benchmark brain refactor(test): split eval-longmemeval slow tests + share engine across e2e half ci(test): bump matrix 6→10, dedicate two slow files, cache bun-install Net user-visible: CI 'Test' check finishes in ~4.5 min instead of ~9 min. Net contributor-visible: new RunOpts.engine seam on runEvalLongMemEval for benchmark suites that want to amortize PGLite cold-create across many calls. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): quarantine hybrid-meta + schema-pack-load-active to serial The 6→10 matrix shard bump in this branch re-shuffled file distribution across shard processes. Two pre-existing tests with hidden cross-file state dependencies surfaced as failures in CI run #77779498812/13: - test/hybrid-meta.test.ts shard 7: gateway state (configured by some other test in the same shard process) survived past the test's `delete process.env.OPENAI_API_KEY` call, so the early-return for expansion didn't fire and `expansion_applied` stayed true. - test/schema-pack-load-active.test.ts shard 8: the schema-pack module's test-injected locator state was left behind by an earlier file, so `loadActivePack` with the default config didn't fall through to the bundled gbrain-base path. Both files pass cleanly solo (verified). The pollution sources are unidentified — bun's reporter only printed 14 of 71 file headers per shard log, hiding the polluters. Rather than spelunk for the source, rename both files to *.serial.test.ts. The serial pass runs them at --max-concurrency=1 in a process that doesn't share state with the parallel matrix shards. Same-wave bookkeeping: - scripts/check-test-isolation.allowlist: drop test/hybrid-meta.test.ts entry (file is now serial, no longer R1-eligible). - scripts/test-weights.json: rename both weight entries to match the new filenames so future matrix LPT runs don't fall back to median. Companion to a7d029d0/2e1c269e/5a749acb of this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
84fed4194a
commit
552ff4ed82
+78
-11
@@ -90,6 +90,10 @@ jobs:
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
|
||||
with:
|
||||
path: ~/.bun/install/cache
|
||||
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
|
||||
- run: bun install
|
||||
- run: bun run verify
|
||||
|
||||
@@ -106,17 +110,74 @@ jobs:
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
|
||||
with:
|
||||
path: ~/.bun/install/cache
|
||||
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
|
||||
- run: bun install
|
||||
- run: bun run test:serial
|
||||
|
||||
slow-eval-longmemeval:
|
||||
# Dedicated runner for the LongMemEval end-to-end test file. The file
|
||||
# was originally 359s. TODO #1 (engine-sharing in runEvalLongMemEval
|
||||
# via RunOpts.engine) cut it to ~200s by amortizing PGLite cold-create
|
||||
# across all 13 runEvalLongMemEval calls in one beforeAll-shared brain.
|
||||
# Pulled out of the matrix (see scripts/test-shard.sh) so a single 200s
|
||||
# atom doesn't dominate a shard's wallclock. Companion file
|
||||
# test/eval-longmemeval.slow.test.ts (the pure-bucket half) stays in
|
||||
# the matrix because it's light (~42s).
|
||||
needs: cache-check
|
||||
if: needs.cache-check.outputs.hit != 'true'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
|
||||
with:
|
||||
path: ~/.bun/install/cache
|
||||
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
|
||||
- run: bun install
|
||||
- run: bun test test/eval-longmemeval-e2e.slow.test.ts --timeout=60000
|
||||
|
||||
slow-entity-resolve-perf:
|
||||
# Dedicated runner for the entity-resolve perf test (~159s, single perf
|
||||
# describe with one test that builds 5000+ pages and asserts the NEW
|
||||
# tryPrefixExpansion shape is 5x faster than the OLD shape — not
|
||||
# subdivisible without weakening the perf guarantee). Pulled out of the
|
||||
# matrix (see scripts/test-shard.sh) so a single 159s atom doesn't
|
||||
# dominate a shard's wallclock. Runs in parallel with the matrix.
|
||||
needs: cache-check
|
||||
if: needs.cache-check.outputs.hit != 'true'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
|
||||
with:
|
||||
path: ~/.bun/install/cache
|
||||
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
|
||||
- run: bun install
|
||||
- run: bun test test/entity-resolve-perf.slow.test.ts --timeout=300000
|
||||
|
||||
test:
|
||||
# Pure matrix shard — no verify, no serial. Each shard runs its slice
|
||||
# of the unit test set under one `bun test` invocation.
|
||||
#
|
||||
# 6 shards (not 8) stays under the GitHub free-tier ~20-job concurrency
|
||||
# budget when multiple PRs land same day: 6 shards + verify + serial +
|
||||
# gitleaks + cache-check + cache-write + test-status = ~12 jobs × 2
|
||||
# concurrent PRs = 24; 8 shards × 2 PRs would queue worse.
|
||||
# 10 shards (was 6) drops per-shard total from 532s → 287s. With the two
|
||||
# dedicated jobs (slow-eval-longmemeval, slow-entity-resolve-perf) also
|
||||
# pulled out, the matrix is bounded by ~287s ≈ 4.8 min. Total CI ≈ max
|
||||
# of matrix + slow-eval (~3.3 min after engine-sharing in TODO #1) +
|
||||
# slow-entity-resolve-perf (~2.6 min) ≈ 4.8 min.
|
||||
#
|
||||
# Concurrency budget: 10 shards + verify + serial + slow-eval +
|
||||
# slow-entity-resolve-perf + gitleaks + cache-check + cache-write +
|
||||
# test-status = ~18 jobs × 2 concurrent PRs = 36. GitHub free-tier
|
||||
# caps at ~20 concurrent jobs, so multi-PR days will see some queue
|
||||
# pressure. Single-PR runs are unaffected.
|
||||
#
|
||||
# Partition policy is weight-aware LPT bin-packing via scripts/sharding.ts
|
||||
# (replaces FNV-1a path hash). Weights live in scripts/test-weights.json,
|
||||
@@ -128,15 +189,19 @@ jobs:
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [1, 2, 3, 4, 5, 6]
|
||||
shard: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
|
||||
steps:
|
||||
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
|
||||
with:
|
||||
bun-version: 1.3.13
|
||||
- uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
|
||||
with:
|
||||
path: ~/.bun/install/cache
|
||||
key: bun-cache-${{ runner.os }}-${{ hashFiles('bun.lock') }}
|
||||
- run: bun install
|
||||
- name: Run test shard ${{ matrix.shard }}/6
|
||||
run: scripts/test-shard.sh ${{ matrix.shard }} 6
|
||||
- name: Run test shard ${{ matrix.shard }}/10
|
||||
run: scripts/test-shard.sh ${{ matrix.shard }} 10
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
# cache-write: ONLY runs when every gated job succeeded. Writes the
|
||||
@@ -147,7 +212,7 @@ jobs:
|
||||
# entry written when tests hadn't actually passed).
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
cache-write:
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, test]
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
|
||||
if: success() && needs.cache-check.outputs.hit != 'true'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
@@ -169,7 +234,7 @@ jobs:
|
||||
# Branch protection (when configured) gates on this single job name.
|
||||
# ──────────────────────────────────────────────────────────────────────
|
||||
test-status:
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, test]
|
||||
needs: [cache-check, gitleaks, verify, serial-tests, slow-eval-longmemeval, slow-entity-resolve-perf, test]
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
@@ -179,15 +244,17 @@ jobs:
|
||||
GITLEAKS="${{ needs.gitleaks.result }}"
|
||||
VERIFY="${{ needs.verify.result }}"
|
||||
SERIAL="${{ needs.serial-tests.result }}"
|
||||
SLOW_EVAL="${{ needs.slow-eval-longmemeval.result }}"
|
||||
SLOW_PERF="${{ needs.slow-entity-resolve-perf.result }}"
|
||||
TEST="${{ needs.test.result }}"
|
||||
echo "cache-check.hit=$HIT"
|
||||
echo "gitleaks=$GITLEAKS verify=$VERIFY serial-tests=$SERIAL test=$TEST"
|
||||
echo "gitleaks=$GITLEAKS verify=$VERIFY serial-tests=$SERIAL slow-eval-longmemeval=$SLOW_EVAL slow-entity-resolve-perf=$SLOW_PERF test=$TEST"
|
||||
if [ "$HIT" = "true" ]; then
|
||||
echo "✓ cache HIT for hash ${{ needs.cache-check.outputs.hash }} — CI green"
|
||||
exit 0
|
||||
fi
|
||||
# Cache miss: every gated job must have succeeded.
|
||||
for r in "$GITLEAKS" "$VERIFY" "$SERIAL" "$TEST"; do
|
||||
for r in "$GITLEAKS" "$VERIFY" "$SERIAL" "$SLOW_EVAL" "$SLOW_PERF" "$TEST"; do
|
||||
if [ "$r" != "success" ]; then
|
||||
echo "✗ gated job did not succeed (got $r) — CI fail"
|
||||
exit 1
|
||||
|
||||
Reference in New Issue
Block a user