mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat: diff-aware E2E test selector Adds scripts/select-e2e.ts: reads git diff vs origin/master, classifies the change set (EMPTY/DOC_ONLY/SRC), and emits the relevant E2E test files on stdout. Fail-closed by design: any unmapped src/ change runs all E2E. - scripts/e2e-test-map.ts: hand-tuned path-glob -> test files map - scripts/select-e2e.ts: pure-function selector with three explicit cases - scripts/run-e2e.sh: accepts optional file list from argv + --dry-run-list - test/select-e2e.test.ts: 24 cases including 3 codex regression guards (skills/, untracked files, unmapped src/) * feat: local CI gate via docker compose Adds bun run ci:local — runs every check GH Actions runs (gitleaks + unit + 29 E2E files) inside a Docker container that bind-mounts the repo. Pure bind-mount + named volumes (gbrain-ci-node-modules, gbrain-ci-bun-cache, gbrain-ci-pg-data) for fast warm restarts. - docker-compose.ci.yml: pgvector/pgvector:pg16 + oven/bun:1 - scripts/ci-local.sh: orchestrator with --diff, --no-pull, --clean - gitleaks runs on host (scoped to working dir + branch commits) - DATABASE_URL unset for unit phase (matches GH Actions split) - git installed in container at startup (oven/bun:1 omits it) - Postgres host port via GBRAIN_CI_PG_PORT env (default 5434) Stronger than PR CI: runs all 29 E2E files vs CI's 2-file Tier 1. * chore: bump version and changelog (v0.23.1) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: document local CI gate for v0.23.1 CLAUDE.md gains key-files entries for docker-compose.ci.yml, scripts/ci-local.sh, scripts/select-e2e.ts + e2e-test-map.ts, and the scripts/run-e2e.sh argv tweak. Pre-ship requirements section now lists the Docker-based local gate as Path A alongside the manual lifecycle. CONTRIBUTING.md tests section adds the bun run ci:local / ci:local:diff / ci:select-e2e block with prerequisites (Docker engine + gitleaks) and the GBRAIN_CI_PG_PORT override. AGENTS.md "Before shipping" promotes ci:local as the easiest path and keeps the manual lifecycle as a fallback. README.md Contributing section points to ci:local for the full gate. CHANGELOG.md untouched — v0.23.1 entry already finalized. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: SHARD=N/M env support in scripts/run-e2e.sh Filters the E2E file list to every M-th file starting at index N (1-indexed). Sequential execution within a shard preserves the TRUNCATE CASCADE no-race property documented at the top of the file. Empty-shard handling under `set -u` uses ${arr[@]:-} fallback. Standalone change; not yet wired up in ci-local.sh. * feat: 4-way parallel E2E shards in ci:local Replaces the single postgres service with 4 (postgres-1..4) on host ports 5434-5437. scripts/ci-local.sh fans 4 workers via xargs -P4 inside the runner container; each pinned to its own DATABASE_URL via SHARD=N/4. Wall-time on a 16-core host: ~6 min sequential -> ~1.5-2 min sharded. Total full-gate wall-time goes from ~25 min to ~3-5 min warm. Also handles git-worktree (Conductor) layouts: when /app/.git is a file instead of a directory, parse the gitdir + commondir and bind-mount the shared host gitdir at its absolute path. Without this, in-container `git ls-files` (used by scripts/check-trailing-newline.sh and friends) exits 128 with "not a git repository". Also runs `git config --global --add safe.directory '*'` inside the container so the root-uid container can read host-uid gitdir without "dubious ownership" rejection. CHANGELOG entry updated to cover the speedup. - docker-compose.ci.yml: 4 pgvector services + per-shard named volumes - scripts/ci-local.sh: parallel xargs orchestration + worktree mount fix - CHANGELOG.md v0.23.1: 4-way sharded wall-time, 36 E2E files, --no-shard flag * chore: regenerate llms-full.txt for v0.23.1 doc updates Required by test/build-llms.test.ts case 4 — committed llms-full.txt must match `bun run build:llms` output. The CHANGELOG + CLAUDE.md updates in this branch shifted bytes; regen catches up. * feat: scripts/run-unit-shard.sh + slow-test convention Tier 1 + Tier 4 plumbing: - scripts/run-unit-shard.sh: SHARD=N/M filter for unit files (excludes test/e2e/*). Excludes *.slow.test.ts (Tier 4 convention) so the fast shard fan-out skips known-slow files; CI's `bun run test` still includes them via default discovery. - scripts/run-slow-tests.sh: companion that runs ONLY *.slow.test.ts. Wired as `bun run test:slow`. - scripts/profile-tests.sh: portable awk parser that extracts the top-N slowest tests from any captured `bun test` output. Wired as `bun run test:profile`. Use it to pick demotion candidates. * feat: PGLite snapshot fixture for ~4.5x faster cold init (Tier 3) scripts/build-pglite-snapshot.ts boots a fresh PGLite, runs the full initSchema() (forward bootstrap + 30 migrations), and dumps the post-init state to test/fixtures/pglite-snapshot.tar plus a SHA-256 schema hash sidecar (.version). Both gitignored — built on demand via `bun run build:pglite-snapshot`. PGLiteEngine.connect() reads GBRAIN_PGLITE_SNAPSHOT env: validates the sidecar hash against the in-process MIGRATIONS hash, loads via PGLite's loadDataDir blob, sets _snapshotLoaded so initSchema() short-circuits. Measured per-file cold init drops from 828ms → 181ms. Bootstrap-correctness tests (bootstrap.test.ts, schema-bootstrap-coverage.test.ts) explicitly delete the env at file top so they keep exercising the cold path they verify. * feat: --classify-only + heartbeat tolerance fix (Tiers 2 + flake fix) - scripts/select-e2e.ts: --classify-only flag emits EMPTY|DOC_ONLY|SRC. Used by ci-local.sh's --diff fast-path to skip the heavy gate when only docs changed. - test/progress.test.ts: startHeartbeat tolerance widened to 1-20 over 200ms (was 2-6 over 85ms). Under 4-way parallel shard load on a contended host, setTimeout's effective quantum balloons and the tight bound flakes. The test still verifies "fires multiple times, stops cleanly" — exact count was never load-bearing. * feat: 4-way unit + E2E sharding in ci-local.sh + CHANGELOG (Tiers 1-4) ci-local.sh ties the four tiers together: - Tier 2: pre-flight diff classification on host. DOC_ONLY exits in ~5s (gitleaks only, no postgres, no container). - Tier 1: guards + typecheck run ONCE before fan-out. xargs -P4 then spawns 4 shards inside the runner container, each running unit phase (env -u DATABASE_URL bash run-unit-shard.sh) followed by E2E phase (DATABASE_URL=postgres-N bash run-e2e.sh) — both sharded N/4. Per-shard logs in /tmp/shard-logs/shard-N.log; printed in shard order at the end. - Tier 3: snapshot fixture built once at runner startup if missing, GBRAIN_PGLITE_SNAPSHOT exported so all shards inherit. - Tier 4: run-unit-shard.sh excludes *.slow.test.ts; run-slow-tests.sh + test:slow npm script handle the demoted set. - --no-shard preserves the legacy single-process flow for debug. package.json: build:pglite-snapshot, test:slow, test:profile scripts. Measured wall-time on 16-core host: 100s warm (down from ~22 min cold single-process). 4 shards × ~640-1024 unit tests each, plus 9 E2E files each. PGLite snapshot saves 4.5× per cold init (828ms → 181ms). CHANGELOG.md updated with measured numbers + four-tier breakdown. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
118 lines
3.8 KiB
YAML
118 lines
3.8 KiB
YAML
# docker-compose.ci.yml
|
|
#
|
|
# Local CI gate with 4-way E2E sharding. Spins up 4 pgvector services + a bun
|
|
# runner that bind-mounts the repo. Used by `bun run ci:local` and
|
|
# `bun run ci:local:diff` (see scripts/ci-local.sh).
|
|
#
|
|
# All services are pulled as `image:` (no build) so `docker compose pull`
|
|
# refreshes everything. The bun version floats with `oven/bun:1` to track CI's
|
|
# `bun-version: latest`. Named volumes isolate the Linux container's deps from
|
|
# the host's darwin-arm64 deps and keep bun + postgres data warm across runs.
|
|
#
|
|
# Why 4 postgres services: bun's E2E suite shares one DB across 36 files and
|
|
# uses TRUNCATE CASCADE in setupDB(). Running files in parallel against ONE DB
|
|
# races (file A's TRUNCATE clobbers file B's fixture import). 4 separate DBs
|
|
# remove the race; we shard the file list 1/4..4/4 and run shards in parallel.
|
|
# Within a shard, files still run sequentially. Total wall-time on a 16-core
|
|
# host: ~6 min sequential -> ~1.5-2 min sharded.
|
|
#
|
|
# Postgres host ports default to 5434-5437 (avoid 5432 manual `gbrain-test-pg`
|
|
# and 5433 sibling-project conflicts). Override BASE port with GBRAIN_CI_PG_PORT;
|
|
# shards take BASE..BASE+3.
|
|
|
|
services:
|
|
postgres-1:
|
|
image: pgvector/pgvector:pg16
|
|
environment:
|
|
POSTGRES_USER: postgres
|
|
POSTGRES_PASSWORD: postgres
|
|
POSTGRES_DB: gbrain_test
|
|
ports:
|
|
- "${GBRAIN_CI_PG_PORT:-5434}:5432"
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
volumes:
|
|
- gbrain-ci-pg-data-1:/var/lib/postgresql/data
|
|
|
|
postgres-2:
|
|
image: pgvector/pgvector:pg16
|
|
environment:
|
|
POSTGRES_USER: postgres
|
|
POSTGRES_PASSWORD: postgres
|
|
POSTGRES_DB: gbrain_test
|
|
ports:
|
|
- "${GBRAIN_CI_PG_PORT_2:-5435}:5432"
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
volumes:
|
|
- gbrain-ci-pg-data-2:/var/lib/postgresql/data
|
|
|
|
postgres-3:
|
|
image: pgvector/pgvector:pg16
|
|
environment:
|
|
POSTGRES_USER: postgres
|
|
POSTGRES_PASSWORD: postgres
|
|
POSTGRES_DB: gbrain_test
|
|
ports:
|
|
- "${GBRAIN_CI_PG_PORT_3:-5436}:5432"
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
volumes:
|
|
- gbrain-ci-pg-data-3:/var/lib/postgresql/data
|
|
|
|
postgres-4:
|
|
image: pgvector/pgvector:pg16
|
|
environment:
|
|
POSTGRES_USER: postgres
|
|
POSTGRES_PASSWORD: postgres
|
|
POSTGRES_DB: gbrain_test
|
|
ports:
|
|
- "${GBRAIN_CI_PG_PORT_4:-5437}:5432"
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U postgres -d gbrain_test"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
volumes:
|
|
- gbrain-ci-pg-data-4:/var/lib/postgresql/data
|
|
|
|
runner:
|
|
image: oven/bun:1
|
|
working_dir: /app
|
|
depends_on:
|
|
postgres-1:
|
|
condition: service_healthy
|
|
postgres-2:
|
|
condition: service_healthy
|
|
postgres-3:
|
|
condition: service_healthy
|
|
postgres-4:
|
|
condition: service_healthy
|
|
# No global DATABASE_URL — scripts/ci-local.sh sets per-shard URL via -e.
|
|
# Unit phase explicitly unsets DATABASE_URL so test/e2e/* gracefully skip.
|
|
volumes:
|
|
- .:/app
|
|
# Linux container's node_modules MUST be isolated from host darwin-arm64.
|
|
# Without this, container `bun install` stomps host node_modules and
|
|
# subsequent `bun test` on host fails with binary-incompat errors.
|
|
- gbrain-ci-node-modules:/app/node_modules
|
|
# Warm install cache across runs.
|
|
- gbrain-ci-bun-cache:/root/.bun/install/cache
|
|
|
|
volumes:
|
|
gbrain-ci-pg-data-1:
|
|
gbrain-ci-pg-data-2:
|
|
gbrain-ci-pg-data-3:
|
|
gbrain-ci-pg-data-4:
|
|
gbrain-ci-node-modules:
|
|
gbrain-ci-bun-cache:
|