admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-30 03:12:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
271 Commits 567 Branches 1 Tag
6dcbb7ed9ba9437e7cb7ddc8d577f3128d4cca6c
Commit Graph
3 Commits
Author SHA1 Message Date
Garry TanGitHubClaude Opus 4.7garrytan-agents
 		84fed4194a v0.41.11.0 feat: conversation retrieval upgrade — production-bar replacement for PR #1406 (#1446) 
* v0.41.11.0 feat: conversation retrieval upgrade — production-bar replacement for PR #1406

Long chat threads stop swallowing your search results. The recall miss
class on long iMessage/Slack imports (60K+ msg history; a chunk that
reads only "Locker 93 code 9494" has no topical anchor because "cabin"
was established 50K messages earlier) gets fixed by walking
conversation/meeting/slack/email pages, splitting into time-windowed
segments (30-min gap or 30-msg cap), prepending a topical/temporal
header, and running through the existing extractFactsFromTurn() so
the resulting anchor-rich facts surface in gbrain search.

This is the production-bar replacement for PR #1406 (which closes
LAST per Codex T6d, AFTER this PR is green). The bug fix survives 1:1;
the wrapping closes 14 load-bearing issues the original PR deferred
or shipped silent bugs around. The wave went through CEO scope review,
3 rounds of spec review, 2 rounds of Codex outside voice grounding
the plan against actual code, and 2 passes of eng review.

Version-slot note: originally planned as v0.41.2.0; master shipped its
own v0.41.2.0 (lens packs) plus v0.41.3-6.0 between plan-time and
ship-time. Re-bumped to v0.41.11.0 (next free slot; v0.41.7-10 claimed
by other open PRs).

Key files (new):
- src/commands/extract-conversation-facts.ts — CLI command with
  --types, --max-cost-usd, --background, --override-disabled,
  --slug, --dry-run, --limit, --since, --force, --sleep,
  --segment-limit, --source-id. Strict per-source core; two-phase
  page enumeration (paginated listPages with 10×25MB cap = 250MB
  worst case); 25MB body cap; page-global row_num accumulator
  (Codex C1 unique-index collision fix); page-level TERMINAL audit
  row after all segments commit (Codex C7 durable extraction marker);
  optional opts.budgetTracker (Codex C5 — nested withBudgetTracker
  REPLACES, so caller-managed scope passes tracker through); reads
  compiled_truth + timeline (F1 — PR silently dropped timeline half);
  honors facts.extraction_enabled kill-switch with --override-disabled
  escape (F2); --types reads cycle config as single source of truth
  (Eng-v2 A2); fingerprint on sourceId only (Eng-v2 A3 — widening
  types doesn't invalidate completion); string-encoded op-checkpoint
  entries "sourceId|slug|endIso" for resume; segment caps tuned
  6500/30 (Eng-v2 T5) to stay under extract.ts MAX_TURN_TEXT_CHARS=8000.
- src/core/cycle/conversation-facts-backfill.ts — cycle phase wrapper
  (default OFF). Iterates listSources() directly; creates ONE
  brain-wide BudgetTracker per tick + wraps the loop in
  withBudgetTracker + passes tracker through opts.budgetTracker so
  core doesn't nest-replace. Two-layer cost AND walltime protection:
  per-source caps ($1, 20min) AND brain-wide caps ($5, 30min).
- test/extract-conversation-facts.test.ts — 27 unit cases (parse,
  segment, render, checkpoint encoding, fingerprint, terminal audit
  row, row_num accumulator, F2 kill-switch, --override-disabled).
- skills/migrations/v0.41.11.0.md — agent-facing migration guide.

Key files (modified):
- src/commands/jobs.ts — register extract-conversation-facts Minion
  handler. NOT in PROTECTED_JOB_NAMES; BudgetExhausted catch + persist
  + mark completed with result.budget_exhausted (NOT a failure).
- src/commands/doctor.ts — computeConversationFactsBacklogCheck (3-state:
  SKIPPED when feature disabled per Eng-v2 C9, OK at backlog=0, WARN
  at >10 with paste-ready remediation step via makeRemediationStep).
  Doctor query is source-scoped (Codex C2 cross-source safety) and
  matches the TERMINAL audit row (Codex C7), not any-fact-for-slug.
- src/commands/sources.ts — runAudit extended with
  facts_backfill_estimate field for cost preview.
- src/cli.ts — CLI_ONLY + CLI_ONLY_SELF_HELP + THIN_CLIENT_REFUSED_COMMANDS
  + dispatch case for extract-conversation-facts.
- src/core/cycle.ts — new CyclePhase 'conversation_facts_backfill';
  PHASE_SCOPE='source' (taxonomy only per cycle.ts:131 — wrapper does
  own multi-source iteration); wired into ALL_PHASES + NEEDS_LOCK_PHASES;
  dispatch block runs between consolidate and embed.
- src/core/migrate.ts — migration v94 adds partial index
  idx_facts_extract_conversation_session ON facts(source_id, source_session)
  WHERE source LIKE 'cli:extract-conversation-facts%' so doctor query
  stays fast on million-fact brains. v14 precedent: transaction:false +
  invalid-index pre-drop on Postgres, plain CREATE INDEX on PGLite.
- src/core/schema-pack/base/gbrain-base.yaml — promote conversation
  (temporal, extractable) and atom (annotation, NOT extractable —
  atoms ARE the extracted form) into base. Flip concept.extractable:
  true semantically (cosmetic on backstop path per Codex T3; the
  original grandfather migration was solving a phantom, dropped).
  Filing rules added for both new types.
- src/core/schema-pack/base/gbrain-recommended.yaml — remove duplicate
  conversation (now inherits via extends: gbrain-base).
- src/core/types.ts — ALL_PAGE_TYPES extended with conversation, atom.
- test/extractable-pack.test.ts — updated parity gate (24 page types
  vs PR's 22; concept + conversation now extractable, atom not).
- test/schema-cli.test.ts — page-count expectation 22→24.
- VERSION + package.json bumped to 0.41.11.0.
- CHANGELOG.md release-summary in the required ELI10-first voice +
  itemized changes section.
- CLAUDE.md Key Files entry for the new modules + architecture notes.
- llms.txt + llms-full.txt regenerated.

Plan + decisions persisted at:
  ~/.claude/plans/system-instruction-you-are-working-linear-unicorn.md
CEO plan at:
  ~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-05-25-conversation-retrieval-upgrade.md

Co-Authored-By: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: align v0.41.11.0 phase ordering + bump hardcoded counts after master merge

Three CI failures from the master merge in this branch:

1. test/phase-scope-coverage.test.ts pinned `ALL_PHASES.length === 19`
   and `Object.keys(PHASE_SCOPE).length === 19`. After merging master's
   v0.41 lens-packs (extract_atoms + synthesize_concepts) + my new
   conversation_facts_backfill phase, the total is 20.

2. test/core/cycle.serial.test.ts had two hardcoded `19` assertions
   (`hookCalls` and `report.phases.length`) tracking the same count.
   Both bumped to 20.

3. cycle.serial's `'default: all 6 phases run in order'` test asserts
   `report.phases.map(p => p.phase) === ALL_PHASES`. My initial commit
   put `conversation_facts_backfill` in ALL_PHASES between consolidate
   and propose_takes, but the runCycle dispatch block runs it AFTER
   the calibration trio (propose_takes / grade_takes / calibration_profile)
   and BEFORE embed. List and dispatch order didn't match, so the
   equality assertion failed.

   Resolution: moved 'conversation_facts_backfill' in ALL_PHASES to
   AFTER 'calibration_profile' so list-order matches dispatch-order.
   The dispatch block placement was correct (and remains correct);
   the list-position comment originally said "AFTER consolidate" but
   the dispatch runs it after the WHOLE consolidate→calibration_profile
   block, not just after consolidate. Comment now reflects reality.

Verified: 61/61 pass across the 3 affected test files (2.9s wallclock).

The CI logs also showed a "(unnamed) [3058.07ms]" failure in shard 1;
unable to reproduce locally (test/scripts/run-unit-parallel.test.ts
passes 6/6 in 1s). Suspected CI-load flakiness under bun's parallel
scheduler. If it persists on the next CI run, will dig in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(ci): orphan sleep cleanup in run-unit-parallel.sh heartbeat

Two CI runs in a row reported `(fail) (unnamed) [~2400ms]` in shard 1
on this PR. Investigation:

- CI's end-of-job cleanup logged: "Terminate orphan process: pid (3344)
  (sleep)" × 6 sleep processes.
- The 6 matches exactly the 6 `runWrapper()` calls in
  test/scripts/run-unit-parallel.test.ts (1 orphan sleep per invocation).
- Each `runWrapper()` spawns scripts/run-unit-parallel.sh, which spawns
  a heartbeat function that runs `while true; do sleep 10; ...; done`
  in the background.
- The wrapper's EXIT trap was `kill "$HB_PID" 2>/dev/null` — kills the
  heartbeat shell, but its currently-running `sleep 10` child gets
  reparented to init/launchd because SIGTERM to a bash shell sleeping
  inside `sleep` doesn't propagate to the sleep child before wait
  returns. Known bash quirk on Linux.
- bun's test runner treats the orphan sleeps as a `(unnamed)` failure
  attributed to the test file that spawned the wrapper.

Fix: pkill children FIRST, then kill heartbeat. If we kill heartbeat
first, its child sleep orphans and pkill -P can no longer find it
(ppid changes to 1). Reorder applied to both the trap AND the normal
shutdown path.

Verified locally: before fix, 6 orphan sleeps after the test ran;
after fix, 0 orphan sleeps. Test still passes 6/6 in ~1s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 15:21:59 -07:00
Garry TanGitHubClaude Opus 4.7
 		ee6b11e563 v0.40.9.0 feat(chunker): .sql indexing via tree-sitter + code-def on SQL DDL (#1173) (#1350) 
* feat(chunker): vendor tree-sitter-sql.wasm + Step 0 grammar inspection tool

Vendored from DerekStride/tree-sitter-sql @ c2e1e08db1ea20dc23bdb8d228a81a8756e9c450,
built with tree-sitter-cli@v0.26.3 + --abi 14 (matches web-tree-sitter 0.22.6's
ABI 13-14 range; default --abi 15 was incompatible). 11 MB binary —
substantially larger than the plan's 400KB-1.4MB estimate (DerekStride's
multi-dialect grammar generates 40MB of parser.c).

tools/inspect-sql-grammar.ts is a one-shot Step 0 script that parsed
9 representative SQL fixtures and surfaced three load-bearing facts:

  1. Top-level node type is `program > statement > <kind>`. Every top-level
     node is `statement`, with the actual statement type as its single
     named child. TOP_LEVEL_TYPES['sql'] = new Set(['statement']) catch-all.
  2. The generic extractSymbolName returns null for EVERY SQL node — needs
     a SQL-specific branch that dives into statement.namedChild(0).
  3. DML emits one statement-chunk per statement (NOT one fat recursive-
     fallback chunk). $$ body parses cleanly. Even invalid SQL ("SELECT
     FROM WHERE") still produces a select-shaped statement, not a parse
     error.

Wave plan: ~/.claude/plans/system-instruction-you-are-working-tender-haven.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(chunker): wire SQL into language manifest + sync walker

Five additive edits to src/core/chunkers/code.ts:
  1. Import G_SQL grammar (DerekStride SHA in inline comment).
  2. Extend SupportedCodeLanguage union with 'sql'.
  3. Register sql entry in LANGUAGE_MANIFEST.
  4. Add .sql case to detectCodeLanguage.
  5. TOP_LEVEL_TYPES['sql'] = Set(['statement']) catch-all per Step 0
     finding that DerekStride wraps every top-level node in `statement`.

Two SQL-aware additions to existing helpers:
  - extractSymbolName: dives into `statement.namedChild(0)` and routes to
    extractSqlSymbolName. DDL kinds (create_table/function/view/index/
    procedure/type/schema/database/trigger + alter_table/view) extract
    target identifier via `name` field with fallback to identifier-shaped
    children. DML kinds (select/insert/update/delete/merge/with) return
    null so chunks emit unnamed.
  - normalizeSymbolType: adds 'table', 'view', 'index', 'procedure',
    'type', 'schema', 'database', 'trigger' branches so chunk headers say
    "table users" instead of "statement users".
  - emit-path passes inner-child type to normalizeSymbolType when the
    outer node is `statement` (SQL only condition).

sync.ts: add '.sql' to CODE_EXTENSIONS so isCodeFilePath routes it to
importCodeFile with page_kind='code'.

Manual verification (bun /tmp/test-sql-chunker2.ts) confirms CREATE TABLE,
CREATE FUNCTION (with $$ body), CREATE INDEX all produce chunks with
correct symbolName + symbolType. Small-sibling merging collapses
short-statement runs into single merged chunks (existing behavior, not
SQL-specific).

Wave plan: ~/.claude/plans/system-instruction-you-are-working-tender-haven.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(sql): unit + e2e + extend findCodeDef DEF_TYPES to cover SQL DDL

Unit tests (test/chunkers/code.test.ts, 8 new cases):
  - detectCodeLanguage now covers all 30 extensions (.sql added)
  - is-case-insensitive extended to .SQL
  - CREATE TABLE / FUNCTION / INDEX / VIEW / ALTER TABLE each extract
    target name into symbolName + map to correct symbolType
  - CREATE FUNCTION with $$ body parses without crashing
  - DML statements (INSERT) emit chunks but with symbolName=null
  - Mixed DDL+DML: per-statement emission, only DDL gets symbolName
  - Header includes "[SQL]" language tag
  - Invalid SQL ("SELECT FROM WHERE") doesn't crash the parser

Sync classifier (test/sync-classifier-widening.test.ts, 1 new case):
  - isCodeFilePath('migrations/001_init.sql') true, case-insensitive

E2E (test/e2e/code-indexing.test.ts, 7 new cases):
  - SQL import produces pages.type='code' + page_kind='code'
  - CREATE TABLE / FUNCTION chunks have correct symbol_name + symbol_type
  - findCodeDef returns CREATE TABLE / FUNCTION / INDEX / VIEW sites by
    name (load-bearing D2 canary — proves SQL is code intelligence,
    not just searchable text)
  - beforeAll timeout bumped to 30s (92-migration replay + 11MB SQL
    grammar load pushes past default 5s)

Source change to make E2E pass (src/commands/code-def.ts):
  - DEF_TYPES extended with 'table', 'view', 'index', 'procedure',
    'schema', 'database', 'trigger'. The chunker's normalizeSymbolType
    already maps create_table → 'table' etc; without this allowlist
    extension the chunks were indexed correctly but invisible to
    `gbrain code-def <name>`. This was the codex F2 missing-piece
    surfaced in /plan-eng-review (D6).

Wave plan: ~/.claude/plans/system-instruction-you-are-working-tender-haven.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.40.9.0 feat(chunker): .sql indexing via tree-sitter, code-def works on SQL DDL (#1173)

Closes #1173. gbrain sync now indexes .sql files; gbrain code-def returns
CREATE TABLE / FUNCTION / VIEW / INDEX / PROCEDURE / TYPE / SCHEMA /
DATABASE / TRIGGER + ALTER TABLE/VIEW sites by name.

Bumps: VERSION + package.json 0.40.8.0 → 0.40.9.0.
Updates: CLAUDE.md (37 grammars, SQL branch documented), llms-full.txt
regenerated. Full release notes in CHANGELOG.md including the 11 MB
binary-size disclosure and the 6 decisions (D1-D6) captured during
/plan-eng-review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(sql): fill remaining coverage gaps — TRIGGER/TYPE/PROCEDURE/SCHEMA + code-refs + idempotency + DML-only file

Unit tests (test/chunkers/code.test.ts, 7 new cases):
  - CREATE TRIGGER extracts name + symbolType=trigger
  - CREATE TYPE (enum) extracts name + symbolType=type
  - CREATE PROCEDURE extracts name + symbolType=procedure
  - CREATE SCHEMA (best-effort — grammar version dependent)
  - Header symbolType reflects inner DDL kind, never the bare 'statement' wrapper
  - Empty SQL input → empty chunk array
  - Whitespace-only SQL → empty chunk array

E2E tests (test/e2e/code-indexing.test.ts, 6 new cases):
  - findCodeRefs returns SQL chunks by substring match (validates the
    ILIKE-based ref path works on SQL with DDL + DML coverage)
  - CREATE TRIGGER + CREATE TYPE chunks land in content_chunks with
    correct symbol_type after import (engine-level regression)
  - findCodeDef on CREATE TYPE returns the chunk (DEF_TYPES allowlist
    regression pin: 'type' was added to DEF_TYPES in the prior commit)
  - findCodeDef on CREATE TRIGGER returns the chunk (DEF_TYPES regression
    pin: 'trigger' is in the allowlist)
  - DML-only file still produces a code page (just with zero
    symbol-named chunks — closes the question codex F14 raised)
  - Re-importing same SQL file is idempotent (content_hash short-circuit
    behaves the same on SQL as it does on TS/Python/Go)

All 63 SQL-related tests pass (chunker + sync classifier + E2E).
The pre-existing master flakes (check-system-of-record.sh, longmemeval
under shard concurrency) pass in isolation — not regressions from this
branch.

Wave plan: ~/.claude/plans/system-instruction-you-are-working-tender-haven.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): root-cause 4 master flakes — GBRAIN_SCAN_ROOT env + .slow rename + budget bumps

Four flakes surfaced during the v0.40.9.0 full unit sweep. All pass in
isolation; all fail under 8-shard parallel CPU contention. Fixes below
hit the actual root cause, not symptoms — no quarantine-and-ignore.

──────────────────────────────────────────────────────────────────────
1. check-system-of-record.sh — "catches violations in scripts/ alongside src/"
──────────────────────────────────────────────────────────────────────
Root cause: under shard load, the test's `spawnSync('git', ['init', '-q'])`
in /tmp/gate-test-* occasionally silently fails (filesystem contention),
so the fakeRepo has no .git dir. The gate then runs `git rev-parse
--show-toplevel` which walks UP past the fakeRepo into our real gbrain
repo, sets ROOT=/real/gbrain/repo, scans the clean real src/+scripts/,
exits 0. The test "expects exit 1 + 'naughty.ts' in stdout" sees exit 0
and empty stdout — fails.

Fix:
- scripts/check-system-of-record.sh: honor `GBRAIN_SCAN_ROOT` env var
  BEFORE the git-rev-parse fallback. Pure additive — production callers
  unchanged, tests get deterministic resolution.
- test/check-system-of-record.test.ts: `runGate` sets
  `GBRAIN_SCAN_ROOT: cwd` in spawnSync env. Closes the flake at the
  cause, not at the symptom (a retry loop would have papered over the
  real bug — the gate's resolution was too clever for its own good).

──────────────────────────────────────────────────────────────────────
2-4. eval-longmemeval.test.ts — 3 timeouts under 8-shard parallel
──────────────────────────────────────────────────────────────────────
Root cause: the file takes ~50s in isolation (full LongMemEval harness
replay with stubbed LLM). Under 8-shard parallel, CPU contention pushes
individual tests past bun's default 60s timeout. 3 tests timed out:
  - JSONL format guard (60s timeout)
  - JSONL key contract (65s timeout)
  - --by-type emits final by_type_summary (60s timeout)

Fix: rename `test/eval-longmemeval.test.ts` → `.slow.test.ts`. This is
exactly what the .slow taxonomy exists for per CLAUDE.md:
  > "*.slow.test.ts → intentional cold-path tests; would dominate the
  >  fast loop's wallclock"

Verified routing:
- Local `bun run test`: skips longmemeval (no flake)
- Local `bun run test:slow`: runs explicitly, 31 pass in 277s
- CI `scripts/test-shard.sh`: still runs (.slow NOT excluded from FNV
  bucketing — verified by dry-run: lands in shard 3/4)

──────────────────────────────────────────────────────────────────────
Adjacent fix: slow wrapper + test-shard.slow.test.ts beforeAll budget
──────────────────────────────────────────────────────────────────────
The longmemeval move surfaced a 4th flake: `test-shard.slow.test.ts`'s
beforeAll shells out 4×`scripts/test-shard.sh --dry-run-list` (~4s solo
each); when longmemeval is now running in the same slow-wrapper invocation
hogging CPU, the 4 sequential dry-runs slip past the 60s beforeAll
timeout.

Fixes:
- scripts/run-slow-tests.sh: bump bun test --timeout 60s → 120s. Slow
  tests are explicit by-name; a generous per-test budget is correct
  posture, not a workaround.
- test/scripts/test-shard.slow.test.ts: bump beforeAll budget 60s → 180s.
  Matches the actual workload under parallel slow-shard execution.

──────────────────────────────────────────────────────────────────────
Verification
──────────────────────────────────────────────────────────────────────
- `bun test test/check-system-of-record.test.ts` — 6 pass (in isolation)
- `bun run test:slow` — 31 pass in 277s (was: 1 fail at 89s before fixes)
- Full `bun run test` re-run in progress; will confirm 0 fail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): two more flake-hardening rounds — shard-aware perf gate + shard cap 600→900

Round 1 caught 4 named flakes; the post-fix sweep surfaced 2 more from the
same flake class (calibration values that were correct when set but are no
longer correct for the larger test suite).

5. longmemeval-trajectory-routing — "perf gate preserved" (3rd-party flake)

Failure: under shard load, test asserts elapsed<10s but real wallclock was
37s. The gate is supposed to catch real harness-layer regressions, not raw
cycle counts; 8-shard CPU contention routinely 3-5x's wallclock.

Fix: mode-aware ceiling. Solo run keeps the tight 10s gate (catches real
algorithmic regressions). Shard run (detected via `$SHARD` env set by the
parallel wrapper) loosens to 60s — still catches >6x regressions but
tolerates parallel contention. Per-test timeout bumped 5s default → 90s.

6. Per-shard wedge-detection too tight (false WEDGED markers)

Shards 5+6 of the prior sweep both got WEDGED markers at the 600s wrapper
cap, but their bun-internal timer shows they actually finished in 620-770s
with 0 failures. The 600s shard cap was calibrated when shards held ~600
tests; suite growth through v0.40.x pushed individual shards to 1100+
tests and 620-770s legitimate wallclock.

Fix: bump GBRAIN_TEST_SHARD_TIMEOUT default 600→900. Real hangs still hit
the 900s cap; fully-completed shards no longer false-kill at 600s. Env
override preserved.

──────────────────────────────────────────────────────────────────────
Cumulative flake hardening (across 2 commits)
──────────────────────────────────────────────────────────────────────
1. check-system-of-record gate — GBRAIN_SCAN_ROOT env override
2. eval-longmemeval (3 tests)   — rename to .slow
3. run-slow-tests.sh             — bump --timeout 60s → 120s
4. test-shard.slow.test.ts       — bump beforeAll 60s → 180s
5. longmemeval perf gate         — shard-mode-aware ceiling 10s/60s
6. Per-shard wedge cap           — bump 600s → 900s

All root-cause fixes; zero retry-loop / quarantine-and-ignore.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): clamp local default shard count 8 → 4 — kills PGLite contention SIGKILLs

Sweep #3 (after the prior 6 hardening fixes + master merge) caught a new
flake class: shard 5 got SIGKILL'd (rc=137) during source-health.test.ts's
92-migration PGLite replay. 8 parallel shards each running their own
PGLite WASM init + 92-migration replay contend severely on shared FS
state — even with the 900s shard cap, shard 5 wedged so hard the wrapper
fell back to SIGKILL.

Root cause: 8-shard parallel was aggressive (we picked detect_cpus on a
12-perf-core M-series, clamped to 8). CI runs 4 via test-shard.sh and is
stable. 8 → 4 trades ~2x local wallclock for reliability + matches CI
fan-out exactly. Override still available via --shards N or SHARDS=N
(clamped at 8 ceiling).

Side benefit: also resolves the 2 .serial.test.ts spawn failures in
sweep #3 — those serial tests run AFTER the parallel pass, so when the
parallel pass leaks PGLite write-locks under heavy contention, the
serial spawn tests inherit the polluted state and timeout on their
own subprocess spawns. Reducing parallel contention upstream cleans up
the FS state by the time serial runs.

──────────────────────────────────────────────────────────────────────
Cumulative flake hardening (3 commits, 7 fixes)
──────────────────────────────────────────────────────────────────────
1. check-system-of-record gate — GBRAIN_SCAN_ROOT env override
2. eval-longmemeval (3 tests)   — rename to .slow
3. run-slow-tests.sh             — bump --timeout 60s → 120s
4. test-shard.slow.test.ts       — bump beforeAll 60s → 180s
5. longmemeval perf gate         — shard-mode-aware ceiling 10s/60s
6. Per-shard wedge cap           — bump 600s → 900s
7. Default local shards          — clamp 8 → 4 (matches CI)

All root-cause fixes; zero quarantine-and-ignore.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): bump shard timeout 900→1500 — fixes 4-shard 968s overshoot

Sweep #4 at the new 4-shard default ran cleanly: 0 failures, 10072 pass.
BUT shard 1 was false-killed at 900s even though its internal completion
was 968s (the same flake pattern as the prior 600→900 bump, just at the
new shard sizing).

Reason: 8→4 shard reduction means each shard now runs 2x more files
(159 vs 80) and 2x more tests (~2420 vs ~1100). Internal wallclock per
shard climbed from 620-770s (8-shard) to 960-1020s (4-shard). The 900s
cap was sized for the prior 8-shard sizing; 4-shard sizing needs more
headroom. 1500s gives ~55% headroom over observed 4-shard wallclock and
catches real hangs that wouldn't complete in 1500s anyway.

──────────────────────────────────────────────────────────────────────
Cumulative flake hardening (4 commits, 8 fixes)
──────────────────────────────────────────────────────────────────────
1. check-system-of-record gate — GBRAIN_SCAN_ROOT env override
2. eval-longmemeval (3 tests)   — rename to .slow
3. run-slow-tests.sh             — bump --timeout 60s → 120s
4. test-shard.slow.test.ts       — bump beforeAll 60s → 180s
5. longmemeval perf gate         — shard-mode-aware ceiling 10s/60s
6. Per-shard wedge cap           — 600s → 900s → 1500s (8→4-shard recalibration)
7. Default local shards          — clamp 8 → 4 (matches CI)
8. (this commit)                 — calibrate cap for new shard sizing

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(test): CI flake — warm-create perf gate ceiling now mode-aware (1500ms solo / 4000ms loaded)

CI test_3 (Ubuntu, run #77585655194) failed on the
test/eval-longmemeval.slow.test.ts > 'warm-create speed gate' p50 assertion.
GHA Ubuntu runners are meaningfully slower than my Apple Silicon dev box
under parallel shard load — the 10-trial loop took 17364ms total which
puts per-trial p50 well above the 1500ms ceiling.

This is the same flake class as D5 in the local sweep hardening
(longmemeval-trajectory-routing perf gate). Apply the same shard-aware
ceiling pattern: 1500ms solo (catches real harness regressions),
4000ms when `$SHARD` (local parallel) OR `$CI` (GHA et al) is set.

Verified solo on Apple Silicon: p50=44ms (well under 1500ms tight gate).
Verified with `CI=true` env: p50=44ms (well under 4000ms loaded gate).
4000ms still catches >50x algorithmic regressions on a 25-44ms baseline.

──────────────────────────────────────────────────────────────────────
Cumulative flake hardening (5 commits, 9 fixes)
──────────────────────────────────────────────────────────────────────
1-8. (prior 4 commits)             — see PR comment #4527950030
9. (this commit) warm-create gate  — shard/CI-mode-aware ceiling

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-24 09:30:57 -07:00
Garry TanGitHubClaude Opus 4.7
 		d97f159793 v0.26.4 test: parallel unit-test loop (12x speedup, failure-first logging) (#605) 
* test: parallel unit-test wrapper + failure-first logging (commit 1/8)

Lay foundation for v0.26.4 parallel test loop:

- scripts/run-unit-parallel.sh: spawns N shards (default min(8, cpu_count))
  via run-unit-shard.sh, captures per-shard logs, post-shard single-writer
  failure-log aggregation at .context/test-failures.log, 10s heartbeat to
  stderr, per-shard 600s timeout (gtimeout/timeout/bg-pid fallback chain),
  loud final banner with absolute path + tail-30 of failures, summary file
  for at-a-glance status. Single writer eliminates concurrent-write hazards
  on the failure log.
- scripts/run-serial-tests.sh: discovers *.serial.test.ts files (concurrency-
  unsafe by design), runs them with --max-concurrency=1. Invoked after the
  parallel pass.
- scripts/run-unit-shard.sh: now accepts --max-concurrency=N (forwarded to
  bun test); --dry-run-list moved into argv parsing alongside; excludes
  *.serial.test.ts in addition to *.slow.test.ts.
- bunfig.toml: trim stale comment about typecheck-chained timeout.
- .gitignore: add .context/ (Conductor workspace artifacts directory; the
  failure log + summary + per-shard logs all live here).

No package.json changes yet (commit 2). No test reorganization yet
(commits 4-7).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: split package.json scripts; bun run test = parallel fast loop (commit 2/8)

Per Codex Tension #4 (verify scope), distinguish three tiers cleanly:

- `bun run test` = fast loop, file-level parallel fan-out via the new wrapper
  (scripts/run-unit-parallel.sh). No pre-checks, no typecheck, no wasm
  compile in the hot path. ~15s of pre-test gates removed.
- `bun run verify` = CI's authoritative gate set: check:jsonb +
  check:progress + check:wasm + typecheck. Matches what
  .github/workflows/test.yml runs on shard 1, no scope drift. The 4
  checks not in CI (privacy, no-legacy-getconnection, trailing-newline,
  exports-count) move to `bun run check:all` for opt-in local use.
- `bun run test:full` = verify + parallel + slow + smart e2e (runs e2e
  only if DATABASE_URL is set; else loud skip notice to stderr per Open
  Item #7). The local equivalent of "everything CI runs."

Adds `bun run test:serial` for the *.serial.test.ts subset (concurrency-
unsafe files run with --max-concurrency=1).

Bumps VERSION + package.json to 0.26.4. Both move together per the CI
version-gate contract in CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: fix-wave for parallel wrapper + tighten privacy gate (commit 3/5)

Wave: makes the new wrapper actually green and tightens the CI gate it
exposed.

Wrapper bug fixes (scripts/run-unit-parallel.sh):
- grep_count helper: avoids the `grep -c | echo 0` double-output bug
  where 0 matches yields a 2-line "0\n0" string and breaks arithmetic.
- bun_summary_count helper: parses Bun's actual end-of-shard summary
  format (`N pass` / `N fail` / `N skip`), not the per-test markers
  (which are `✓` / `(fail)`, never `(pass)` / `(skip)`).
- Heartbeat now reads `^\s+✓` (Bun's per-test pass marker) for live
  progress mid-run; final summary still uses the summary-line counts
  for accuracy.

Privacy gate tightening:
- Move scripts/check-privacy.sh into `bun run verify` (was previously
  only in the now-removed `bun run test` chain). Without this, after
  commit 2 the privacy check ran in nothing automatic.
- .github/workflows/test.yml now calls `bun run verify` instead of
  inlining the gate list. Single source of truth for "what's the ship
  gate." This is what verify == CI was supposed to mean per Codex T#4.
- Pre-existing `Wintermute` references in src/core/mounts-cache.ts:6
  and :324 caught by the now-running gate; replaced with `your OpenClaw`
  per CLAUDE.md privacy rule (verify gate now passes on master HEAD).
- test/privacy-script-wired.test.ts updated: regression guard now
  asserts verify includes check:privacy AND that test.yml runs
  `bun run verify`, replacing the obsolete "test script includes
  check-privacy.sh" assertion.

Quarantine 2 cross-file-contention flakes:
- test/brain-registry.test.ts: 28 tests pass alone (41ms); 1 test
  ("empty/null/undefined id routes to host") fails when run alongside
  other files in the same shard. Renamed → *.serial.test.ts so it
  runs in scripts/run-serial-tests.sh's serial pass after the parallel
  pass completes.
- test/reconcile-links.test.ts: 6 tests pass alone (1s); a beforeEach
  hook times out (~896s) under cross-file contention. Same treatment.

Both flakes are bun-process-level shared-state leaks (PGLite singletons
or top-level imports). Fixing them properly is the v0.27.0+ intra-file
parallelism project (TODO P0 — see commit 5).

Measurement after this commit:
  bun run test = 94s (was 18 min sequential)
  3639 pass, 0 fail, 0 skip across 8 parallel shards + 34 serial tests
  Failure-log + heartbeat + summary all working

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: regression tests for parallel wrapper + serial-test contracts (commit 4/5)

Three regression suites pin the v0.26.4 contracts. Without these,
future refactors of the wrapper or shard scripts could silently
regress the work in commits 1-3.

test/scripts/run-unit-shard.test.ts (4 cases — gap b):
- Asserts the unit-shard `--dry-run-list` output excludes every
  *.slow.test.ts and *.serial.test.ts file, plus the test/e2e/ subtree.
- Catches a future `find` expression that drops one of the `-not -name`
  clauses and silently un-quarantines slow/serial files into the
  parallel pass.

test/scripts/serial-files.test.ts (3 cases — gap e):
- Every checked-in *.serial.test.ts (via `git ls-files`) is listed by
  scripts/run-serial-tests.sh's `--dry-run-list`.
- The script's source contains `bun test --max-concurrency=1` (the
  serial-pass guarantee that quarantined files don't run intra-file
  concurrent and reintroduce the contention they were quarantined for).
- Disjoint set: a file is never in both the unit-shard list AND the
  serial list — pins the carve-out contract.

test/scripts/run-unit-parallel.test.ts (6 cases — gaps a + d):
- Exit-code propagation (a): wrapper exits non-zero when ANY shard
  has a failing test; exits zero when all pass. The hardest contract
  to silently break in a fan-out wrapper (`for ... &; wait` returns
  the LAST child's status, not any failure's).
- Failure-log contract (d): on failure, .context/test-failures.log
  exists, is non-empty, contains the `--- shard N:` prefix and the
  failing test's describe text. Stderr banner contains the absolute
  log path. On success, the log is cleared (no stale content).
- Summary file format: `shard N/M: pass=X fail=Y skip=Z rc=W` per
  shard, machine-parseable for future tooling.

The wrapper test runs against a 4-file tempdir (3 pass + 1 fail) so
it executes in ~500ms; spawning the wrapper against the real test
suite would take ~90s and isn't worth the cost in a regression suite.

All 13 cases pass on first run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(v0.26.4): testing tier docs + CHANGELOG + intra-file P0 TODO (commit 5/5)

Closes the v0.26.4 ship.

CLAUDE.md Testing section rewritten:
- New tier table: test (fast loop, 85s) / verify (CI gates, 12s) /
  test:full (everything local) / test:slow / test:serial / test:e2e /
  check:all. Each row names its scope, wallclock, and when to use.
- Intentional CI vs local divergence section: CI matrix (test-shard.sh,
  hash-bucketed, includes slow) vs local fast loop (run-unit-shard.sh,
  round-robin, excludes slow + serial). Codex correctly flagged that a
  parity test would always fail by design — this is the documentation
  that explains why.
- Failure-first logging contract: .context/test-failures.log format,
  stderr banner, summary file, wedge handling.
- File taxonomy: *.test.ts / *.slow.test.ts / *.serial.test.ts /
  test/e2e/. Names the two currently-quarantined files and points at the
  intra-file P0 TODO for the proper fix.

CHANGELOG.md `## [0.26.4]` entry per voice rules:
- Two-line headline: "bun run test finishes in 85 seconds. Was 18
  minutes." + failure-log directive.
- Lead paragraph names what shipped and why.
- Numbers-that-matter table: BEFORE / AFTER / Δ for wallclock, pre-test
  gates, failure visibility, shards, pipe-survival.
- "What this means for you" closing tied to the inner-loop user.
- "To take advantage of v0.26.4" block per the v0.13+ self-repair
  template (gbrain upgrade + contributor steps).
- Itemized changes by area (new scripts, script extensions, package.json
  tier split, CI tightening, failure-first logging, quarantine, regression
  tests, bunfig).
- "What did NOT ship" section names the intra-file project + E2E
  template-DB project as P0/P1 follow-ups with concrete acceptance
  criteria.
- Process section names the codex review + scope-correction loop
  honestly: "snapped back to ship today once empirical measurement showed
  Bun's --max-concurrency does nothing on tests not marked
  test.concurrent()."
- For-contributors note on portability + single-writer + fallback paths.

TODOS.md adds two P-rated entries:
- P0: intra-file parallelism via --concurrent flag. Sweep ~58 PGLite
  sites + ~40 env mutations + 2 mock.module sites. Target: bun run test
  < 30s. ~1-2 weeks. Detailed acceptance criteria. References Codex
  findings and plan-file rationale.
- P1: E2E parallelism via Postgres template databases. CREATE DATABASE
  TEMPLATE gbrain_template per test file. ~1-2 days.

llms.txt + llms-full.txt regenerated via `bun run build:llms` to absorb
the CLAUDE.md changes (per CLAUDE.md's "After any release ship that
touches the Key Files annotations in CLAUDE.md, run bun run build:llms"
rule). The build-llms regression test was firing in shard 7 of the
parallel pass — caught the drift, regeneration cleared it. Final
measurement after fix: 94s wallclock, 3652 pass, 0 fail across 8
parallel shards + 34 serial tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 20:16:15 -07:00