2 Commits

Include renames

Author	SHA1	Message	Date
3 people	8ab733471b	v0.41.17.0 feat: --workers N on every bulk command + facts dim doctor parity (#1519) ... * feat(worker-pool): shared sliding pool + bounded semaphore + PGLite-clamp wrapper T1 + T2 of the v0.41.16.0 workers cathedral. New src/core/worker-pool.ts is the canonical primitive every --workers N bulk command in this wave (and future bulk commands) builds on. Atomic-claim invariant enforced by scripts/check-worker-pool-atomicity.sh (wired into bun run verify). BudgetExhausted bypass + AbortSignal composition baked into the helper so budget caps are a structural ceiling under concurrency, not a per-caller convention. The new resolveWorkersWithClamp wrapper composes existing autoConcurrency with PGLite-clamp + per-(command, requested) stderr dedup. Deliberately NOT a modification to shared autoConcurrency (silent today, used by sync + import); embed.ts keeps GBRAIN_EMBED_CONCURRENCY || 20 default per codex #13. 23 + 12 + 9 = 44 hermetic tests pin every contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: structural + dim-check regression suites for v0.41.16.0 wave - test/embed-helper-migration.test.ts (T3): asserts embed.ts's two sliding-pool sites are migrated to runSlidingPool, pre-migration shapes (let nextIdx = 0, Promise.all(Array.from(...))) are gone, GBRAIN_EMBED_CONCURRENCY || 20 default preserved, failureLabel threads page.slug. Per codex #16/#17 these are invariant assertions, not byte-equality on progress event ORDERING. - test/embedding-dim-check-facts.test.ts (T6): readFactsEmbeddingDim covers vector(N) + halfvec(N), halfvec-before-vector regex ordering pinned (codex #19), buildFactsAlterRecipe emits DROP INDEX + ALTER USING + CREATE INDEX (codex #18, not bare REINDEX), FactsEmbeddingDimMismatchError tagged class shape, assertFactsEmbeddingDimMatchesConfig PGLite skip + Postgres absent- column skip, doctor check + insert-cast wiring assertions. - test/extract-conversation-facts-workers.test.ts (T5): helper exports (extractConversationFactsLockId, PER_PAGE_LOCK_TTL_MINUTES), structural wiring (runSlidingPool, resolveWorkersWithClamp, withRefreshingLock, LockUnavailableError, delete-orphans-first before segment loop, preflight before pool, exit 3 when lock_skipped > 0), Minion handler round-trip. - test/extract-workers.test.ts (T7): --workers wiring on all 3 inner fs-walk loops (extractForSlugs, extractLinksFromDir, extractTimelineFromDir) + CLI parse + opts threading through runExtractCore. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.41.16.0 → v0.41.17.0 (queue collision with PR #1510) PR #1510 (garrytan/dynamic-regex-conversation-formats) claimed v0.41.16.0 on master in parallel. Advancing this wave to v0.41.17.0 so both can land cleanly. Pure mechanical version bump: - VERSION + package.json → 0.41.17.0 - CHANGELOG.md header + "To take advantage of v0.41.17.0" block - TODOS.md section header + v0.41.18+ forward references - CLAUDE.md inline version tags - Regenerated llms-full.txt / llms.txt No code changes. The actual workers cathedral feature set is unchanged from the two prior commits in this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): search-image-column probes column dim at runtime CI shard 5 failed on `searchVector column routing (v0.27.1)` with: error: expected 1280 dimensions, not 1536 The test had a hardcoded `fakeText1536` helper that seeded chunks at 1536-d vectors. Master's default embedding model switched from OpenAI text-embedding-3-large (1536) to ZeroEntropy zembed-1 (1280) so a fresh PGLite brain on CI now sizes content_chunks.embedding at 1280; the test's 1536-d INSERT trips pgvector's CheckExpectedDim. Fix: probe `content_chunks.embedding` width via `readContentChunksEmbeddingDim(engine)` in `beforeAll`, store in `TEXT_DIM`, and build `fakeTextDefault(seed)` at that width. The test now passes regardless of which default ships (the model has flipped twice and may flip again). Local dev (1536 from older config) and CI fresh-install (1280 from new default) both pass. Image-side vectors stay at 1024 (matches Voyage multimodal-3 + the column's fixed width on the image side). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): bump PGLite hook timeout for shard-4 deep-process files facts-anti-loop.test.ts and ingest-capture.test.ts were timing out in CI shard 4 with "beforeEach/afterEach hook timed out" after the v0.41.16.0 master merge brought migration count to 99. When these files run deep in a shard process that has already created ~20 PGLite engines, the WASM cold-start + 95-migration replay legitimately exceeds bun's 5s default hook timeout (observed 5.6s and 7.3s locally when reproducing). Bun's --timeout=60000 from scripts/test-shard.sh covers TEST timeouts but NOT hook timeouts; those default to 5s and must be set per-hook via the optional 2nd arg to beforeAll/afterAll. Reproduced locally by running the first 21 shard-4 files via head -21 /tmp/shard4-list.txt | xargs bun test → 179 pass, 2 fail (both with hook-timeout error) After fix: → 198 pass, 0 fail (the 4 anti-loop + 15 ingest-capture tests recover) Full shard 4 with fix: 955 pass, 0 fail. Full shard 5 with fix: 1261 pass, 0 fail. Also added a defensive diagnostic to the two put_page tests: if facts_backstop is missing in the response payload, throw with the full payload + isError so future failures surface the actual handler error instead of a bare "expected {...} got undefined" assertion. No-op when the test passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-26 18:29:03 -07:00
3 people	94aaf7e396	v0.40.1.0 Track D — eval infrastructure (catch retrieval regressions, prove answer-quality wins) (#1298) ... * feat(eval-longmemeval): --by-type flag + question field + resume-replace Per-question JSONL row gains `question`, `question_type`, and (when ground truth is available) `recall_hit` — additive fields that existing consumers (LongMemEval's `evaluate_qa.py`) ignore. New `--by-type` flag emits a `{kind:"by_type_summary", recall_by_type, aggregate}` line at the end of the output, resume-safe: rebuilt from existing rows so the final aggregate covers cumulative resumed questions, prior summary at the tail replaced rather than appended. New `--by-type-floor F` exits non-zero per breached question_type. Empty-bucket guard emits null rate not NaN. Exports `buildByTypeSummary` + `emitByTypeSummary` + `seedRecallByTypeFromFile` for unit testing. * feat(eval-cross-modal): --batch flag + semaphore + DI seam Adds `--batch <jsonl> [--limit N] [--concurrent N] [--max-usd FLOAT] [--yes]` to the existing eval cross-modal command. Mutually exclusive with --task. Reads LongMemEval-shape JSONL output, filters by_type_summary rows automatically, fans out via a new `runWithLimit<T>` semaphore primitive (default --concurrent 3 x 3 model slots = 9 simultaneous calls; below tier-1 rate limits on all 3 providers). Pre-flight cost estimate refuses past --max-usd (default $5) unless --yes. Per-question receipts written to a per-batch tempdir + deleted at end of run so ~/.gbrain/eval-receipts/ stays clean; summary receipt inlines verdicts. Exit precedence (new batch-level policy, not inherited from aggregate.ts): ERROR > FAIL > INCONCLUSIVE > PASS — any per-question runtime error exits 2. New `runEvalCrossModal(args, opts?: {runEval?})` DI seam mirrors the existing eval-longmemeval pattern. Tests pass a stub runEval so unit tests don't need API keys; gateway availability check is also skipped when opts.runEval is provided. Pinned by 17 cases. * test: hermetic qrels retrieval gate against synthetic basis-vector corpus Adds test/eval-replay-gate.test.ts as a unit-shard test (NOT under test/e2e/ — the unit-shard CI matrix runs every PR via bun test; test/e2e/ is fixed-file). Seeds a PGLite engine with synthetic placeholder-name pages whose embeddings are basis vectors (same pattern as test/e2e/search-quality.test.ts:23-28) so retrieval is hermetic — no API keys, no DATABASE_URL, fully deterministic. The qrels fixture at test/fixtures/eval-baselines/qrels-search.json has 12 hand-curated queries; each maps to a ranked list of relevant slugs + `first_relevant_slug` (expected top-1). For each query, the gate asserts `top1_match_rate >= 0.80` AND `recall_at_10 >= 0.85`. Env-overridable floors via GBRAIN_REPLAY_GATE_TOP1_FLOOR / GBRAIN_REPLAY_GATE_RECALL_FLOOR through withEnv(). Gate-fire prints per-query HIT/miss + recall to stderr. When ranking changes intentionally move expected slugs, edit qrels-search.json directly with a 'Why:' line in the commit body — documented in docs/eval-bench.md. scripts/check-test-real-names.sh allowlist gains 6 entries for the privacy-grep regression guard inside the test, which must literally spell the names it forbids to assert they're NOT in the fixture (same meta-rule exception as skillpack-harvest privacy tests). * feat(autopilot): opt-in nightly cross-modal quality probe + doctor check Composes `gbrain eval longmemeval --by-type` + `gbrain eval cross-modal --batch` into a 24h-cadenced quality check. Default DISABLED — opt-in via `gbrain config set autopilot.nightly_quality_probe.enabled true` so new users don't discover background API spend. src/core/cycle/nightly-quality-probe.ts ships the phase implementation with a full NightlyProbeDeps DI surface (isEnabled, hasEmbeddingProvider, resolveMaxUsd, resolveRepoRoot, runLongMemEval, runCrossModalBatch, now) so tests stub every external effect — no PGLite, no real LLM calls. Pure `shouldRunNightly(now, recentEvents, windowMs?)` rate-limit fn. src/core/audit-quality-probe.ts is the ISO-week-rotated JSONL writer (mirrors audit-slug-fallback.ts; honors GBRAIN_AUDIT_DIR). One event per run: outcome (pass/fail/inconclusive/error/budget_exceeded/rate_limited/ no_embedding_key), exit code, pass/fail/error counts, est_cost_usd, fixture_sha8. src/commands/doctor.ts gains a `nightly_quality_probe_health` check: SKIPPED with paste-ready enable command when disabled; OK with timestamp when all PASS in last 7 days; WARN with per-outcome counts when any FAIL/ERROR/BUDGET_EXCEEDED. Extracted as pure `computeNightlyQualityProbeHealthCheck(probeEnabled, events)` for unit testing. test/fixtures/longmemeval-nightly.jsonl is a 10-question placeholder dataset (synthetic names only) distinct from the existing 5-question mini fixture so the probe has consistent regression signal. Real expected cost: ~$0.35/night = ~$10.50/month. Worst-case at default $5 cap: $150/month. Pinned by 21 cases in test/nightly-quality-probe.test.ts covering the rate-limit pure function, every outcome branch, and all 7 branches of the doctor check. Autopilot scheduler wiring deferred to v0.41+ — the phase is callable in isolation today (via the DI surface); cycle-loop dispatcher integration filed in TODOS.md as a follow-up. * docs: document Track D eval surfaces + file v0.41+ follow-up TODOs docs/eval-bench.md gains a 'v0.40.1.0 Track D — Eval infrastructure' section covering: --by-type usage + resume-replace semantics, the hermetic qrels gate workflow + 'Why:' commit-body refresh convention, --batch end-to-end with cost-bound + concurrency knobs, and the opt-in nightly probe enable workflow + cost ceiling. TODOS.md files two follow-ups: - v0.41+: contributor-mode CI capture for BrainBench-Real replay gate (the deferred original Task 2 design — replay against real captured queries is more valuable than synthetic qrels long-term, but needs CI secret + nightly capture pipeline + commit automation; deferred to a dedicated wave) - v0.41+: wire the nightly quality probe into autopilot scheduling (phase callable in isolation today; cycle-loop dispatcher integration is a ~3-hour follow-up) CLAUDE.md Key Files annotations extended for the four lanes: eval-longmemeval gains the --by-type description, eval-cross-modal gains the --batch + DI seam description, new entries for the qrels gate test + the nightly probe + audit-quality-probe writer. * chore: bump version and changelog (v0.40.1.0) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(eval): close 4 codex-flagged eval-integrity bugs Codex adversarial review on the Track D wave found 4 real ways the new eval-gate code could silently bypass its gates. Each fix below either counts what was previously dropped, fails fast on a parser edge case, or enforces a gate that was previously skipped on an early-return path. CDX-1: cross-modal --batch silently dropped failed/corrupt LongMemEval rows. `gbrain eval longmemeval` emits {error:..., hypothesis:''} when runOneQuestion throws; the batch reader's missing-field skip threw those rows away, shrinking the denominator. A green eval on a subset is now impossible: - eval-longmemeval.ts: error rows now carry `question` + `question_type` so the batch consumer can identify them as upstream failures, not skip them as malformed. - eval-cross-modal.ts: readBatchRows now returns {rows, upstream_errors, malformed_count}. Upstream errors fold into per_question with verdict 'upstream_error'. BatchSummary gains `upstream_error_count` and `malformed_count`. ERROR exit precedence widens to include both, so any upstream failure exits 2. CDX-2: --limit 0 was a direct CI bypass — zero-row check fired before slicing, then the empty result fell through to verdict='pass'. Fixed with a hard `limit >= 1` check. CDX-3: --resume-from + --by-type-floor was a real gate skip. When a prior run had every question answered, the early "nothing to do" return fired BEFORE summary emission and floor enforcement. Now the no-op resume path still seeds recallByType from the existing file, emits the by_type_summary at the tail, and runs the floor gate. CDX-5: doctor nightly_quality_probe_health only flagged fail / error / budget_exceeded as warn. no_embedding_key / rate_limited / inconclusive were silently reported as PASS — hiding misconfigurations and queue backpressure. The bad-event filter is now `outcome !== 'pass'`, and the counts string surfaces every bucket so the operator sees exactly what went wrong. scripts/check-privacy.sh: adds test/eval-replay-gate.test.ts to the allowlist (the qrels test's privacy-grep regression guard literally names what it forbids, same meta-rule exception as the existing test/recency-decay.test.ts + skillpack-harvest allowlist entries). Pinned by 8 new regression cases across eval-longmemeval (CDX-3), eval-cross-modal-batch (CDX-1 + CDX-2), and nightly-quality-probe (CDX-5). 76 Track D tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-22 22:16:25 -07:00