mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(skillopt): foundation modules — types, lr-schedule, benchmark, score, audit, lock
* feat(skillopt): edit primitives — apply-edits (D5+D9), rejected-buffer LRU, version-store (D8 history-intent-first)
* feat(skillopt): rollout (D2 gateway.toolLoop + D13 read-only allowlist), reflect (D7 two calls), validate-gate (D12 median+epsilon, D4 parallel), preflight (D3), bundled-skill-gate (D16)
* feat(skillopt): orchestrator (D6 slow-update, D10 ASCII diagrams, D11 caching), checkpoint, bootstrap (D15 sentinel), CLI dispatch + help
* feat(skillopt): cycle phase (F1 dream-loop wiring), PROTECTED_JOB_NAMES + MCP op (F6 admin scope + allowlist) + Minion handler (F7 --background)
* feat(skillopt): full cathedral — --all batch (F4), --target-models fleet (F5), write-capture (F10), held-out scaffold (F11), adversarial suite 41 cases (F2), E2E PGLite (F3), meta-skill bundle (T7), reflect+judge evals (F8+F9), docs (T10)
* chore: bump version to v0.42.0.0 (MINOR — significant new feature)
* fix(skillopt): wire trajectories from forward gate to reflect + fix parseEditsResponse parser misuse
Two related v0.42.0.0 bugs that conspired to make `runSkillOpt` structurally
unable to accept any candidate edit. Either alone would have killed self-evolution;
together they made the loop a no-op for every input.
**Bug 1 (orchestrator gap):** `runOptimizationLoop` in orchestrator.ts called
`runReflect({successes: [], failures: []})` with hardcoded empty arrays. The
forward gate's `scoredRollouts` were computed then voided. `runReflect`
short-circuits both modes when their batches are empty, so the optimizer was
never asked to propose an edit. Every step hit the no_edits_applied branch.
Fix: add `scoredRollouts: ScoredRollout[]` to `GateResult` and
`runsPerTask?: number` to `ValidateGateOpts`. Forward pass uses
`runsPerTask: 1`; orchestrator partitions returned rollouts by `score >= 0.5`
and threads real successes + failures into `runReflect`.
**Bug 2 (parser misuse):** `parseEditsResponse` in reflect.ts routed every
optimizer response through `parseJudgeJson` first. `parseJudgeJson` looks for
a `score` key (it's a judge-output parser, not an edits parser) and returns
null for any JSON without one — including the well-formed `{"edits": [...]}`
the optimizer is contractually required to emit. The function then early-
returned `[]` and the actual `tryExtractEdits` path on the next line was
unreachable dead code.
Fix: drop the wrong-typed guard. `parseEditsResponse` now calls
`tryExtractEdits` directly. Export it so `reflect.test.ts` can pin the
contract independently of the chat transport.
**Why this slipped through 152 prior skillopt tests:** zero unit coverage
of `parseEditsResponse` or `runReflect`. The existing E2E `all-reject` case
asserted no_improvement (which was true for the wrong reason — empty edits,
not gate rejection). Both bugs were structurally invisible to the existing
test surface.
**New coverage:**
- `test/skillopt/reflect.test.ts` (15 cases):
- 8 `parseEditsResponse` cases including the IRON-RULE regression pin
for the v0.42.0.1 fix (`{"edits": [...]}` JSON must survive the parser).
- 7 `runReflect` D7 contract cases: both modes fire, empty-batch skips,
additive token usage, one-mode-throws-other-still-works, rejected-buffer
flows into anti-bias prompt.
- Documents the trailing-comma limitation as an explicit out-of-scope pin
(so a future tightening of `tryExtractEdits` lights this test up
intentionally).
- `test/e2e/skillopt-loop.serial.test.ts` (7 cases):
- HAPPY PATH: stubbed `gateway.chat` acts as both target agent (emits
sections based on skill content) and optimizer (proposes a real
add-Citations edit). Drives `runSkillOpt` end-to-end against PGLite.
Asserts outcome=accepted, SKILL.md mutated with new section,
frontmatter preserved (D5), history has one committed row,
best.md mirrors disk, delta > epsilon, receipt fields populated.
- 5 broken cases (each isolates a distinct orchestrator-visible failure):
1. Below-baseline regression: optimizer proposes a destructive edit;
gate rejects with reason=below_baseline; SKILL.md unchanged;
rejected-buffer captures the bad edit for anti-bias context.
2. Malformed reflect JSON: orchestrator degrades gracefully to
no_improvement without crashing.
3. Anchor-not-found: applyEditBatch rejects all; sel gate skipped;
rejected-buffer captures with reason=apply_failed.
4. Budget exhausted mid-step: outcome=aborted, no pending rows survive.
5. Converged-skill re-run: starting from already-perfect skill →
no_improvement (no thrash on a well-tuned starting point).
- IDEMPOTENT RE-RUN: drive runSkillOpt twice in sequence. Run 1 accepts.
Run 2 sees improved baseline, no failures, returns no_improvement.
SKILL.md byte-identical to post-run-1; history still has exactly 1
committed row. Proves stability at the fixed point.
All hermetic (no DATABASE_URL, no API keys). PGLite in-memory engine,
tempdir SKILL.md + benchmark, stubbed gateway.chat via
`__setChatTransportForTests`. `.serial.test.ts` because the stub installs
module state and the loop walks shared disk state across epochs.
Test counts after fix: 174 skillopt-surface tests pass (149 pre-existing
unit + 15 new reflect unit + 3 existing E2E + 7 new E2E). Typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cycle): align ALL_PHASES skillopt position with actual dispatch order
v0.42.0.0 added skillopt to ALL_PHASES right after `patterns` (line 127), but
the dispatch block in runCycle (line ~1912) actually runs skillopt between
`conversation_facts_backfill` and `embed`. The two were inconsistent, and the
serial test `report.phases.map(p => p.phase)).toEqual(ALL_PHASES)` was failing
on master because of it.
A second pre-existing failure: the two phase-count assertions in
`test/core/cycle.serial.test.ts` still said `toBe(20)` even though
ALL_PHASES grew to 21 when skillopt was added. The author bumped the array
but forgot the test.
Two fixes, one commit:
1. Move `'skillopt'` in ALL_PHASES from after `patterns` to between
`conversation_facts_backfill` and `embed`, matching where runCycle
actually dispatches it. Runtime behavior is unchanged — only the
declaration order moves. Updated the surrounding comment to call out
the position invariant and reference the test that pins it.
2. Update both `toBe(20)` assertions in cycle.serial.test.ts to `toBe(21)`
with a v0.42.0.0 history line in the running comments.
Why declaration follows runtime (not the other way around): the comment
intent ("Runs AFTER patterns — graph-fresh") is still satisfied because
"after the entire main graph-mutating cluster" is strictly fresher than
"right after patterns". No design intent is lost.
Test result: cycle.serial.test.ts is now 28/28 (was 27/28 on master + my
prior commit). Skillopt suite still 174/174.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): bump PHASE_SCOPE assertion to 21 + fix skill-optimizer Anti-Patterns case
Two CI failures pre-existing on this branch since the v0.42.0.0 skillopt
cathedral landed; master is green because skillopt didn't exist there yet.
1. test/phase-scope-coverage.test.ts asserted ALL_PHASES.length === 20.
skillopt is the 21st phase. Bumped to 21 with v0.42.0.0 history line
in the comment chain. Sibling fix to the cycle.serial.test.ts bump
in commit 08ad2468.
2. skills/skill-optimizer/SKILL.md had `## Anti-patterns` (lowercase p).
skills-conformance.test.ts asserts `## Anti-Patterns` (capital P) as
the required section header. Single-character rename.
Local: 174 skillopt-surface tests + 6 phase-scope tests + 249 skills-
conformance tests all green. Typecheck clean.
Remaining CI delta: 5 put_page facts backstop failures in shard 10 that
reproduce only on Linux CI, not locally even with empty env / cleared
HOME / max-concurrency=1. The error surface is `r.isError === true` with
no further detail captured in the bun:test output. Pushing these 2 fixes
first to narrow the CI signal; will instrument if the 5 persist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(e2e): align dream-cycle-phase-order + onboard-full-flow with v0.41/v0.42 reality
Two stale E2E assertion files surfaced by a full local E2E run against
real Postgres (the gbrain-test-pg container on port 5434). Neither file
is in the CI E2E job (CI only runs mechanical.test.ts + mcp.test.ts +
skills.test.ts + zeroentropy-live.test.ts), so the drift has been latent.
1. `test/e2e/dream-cycle-phase-order-pglite.test.ts`
EXPECTED_PHASES was missing 4 phases that landed in master since the
list was last revised:
- extract_atoms (v0.41 T9 — atom extraction, after extract_facts)
- synthesize_concepts (v0.41 T9 — concept synthesis, after patterns)
- conversation_facts_backfill (v0.41.11.0, after calibration_profile)
- skillopt (v0.42.0.0 — self-evolving skills, between
conversation_facts_backfill and embed)
Updated to 21 entries in the actual runtime dispatch order (matches
ALL_PHASES exactly). 5/5 tests in the file pass after.
2. `test/e2e/onboard-full-flow.test.ts`
`runAllOnboardChecks` shape test asserted exactly 4 checks; v0.42's
type-unification cathedral (PR #1542, T13-T15) added 3 more
(`pack_upgrade_available`, `type_proliferation`, `dangling_aliases`)
for a total of 7. And `empty brain returns 0 remediations` regressed
because `pack_upgrade_available` can emit a manual_only remediation
on brains where gbrain-base@1.x is active and gbrain-base-v2 is
registered as a successor. Tightened that assertion to `total <= 1`
AND kept a per-check guard asserting takes_count remediations stay 0
(the original test's load-bearing claim — A12 two-gate consent).
13/13 tests in the file pass after.
Honest scope: 4 other E2E files still fail locally after this commit
(cycle.test.ts, dream.test.ts, phantom-redirect.test.ts,
sync-lock-recovery.test.ts), each for a distinct pre-existing master
bug unrelated to v0.42 skillopt work:
- cycle.test.ts (5 fails): PostgresEngine.getConfig falls back to
db.getConnection() singleton via the `get sql()` getter when no
poolSize is set; the new conversation_facts_backfill phase chain
hits this fallback even though the test's setupDB() connects both
the singleton AND the engine. Race condition between the test's
singleton lifecycle and the phase's getConfig call. Deeper fix
needed in PostgresEngine.getConfig (use this._sql directly with
explicit fallback only on user-driven CLI paths).
- dream.test.ts (1 fail): expects "concepts/testing" slug to appear
in dream cycle output, gets empty array. Related to v0.42 concept
type-unification semantics.
- phantom-redirect.test.ts (2 fails): concurrent-sync race +
postgres-js text-string embedding survival. Master-level data-path
bug; would need its own fix wave.
- sync-lock-recovery.test.ts (1 fail): `gbrain sync --break-lock
--all` exits 0 but test expects 1 with a shell-loop hint. CLI
behavior changed in a master commit; need to either restore the
refusal behavior or update the assertion.
None of these 4 block CI (E2E job doesn't run them). Filed as a
TODOS.md entry for a follow-up wave; the 2 in this commit are the
ones that mirror v0.42 work landing.
Local: 130/136 E2E files green, 927/940 tests pass (was 925/940
before these fixes; the 2 files this commit fixes added 7 newly-
passing tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): quarantine query-cache-knobs-hash.test.ts to serial runner
CI shard 10 (commit 4d721077) failed 5 tests in the
`SemanticQueryCache cross-mode isolation (CDX-4 hotfix)` describe block,
all ~7-34ms each, all expecting writes/reads to round-trip through one
shared PGLite engine + a `beforeEach DELETE FROM query_cache`. Passes
9/9 locally; fails 5/9 on Linux CI under bun's default in-file
max-concurrency=4.
Classic intra-file concurrency race shape: test A's `beforeEach`
clears the table → test A's `store` writes a row → test B's
`beforeEach` (concurrent with A's `store`) clears the table → test A's
follow-up COUNT query returns 0. Same root cause that quarantined
`embed-stale.test.ts`, `brain-allowlist.test.ts`, and
`schema-pack-find-pack-successors.test.ts` to the serial runner in
prior fix waves (documented in v0.41.22.0 CI fix wave).
Fix: rename to `query-cache-knobs-hash.serial.test.ts` so the v0.26.7
serial-tests runner picks it up at `max-concurrency=1`. Tests still
exercise the actual cache logic — no test deleted, no production code
changed. The describe block's `beforeAll` engine + `beforeEach`
TRUNCATE pattern works correctly at serial concurrency.
Local: 12/12 in this file + 52/52 in the serial runner. Production
SemanticQueryCache code is untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(heavy): frontmatter_scan_wallclock — opt into --no-embedding so CI runners work
Heavy tests workflow run 26542447602 (commit 483a5577) failed on the
first heavy script:
[fm_wallclock] FAIL: gbrain init exited non-zero
No embedding provider configured. Set one of:
OPENAI_API_KEY / ZEROENTROPY_API_KEY / VOYAGE_API_KEY
Or defer setup: gbrain init --pglite --no-embedding
The v0.37 D9 hard-require landed in init.ts: `gbrain init --pglite` now
refuses to proceed without an embedding provider configured. The
heavy-tests GitHub workflow doesn't pipe any embedding API keys
(deliberate — the heavy tests measure ops shape, not LLM behavior), so
every CI invocation now blocks at step 2 of this script.
The script's whole purpose is measuring `gbrain doctor`'s
frontmatter-scan wallclock — it never embeds, never calls
`gbrain embed`, never queries vectors. The right fix is to opt out of
the provider requirement via the same `--no-embedding` flag init.ts
already exposes for this exact "deferred setup" case.
Verified locally:
TMP=$(mktemp -d); GBRAIN_HOME="$TMP" \
bun run src/cli.ts init --pglite --yes --no-embedding
# exit 0, brain initialized.
No production code change. One-line + comment in the script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(heavy): sync_lock_regression — pass --no-embed so CI runs measure lock contention, not key absence
Heavy tests workflow run 26542545802 (commit 7962d312, after the
previous fm_wallclock fix) failed at the next heavy script in the chain:
[sync_lock_regression] outcomes: winners=0 losers=0 unknown=4
[sync_lock_regression] FAIL: expected 1 winner, got 0
[sync_lock_regression] FAIL: expected 3 lock-busy losers, got 0
Each of the 4 parallel `gbrain sync` invocations failed for the same
reason — none of them ever even got to the lock-acquire step:
Embedding model "zeroentropyai:zembed-1" requires ZEROENTROPY_API_KEY.
Re-run with --no-embed to import-only and embed later once the key is set.
The CI runner doesn't pipe any embedding-provider API keys (deliberate —
heavy tests measure ops shape, not LLM behavior), and sync now hard-fails
when its embed step can't reach a configured provider.
This script measures the writer-lock race shape — `gbrain-sync` row in
`gbrain_cycle_locks`, exactly-one-winner semantics, N-1 fail-fast losers
with "Another sync is in progress", zero leaked rows post-run. It never
needed embeddings; the original write predates the hard-require landing.
Fix: pass `--no-embed` to the sync invocation. Same kind of fix as
fm_wallclock (commit 7962d312) but on the sync side rather than init.
No production code touched. One-line change in the bash script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(heavy): sync_lock_regression — register source via psql + use --repo + tolerate doctor warns
Heavy tests run 26542638471 (commit 60145eee, after the --no-embed
fix) failed at the same script but at a downstream step:
> Source "default" has no local_path. Run: gbrain sources add default --path <path>
Three independent bugs in the script that all surfaced at once after
v0.41's source-registry landed:
1. `gbrain config set sync.repo_path` is the legacy way; sync now
reads `sources.local_path` first. Replaced with an upsert into the
sources table via psql:
INSERT INTO sources (id, name, local_path)
VALUES ('default', 'default', $BRAIN_DIR)
ON CONFLICT (id) DO UPDATE SET local_path = EXCLUDED.local_path
Kept the legacy `config set sync.repo_path` line too as
belt-and-suspenders for any downstream caller that still reads it.
2. `gbrain sync --dir <path>` is silently ignored; sync's CLI parser
recognizes `--repo`, not `--dir`. Switched to `--repo`.
3. `bun run src/cli.ts doctor --json` at the top (used to apply
migrations as a side effect) exits non-zero whenever ANY check
warns — including the new "no embedding provider configured"
warning on a fresh CI runner. The script's `set -e` aborted at
line 53 before reaching any of the sync invocations. Added `|| true`
since the migration runs regardless of doctor's exit verdict.
Verified locally — `DATABASE_URL=... bash tests/heavy/sync_lock_regression.sh`
output:
[sync 1] rc= (lock-busy: 'Another sync is in progress')
[sync 2] rc=0 (winner)
[sync 3] rc= (lock-busy: 'Another sync is in progress')
[sync 4] rc= (lock-busy: 'Another sync is in progress')
outcomes: winners=1 losers=3 unknown=0
post-run gbrain_cycle_locks(gbrain-sync) row count: 0
OK — 1 winner, 3 lock-busy losers, no leaked lock rows.
Production code untouched. All three fixes are in the bash script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(skillopt): hands-on tutorial for auto-improving a skill + discoverability
There was no tutorial for skillopt — only a reference guide
(docs/guides/skillopt.md) that opens at --bootstrap-from-routing and
assumes you already understand benchmarks, and an agent-facing SKILL.md.
README had ZERO skillopt mention. The one thing a user must hand-author
(the benchmark JSONL) was taught nowhere with a worked example.
New: docs/tutorials/improving-skills-with-skillopt.md — Diataxis tutorial
(learning-oriented), copy-pasteable end to end:
1. mental model in two sentences (SKILL.md is the trainable param, the
agent is frozen)
2. write your first benchmark from scratch — a complete 15-task rule-judge
starter you paste and run, with the full check-op table
(contains/regex/section_present/max_chars/min_citations/tool_called/
tool_not_called)
3. --dry-run cost preview (and that it exits 2 by convention, not failure)
4. real run + reading accepted(0)/no_improvement(1)/aborted(2) with the
actual stderr output shape
5. where output lands (best.md, versions/, history.json, rejected.json,
audit jsonl)
6. accept/reject — bundled vs user skills, --no-mutate vs
--allow-mutate-bundled
7. iterate by sharpening the benchmark
The load-bearing fix the tutorial makes that the reference guide got wrong:
the DEFAULT --split 4:1:5 needs ~50 tasks before it runs (sel = N/10, floor
5). A first-time author writing 10-15 tasks hits `D_sel has N task(s)
(need >=5)` and bounces. The tutorial ships 15 tasks + `--split 1:1:1`
(clean 5/5/5) so the copy-paste path actually works. Verified against the
real loadBenchmark + splitBench: the exact shipped block parses 15 unique
tasks and splits 5/5/5 with sel>=5; the system's own error message confirms
"need ~50 total for 4:1:5".
Discoverability (Diataxis cross-linking):
- README.md tutorials section: new entry (was zero skillopt mention)
- docs/tutorials/README.md: added under ## Shipped
- docs/guides/skillopt.md: "New to this? Start with the tutorial" callout
Every claim devex-verified against source: exit-code map from
skillopt.ts (accepted:0/no_improvement:1/aborted:2/errored:2), stderr
format from skillopt.ts:286-292, check ops from score.ts, output paths
from SKILL.md, split math from benchmark.ts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: regenerate llms-full.txt after skillopt tutorial + README edit
Refreshes the inlined doc bundle so the committed llms-full.txt matches
fresh `bun run build:llms` output (test/build-llms.test.ts drift guard).
Picks up the README tutorials-section edit from c39dbdb1. The new tutorial
file itself isn't curated into scripts/llms-config.ts (the bundle curates
a fixed doc set, not every tutorial) — this is purely the README delta.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ci): stop embed-preflight leaking gateway config into facts-backstop shard
CI shard 10 failed 5 `put_page facts backstop` tests with:
[embed(openai:text-embedding-3-small)] Incorrect API key provided: sk-test
(captured by the diagnostic stderr added in a prior commit). Root cause is
a cross-file module-state leak, not a logic bug:
- `embed-preflight.test.ts` calls `configureGateway({env:{OPENAI_API_KEY:
'sk-test'}})` to drive credential-validation scenarios. It resets the
gateway `beforeEach` but never AFTER its last test, so it leaves the
gateway configured with `sk-test`.
- bun runs every file in a shard inside ONE process. The residual config
bleeds into the next file. When `facts-backstop-gating.test.ts` lands in
the same shard, its put_page calls see `isAvailable('embedding') === true`
(the key is *present*, just invalid), so put_page attempts a real embed
and 401s before the backstop gating even runs.
- It's intermittent across master merges because shard bin-packing changes
which files co-locate. (It "resolved" after the v107 merge earlier for
exactly this reason, then came back.)
R1/R2 test-isolation lint doesn't catch this — it's `configureGateway`
module state, not `process.env` or `mock.module`.
Two fixes, both using the gateway's own `resetGateway()` seam (no
process.env, R-compliant):
1. embed-preflight.test.ts — `afterAll(() => resetGateway())` so the leaker
cleans up after the whole file. Primary fix; also protects any OTHER
shard-mate that reads gateway state.
2. facts-backstop-gating.test.ts — `beforeEach(() => resetGateway())` so the
suite is deterministic regardless of ambient gateway config. Defense in
depth: isAvailable('embedding') is now reliably false → put_page uses
noEmbed → the import never embeds → only the backstop gating (the suite's
actual subject) is exercised.
Verified: running leaker+victim in one process (the shard repro) goes
16/16; full shard 10 goes 1208/1208 (was 5 fail in CI). Typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(skillopt): make benchmark authoring an agent job, not a human chore
The prior tutorial taught a human to hand-write a 15-task benchmark — but
nobody does that. The real workflow is: user says "make skill X better,"
the AGENT authors the benchmark and runs the optimizer. The agent-facing
dispatcher didn't actually cover that.
Gap found: skill-optimizer/SKILL.md documented exactly one authoring path,
`--bootstrap-from-routing`, which (a) requires a pre-existing
routing-eval.jsonl (bootstrap-benchmark.ts:57-63 refuses without it) and
(b) generates tasks from ROUTING fixtures — which test dispatch ("does
this phrasing pick this skill"), not output quality. So an agent told to
improve a skill with no benchmark had no documented way to author a
*quality* benchmark; it'd have to reinvent the JSONL format the human
tutorial teaches.
Two fixes:
1. skills/skill-optimizer/SKILL.md — new "Authoring the benchmark yourself
(the common case)" section: read the target SKILL.md, generate ~15
realistic tasks, attach rule judges (contains/max_chars/min_citations/
section_present/regex/tool_called), write the JSONL, run with
`--split 1:1:1` (the default 4:1:5 needs ~50 tasks). Decision-tree row
"New skill, no benchmark" now says "Author one" instead of pointing at
bootstrap-from-routing; the bootstrap row is reframed as a head-start
that only applies when routing fixtures exist and notes routing tasks
test dispatch, not quality.
2. docs/tutorials/improving-skills-with-skillopt.md — new "The easiest
path: ask your agent" section up top. Tells humans to just tell their
agent "improve my X skill — write a benchmark first," and frames the
manual walkthrough as "read this when you want to understand or
hand-curate what the agent is doing."
Verified: conformance 249/0, resolver 99/0, build-llms drift guard 7/0,
cross-link resolves.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(skillopt): --bootstrap-from-skill starter benchmark generator
Generate a quality benchmark from a skill's SKILL.md directly, no
routing-eval.jsonl required. One LLM call emits JSONL tasks (each with rule
judges) that the agent reviews + strengthens before optimizing.
- runBootstrapFromSkill: JSONL output parsed line-by-line with skip-bad-line
salvage (a truncated final line drops, the rest survive); a task is kept only
when >=2 valid rule checks survive; provider errors propagate instead of
collapsing to bootstrap_empty.
- --bootstrap-tasks N (default 15, cap 50); maxTokens scales with the count.
- Extracted assertBenchmarkAbsent + readSkillBodyOrThrow shared with the routing
bootstrap; hardened runBootstrap's routing-eval parse to skip malformed lines.
- CLI: --bootstrap-from-skill short-circuit + 6-way mutual exclusion; parseFlags
exported for unit tests. The benchmark-not-found hint + --help now point here.
- The generator's REVIEW line prints the paste-ready
`--bootstrap-reviewed --split 1:1:1` next command (the default 4:1:5 split
refuses a 15-task starter at D_sel >= 5).
- 20 hermetic cases incl. round-trip into loadBenchmark + splitBench(1:1:1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(skillopt): make --bootstrap-from-skill the primary no-benchmark path
The agent runs --bootstrap-from-skill, strengthens the generated judges (they
are weak drafts), deletes the sentinel, then runs --bootstrap-reviewed
--split 1:1:1. Freehand authoring is demoted to the fallback for the rare skill
the generator can't draft well. Updates the Iron Law, decision tree, and
anti-patterns to cover both bootstrap modes and the 15-task / --split 1:1:1
gotcha.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(release): v0.42.1.0 --bootstrap-from-skill
VERSION + package.json -> 0.42.1.0, CHANGELOG entry, CLAUDE.md skillopt
annotation, regenerated llms-full.txt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: surface --bootstrap-from-skill in README + skillopt reference
- docs/guides/skillopt.md: 30-second pitch leads with --bootstrap-from-skill;
flag table adds --bootstrap-from-skill + --bootstrap-tasks rows.
- README.md: skillopt tutorial pointer mentions generating a starter benchmark.
- Regenerated llms-full.txt (README is in the bundle).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ci): bump FULL_SIZE_BUDGET 700KB→750KB for legitimate CLAUDE.md growth
The skillopt wave annotations + merged v0.41.34-36 master releases pushed
llms-full.txt to 700,423 bytes — 423 over the 700KB cap — failing the
build-llms size-budget test on CI shard 6. CLAUDE.md is ~540KB (77% of the
bundle) and is the whole point of the one-fetch artifact, so it stays inlined;
the budget tracks its per-release growth. 750KB still fits 200k+ context models.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
162 lines
6.9 KiB
TypeScript
162 lines
6.9 KiB
TypeScript
/**
|
|
* v0.41.6.0 D1 — embedding credential preflight.
|
|
*
|
|
* Pure-function tests; uses the gateway's configureGateway / resetGateway
|
|
* test seam to drive different recipe / env shapes without touching
|
|
* process.env.
|
|
*/
|
|
import { describe, test, expect, beforeEach, afterAll } from 'bun:test';
|
|
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
|
|
import {
|
|
validateEmbeddingCreds,
|
|
formatEmbeddingCredsError,
|
|
EmbeddingCredentialError,
|
|
} from '../src/core/embed-preflight.ts';
|
|
import type { AIGatewayConfig } from '../src/core/ai/types.ts';
|
|
|
|
// This file calls configureGateway() to drive credential-validation
|
|
// scenarios. configureGateway mutates module-level gateway state (_config).
|
|
// beforeEach resets BEFORE each test, but the LAST test leaves its config
|
|
// behind — and bun runs every file in a shard inside ONE process, so that
|
|
// residue (e.g. OPENAI_API_KEY: 'sk-test') bleeds into the next file's
|
|
// isAvailable('embedding') check. That's what made facts-backstop-gating
|
|
// fail intermittently (bin-pack-dependent) on CI shard 10. Clean up after
|
|
// the whole file so the gateway is pristine for whatever runs next.
|
|
afterAll(() => { resetGateway(); });
|
|
|
|
function baseConfig(overrides: Partial<AIGatewayConfig> = {}): AIGatewayConfig {
|
|
return {
|
|
embedding_model: 'openai:text-embedding-3-small',
|
|
embedding_dimensions: 1536,
|
|
chat_model: 'anthropic:claude-sonnet-4-6',
|
|
expansion_model: 'anthropic:claude-haiku-4-5',
|
|
env: {},
|
|
base_urls: {},
|
|
...overrides,
|
|
};
|
|
}
|
|
|
|
describe('validateEmbeddingCreds', () => {
|
|
beforeEach(() => { resetGateway(); });
|
|
|
|
test('passes when OPENAI_API_KEY is present and openai model is configured', () => {
|
|
configureGateway(baseConfig({ env: { OPENAI_API_KEY: 'sk-test' } }));
|
|
expect(() => validateEmbeddingCreds()).not.toThrow();
|
|
});
|
|
|
|
test('throws EmbeddingCredentialError with reason=missing_env when OPENAI_API_KEY is unset', () => {
|
|
configureGateway(baseConfig({ env: {} }));
|
|
let caught: unknown;
|
|
try { validateEmbeddingCreds(); } catch (e) { caught = e; }
|
|
expect(caught).toBeInstanceOf(EmbeddingCredentialError);
|
|
const e = caught as EmbeddingCredentialError;
|
|
expect(e.diagnosis.ok).toBe(false);
|
|
if (!e.diagnosis.ok) {
|
|
expect(e.diagnosis.reason).toBe('missing_env');
|
|
if (e.diagnosis.reason === 'missing_env') {
|
|
expect(e.diagnosis.missingEnvVars).toEqual(['OPENAI_API_KEY']);
|
|
expect(e.diagnosis.provider).toBe('openai');
|
|
}
|
|
}
|
|
});
|
|
|
|
test('throws missing_env for voyage when VOYAGE_API_KEY is unset', () => {
|
|
configureGateway(baseConfig({ embedding_model: 'voyage:voyage-3-large', env: {} }));
|
|
let caught: unknown;
|
|
try { validateEmbeddingCreds(); } catch (e) { caught = e; }
|
|
expect(caught).toBeInstanceOf(EmbeddingCredentialError);
|
|
const e = caught as EmbeddingCredentialError;
|
|
if (!e.diagnosis.ok && e.diagnosis.reason === 'missing_env') {
|
|
expect(e.diagnosis.missingEnvVars).toEqual(['VOYAGE_API_KEY']);
|
|
expect(e.diagnosis.provider).toBe('voyage');
|
|
} else { expect('expected missing_env').toBe(JSON.stringify(e.diagnosis)); }
|
|
});
|
|
|
|
test('throws missing_env for google when GOOGLE_GENERATIVE_AI_API_KEY is unset', () => {
|
|
configureGateway(baseConfig({ embedding_model: 'google:text-embedding-004', env: {} }));
|
|
let caught: unknown;
|
|
try { validateEmbeddingCreds(); } catch (e) { caught = e; }
|
|
expect(caught).toBeInstanceOf(EmbeddingCredentialError);
|
|
const e = caught as EmbeddingCredentialError;
|
|
if (!e.diagnosis.ok && e.diagnosis.reason === 'missing_env') {
|
|
expect(e.diagnosis.missingEnvVars).toEqual(['GOOGLE_GENERATIVE_AI_API_KEY']);
|
|
} else { expect('expected missing_env').toBe(JSON.stringify(e.diagnosis)); }
|
|
});
|
|
|
|
test('throws no_touchpoint when configured embedding_model points at anthropic', () => {
|
|
configureGateway(baseConfig({
|
|
embedding_model: 'anthropic:claude-3-5-sonnet',
|
|
env: { ANTHROPIC_API_KEY: 'sk-ant-test' },
|
|
}));
|
|
let caught: unknown;
|
|
try { validateEmbeddingCreds(); } catch (e) { caught = e; }
|
|
expect(caught).toBeInstanceOf(EmbeddingCredentialError);
|
|
const e = caught as EmbeddingCredentialError;
|
|
if (!e.diagnosis.ok) {
|
|
expect(e.diagnosis.reason).toBe('no_touchpoint');
|
|
}
|
|
});
|
|
|
|
test('throws unknown_provider when embedding_model uses unknown provider', () => {
|
|
configureGateway(baseConfig({ embedding_model: 'fakeprovider:embed-1', env: {} }));
|
|
let caught: unknown;
|
|
try { validateEmbeddingCreds(); } catch (e) { caught = e; }
|
|
expect(caught).toBeInstanceOf(EmbeddingCredentialError);
|
|
const e = caught as EmbeddingCredentialError;
|
|
if (!e.diagnosis.ok) {
|
|
expect(e.diagnosis.reason).toBe('unknown_provider');
|
|
}
|
|
});
|
|
|
|
test('throws no_gateway_config when gateway was not configured', () => {
|
|
// resetGateway() in beforeEach already cleared _config.
|
|
let caught: unknown;
|
|
try { validateEmbeddingCreds(); } catch (e) { caught = e; }
|
|
expect(caught).toBeInstanceOf(EmbeddingCredentialError);
|
|
const e = caught as EmbeddingCredentialError;
|
|
if (!e.diagnosis.ok) {
|
|
expect(e.diagnosis.reason).toBe('no_gateway_config');
|
|
}
|
|
});
|
|
});
|
|
|
|
describe('formatEmbeddingCredsError', () => {
|
|
beforeEach(() => { resetGateway(); });
|
|
|
|
test('missing_env produces paste-ready hint naming the env var + --no-embed option', () => {
|
|
configureGateway(baseConfig({ env: {} }));
|
|
let e: EmbeddingCredentialError;
|
|
try { validateEmbeddingCreds(); throw new Error('expected throw'); }
|
|
catch (err) { e = err as EmbeddingCredentialError; }
|
|
expect(e!.userMessage).toContain('OPENAI_API_KEY');
|
|
expect(e!.userMessage).toContain('--no-embed');
|
|
expect(e!.userMessage).toContain('export OPENAI_API_KEY');
|
|
});
|
|
|
|
test('openai-missing message suggests switching to voyage (not openai)', () => {
|
|
configureGateway(baseConfig({ env: {} }));
|
|
let e: EmbeddingCredentialError;
|
|
try { validateEmbeddingCreds(); throw new Error('expected throw'); }
|
|
catch (err) { e = err as EmbeddingCredentialError; }
|
|
// Don't tell user to switch to the provider they already have.
|
|
expect(e!.userMessage).toContain('voyage');
|
|
expect(e!.userMessage).not.toMatch(/Switch providers:.*openai:/);
|
|
});
|
|
|
|
test('voyage-missing message suggests switching to openai', () => {
|
|
configureGateway(baseConfig({ embedding_model: 'voyage:voyage-3-large', env: {} }));
|
|
let e: EmbeddingCredentialError;
|
|
try { validateEmbeddingCreds(); throw new Error('expected throw'); }
|
|
catch (err) { e = err as EmbeddingCredentialError; }
|
|
expect(e!.userMessage).toContain('VOYAGE_API_KEY');
|
|
expect(e!.userMessage).toContain('openai:text-embedding-3-small');
|
|
});
|
|
|
|
test('no_model_configured returns empty-string for ok diagnosis', () => {
|
|
configureGateway(baseConfig({ env: { OPENAI_API_KEY: 'sk-test' } }));
|
|
expect(formatEmbeddingCredsError({
|
|
ok: true, model: 'openai:text-embedding-3-small', provider: 'openai', recipeId: 'openai',
|
|
})).toBe('');
|
|
});
|
|
});
|