mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.28.4 feat(skillpack): enhance skillify with cross-modal eval quality gate (#674)
* feat(skillpack): enhance skillify with cross-modal eval quality gate Updates skillify from v1.0.0 to v2.0.0 with the key innovation: cross-modal evaluation runs BEFORE tests (step 3) to establish quality, then tests lock in the proven-good behavior. Key changes: - 11-item checklist (was 10) - adds cross-modal eval as step 3 - Cross-modal eval uses 3 models to score output on 5 dimensions - Quality gate: all dimensions ≥ 7 average before proceeding to tests - Prevents locking in mediocrity through tests-first approach - References cross-modal-review skill for eval pipeline - Updated all gbrain-specific paths (bun test, scripts/*.ts) - Maintains compatibility with gbrain check-resolvable workflow The meta-skill for turning raw features into properly-skilled, tested, resolvable capabilities. Cross-modal eval ensures output quality before tests cement the behavior. * feat: skillify hardened via 2 cross-modal eval cycles (8.1/10) Applied top improvements from GPT-5.5 + Opus 4-7 + DeepSeek V4 Pro: - Named 3 frontier models explicitly with provider table - Inlined eval prompt template with CONTEXT param + scoring calibration - Defined aggregation math: mean >= 7 AND no single dim < 5 - Added eval receipt JSON schema - Structured 3-cycle fix loop with before/after delta tracking - Added worked example (summarize-pr, end-to-end) - Added cost guardrails (skip < 200 tokens, max 9 API calls) - Added representative input selection rule - Added SKILL.md frontmatter template (copy-paste ready) - Added Phase 0 decision gate (is this worth skillifying?) Also includes cross-modal-eval runner recipe with robust JSON parsing for LLMs that return malformed JSON (3-tier repair). * chore(recipes): remove cross-modal-eval.mjs Superseded by `gbrain eval cross-modal` (next commit). The .mjs script was the original PR's hand-rolled provider stack; the replacement reuses src/core/ai/gateway.ts so config/auth/model-aliasing comes from the canonical recipe registry instead of a parallel stack. No code references the .mjs (it was invoked by skill prose only), so this delete is independently safe to bisect through. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): cross-modal-eval core module + unit tests Pure-logic foundation for the new `gbrain eval cross-modal` command (wired in the next commit). All five modules are self-contained — no CLI surface, no I/O outside the receipt writer's mkdirSync. Imported from src/core/ai/gateway.ts at runtime via gwChat (no config impact at load time). Modules: - json-repair.ts: parseModelJSON 4-strategy fallback chain. Adversarial nuclear-option throws rather than fabricating scores (Q6 + Q3 in plan). - aggregate.ts: verdict logic. PASS = (>=2 successes) AND (every dim mean >= 7) AND (every dim min across models >= 5). INCONCLUSIVE when <2/3 models returned parseable scores — closes the v1 .mjs `Object.values({}).every(...) === true` empty-array silent-PASS bug (Q2 + Q3). - receipt-name.ts: receipt filename binds (slug, sha8 of SKILL.md) so `gbrain skillify check` can detect stale audits (T10 in plan). - receipt-write.ts: thin wrapper over writeFileSync that auto-mkdirs the parent directory. Standalone module because gbrainPath() does NOT auto-mkdir (T5 plan correction — Codex caught this). - runner.ts: orchestrator. Promise.allSettled across 3 slots per cycle; up to 3 cycles; stops early on PASS or INCONCLUSIVE. Default slots: openai:gpt-4o / anthropic:claude-opus-4-7 / google:gemini-1.5-pro. estimateCost() exports a small per-model pricing table (drifts; refresh alongside model-family bumps). Tests (32 cases total, all green): - json-repair.test.ts: 10 cases (clean JSON, fences, trailing commas, single quotes, embedded newlines, mismatched braces, nuclear-option success + adversarial throws, empty input, numeric-shorthand scores). - aggregate.test.ts: 8 cases pinning Q2/Q3/dedup. The 0-of-3 INCONCLUSIVE case is the regression guard for the v1 silent-PASS bug. - cli.test.ts: 12 cases on receipt-name / receipt-write / GBRAIN_HOME isolation. Uses withEnv() helper for env mutation (R1 isolation rule). Verifies bisect-clean: typecheck passes, all 32 unit cases green. The runner.ts import of gateway.chat() is dead until commit 3 wires the CLI surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): wire `gbrain eval cross-modal` CLI subcommand User-facing surface for the multi-model quality gate. Three different- provider frontier models score the OUTPUT against the TASK on a 5-dim rubric. Verdict drives exit code: 0 PASS, 1 FAIL, 2 INCONCLUSIVE (<2/3 models returned parseable scores per Q3 in plan). Wiring touches three files: - src/commands/eval-cross-modal.ts (new, ~290 lines) CLI handler. Self-configures the AI gateway from loadConfig() + process.env so it works without `gbrain init` (the cli.ts no-DB branch bypasses connectEngine()). Defaults: cycles=3 in TTY, cycles=1 in non-TTY (T11 partial cost guardrail — limits scripted bulk spend; full --budget-usd hard cap is a v0.27.x TODO). Prints estimated max-cost-per-cycle to stderr before each run. Uses gbrainPath('eval-receipts') for receipt directory. - src/cli.ts (no-DB dispatch branch, 5-line addition) Special-cases `eval cross-modal` BEFORE the existing handleCliOnly path that requires connectEngine(). Mirrors the `dream` no-DB pattern but doesn't even attempt the connect — the command never touches the DB. New users can run the gate before `gbrain init` (T3 in plan). - src/commands/eval.ts (sub-subcommand dispatch) Adds `cross-modal` alongside `export`/`prune`/`replay`. The cli.ts branch takes precedence in the user-facing path; this branch only fires when callers re-enter runEvalCommand with an existing engine. Engine is intentionally unused — the handler self-routes. - test/e2e/cross-modal-eval.test.ts (new, 4 cases) Mocked-fetch E2E. Lives at test/e2e/* (NOT *.serial.test.ts) per plan T8: test/e2e/* is exempt from the test-isolation lint and already runs serially via scripts/run-e2e.sh, so the mock.module() call doesn't need a quarantine rename. Cases: PASS / FAIL (mean<7) / FAIL (min<5 — Q2 floor) / INCONCLUSIVE (2 mock 5xx — Q3 contract). The runner from commit 2 now has live callers. typecheck passes; the 4 E2E cases all green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(skillify): add informational 11th item (cross-modal eval) Promotes the skillify contract from 10 to 11 items. The 11th item (cross-modal eval) is `required:false` per T7 in the plan — a missing or stale receipt surfaces in the audit output but does not fail the gate. Existing skills keep their current required-score; the bump is additive, not breaking. Changes: - src/commands/skillify.ts Header jsdoc updated 10-item -> 11-item. No code-flow changes. - src/commands/skillify-check.ts (the per-skill audit; not src/commands/skillpack-check.ts which is a different command — plan T6 corrected the conflation in the original plan) New informational item at position 11. Reuses findReceiptForSkill() helper from src/core/cross-modal-eval/receipt-name.ts to detect: * found — receipt matches current SKILL.md sha-8 * stale — receipt exists for an older SKILL.md * missing — no receipt yet Audit output cases pass through to existing pretty/JSON formats. - src/core/skillify/templates.ts Scaffolded SKILL.md now includes a "Phase 3: Cross-modal eval (informational)" section with copy-paste `gbrain eval cross-modal` invocation, pass criteria, and receipt-naming convention. Helps new skill authors discover the gate. - test/skillify-scaffold.test.ts New T9 case verifies the scaffold emits the Phase 3 section, points at the correct command, documents the receipt path, and appends exactly one resolver row. Replaces the original plan's `gbrain skillify scaffold demo-eleven` shell verification (which Codex caught as invalid + repo-mutating). Verifies: typecheck passes; scaffold test 19/19 (was 18, +1 T9 case). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: skillify v1.1.0 + cross-modal-eval references Documentation catches up with the new behavior shipped in commits 1-4. - skills/skillify/SKILL.md (1.0.0 -> 1.1.0) Full rewrite. Frontmatter version is additive (T7 in plan); the 11th item is informational, not breaking. Phase 3 now points at `gbrain eval cross-modal` with copy-paste invocation, default slot table, pass criteria, receipt-naming convention, cycles + cost guardrails (T11 partial cap), provider configuration via the AI gateway, and the cycle-1/2/3 fix loop. Adds Output Format section (skills-conformance.test.ts requires it). Drops the original `(or lib/cross-modal-eval.ts)` parenthetical (Q5 plan correction — that path never existed). - skills/cross-modal-review/SKILL.md Adds 4-line Relationship section pointing at `gbrain eval cross-modal` (D3 plan reciprocal). Distinguishes the manual second-opinion gate (this skill) from the automated multi-model score-and-iterate gate (the new command). - CLAUDE.md Key Files entries for src/commands/eval-cross-modal.ts and the five new src/core/cross-modal-eval/* modules. Commands list gains the `gbrain eval cross-modal` entry under v0.27.x. Notes the non-TTY default 1-cycle behavior + the gbrainPath('eval- receipts') resolution. - TODOS.md Four v0.27.x follow-ups filed under a new "cross-modal-eval" section: full --budget-usd cap (T11 follow-up), subagent integration (recovers cross-process rate-leases T4 deferred), skill adoption telemetry (revisit T7=C with data after 30 days), docs/cross-modal-eval.md user guide. - llms-full.txt Regenerated via `bun run build:llms` to match the CLAUDE.md edits — sync guard at test/build-llms.test.ts requires this. Verifies: typecheck passes; skills-conformance 199/199 green; build-llms 7/7 green; full unit fast loop 3861/3861 green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.28.4) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
garrytan-agents
co-authored by
garrytan-agents
Garry Tan
Claude Opus 4.7
garrytan-agents
Garry Tan
Claude Opus 4.7
parent
e744eda66c
commit
a1a2671c21
@@ -2,6 +2,69 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.28.4] - 2026-05-06
|
||||
|
||||
## **`gbrain eval cross-modal` — three frontier models score your skill output BEFORE tests cement it.**
|
||||
## **Different providers, different blind spots. Pass criterion: every dim mean >= 7 AND no model scored any dim < 5. INCONCLUSIVE when fewer than 2 of 3 evaluators returned parseable scores.**
|
||||
|
||||
Cross-modal eval is the new Phase 3 in the skillify checklist (now 11 items, up from 10). Run it against any skill output before you write tests. Three frontier models from three different providers score the output on five documented dimensions in parallel. Receipts bind to a SHA-8 of the SKILL.md content so `gbrain skillify check` can tell whether the audit is current or stale. Required:false in v0.28.4, informational only, so existing skills don't retroactively fail their audits.
|
||||
|
||||
The original v0.27 PR added a hand-rolled `.mjs` script with three correctness bugs: hardcoded `/data/.env` (cloud-sandbox path; failed on every normal Mac), all-models-fail returned PASS because `Object.values({}).every(...)` is `true` for an empty array, and the documented "no single model < 5" floor was never implemented. Plan-eng-review caught those plus structural drift between SKILL.md and the gbrain CLI. Codex consult-mode caught five more I missed: the script rolled a parallel provider stack instead of reusing `src/core/ai/gateway.ts`; the `gbrain eval` dispatch path required `connectEngine()` so first-run users couldn't run the gate; the `rate-leases` helper requires a `minion_jobs.id` that a CLI eval doesn't have; the proposed `gbrainPath` semantics were wrong; the conformance test required an `## Output Format` section the rewrite dropped. All fixed. 25 decisions resolved across the two review rounds.
|
||||
|
||||
### What you get
|
||||
|
||||
| Capability | Before | After |
|
||||
|---|---|---|
|
||||
| Multi-model quality gate | Hand-rolled `.mjs` script with `/data/.env` hardcoded | `gbrain eval cross-modal --task "..." --output skills/<slug>/SKILL.md` |
|
||||
| Provider config | Parallel stack with raw `fetch` + custom env loader | Reuses `src/core/ai/gateway.ts:chat()`; configure via `gbrain config` or env vars |
|
||||
| Pass criterion | Mean >= 7 only | Mean >= 7 AND no model scored any dim < 5 |
|
||||
| All-models-fail bug | Silent PASS (empty-array `.every()` = true) | INCONCLUSIVE (exit 2) when < 2/3 succeed |
|
||||
| Default model identifiers | `gpt-5.5`, `claude-opus-4-7`, `deepseek-ai/DeepSeek-V4-Pro` (two of three fictional) | `openai:gpt-4o`, `anthropic:claude-opus-4-7`, `google:gemini-1.5-pro` (all real, all addressable) |
|
||||
| Receipt path | `/tmp/cross-modal-eval-<ts>.json` (Windows-broken; cleared on reboot) | `~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json` (honors `GBRAIN_HOME`; sha-8 detects stale) |
|
||||
| Onboarding | Required `gbrain init` first | No-DB CLI branch, runs before any brain exists |
|
||||
| Cycle default | Always 3 (cost runaway risk in CI loops) | 3 in TTY, 1 in non-TTY; cost-estimate prints to stderr before each run |
|
||||
| Tests for the gate logic | Zero | 32 unit + 4 mocked-fetch E2E (verdict / floor / inconclusive / dedup pinned) |
|
||||
|
||||
### What this means for you
|
||||
|
||||
If you're skillifying a feature and want to lock quality in BEFORE writing tests:
|
||||
|
||||
```bash
|
||||
gbrain eval cross-modal \
|
||||
--task "Skillify SKILL.md teaches the 11-item meta-skill checklist" \
|
||||
--output skills/skillify/SKILL.md
|
||||
```
|
||||
|
||||
Three frontier models score in parallel. Total wallclock is 30-60s per cycle. Cost estimate prints before each run. Exit 0 = PASS, 1 = FAIL, 2 = INCONCLUSIVE (don't trust the verdict; rerun). Receipt lands at `~/.gbrain/.gbrain/eval-receipts/skillify-<sha8>.json` so `gbrain skillify check` can show its status the next time you run an audit.
|
||||
|
||||
If you contribute to gbrain itself: `skills/skillify/SKILL.md` is the canonical 11-item checklist. `skills/cross-modal-review/SKILL.md` is the manual second-opinion gate (one model reviews work in flow). The two are complementary; both have a Relationship section pointing at each other so you know which to use when.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- `gbrain eval cross-modal` (new CLI subcommand). Reuses `src/core/ai/gateway.ts:chat()` for provider config + auth + model aliasing. No-DB dispatch in `src/cli.ts` mirrors the `dream` pattern so first-run users (no `gbrain init` yet) can run the gate.
|
||||
- `src/core/cross-modal-eval/{json-repair,aggregate,runner,receipt-name,receipt-write}.ts` (new) — pure-logic foundation. JSON repair has a 4-strategy fallback chain; nuclear-option throws rather than fabricates scores. Aggregate enforces both pass criteria (mean >= 7 AND min >= 5) and the >=2/3-successes rule.
|
||||
- `skills/skillify/SKILL.md` — bumped to v1.1.0. Phase 3 documents the gate; Anti-Patterns section warns against single-provider correlated blind spots and budget-model evaluation.
|
||||
- `skills/cross-modal-review/SKILL.md` — Relationship section added pointing at the new command.
|
||||
- `src/commands/skillify-check.ts` — informational 11th item surfaces receipt status (`found` / `stale` / `missing`). Not blocking; existing skills keep their required-score.
|
||||
- `src/core/skillify/templates.ts` — scaffolded SKILL.md now includes a Phase 3 section so new skill authors discover the gate.
|
||||
- `recipes/cross-modal-eval/cross-modal-eval.mjs` — DELETED. Behavior moved to the new command.
|
||||
- `test/cross-modal-eval-{json-repair,aggregate,cli}.test.ts` (new) — 32 unit cases.
|
||||
- `test/e2e/cross-modal-eval.test.ts` (new, mocked fetch) — 4 verdict-contract cases.
|
||||
- `test/skillify-scaffold.test.ts` — extended with the 11-item assertion (replaces the original plan's mutating shell-out verification).
|
||||
- `CLAUDE.md` — Key Files entries for the new module + Commands list under v0.27.x. `TODOS.md` files four follow-ups (full `--budget-usd` cap, subagent integration to recover rate-leases, adoption telemetry, user guide).
|
||||
|
||||
## To take advantage of v0.28.4
|
||||
|
||||
`gbrain upgrade` should do this automatically. The new command is wired through the existing CLI; nothing schema-level changed. To verify:
|
||||
|
||||
```bash
|
||||
gbrain eval cross-modal --help
|
||||
# If you have OPENAI_API_KEY + ANTHROPIC_API_KEY + GOOGLE_GENERATIVE_AI_API_KEY in your shell:
|
||||
gbrain eval cross-modal \
|
||||
--task "Sample task description" \
|
||||
--output skills/skillify/SKILL.md
|
||||
```
|
||||
|
||||
## [0.28.3] - 2026-05-06
|
||||
|
||||
## **`gbrain integrations show restart-sweep` — detect dropped Telegram messages after OpenClaw gateway restarts.**
|
||||
|
||||
@@ -67,7 +67,13 @@ strict behavior when unset.
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. v0.27.x adds `gbrain eval cross-modal` to the dispatch (the user-facing path is the cli.ts no-DB branch — `src/commands/eval.ts:cross-modal` only fires when callers re-enter with an existing engine).
|
||||
- `src/commands/eval-cross-modal.ts` (v0.27.x) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes per Q3=A in plans/radiant-napping-lerdorf.md). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (T11=B partial cost guardrail). Receipts land at `gbrainPath('eval-receipts')/<slug>-<sha8-of-output>.json`. The full `--budget-usd` cap is a v0.27.x follow-up TODO.
|
||||
- `src/core/cross-modal-eval/json-repair.ts` (v0.27.x) — `parseModelJSON(raw)` named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes.
|
||||
- `src/core/cross-modal-eval/aggregate.ts` (v0.27.x) — pure verdict logic. Pass criterion: `(successes >= 2) AND (every dim mean >= 7) AND (every dim min across models >= 5)` (Q2=A floor). Inconclusive when <2/3 models returned parseable scores (Q3=A regression guard for the v1 .mjs `Object.values({}).every(...) === true` empty-array PASS bug).
|
||||
- `src/core/cross-modal-eval/runner.ts` (v0.27.x) — orchestrator. Each cycle runs `Promise.allSettled([gwChat(slotA), gwChat(slotB), gwChat(slotC)])` (T4=A — bare allSettled, no rate-leases for the CLI path; minion-integration TODO recovers cross-process concurrency). Stops early on PASS or INCONCLUSIVE; runs up to 3 cycles. Default slots: `openai:gpt-4o` / `anthropic:claude-opus-4-7` / `google:gemini-1.5-pro`. `estimateCost()` exports a small per-model pricing table (drifts; refresh alongside model-family bumps).
|
||||
- `src/core/cross-modal-eval/receipt-name.ts` (v0.27.x) — receipt filename binds (slug, SKILL.md sha-8). `findReceiptForSkill(skillPath, receiptDir)` returns `'found' | 'stale' | 'missing'` (T10=A). Skillify-check item 11 surfaces the status as informational (T7=C); the audit does NOT fail on missing/stale receipts.
|
||||
- `src/core/cross-modal-eval/receipt-write.ts` (v0.27.x) — wraps `fs.writeFileSync` with `mkdirSync({recursive:true})` ahead of every write (T5 correction; `gbrainPath()` does NOT auto-mkdir).
|
||||
- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
@@ -265,6 +271,7 @@ Key commands added in v0.25.0:
|
||||
- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay.
|
||||
- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
|
||||
- `gbrain eval replay --against FILE.ndjson [--limit N] [--top-regressions K] [--json] [--verbose]` — contributor-facing dev loop. Reads a captured NDJSON snapshot, re-runs each `query` / `search` op against the current brain, computes mean set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. JSON mode (`schema_version: 1`) for CI gating; human mode prints a regression table sorted worst-first. Closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `gbrain eval cross-modal --task "..." --output <path> [--cycles N] [--slot-a-model ID] [--slot-b-model ID] [--slot-c-model ID] [--receipt-dir DIR] [--json]` (v0.27.x) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on 5 documented dimensions. Pass criterion: every dim mean >=7 AND no model scored any dim <5. Exit codes: 0 PASS, 1 FAIL, 2 INCONCLUSIVE (<2/3 models returned parseable scores). Default cycles=3 in TTY, **cycles=1 in non-TTY** (limits accidental scripted bulk spend). Default slots: `openai:gpt-4o` / `anthropic:claude-opus-4-7` / `google:gemini-1.5-pro` — refresh alongside model-family bumps. Receipts land at `~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8-of-output>.json` (gbrainPath honors GBRAIN_HOME). Bypasses `connectEngine()` via the cli.ts no-DB branch — runs cleanly before `gbrain init`. Reuses `src/core/ai/gateway.ts:chat()` for config/auth (no parallel provider stack). Cost-estimate prints to stderr before each cycle (T11=B partial cost guardrail; full `--budget-usd N` is a follow-up TODO).
|
||||
- `gbrain doctor` gains an `eval_capture` check: reads `eval_capture_failures` for the last 24h, groups by reason, warns when non-zero. Cross-process visibility (doctor runs in a separate process from MCP). Pre-v31 brains get `Skipped (table unavailable)` — non-fatal.
|
||||
- Config addition: `eval: { capture?: boolean, scrub_pii?: boolean }` in `~/.gbrain/config.json`. **File-plane only** — `gbrain config set` writes the DB plane and does NOT control capture.
|
||||
- **`GBRAIN_CONTRIBUTOR_MODE=1` env var** is the contributor-facing toggle. Capture is **off by default** as of v0.25.0; production users get a quiet brain. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Documented in README.md, CONTRIBUTING.md, and `docs/eval-bench.md`.
|
||||
|
||||
@@ -1,5 +1,55 @@
|
||||
# TODOS
|
||||
|
||||
## cross-modal-eval (v0.27.x follow-ups from PR #674 plan)
|
||||
|
||||
### `--budget-usd` hard cap + per-call cost telemetry (T11=B follow-up)
|
||||
**Priority:** P2
|
||||
|
||||
**What:** `gbrain eval cross-modal` ships in v0.27.x with a partial cost guardrail: default `--cycles 1` in non-TTY plus a stderr cost-estimate printed before each run. The full `--budget-usd N` hard cap (refuse to start the next cycle if estimated spend would exceed) and per-call actual-cost telemetry written into the receipt are intentionally deferred.
|
||||
|
||||
**Why:** Codex pushback on the original P2=B "defer everything" decision was right — even with `>=2/3` success required for a verdict (Q3=A), 3 cycles × 3 calls = 9 frontier calls per run, repeated across N skills if anyone scripts a bulk audit. The TTY/non-TTY cycle default catches the worst case; the hard cap catches the next class of mistakes.
|
||||
|
||||
**Pros:** Deterministic spend ceiling. Real per-call cost in the receipt drives a feedback loop that lets us refine the price-table constant in `src/core/cross-modal-eval/runner.ts:estimateCost`. Future bulk-audit integrations get a safety net by default.
|
||||
**Cons:** ~80 lines of pricing-table + parsing + threading. Pricing values drift; the file becomes a small maintenance burden between model-family bumps.
|
||||
**Context:** Pricing table lives at `src/core/cross-modal-eval/runner.ts:estimateCost`. Once we have real telemetry from a few weeks of usage, we can switch the table to "last observed" instead of "list price" and get more accurate caps. v0.27.x candidate.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
### Subagent integration (recovers cross-process rate-leases — T4 deferred)
|
||||
**Priority:** P2
|
||||
|
||||
**What:** Wire `gbrain eval cross-modal` to be invokable as a `gbrain agent run` child job. Today the CLI runs synchronously and bypasses `src/core/minions/rate-leases.ts` because the lease helper requires a `minion_jobs.id` that the CLI path doesn't have (T4=A in plans/radiant-napping-lerdorf.md).
|
||||
|
||||
**Why:** Cross-process concurrency cap. A user running `gbrain eval cross-modal` in one terminal alongside `gbrain agent run` in another can hit Anthropic 429s due to combined load. As a minion job, the eval gets the rate-lease behavior for free, plus stagger / quiet-hours / retry surface from the existing Minions queue.
|
||||
|
||||
**Pros:** No new helper API; reuses what's already there. Closes the cross-process gap that today's `Promise.allSettled` design intentionally leaves open.
|
||||
**Cons:** Requires a job handler registration + receipt-path threading through job context. Probably ~150 lines plus tests. Behavior parity (verdict / receipt shape) needs to be pinned with a parametrized test.
|
||||
**Context:** Pattern is the same as `src/core/minions/handlers/subagent.ts`. v0.27.x candidate.
|
||||
**Depends on:** Nothing.
|
||||
|
||||
### Skill adoption telemetry (revisit T7=C with data)
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Track how many skills land cross-modal eval receipts. If adoption stalls at, say, <30% of skills after 30 days, consider flipping the 11th item from `required:false` (T7=C, current) to `required:true` (T7=A) in v0.28.x.
|
||||
|
||||
**Why:** T7=C ships the gate as informational so existing audits don't regress. The forcing function is documentation alone. We don't yet know if that's enough.
|
||||
|
||||
**Pros:** Data-driven decision instead of guessing. Lightweight: count receipt files in `gbrainPath('eval-receipts')` against the count of skills under `skills/*/SKILL.md`.
|
||||
**Cons:** "Adoption stalled" is a judgment call without a baseline. Could become a debate.
|
||||
**Context:** New check in `gbrain doctor` would surface the count. v0.28.x candidate.
|
||||
**Depends on:** None.
|
||||
|
||||
### `docs/cross-modal-eval.md` user guide
|
||||
**Priority:** P3
|
||||
|
||||
**What:** Add a user-facing guide. Cover the gateway-config flow, receipt forensics, the `<slug>-<sha8>.json` filename convention, default models + how to override them, the relationship to `skills/cross-modal-review/SKILL.md`, and worked examples on a real skill.
|
||||
|
||||
**Why:** SKILL.md teaches the workflow but lives under `skills/skillify/`. CLAUDE.md "Key files" entries are agent-facing, not human-facing. A `docs/cross-modal-eval.md` is the natural home for "I'm a user, how do I use this command?" answers.
|
||||
|
||||
**Pros:** Discoverable from CLAUDE.md "Key files" reference. Mirrors `docs/eval-bench.md` precedent.
|
||||
**Cons:** Doc-write task; ~250 lines of prose.
|
||||
**Context:** v0.27.x candidate.
|
||||
**Depends on:** None.
|
||||
|
||||
## /health endpoint hardening (v0.28.1 follow-up)
|
||||
|
||||
### Cancel `engine.getStats()` when /health times out
|
||||
|
||||
+8
-1
@@ -164,7 +164,13 @@ strict behavior when unset.
|
||||
- `src/core/search/eval.ts` — Retrieval eval harness: P@k, R@k, MRR, nDCG@k metrics + runEval() orchestrator
|
||||
- `src/core/search/source-boost.ts` (v0.22.0) — Source-type boost map keyed by slug prefix. `DEFAULT_SOURCE_BOOSTS` (originals/ 1.5, concepts/ 1.3, writing/ 1.4, people/companies/deals/ 1.2, daily/ 0.8, media/x/ 0.7, wintermute/chat/ 0.5) and `DEFAULT_HARD_EXCLUDES` (test/, archive/, attachments/, .raw/). `parseSourceBoostEnv` / `parseHardExcludesEnv` parse comma-separated `prefix:factor` pairs from `GBRAIN_SOURCE_BOOST` / `GBRAIN_SEARCH_EXCLUDE` env vars. `resolveBoostMap` and `resolveHardExcludes` merge defaults + env + caller `SearchOpts.exclude_slug_prefixes`/`include_slug_prefixes`.
|
||||
- `src/core/search/sql-ranking.ts` (v0.22.0) — Pure SQL string builders. `buildSourceFactorCase(slugColumn, boostMap, detail)` emits a CASE expression with longest-prefix-match wins (returns literal `'1.0'` when `detail === 'high'` for temporal-bypass parity with COMPILED_TRUTH_BOOST). `buildHardExcludeClause(slugColumn, prefixes)` emits `NOT (col LIKE 'p1%' OR col LIKE 'p2%')` — OR-chain wrapped in NOT, NOT `NOT LIKE ALL/ANY` (those quantifiers don't express set-exclusion). LIKE meta-character escape covers all three of `%`, `_`, AND `\` (backslash matters because it's Postgres LIKE's default escape char). Single-quote doubling on SQL string literals so injection-style inputs are inert text.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow.
|
||||
- `src/commands/eval.ts` — `gbrain eval` command: single-run table + A/B config comparison. v0.25.0 adds sub-subcommand dispatch on `args[0]` so `gbrain eval export` + `gbrain eval prune` + `gbrain eval replay` route into session-capture handlers; bare `gbrain eval --qrels …` fall-through preserves the legacy IR-metrics flow. v0.27.x adds `gbrain eval cross-modal` to the dispatch (the user-facing path is the cli.ts no-DB branch — `src/commands/eval.ts:cross-modal` only fires when callers re-enter with an existing engine).
|
||||
- `src/commands/eval-cross-modal.ts` (v0.27.x) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on a 5-dim list. Verdict `pass` (exit 0) / `fail` (exit 1) / `inconclusive` (exit 2; <2/3 model successes per Q3=A in plans/radiant-napping-lerdorf.md). Reuses `src/core/ai/gateway.ts:chat()` so config/auth/aliasing comes from the gateway recipe registry — no parallel provider stack. Self-configures the gateway (`configureGateway(loadConfig() + process.env)`) since the cli.ts dispatch bypasses `connectEngine()`. Default cycles 3 in TTY, 1 in non-TTY (T11=B partial cost guardrail). Receipts land at `gbrainPath('eval-receipts')/<slug>-<sha8-of-output>.json`. The full `--budget-usd` cap is a v0.27.x follow-up TODO.
|
||||
- `src/core/cross-modal-eval/json-repair.ts` (v0.27.x) — `parseModelJSON(raw)` named export with a 4-strategy fallback chain (direct parse → fence-strip → trailing-comma + single-quote + embedded-newline repair → regex nuclear option). Adversarial input throws rather than fabricating scores — the aggregator treats a throw as "this model contributed nothing this cycle" so the gate stays correct at >=2/3 successes.
|
||||
- `src/core/cross-modal-eval/aggregate.ts` (v0.27.x) — pure verdict logic. Pass criterion: `(successes >= 2) AND (every dim mean >= 7) AND (every dim min across models >= 5)` (Q2=A floor). Inconclusive when <2/3 models returned parseable scores (Q3=A regression guard for the v1 .mjs `Object.values({}).every(...) === true` empty-array PASS bug).
|
||||
- `src/core/cross-modal-eval/runner.ts` (v0.27.x) — orchestrator. Each cycle runs `Promise.allSettled([gwChat(slotA), gwChat(slotB), gwChat(slotC)])` (T4=A — bare allSettled, no rate-leases for the CLI path; minion-integration TODO recovers cross-process concurrency). Stops early on PASS or INCONCLUSIVE; runs up to 3 cycles. Default slots: `openai:gpt-4o` / `anthropic:claude-opus-4-7` / `google:gemini-1.5-pro`. `estimateCost()` exports a small per-model pricing table (drifts; refresh alongside model-family bumps).
|
||||
- `src/core/cross-modal-eval/receipt-name.ts` (v0.27.x) — receipt filename binds (slug, SKILL.md sha-8). `findReceiptForSkill(skillPath, receiptDir)` returns `'found' | 'stale' | 'missing'` (T10=A). Skillify-check item 11 surfaces the status as informational (T7=C); the audit does NOT fail on missing/stale receipts.
|
||||
- `src/core/cross-modal-eval/receipt-write.ts` (v0.27.x) — wraps `fs.writeFileSync` with `mkdirSync({recursive:true})` ahead of every write (T5 correction; `gbrainPath()` does NOT auto-mkdir).
|
||||
- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
@@ -362,6 +368,7 @@ Key commands added in v0.25.0:
|
||||
- `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay.
|
||||
- `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s.
|
||||
- `gbrain eval replay --against FILE.ndjson [--limit N] [--top-regressions K] [--json] [--verbose]` — contributor-facing dev loop. Reads a captured NDJSON snapshot, re-runs each `query` / `search` op against the current brain, computes mean set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. JSON mode (`schema_version: 1`) for CI gating; human mode prints a regression table sorted worst-first. Closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `gbrain eval cross-modal --task "..." --output <path> [--cycles N] [--slot-a-model ID] [--slot-b-model ID] [--slot-c-model ID] [--receipt-dir DIR] [--json]` (v0.27.x) — multi-model quality gate. Three different-provider frontier models score the OUTPUT against the TASK on 5 documented dimensions. Pass criterion: every dim mean >=7 AND no model scored any dim <5. Exit codes: 0 PASS, 1 FAIL, 2 INCONCLUSIVE (<2/3 models returned parseable scores). Default cycles=3 in TTY, **cycles=1 in non-TTY** (limits accidental scripted bulk spend). Default slots: `openai:gpt-4o` / `anthropic:claude-opus-4-7` / `google:gemini-1.5-pro` — refresh alongside model-family bumps. Receipts land at `~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8-of-output>.json` (gbrainPath honors GBRAIN_HOME). Bypasses `connectEngine()` via the cli.ts no-DB branch — runs cleanly before `gbrain init`. Reuses `src/core/ai/gateway.ts:chat()` for config/auth (no parallel provider stack). Cost-estimate prints to stderr before each cycle (T11=B partial cost guardrail; full `--budget-usd N` is a follow-up TODO).
|
||||
- `gbrain doctor` gains an `eval_capture` check: reads `eval_capture_failures` for the last 24h, groups by reason, warns when non-zero. Cross-process visibility (doctor runs in a separate process from MCP). Pre-v31 brains get `Skipped (table unavailable)` — non-fatal.
|
||||
- Config addition: `eval: { capture?: boolean, scrub_pii?: boolean }` in `~/.gbrain/config.json`. **File-plane only** — `gbrain config set` writes the DB plane and does NOT control capture.
|
||||
- **`GBRAIN_CONTRIBUTOR_MODE=1` env var** is the contributor-facing toggle. Capture is **off by default** as of v0.25.0; production users get a quiet brain. Resolution order: explicit `eval.capture` config wins both directions, then env var, then off. Documented in README.md, CONTRIBUTING.md, and `docs/eval-bench.md`.
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.28.3",
|
||||
"version": "0.28.4",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -26,6 +26,14 @@ mutating: false
|
||||
> **Convention:** see [conventions/cross-modal.yaml](../conventions/cross-modal.yaml)
|
||||
> for the review pairs and refusal routing chain.
|
||||
|
||||
> **Relationship to `gbrain eval cross-modal`:** This skill is the manual
|
||||
> mid-flow gate (one model reviews work product before commit, with refusal
|
||||
> routing). The `gbrain eval cross-modal` command (v0.27.x) is a sibling
|
||||
> surface: 3 different-provider frontier models score-and-iterate on a
|
||||
> documented dimension list *before* tests cement behavior. Use this skill
|
||||
> for ad-hoc second opinions; use `gbrain eval cross-modal` for the
|
||||
> skillify Phase 3 quality gate. The two are complementary, not redundant.
|
||||
|
||||
## Contract
|
||||
|
||||
This skill guarantees:
|
||||
|
||||
+276
-151
@@ -1,16 +1,12 @@
|
||||
---
|
||||
name: skillify
|
||||
version: 1.0.0
|
||||
version: 1.1.0
|
||||
description: |
|
||||
The meta skill. Turn any raw feature or script into a properly-skilled,
|
||||
tested, resolvable, evaled unit of agent-visible capability. Use when
|
||||
the user says "skillify this", "is this a skill?", "make this proper",
|
||||
or after a new feature is built without the full skill infrastructure.
|
||||
|
||||
Paired with `gbrain check-resolvable`, skillify gives a user-controllable
|
||||
equivalent of Hermes' auto-skill-creation: you build, skillify checks the
|
||||
checklist, check-resolvable verifies nothing is orphaned. The human keeps
|
||||
judgment; the tooling keeps the checklist honest.
|
||||
The meta skill. Turn any raw feature into a properly-skilled, tested,
|
||||
resolvable unit of agent capability. Cross-modal eval is the recommended
|
||||
Phase 3 quality gate: 3 frontier models from different providers critique
|
||||
the output, you iterate to quality, THEN write tests that lock in the
|
||||
proven-good behavior.
|
||||
triggers:
|
||||
- "skillify this"
|
||||
- "skillify"
|
||||
@@ -19,167 +15,296 @@ triggers:
|
||||
- "add tests and evals for this"
|
||||
- "check skill completeness"
|
||||
tools:
|
||||
- search
|
||||
- list_pages
|
||||
mutating: false
|
||||
- exec
|
||||
- read
|
||||
- write
|
||||
mutating: true
|
||||
---
|
||||
|
||||
# Skillify — The Meta Skill
|
||||
|
||||
> **Relationship to `/cross-modal-review`:** That skill is the manual mid-flow
|
||||
> "second opinion" gate (one model reviews work product before commit). This
|
||||
> skill's Phase 3 below uses `gbrain eval cross-modal` instead — three
|
||||
> different-provider frontier models score-and-iterate on a documented
|
||||
> dimension list *before* tests cement behavior. Use `/cross-modal-review`
|
||||
> for ad-hoc second opinions; use Phase 3 here when skillifying a feature.
|
||||
|
||||
## Contract
|
||||
|
||||
A feature is "properly skilled" when all ten checklist items are present:
|
||||
A feature is "properly skilled" when all 11 checklist items pass. Item 3
|
||||
(cross-modal eval) is informational in v1.1.0 — it does not gate the
|
||||
skillpack-check audit, but a missing or stale receipt is surfaced so the
|
||||
user knows where the gate stands.
|
||||
|
||||
1. `SKILL.md` — skill file with YAML frontmatter, triggers, contract, phases.
|
||||
2. Code — deterministic script if applicable.
|
||||
3. Unit tests — cover every branch of deterministic logic.
|
||||
4. Integration tests — exercise live endpoints, not just in-memory shape.
|
||||
5. LLM evals — quality/correctness cases if the feature includes any LLM call.
|
||||
6. Resolver trigger — `skills/RESOLVER.md` entry with the trigger patterns
|
||||
the user actually types.
|
||||
7. Resolver trigger eval — test that feeds trigger phrases to the resolver
|
||||
and asserts they route to this skill, not the old pre-skillify path.
|
||||
8. Check-resolvable — `gbrain check-resolvable` passes (skill is reachable,
|
||||
MECE against its siblings, no DRY violations).
|
||||
9. E2E test — exercises the full pipeline from user turn to side effect.
|
||||
10. Brain filing — if the feature writes brain pages, `brain/RESOLVER.md`
|
||||
has an entry for the directory so the pages aren't orphaned.
|
||||
## The Checklist
|
||||
|
||||
## Trigger
|
||||
|
||||
- "skillify this" / "skillify" / "is this a skill?" / "make this proper"
|
||||
- "add tests and evals for this"
|
||||
- After building any new feature that touches user-facing behavior
|
||||
- When you grep the repo and notice a script with no SKILL.md next to it
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Audit what exists
|
||||
|
||||
For the feature being skillified, answer:
|
||||
|
||||
- **Feature name**: what does it do in one line?
|
||||
- **Code path**: where does the implementation live (file path)?
|
||||
- **Checklist status**: run `gbrain skillify check <path>` (preferred)
|
||||
or the legacy `scripts/skillify-check.ts <path>` shim. Both produce
|
||||
the same 10-item scorecard. Note which items are missing.
|
||||
|
||||
### Phase 2: Create missing pieces in order
|
||||
|
||||
**Fast path — brand-new skill:** run `gbrain skillify scaffold <name>
|
||||
--description "..." [--triggers "p1,p2,p3"] [--writes-pages --writes-to
|
||||
"people/,companies/"]`. This creates all 5 stub files atomically and
|
||||
appends an idempotent resolver row. Every scaffolded file carries the
|
||||
`SKILLIFY_STUB` sentinel; `gbrain check-resolvable --strict` will fail
|
||||
CI until you replace the stubs with real content.
|
||||
|
||||
**Manual path — extending an existing skill:** work the list top-down.
|
||||
Each earlier item constrains what later items look like (the SKILL.md
|
||||
contract determines what tests assert; tests determine what evals gate;
|
||||
the resolver entry determines what trigger-eval checks).
|
||||
|
||||
1. Write `SKILL.md` first. Frontmatter must include `name`, `version`,
|
||||
`description`, `triggers[]`, `tools[]`, `mutating`. Body has at minimum
|
||||
Contract, Phases, and Output Format sections.
|
||||
2. Extract deterministic code into a script if applicable (scripts/*.ts
|
||||
for gbrain; host projects may use .mjs / .py / whatever their runtime
|
||||
uses).
|
||||
3. Write unit tests for every branch of the script. Mock external calls
|
||||
(LLM, DB, network) so tests run fast and deterministic.
|
||||
4. Add integration tests that hit real endpoints. These catch bugs the
|
||||
unit tests' mocks hide (see the `files-test-reimplements-production`
|
||||
learning: reimplementation in tests lets production vulnerabilities
|
||||
slip through).
|
||||
5. Add LLM evals if the feature includes any LLM call. Even a three-case
|
||||
eval (happy / edge / adversarial) is cheap insurance against prompt
|
||||
regressions.
|
||||
6. Add the resolver trigger to `skills/RESOLVER.md`. Use the trigger
|
||||
patterns the user ACTUALLY types, not what you think they should type.
|
||||
7. Add a resolver trigger eval that feeds those patterns in and asserts
|
||||
they route to the new skill.
|
||||
8. Run `gbrain check-resolvable` (auto-detects skill trees) or
|
||||
`gbrain check-resolvable --skills-dir <path>` for custom locations.
|
||||
OpenClaw workspaces are auto-detected from
|
||||
`~/.openclaw/workspace/skills/`. The check validates reachability (is
|
||||
the skill mentioned from RESOLVER.md?), MECE overlap (does it duplicate
|
||||
an existing skill's trigger?), gap detection (are there user intents
|
||||
that fall through the resolver with no match?), and DRY. If it fails,
|
||||
fix the skill (or extend an existing one instead of creating a
|
||||
duplicate).
|
||||
9. Add an E2E smoke test. For gbrain: submit a Minion job or run a CLI
|
||||
invocation end-to-end against a fixture brain; assert side effects.
|
||||
10. Update `brain/RESOLVER.md` if the skill writes brain pages. Orphaned
|
||||
brain pages are worse than no brain pages.
|
||||
|
||||
### Phase 3: Verify
|
||||
|
||||
Run each of these and confirm green:
|
||||
|
||||
```bash
|
||||
# Unit tests
|
||||
bun test test/<skill-name>.test.ts
|
||||
|
||||
# Integration tests (when applicable)
|
||||
bun run test:e2e
|
||||
|
||||
# Resolver reachability + MECE + DRY
|
||||
gbrain check-resolvable
|
||||
|
||||
# Conformance tests (skill YAML + required sections)
|
||||
bun test test/skills-conformance.test.ts
|
||||
```
|
||||
□ 1. SKILL.md — skill file with frontmatter + contract + phases
|
||||
□ 2. Code — deterministic script if applicable
|
||||
□ 3. Cross-modal eval — 3 frontier models from 3 providers; informational
|
||||
□ 4. Unit tests — cover every branch of deterministic logic
|
||||
□ 5. Integration tests — exercise live endpoints
|
||||
□ 6. LLM evals — quality/correctness cases for LLM-involving steps
|
||||
□ 7. Resolver trigger — entry in skills/RESOLVER.md with real user trigger phrases
|
||||
□ 8. Resolver eval — test that triggers route to this skill
|
||||
□ 9. Check-resolvable — DRY + MECE audit, no orphans
|
||||
□ 10. E2E test — smoke test: trigger → side effect
|
||||
□ 11. Brain filing — if it writes pages, entry in brain/RESOLVER.md
|
||||
```
|
||||
|
||||
## Quality gates
|
||||
## Phase 0: Should This Be a Skill?
|
||||
|
||||
A feature is NOT properly skilled until:
|
||||
Before skillifying, check:
|
||||
- Will this be invoked 2+ times? (One-off work ≠ skill)
|
||||
- Is there >20 lines of logic? (Trivial helpers don't need full infrastructure)
|
||||
- Does it have a clear trigger phrase a user would actually say?
|
||||
|
||||
- All tests pass (unit + integration + evals).
|
||||
- It appears in `skills/RESOLVER.md` with accurate trigger patterns.
|
||||
- The resolver trigger eval confirms patterns route to the new skill.
|
||||
- `gbrain check-resolvable` shows no orphaned skills, no MECE overlaps,
|
||||
no DRY violations.
|
||||
- If it writes brain pages, `brain/RESOLVER.md` has the directory.
|
||||
If no to all three, it's a script, not a skill. Move on.
|
||||
|
||||
## Anti-Patterns
|
||||
## Phase 1: Audit
|
||||
|
||||
- ❌ Code with no SKILL.md — invisible to the resolver; the agent will
|
||||
never run it.
|
||||
- ❌ SKILL.md with no tests — untested contract; one prompt change
|
||||
regresses silently.
|
||||
- ❌ Tests that reimplement production code — the reimplementation's
|
||||
bugs don't catch production's bugs (the `files-test-reimplements-
|
||||
production` lesson).
|
||||
- ❌ Resolver entry that uses internal jargon the user never types —
|
||||
trigger patterns must mirror real user language.
|
||||
- ❌ Feature that writes to brain without a `brain/RESOLVER.md` entry —
|
||||
orphaned pages the agent will never find.
|
||||
- ❌ Deterministic logic in LLM space — should be a script.
|
||||
- ❌ LLM judgment in deterministic space — should be an eval.
|
||||
```
|
||||
Feature: [name]
|
||||
Code: [path]
|
||||
Missing items: [check each of the 11]
|
||||
```
|
||||
|
||||
## Why skillify + check-resolvable is the right pair
|
||||
## Phase 2: Write SKILL.md + Code (items 1-2)
|
||||
|
||||
Hermes and similar agent frameworks auto-create skills as a background
|
||||
behavior. That's fine until you don't know what the agent shipped —
|
||||
checklists decay, tests drift, resolver entries get stale.
|
||||
### SKILL.md frontmatter template (copy-paste):
|
||||
|
||||
Gbrain ships the same capability as two user-controlled tools:
|
||||
```yaml
|
||||
---
|
||||
name: my-skill
|
||||
version: 1.0.0
|
||||
description: |
|
||||
One paragraph. What it does, when to use it.
|
||||
triggers:
|
||||
- "trigger phrase users actually say"
|
||||
- "another real trigger"
|
||||
tools:
|
||||
- exec
|
||||
- read
|
||||
- write
|
||||
mutating: false # true if it writes to brain/disk
|
||||
---
|
||||
```
|
||||
|
||||
- `/skillify` builds the checklist and helps you fill in the gaps.
|
||||
- `gbrain check-resolvable` validates the whole skill tree: reachability,
|
||||
MECE, DRY, gap detection, orphaned skills.
|
||||
Body must include: **Contract** (what it guarantees), **Phases** (step-by-step), **Output Format** (what it produces).
|
||||
|
||||
You decide when and what. The human keeps judgment. The tooling keeps the
|
||||
checklist honest. In practice this combo produces zero orphaned skills,
|
||||
every feature with tests + evals + resolver triggers + evals of the
|
||||
triggers.
|
||||
Extract deterministic code into `scripts/*.ts`.
|
||||
|
||||
## Phase 3: Cross-Modal Eval (item 3) — THE QUALITY GATE
|
||||
|
||||
### Why this comes before tests
|
||||
|
||||
Tests lock in behavior. If the behavior is mediocre, tests lock in mediocrity.
|
||||
Cross-modal eval proves the quality bar FIRST, then tests cement it.
|
||||
|
||||
### Step 1: Pick a representative input
|
||||
|
||||
Choose the input that exercises the skill's hardest documented use case. If
|
||||
unsure: use the primary trigger example from SKILL.md, or the most complex
|
||||
real-world input from the last 7 days of memory files.
|
||||
|
||||
### Step 2: Run the skill, capture output
|
||||
|
||||
Run the skill on the representative input. The OUTPUT FILE is what gets
|
||||
evaluated.
|
||||
|
||||
### Step 3: Run the eval gate
|
||||
|
||||
```bash
|
||||
gbrain eval cross-modal \
|
||||
--task "What this skill is supposed to accomplish" \
|
||||
--output skills/<slug>/SKILL.md
|
||||
```
|
||||
|
||||
The command runs 3 frontier models from 3 different providers in parallel,
|
||||
scores the OUTPUT against the TASK on 5 documented dimensions, and writes a
|
||||
receipt under `~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json` (the
|
||||
sha-8 binds the receipt to the current SKILL.md content — re-running after
|
||||
edits writes a new receipt).
|
||||
|
||||
**Default models** (override per slot via `--slot-a-model`, `--slot-b-model`,
|
||||
`--slot-c-model`):
|
||||
|
||||
| Slot | Default | Provider |
|
||||
|------|---------|----------|
|
||||
| A | `openai:gpt-4o` | OpenAI |
|
||||
| B | `anthropic:claude-opus-4-7` | Anthropic |
|
||||
| C | `google:gemini-1.5-pro` | Google |
|
||||
|
||||
**These MUST be frontier models from DIFFERENT providers.** Using a single
|
||||
provider's family or budget models defeats the purpose — different families
|
||||
have less correlated blind spots. Refresh the list when a new model
|
||||
generation ships.
|
||||
|
||||
**Pass criteria (BOTH must be true):**
|
||||
|
||||
1. Every dimension's mean across successful models ≥ 7.
|
||||
2. No single model scored any dimension < 5 (the floor).
|
||||
|
||||
**Inconclusive:** fewer than 2 of 3 models returned parseable scores.
|
||||
Receipt is still written (forensics) but the gate is not authoritative.
|
||||
Exit code 2; CI wrappers should treat this as "did not run cleanly", not
|
||||
"failed quality gate".
|
||||
|
||||
### Step 4: Cycle until you pass (≤3 cycles)
|
||||
|
||||
```
|
||||
CYCLE 1:
|
||||
Eval → scores + top 10 improvements
|
||||
IF pass: → done, write tests
|
||||
ELSE:
|
||||
Apply top 10 improvements to the actual file
|
||||
Log: which improvements applied, what changed
|
||||
|
||||
CYCLE 2:
|
||||
Re-eval the FIXED output (same 3 models, same dimensions)
|
||||
Compare: before/after scores per dimension (track delta)
|
||||
IF pass: → done, write tests
|
||||
ELSE: apply remaining improvements + new ones
|
||||
|
||||
CYCLE 3 (final):
|
||||
Re-eval
|
||||
IF pass: → ship
|
||||
ELSE: → ship with KNOWN_GAPS section listing:
|
||||
- Which dimensions are still below 7
|
||||
- Which improvements couldn't be resolved
|
||||
- Why (e.g., "would require architectural change")
|
||||
```
|
||||
|
||||
### Cycles + cost guardrails
|
||||
|
||||
- Default `--cycles 3` in TTY, `--cycles 1` in non-TTY (limits scripted
|
||||
bulk spend in CI loops).
|
||||
- The command prints an estimated max-cost-per-cycle from a small pricing
|
||||
constant before each run. Real cost varies with prompt size; treat the
|
||||
estimate as a ceiling for default `--max-tokens 4000`.
|
||||
- A `--budget-usd N` hard cap is a v0.27.x follow-up TODO.
|
||||
|
||||
### Provider configuration
|
||||
|
||||
Models resolve through the gbrain AI gateway. Configure once with:
|
||||
|
||||
```bash
|
||||
gbrain providers test # see what's configured
|
||||
gbrain config # set keys
|
||||
```
|
||||
|
||||
Or set env vars: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`,
|
||||
`GOOGLE_GENERATIVE_AI_API_KEY`, `TOGETHER_API_KEY`, etc. The gateway reads
|
||||
from `~/.gbrain/config.json` plus `process.env`.
|
||||
|
||||
### Cost expectations
|
||||
|
||||
3 cycles × 3 models = 9 frontier calls max per run. With Opus-class +
|
||||
GPT-4o-class + Gemini-1.5-Pro, expect $1–3 per full run on default
|
||||
`--max-tokens 4000`. Receipts include the per-call model identifiers so
|
||||
you can audit retroactively.
|
||||
|
||||
### Skip cross-modal eval when:
|
||||
|
||||
- Output is < 200 tokens (trivial — not worth 9 API calls).
|
||||
- The skill is a thin wrapper around a single API call (one cycle is enough).
|
||||
|
||||
## Phase 4: Tests (items 4-6)
|
||||
|
||||
NOW that eval has proven quality, write tests that lock it in:
|
||||
|
||||
**Unit tests** — every branch of deterministic logic. Mock external calls.
|
||||
**Integration tests** — hit real endpoints. Catch bugs mocks hide.
|
||||
**LLM evals** — quality/correctness for LLM steps. Lighter than cross-modal eval — test specific behaviors.
|
||||
|
||||
## Phase 5: Resolver + Check-Resolvable (items 7-9)
|
||||
|
||||
1. Add to skills/RESOLVER.md with trigger phrases users ACTUALLY type
|
||||
2. Resolver eval: feed triggers, assert correct routing
|
||||
3. Check-resolvable:
|
||||
- Skill reachable from skills/RESOLVER.md (not orphaned)
|
||||
- No MECE overlap with other skills
|
||||
- No DRY violations (shared logic in lib/, not copy-pasted)
|
||||
- No ambiguous trigger routing
|
||||
|
||||
## Phase 6: E2E + Brain Filing (items 10-11)
|
||||
|
||||
- E2E smoke: full pipeline from trigger to side effect
|
||||
- Brain filing: add to brain/RESOLVER.md if the skill writes brain pages
|
||||
|
||||
## Phase 7: Verify
|
||||
|
||||
```bash
|
||||
bun test test/<skill>.test.ts # unit tests
|
||||
gbrain skillify check skills/<slug>/scripts/<slug>.mjs --json | \
|
||||
jq '.[] | .items[] | select(.name | contains("Cross-modal"))'
|
||||
ls ~/.gbrain/.gbrain/eval-receipts/ # receipt landed
|
||||
gbrain check-resolvable --json | jq .ok # resolver clean
|
||||
```
|
||||
|
||||
## Worked Example: Skillifying a "summarize-pr" Feature
|
||||
|
||||
```
|
||||
Phase 0: Yes — invoked weekly, 50+ lines, clear trigger "summarize this PR"
|
||||
Phase 1: Audit → SKILL.md missing, no tests, no resolver entry. Score: 1/11
|
||||
Phase 2: Write SKILL.md + extract script to scripts/summarize-pr.ts
|
||||
Phase 3: Cross-modal eval cycle 1 →
|
||||
GPT-4o: goal=6, depth=5, specificity=4 → "misses file-level diffs"
|
||||
Opus 4.7: goal=7, depth=6, specificity=5 → "no test plan in summary"
|
||||
Gemini 1.5 Pro: goal=6, depth=5, specificity=5 → "template feels generic"
|
||||
Aggregate: goal=6.3 FAIL, depth=5.3 FAIL
|
||||
Top improvements: add file-level changes, include test plan, use PR context
|
||||
→ Apply fixes → Cycle 2: goal=8, depth=7.5, specificity=7 → PASS
|
||||
Phase 4: Write 12 unit tests locking in the improved behavior
|
||||
Phase 5: Add "summarize this PR" trigger to skills/RESOLVER.md
|
||||
Phase 6: E2E test: feed a real PR URL → verify brain page created
|
||||
Phase 7: All green. Score: 11/11
|
||||
```
|
||||
|
||||
## Quality Gates
|
||||
|
||||
NOT properly skilled until:
|
||||
|
||||
- All required items pass (1-2, 4-10; 11 only when applicable).
|
||||
- Cross-modal eval (item 3) has a current receipt OR is explicitly waived
|
||||
with rationale (item 3 is informational; not blocking, but a missing
|
||||
receipt is visible in the audit).
|
||||
- All tests pass (unit + integration + LLM evals).
|
||||
- Resolver entry exists with real trigger phrases.
|
||||
- Check-resolvable shows no orphans, overlaps, or DRY violations.
|
||||
- Brain filing if applicable.
|
||||
|
||||
## Output Format
|
||||
|
||||
A skillify run produces, in order:
|
||||
Skillify produces three durable artifacts per skill:
|
||||
|
||||
1. An audit printout listing which of the 10 items exist and which are
|
||||
missing for the target feature.
|
||||
2. The files created to close each gap (SKILL.md, test files, resolver
|
||||
entries).
|
||||
3. The final `gbrain check-resolvable` output confirming reachability.
|
||||
4. A one-line summary of the resulting skill completeness score (N/10).
|
||||
1. **The skill tree on disk.** `skills/<slug>/SKILL.md`, `scripts/<slug>.mjs`,
|
||||
`routing-eval.jsonl`, plus a `test/<slug>.test.ts` skeleton. Generated by
|
||||
`gbrain skillify scaffold <name>` and refined by the human/agent into a
|
||||
real implementation.
|
||||
2. **A cross-modal eval receipt** at
|
||||
`~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json`. The sha-8 binds the
|
||||
receipt to the current `SKILL.md` content. `gbrain skillify check`
|
||||
surfaces the status (`found` / `stale` / `missing`) as informational.
|
||||
3. **An audit verdict** from `gbrain skillify check`: `properly skilled` |
|
||||
`close — create: <missing items>` | `needs skillify — run /skillify on
|
||||
<target>`. Score is `<passed>/<total>`. Required items gate the verdict;
|
||||
item 11 (cross-modal eval) is informational and never blocks PASS.
|
||||
|
||||
JSON output (`gbrain skillify check --json`) includes the same fields plus
|
||||
the per-item detail string, so agents can route on the structured envelope
|
||||
without parsing prose.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
- ❌ Writing tests before cross-modal eval (locks in mediocrity)
|
||||
- ❌ Using budget models for eval (C student grading A student)
|
||||
- ❌ Using a single provider's family for all 3 slots (correlated blind spots)
|
||||
- ❌ Skipping eval "because the output looks fine" (your judgment isn't 3 models)
|
||||
- ❌ Eval without fix cycle (vanity metrics)
|
||||
- ❌ Code with no SKILL.md (invisible to resolver)
|
||||
- ❌ Tests that reimplement production code (masks real bugs)
|
||||
- ❌ Resolver entry with internal jargon (must mirror real user language)
|
||||
- ❌ Two skills doing the same thing (merge or kill one)
|
||||
- ❌ Running cross-modal eval on trivial outputs (< 200 tokens, not worth 9 API calls)
|
||||
|
||||
+10
@@ -455,6 +455,16 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
return;
|
||||
}
|
||||
|
||||
// `eval cross-modal` is a pure API-call command — no DB, no brain. Bypass
|
||||
// connectEngine entirely so first-run users (no `gbrain init` yet) can
|
||||
// run the quality gate. Mirrors the dream/doctor no-DB pattern but
|
||||
// doesn't even attempt the connect (T3=A in plans/radiant-napping-lerdorf.md).
|
||||
// The handler self-configures the AI gateway from loadConfig() + process.env.
|
||||
if (command === 'eval' && args[0] === 'cross-modal') {
|
||||
const { runEvalCrossModal } = await import('./commands/eval-cross-modal.ts');
|
||||
process.exit(await runEvalCrossModal(args.slice(1)));
|
||||
}
|
||||
|
||||
// All remaining CLI-only commands need a DB connection
|
||||
const engine = await connectEngine();
|
||||
try {
|
||||
|
||||
@@ -0,0 +1,358 @@
|
||||
/**
|
||||
* gbrain eval cross-modal — multi-model quality gate (v0.27.x).
|
||||
*
|
||||
* Three different-provider frontier models score the OUTPUT against the TASK
|
||||
* on a fixed dimension list. Verdict: PASS (exit 0) / FAIL (exit 1) /
|
||||
* INCONCLUSIVE (exit 2; <2/3 model successes).
|
||||
*
|
||||
* Reuses `src/core/ai/gateway.ts` for provider config + auth (T1+T2). Bypasses
|
||||
* `connectEngine()` via the cli.ts no-DB branch (T3=A) so onboarding works
|
||||
* before `gbrain init`. Receipts are bound to (slug, SKILL.md sha-8) so
|
||||
* `gbrain skillify check` can detect stale audits (T10=A).
|
||||
*
|
||||
* Cost guardrails (T11=B):
|
||||
* - Default cycles = 3 in TTY, 1 in non-TTY (limits scripted bulk spend).
|
||||
* - Cost-estimate prints to stderr before each cycle.
|
||||
* - `--budget-usd` hard cap is a v0.27.x follow-up TODO.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync } from 'fs';
|
||||
|
||||
import { gbrainPath, loadConfig } from '../core/config.ts';
|
||||
import { configureGateway, isAvailable } from '../core/ai/gateway.ts';
|
||||
import {
|
||||
DEFAULT_DIMENSIONS,
|
||||
DEFAULT_SLOTS,
|
||||
estimateCost,
|
||||
runEval,
|
||||
} from '../core/cross-modal-eval/runner.ts';
|
||||
import type {
|
||||
ProgressEvent,
|
||||
RunEvalResult,
|
||||
SlotConfig,
|
||||
} from '../core/cross-modal-eval/runner.ts';
|
||||
|
||||
const HELP = `gbrain eval cross-modal — multi-model quality gate
|
||||
|
||||
USAGE:
|
||||
gbrain eval cross-modal --task "<description>" --output <path-or-skill-slug> [flags]
|
||||
|
||||
REQUIRED:
|
||||
--task "..." What the OUTPUT was meant to achieve.
|
||||
--output <path> File whose content gets scored. Pass a skill slug
|
||||
shortcut (e.g. \`--output skills/my-skill/SKILL.md\`)
|
||||
to bind the receipt to that skill (T10).
|
||||
|
||||
FLAGS:
|
||||
--slug <name> Receipt filename slug. Defaults to inferred slug
|
||||
from --output path (skills/<slug>/SKILL.md → <slug>),
|
||||
or a content sha for ad-hoc inputs.
|
||||
--dimensions "d1,d2,..." Comma-separated dimension list. Default: 5 standard
|
||||
dimensions (goal, depth, sourcing, specificity, useful).
|
||||
--cycles N 1-3. Default: 3 in TTY, 1 in non-TTY (T11). Each
|
||||
cycle is 3 model calls; verdict aggregates over them.
|
||||
--slot-a-model <id> Override default 'openai:gpt-4o'.
|
||||
--slot-b-model <id> Override default 'anthropic:claude-opus-4-7'.
|
||||
--slot-c-model <id> Override default 'google:gemini-1.5-pro'.
|
||||
--receipt-dir <path> Default: gbrainPath('eval-receipts').
|
||||
--max-tokens N Output token budget per call. Default: 4000.
|
||||
--json Emit final aggregate as JSON to stdout (progress to stderr).
|
||||
--help, -h Show this help.
|
||||
|
||||
EXIT CODES:
|
||||
0 PASS — every dim mean >=7 AND no model scored any dim <5.
|
||||
1 FAIL — at least one dim mean <7 OR at least one model scored a dim <5.
|
||||
2 INCONCLUSIVE — fewer than 2/3 models returned parseable scores. Receipt
|
||||
is still written for forensics; the gate is not authoritative.
|
||||
|
||||
CONFIGURATION:
|
||||
Models resolve via the gbrain AI gateway. Configure with:
|
||||
gbrain providers test # see what's configured
|
||||
gbrain config # set keys
|
||||
Or set env vars: OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY,
|
||||
TOGETHER_API_KEY, etc. The gateway reads from \`~/.gbrain/config.json\` plus
|
||||
process.env.
|
||||
|
||||
EXAMPLES:
|
||||
gbrain eval cross-modal \\
|
||||
--task "Skillify SKILL.md teaches the 11-item meta-skill checklist" \\
|
||||
--output skills/skillify/SKILL.md
|
||||
|
||||
gbrain eval cross-modal \\
|
||||
--task "PR description sells the value of cross-modal eval" \\
|
||||
--output /tmp/pr-description.md \\
|
||||
--cycles 1
|
||||
`;
|
||||
|
||||
interface ParsedArgs {
|
||||
help: boolean;
|
||||
task?: string;
|
||||
output?: string;
|
||||
slug?: string;
|
||||
dimensions?: string[];
|
||||
cycles?: number;
|
||||
slotAModel?: string;
|
||||
slotBModel?: string;
|
||||
slotCModel?: string;
|
||||
receiptDir?: string;
|
||||
maxTokens?: number;
|
||||
json: boolean;
|
||||
}
|
||||
|
||||
function parseArgs(args: string[]): ParsedArgs {
|
||||
const out: ParsedArgs = { help: false, json: false };
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const arg = args[i]!;
|
||||
const next = args[i + 1];
|
||||
switch (arg) {
|
||||
case '--help':
|
||||
case '-h':
|
||||
out.help = true;
|
||||
break;
|
||||
case '--task':
|
||||
if (next === undefined) break;
|
||||
out.task = next;
|
||||
i++;
|
||||
break;
|
||||
case '--output':
|
||||
if (next === undefined) break;
|
||||
out.output = next;
|
||||
i++;
|
||||
break;
|
||||
case '--slug':
|
||||
if (next === undefined) break;
|
||||
out.slug = next;
|
||||
i++;
|
||||
break;
|
||||
case '--dimensions':
|
||||
if (next === undefined) break;
|
||||
out.dimensions = next.split(',').map(s => s.trim()).filter(Boolean);
|
||||
i++;
|
||||
break;
|
||||
case '--cycles':
|
||||
if (next === undefined) break;
|
||||
out.cycles = parseIntStrict(next);
|
||||
i++;
|
||||
break;
|
||||
case '--slot-a-model':
|
||||
if (next === undefined) break;
|
||||
out.slotAModel = next;
|
||||
i++;
|
||||
break;
|
||||
case '--slot-b-model':
|
||||
if (next === undefined) break;
|
||||
out.slotBModel = next;
|
||||
i++;
|
||||
break;
|
||||
case '--slot-c-model':
|
||||
if (next === undefined) break;
|
||||
out.slotCModel = next;
|
||||
i++;
|
||||
break;
|
||||
case '--receipt-dir':
|
||||
if (next === undefined) break;
|
||||
out.receiptDir = next;
|
||||
i++;
|
||||
break;
|
||||
case '--max-tokens':
|
||||
if (next === undefined) break;
|
||||
out.maxTokens = parseIntStrict(next);
|
||||
i++;
|
||||
break;
|
||||
case '--json':
|
||||
out.json = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function parseIntStrict(s: string): number {
|
||||
const m = String(s).trim();
|
||||
if (!/^\d+$/.test(m)) {
|
||||
throw new Error(`expected positive integer, got: ${s}`);
|
||||
}
|
||||
return parseInt(m, 10);
|
||||
}
|
||||
|
||||
function inferSlugFromOutputPath(path: string): string | undefined {
|
||||
// skills/<slug>/SKILL.md or .../skills/<slug>/...
|
||||
const m = path.replace(/\\/g, '/').match(/(?:^|\/)skills\/([^/]+)\/SKILL\.md$/);
|
||||
return m ? m[1] : undefined;
|
||||
}
|
||||
|
||||
function isTTY(): boolean {
|
||||
return Boolean(process.stdout.isTTY);
|
||||
}
|
||||
|
||||
/**
|
||||
* Configure the AI gateway from `~/.gbrain/config.json` + process.env.
|
||||
*
|
||||
* Mirrors the body of `cli.ts:connectEngine()` minus the DB connect — we call
|
||||
* this from the no-DB branch so the gateway is ready when runEval starts.
|
||||
* Returns true on success; false (and prints a hint) when no config is found.
|
||||
*/
|
||||
function configureGatewayForCli(): boolean {
|
||||
const config = loadConfig();
|
||||
if (!config) {
|
||||
// No config file is fine for the eval command — env vars alone may serve.
|
||||
// We still call configureGateway so gateway recipes can read the env map.
|
||||
configureGateway({
|
||||
embedding_model: undefined,
|
||||
embedding_dimensions: undefined,
|
||||
expansion_model: undefined,
|
||||
chat_model: undefined,
|
||||
chat_fallback_chain: undefined,
|
||||
base_urls: undefined,
|
||||
env: { ...process.env },
|
||||
});
|
||||
return true;
|
||||
}
|
||||
configureGateway({
|
||||
embedding_model: config.embedding_model,
|
||||
embedding_dimensions: config.embedding_dimensions,
|
||||
expansion_model: config.expansion_model,
|
||||
chat_model: config.chat_model,
|
||||
chat_fallback_chain: config.chat_fallback_chain,
|
||||
base_urls: config.provider_base_urls,
|
||||
env: { ...process.env },
|
||||
});
|
||||
return true;
|
||||
}
|
||||
|
||||
export async function runEvalCrossModal(args: string[]): Promise<number> {
|
||||
const parsed = parseArgs(args);
|
||||
|
||||
if (parsed.help) {
|
||||
process.stdout.write(HELP);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!parsed.task) {
|
||||
process.stderr.write('Error: --task "<description>" is required\n\n');
|
||||
process.stderr.write(HELP);
|
||||
return 1;
|
||||
}
|
||||
if (!parsed.output) {
|
||||
process.stderr.write('Error: --output <path> is required\n\n');
|
||||
process.stderr.write(HELP);
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (!existsSync(parsed.output)) {
|
||||
process.stderr.write(`Error: --output path not found: ${parsed.output}\n`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
const outputContent = readFileSync(parsed.output, 'utf-8');
|
||||
if (outputContent.trim().length === 0) {
|
||||
process.stderr.write(`Error: --output file is empty: ${parsed.output}\n`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
const slug = parsed.slug ?? inferSlugFromOutputPath(parsed.output);
|
||||
const cycles = parsed.cycles ?? (isTTY() ? 3 : 1);
|
||||
const dimensions = parsed.dimensions ?? DEFAULT_DIMENSIONS;
|
||||
const receiptDir = parsed.receiptDir ?? gbrainPath('eval-receipts');
|
||||
const maxTokens = parsed.maxTokens ?? 4000;
|
||||
|
||||
const slots: SlotConfig[] = [
|
||||
{ id: 'A', model: parsed.slotAModel ?? DEFAULT_SLOTS[0]!.model },
|
||||
{ id: 'B', model: parsed.slotBModel ?? DEFAULT_SLOTS[1]!.model },
|
||||
{ id: 'C', model: parsed.slotCModel ?? DEFAULT_SLOTS[2]!.model },
|
||||
];
|
||||
|
||||
// Configure the AI gateway. Without this, every chat() call throws
|
||||
// "AI gateway is not configured" because the cli.ts no-DB branch skips
|
||||
// connectEngine (T3=A).
|
||||
configureGatewayForCli();
|
||||
|
||||
// Probe whether the gateway can serve `chat`. If not, we can't run.
|
||||
if (!isAvailable('chat')) {
|
||||
process.stderr.write(
|
||||
'Error: AI gateway has no usable chat provider. ' +
|
||||
'Configure one of OPENAI_API_KEY / ANTHROPIC_API_KEY / GOOGLE_GENERATIVE_AI_API_KEY ' +
|
||||
'in your shell or run `gbrain config` to set keys.\n',
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Cost estimate (T11=B).
|
||||
const cost = estimateCost(slots, cycles, maxTokens);
|
||||
process.stderr.write(
|
||||
`[eval cross-modal] estimated cost: ~$${cost.perCycleUSD.toFixed(2)}/cycle, ` +
|
||||
`~$${cost.perRunMaxUSD.toFixed(2)} max for ${cycles} cycle(s).\n`,
|
||||
);
|
||||
for (const note of cost.notes) {
|
||||
process.stderr.write(`[eval cross-modal] note: ${note}\n`);
|
||||
}
|
||||
|
||||
// Progress reporter (stderr only).
|
||||
const onProgress = (ev: ProgressEvent) => {
|
||||
switch (ev.kind) {
|
||||
case 'cycle_start':
|
||||
process.stderr.write(`[eval cross-modal] cycle ${ev.cycle}/${ev.total} starting...\n`);
|
||||
break;
|
||||
case 'slot_done': {
|
||||
const status = ev.ok ? 'ok' : 'failed';
|
||||
process.stderr.write(
|
||||
`[eval cross-modal] slot ${ev.slotId} (${ev.modelId}) ${status} in ${ev.ms}ms\n`,
|
||||
);
|
||||
break;
|
||||
}
|
||||
case 'cycle_end':
|
||||
process.stderr.write(`[eval cross-modal] cycle ${ev.cycle} verdict: ${ev.verdict}\n`);
|
||||
break;
|
||||
}
|
||||
};
|
||||
|
||||
let result: RunEvalResult;
|
||||
try {
|
||||
result = await runEval({
|
||||
task: parsed.task,
|
||||
output: outputContent,
|
||||
slug,
|
||||
dimensions,
|
||||
slots,
|
||||
cycles,
|
||||
receiptDir,
|
||||
maxTokens,
|
||||
onProgress,
|
||||
});
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[eval cross-modal] runtime error: ${msg}\n`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Final summary to stderr (always) + JSON to stdout (when --json).
|
||||
const verdict = result.finalAggregate.verdict;
|
||||
process.stderr.write('\n');
|
||||
process.stderr.write(`[eval cross-modal] ${result.finalAggregate.verdictMessage}\n`);
|
||||
process.stderr.write(`[eval cross-modal] receipt: ${result.finalReceiptPath}\n`);
|
||||
|
||||
if (parsed.json) {
|
||||
process.stdout.write(
|
||||
JSON.stringify(
|
||||
{
|
||||
verdict,
|
||||
aggregate: result.finalAggregate,
|
||||
cycles: result.cycles.map(c => ({
|
||||
cycle: c.cycle,
|
||||
receipt_path: c.receipt_path,
|
||||
verdict: c.aggregate.verdict,
|
||||
overall: c.aggregate.overall,
|
||||
})),
|
||||
finalReceiptPath: result.finalReceiptPath,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
process.stdout.write('\n');
|
||||
}
|
||||
|
||||
if (verdict === 'pass') return 0;
|
||||
if (verdict === 'inconclusive') return 2;
|
||||
return 1;
|
||||
}
|
||||
@@ -37,6 +37,14 @@ export async function runEvalCommand(engine: BrainEngine, args: string[]): Promi
|
||||
const { runEvalReplay } = await import('./eval-replay.ts');
|
||||
return runEvalReplay(engine, args.slice(1));
|
||||
}
|
||||
if (sub === 'cross-modal') {
|
||||
// No-DB sub-subcommand. The cli.ts dispatcher routes the user-facing
|
||||
// path before connectEngine, so this branch only fires when callers
|
||||
// already have an engine and re-enter via runEvalCommand. Engine is
|
||||
// intentionally unused.
|
||||
const { runEvalCrossModal } = await import('./eval-cross-modal.ts');
|
||||
process.exit(await runEvalCrossModal(args.slice(1)));
|
||||
}
|
||||
|
||||
const opts = parseArgs(args);
|
||||
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
/**
|
||||
* gbrain skillify check — 10-item post-task audit.
|
||||
* gbrain skillify check — 11-item post-task audit.
|
||||
*
|
||||
* Promoted from `scripts/skillify-check.ts` (D-CX-2). The legacy
|
||||
* script stays as a thin shim so existing callers keep working, but
|
||||
* the CLI entry point is now `gbrain skillify check`.
|
||||
*
|
||||
* 10-item checklist (essay Step 3-10):
|
||||
* 11-item checklist (essay Step 3-10 + v0.27.x cross-modal eval):
|
||||
* 1. SKILL.md exists
|
||||
* 2. Code file exists at target path
|
||||
* 3. Unit tests exist
|
||||
@@ -16,12 +16,24 @@
|
||||
* 8. check-resolvable gate (runs `gbrain check-resolvable --json`)
|
||||
* 9. E2E smoke (required copy of #4 for required-gate semantics)
|
||||
* 10. Brain filing (only when the script writes pages)
|
||||
* 11. Cross-modal eval (INFORMATIONAL; required:false). Looks for a
|
||||
* receipt at `gbrainPath('eval-receipts')/<slug>-<sha8>.json`
|
||||
* bound to the current SKILL.md content hash (T10=A,T7=C in
|
||||
* plans/radiant-napping-lerdorf.md). A missing or stale receipt
|
||||
* surfaces as a non-blocking note, not a failure.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync, readdirSync, statSync } from 'fs';
|
||||
import { basename, dirname, join, resolve } from 'path';
|
||||
import { spawnSync } from 'child_process';
|
||||
|
||||
import { gbrainPath } from '../core/config.ts';
|
||||
import {
|
||||
describeReceiptStatus,
|
||||
findReceiptForSkill,
|
||||
type ReceiptStatus,
|
||||
} from '../core/cross-modal-eval/receipt-name.ts';
|
||||
|
||||
interface CheckItem {
|
||||
name: string;
|
||||
passed: boolean;
|
||||
@@ -259,6 +271,18 @@ function runSkillifyCheckTarget(target: string, root: string): CheckResult {
|
||||
),
|
||||
);
|
||||
|
||||
// Item 11: cross-modal eval (informational, T7=C). The receipt is bound
|
||||
// to (slug, sha8 of SKILL.md). The audit doesn't fail on a missing or
|
||||
// stale receipt — it just surfaces the status.
|
||||
const crossModalReceipt = lookupCrossModalReceipt(skillMd, skillName);
|
||||
items.push(
|
||||
checkOptional(
|
||||
'Cross-modal eval (informational)',
|
||||
crossModalReceipt.passed,
|
||||
crossModalReceipt.detail,
|
||||
),
|
||||
);
|
||||
|
||||
const passed = items.filter(i => i.passed).length;
|
||||
const total = items.length;
|
||||
const missing = items.filter(i => !i.passed && i.required).map(i => i.name);
|
||||
@@ -275,6 +299,46 @@ function runSkillifyCheckTarget(target: string, root: string): CheckResult {
|
||||
return { path: target, skillName, items, score: passed, total, recommendation };
|
||||
}
|
||||
|
||||
/**
|
||||
* Item 11 helper: look up the cross-modal eval receipt for this skill.
|
||||
* `passed` is true when a current-sha receipt exists. Stale or missing
|
||||
* receipts return passed=true *for the audit* — item 11 is informational
|
||||
* (T7=C) — but the detail string makes the status visible.
|
||||
*
|
||||
* Reads the receipt from `gbrainPath('eval-receipts')` (T5 correction:
|
||||
* this resolves to <GBRAIN_HOME>/.gbrain/eval-receipts/, NOT the legacy
|
||||
* <GBRAIN_HOME>/eval-receipts/ that the original plan claimed).
|
||||
*/
|
||||
function lookupCrossModalReceipt(
|
||||
skillMdPath: string,
|
||||
skillName: string,
|
||||
): { passed: boolean; detail: string } {
|
||||
if (!existsSync(skillMdPath)) {
|
||||
return { passed: true, detail: 'no SKILL.md — skipping cross-modal eval check' };
|
||||
}
|
||||
let status: ReceiptStatus;
|
||||
try {
|
||||
status = findReceiptForSkill(skillMdPath, gbrainPath('eval-receipts'));
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
return { passed: true, detail: `receipt lookup failed: ${msg}` };
|
||||
}
|
||||
switch (status.status) {
|
||||
case 'found':
|
||||
return { passed: true, detail: describeReceiptStatus(skillName, status) };
|
||||
case 'stale':
|
||||
return {
|
||||
passed: false, // visually marked as not-yet-rerun but item is required:false
|
||||
detail: describeReceiptStatus(skillName, status),
|
||||
};
|
||||
case 'missing':
|
||||
return {
|
||||
passed: false,
|
||||
detail: describeReceiptStatus(skillName, status),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function recentlyModified(root: string, days: number = 7): string[] {
|
||||
const candidates: string[] = [];
|
||||
const cutoff = Date.now() - days * 24 * 60 * 60 * 1000;
|
||||
|
||||
@@ -2,15 +2,17 @@
|
||||
* gbrain skillify <scaffold|check> — W4 CLI namespace.
|
||||
*
|
||||
* `scaffold`: creates 5 stub files for a new skill. Mechanical only.
|
||||
* `check`: 10-item audit of an existing skill. Promoted from
|
||||
* `scripts/skillify-check.ts` (D-CX-2). The legacy script
|
||||
* remains as a thin shim that invokes this subcommand.
|
||||
* `check`: 11-item audit of an existing skill (item 11, cross-modal
|
||||
* eval, is informational; T7=C in plans/radiant-napping-lerdorf.md).
|
||||
* Promoted from `scripts/skillify-check.ts` (D-CX-2). The
|
||||
* legacy script remains as a thin shim that invokes this
|
||||
* subcommand.
|
||||
*
|
||||
* The markdown skill at `skills/skillify/SKILL.md` orchestrates the
|
||||
* full 10-step loop (essay's "skillify it!"): scaffold → fill in the
|
||||
* body → run check → run check-resolvable → run tests → commit.
|
||||
* The CLI primitives do the mechanical steps; the skill carries the
|
||||
* judgment steps.
|
||||
* full 11-step loop (essay's "skillify it!"): scaffold → fill in the
|
||||
* body → run cross-modal eval → run check → run check-resolvable →
|
||||
* run tests → commit. The CLI primitives do the mechanical steps;
|
||||
* the skill carries the judgment steps.
|
||||
*/
|
||||
|
||||
import { isAbsolute, resolve as resolvePath } from 'path';
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
/**
|
||||
* cross-modal-eval/aggregate — verdict logic for one cycle.
|
||||
*
|
||||
* Inputs: per-slot results from the 3 frontier models (each either a parsed
|
||||
* scores object or a captured error). Output: verdict + dim averages +
|
||||
* top improvements + verdict prose.
|
||||
*
|
||||
* Pass criterion (Q2 + Q3):
|
||||
* - At least 2 of 3 model calls succeeded with parseable scores.
|
||||
* - Every dimension's mean across successful models is >= 7.
|
||||
* - For every dimension, no successful model scored < 5 (the floor).
|
||||
*
|
||||
* Inconclusive (Q3): fewer than 2 models succeeded.
|
||||
* `Object.values({}).every(...) === true`, so an empty scores map would
|
||||
* silently PASS without this guard. Test 6 in aggregate.test.ts is the
|
||||
* regression guard.
|
||||
*/
|
||||
|
||||
import type { ParsedModelResult } from './json-repair.ts';
|
||||
|
||||
export type SlotResult =
|
||||
| { ok: true; modelId: string; parsed: ParsedModelResult }
|
||||
| { ok: false; modelId: string; error: string };
|
||||
|
||||
export interface AggregateInput {
|
||||
/** One entry per slot (typically 3). */
|
||||
slots: SlotResult[];
|
||||
}
|
||||
|
||||
export interface DimensionRoll {
|
||||
/** Mean across successful models. */
|
||||
mean: number;
|
||||
/** Minimum across successful models (the floor). */
|
||||
min: number;
|
||||
/** All raw scores from successful models, in slot order. */
|
||||
scores: number[];
|
||||
/** Pass=false reason if this dim fails. */
|
||||
failReason?: 'mean_below_7' | 'min_below_5';
|
||||
}
|
||||
|
||||
export interface AggregateResult {
|
||||
/** Verdict: 'pass' | 'fail' | 'inconclusive' (Q3=A). */
|
||||
verdict: 'pass' | 'fail' | 'inconclusive';
|
||||
/** Number of slots that returned parseable scores. */
|
||||
successes: number;
|
||||
/** Number of slots that errored or returned unparseable output. */
|
||||
failures: number;
|
||||
/** Per-dimension roll-up; undefined if inconclusive. */
|
||||
dimensions: Record<string, DimensionRoll>;
|
||||
/** Mean of dimension means; undefined if inconclusive. */
|
||||
overall: number | undefined;
|
||||
/** Top 10 deduplicated improvements across all successful models. */
|
||||
topImprovements: string[];
|
||||
/** Slot-level error notes (carried through to receipt). */
|
||||
errors: Array<{ modelId: string; error: string }>;
|
||||
/** Human-readable one-liner for stderr / receipt verdict prose. */
|
||||
verdictMessage: string;
|
||||
}
|
||||
|
||||
const PASS_MEAN_THRESHOLD = 7;
|
||||
const PASS_FLOOR_THRESHOLD = 5;
|
||||
const MIN_SUCCESSES_FOR_VERDICT = 2;
|
||||
const TOP_IMPROVEMENTS_CAP = 10;
|
||||
const DEDUP_PREFIX_LEN = 40;
|
||||
|
||||
export function aggregate(input: AggregateInput): AggregateResult {
|
||||
const successes = input.slots.filter(s => s.ok);
|
||||
const failures = input.slots.filter(s => !s.ok) as Array<
|
||||
Extract<SlotResult, { ok: false }>
|
||||
>;
|
||||
const errors = failures.map(f => ({ modelId: f.modelId, error: f.error }));
|
||||
|
||||
if (successes.length < MIN_SUCCESSES_FOR_VERDICT) {
|
||||
return {
|
||||
verdict: 'inconclusive',
|
||||
successes: successes.length,
|
||||
failures: failures.length,
|
||||
dimensions: {},
|
||||
overall: undefined,
|
||||
topImprovements: [],
|
||||
errors,
|
||||
verdictMessage:
|
||||
`INCONCLUSIVE: only ${successes.length} of ${input.slots.length} models returned ` +
|
||||
`parseable scores (need >=${MIN_SUCCESSES_FOR_VERDICT}). See receipt for per-slot errors.`,
|
||||
};
|
||||
}
|
||||
|
||||
// Roll up per-dimension across successful slots.
|
||||
const dimensions: Record<string, DimensionRoll> = {};
|
||||
const allDimNames = new Set<string>();
|
||||
for (const s of successes) {
|
||||
if (s.ok) {
|
||||
for (const dim of Object.keys(s.parsed.scores)) {
|
||||
allDimNames.add(dim);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const dim of allDimNames) {
|
||||
const scores: number[] = [];
|
||||
for (const s of successes) {
|
||||
if (s.ok) {
|
||||
const entry = s.parsed.scores[dim];
|
||||
if (entry && Number.isFinite(entry.score)) scores.push(entry.score);
|
||||
}
|
||||
}
|
||||
if (scores.length === 0) continue;
|
||||
const mean = scores.reduce((a, b) => a + b, 0) / scores.length;
|
||||
const min = Math.min(...scores);
|
||||
const roll: DimensionRoll = { mean: round1(mean), min, scores };
|
||||
if (roll.mean < PASS_MEAN_THRESHOLD) roll.failReason = 'mean_below_7';
|
||||
else if (roll.min < PASS_FLOOR_THRESHOLD) roll.failReason = 'min_below_5';
|
||||
dimensions[dim] = roll;
|
||||
}
|
||||
|
||||
const dimRolls = Object.values(dimensions);
|
||||
const overall =
|
||||
dimRolls.length > 0
|
||||
? round1(dimRolls.reduce((a, b) => a + b.mean, 0) / dimRolls.length)
|
||||
: 0;
|
||||
const allDimsPass = dimRolls.every(d => !d.failReason);
|
||||
const verdict: 'pass' | 'fail' = allDimsPass ? 'pass' : 'fail';
|
||||
|
||||
const topImprovements = dedupImprovements(
|
||||
successes.flatMap(s => (s.ok ? s.parsed.improvements : [])),
|
||||
).slice(0, TOP_IMPROVEMENTS_CAP);
|
||||
|
||||
const verdictMessage =
|
||||
verdict === 'pass'
|
||||
? `PASS: every dimension mean >=${PASS_MEAN_THRESHOLD} and min >=${PASS_FLOOR_THRESHOLD} ` +
|
||||
`across ${successes.length}/${input.slots.length} models. Overall ${overall}/10.`
|
||||
: describeFailure(dimensions, successes.length, input.slots.length, overall);
|
||||
|
||||
return {
|
||||
verdict,
|
||||
successes: successes.length,
|
||||
failures: failures.length,
|
||||
dimensions,
|
||||
overall,
|
||||
topImprovements,
|
||||
errors,
|
||||
verdictMessage,
|
||||
};
|
||||
}
|
||||
|
||||
function describeFailure(
|
||||
dimensions: Record<string, DimensionRoll>,
|
||||
successes: number,
|
||||
total: number,
|
||||
overall: number,
|
||||
): string {
|
||||
const failed = Object.entries(dimensions).filter(([, d]) => d.failReason);
|
||||
if (failed.length === 0) {
|
||||
return `FAIL: aggregate failure with no dimension flagged (likely zero dimensions returned).`;
|
||||
}
|
||||
const reasons = failed
|
||||
.map(([name, d]) => {
|
||||
if (d.failReason === 'mean_below_7') {
|
||||
return `${name} mean=${d.mean} (<${PASS_MEAN_THRESHOLD})`;
|
||||
}
|
||||
return `${name} min=${d.min} (<${PASS_FLOOR_THRESHOLD}; scores=[${d.scores.join(', ')}])`;
|
||||
})
|
||||
.join('; ');
|
||||
return `FAIL across ${successes}/${total} models. Overall ${overall}/10. Failing: ${reasons}.`;
|
||||
}
|
||||
|
||||
function dedupImprovements(items: string[]): string[] {
|
||||
const seen = new Set<string>();
|
||||
const out: string[] = [];
|
||||
for (const item of items) {
|
||||
const key = item.slice(0, DEDUP_PREFIX_LEN).toLowerCase().replace(/\s+/g, ' ').trim();
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
out.push(item);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function round1(n: number): number {
|
||||
return Math.round(n * 10) / 10;
|
||||
}
|
||||
@@ -0,0 +1,158 @@
|
||||
/**
|
||||
* cross-modal-eval/json-repair — best-effort JSON parser for LLM output.
|
||||
*
|
||||
* Frontier models routinely return:
|
||||
* - Plain JSON
|
||||
* - JSON wrapped in ```json fences
|
||||
* - JSON with trailing commas before } or ]
|
||||
* - JSON with embedded newlines inside strings
|
||||
* - JSON with single quotes used as string delimiters
|
||||
*
|
||||
* Four-strategy fallback chain. The "nuclear option" extracts scores via
|
||||
* regex when none of the above parses succeed; if even that fails to find
|
||||
* any dimension scores, we throw rather than fabricate.
|
||||
*
|
||||
* The aggregator (aggregate.ts) treats a throw here as "this model
|
||||
* contributed nothing this cycle" — the model is excluded from the verdict
|
||||
* but the gate can still PASS at >=2/3 successes.
|
||||
*/
|
||||
|
||||
export interface ParsedScore {
|
||||
score: number;
|
||||
feedback?: string;
|
||||
}
|
||||
|
||||
export interface ParsedModelResult {
|
||||
scores: Record<string, ParsedScore>;
|
||||
overall?: number;
|
||||
improvements: string[];
|
||||
/** True when the result was reconstructed via the regex nuclear option. */
|
||||
_repaired?: boolean;
|
||||
}
|
||||
|
||||
const FENCE_RE = /```(?:json)?\s*\n?([\s\S]*?)```/i;
|
||||
|
||||
export function parseModelJSON(raw: string): ParsedModelResult {
|
||||
if (typeof raw !== 'string' || !raw.trim()) {
|
||||
throw new Error('parseModelJSON: empty or non-string input');
|
||||
}
|
||||
|
||||
// Strategy 1: strip markdown fences if present, then JSON.parse.
|
||||
const cleaned = stripFences(raw).trim();
|
||||
const direct = tryParse(cleaned);
|
||||
if (direct) return shape(direct);
|
||||
|
||||
// Strategy 2: extract the first {...} object substring.
|
||||
const match = cleaned.match(/\{[\s\S]*\}/);
|
||||
if (!match) {
|
||||
throw new Error('parseModelJSON: no JSON object found in input');
|
||||
}
|
||||
const obj = match[0];
|
||||
|
||||
const second = tryParse(obj);
|
||||
if (second) return shape(second);
|
||||
|
||||
// Strategy 3: repair common LLM-JSON mistakes.
|
||||
const fixed = repairJson(obj);
|
||||
const third = tryParse(fixed);
|
||||
if (third) return shape(third);
|
||||
|
||||
// Strategy 4: nuclear option — regex-extract scores + improvements.
|
||||
const reconstructed = regexNuclearOption(obj);
|
||||
if (reconstructed) return reconstructed;
|
||||
|
||||
throw new Error('parseModelJSON: all repair strategies failed');
|
||||
}
|
||||
|
||||
function stripFences(s: string): string {
|
||||
const m = s.match(FENCE_RE);
|
||||
return m ? m[1]! : s;
|
||||
}
|
||||
|
||||
function tryParse(s: string): unknown | null {
|
||||
try {
|
||||
return JSON.parse(s);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function repairJson(s: string): string {
|
||||
return (
|
||||
s
|
||||
// Trailing commas before } or ]
|
||||
.replace(/,(\s*[}\]])/g, '$1')
|
||||
// Single-quoted string values used as delimiters around keys/values
|
||||
// (only between structural punctuation, to avoid touching apostrophes
|
||||
// inside legitimate double-quoted strings).
|
||||
.replace(/(?<=[:{,\[]\s*)'([^']*?)'(?=\s*[,}\]:])/g, '"$1"')
|
||||
// Unescaped newlines inside double-quoted strings — replace with \n.
|
||||
.replace(/("(?:[^"\\]|\\.)*?)\n((?:[^"\\]|\\.)*?")/g, '$1\\n$2')
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Last-resort: scan for `"<dim>": { ... "score": N }` patterns and any
|
||||
* numbered `"N. ..."` improvement strings. Throws if zero scores are
|
||||
* recoverable (better than fabricating a fake PASS).
|
||||
*/
|
||||
function regexNuclearOption(obj: string): ParsedModelResult | null {
|
||||
const scores: Record<string, ParsedScore> = {};
|
||||
const scoreRe = /["']?(\w[\w_-]*)["']?\s*:\s*\{[^}]*?["']?score["']?\s*:\s*(\d+(?:\.\d+)?)/g;
|
||||
for (const m of obj.matchAll(scoreRe)) {
|
||||
const dim = m[1]!;
|
||||
const num = Number(m[2]);
|
||||
if (Number.isFinite(num)) scores[dim] = { score: num };
|
||||
}
|
||||
|
||||
if (Object.keys(scores).length === 0) return null;
|
||||
|
||||
const improvements: string[] = [];
|
||||
const impRe = /"(\d+\.\s[^"]{10,})"/g;
|
||||
for (const m of obj.matchAll(impRe)) {
|
||||
improvements.push(m[1]!);
|
||||
}
|
||||
|
||||
const overallMatch = obj.match(/["']?overall["']?\s*:\s*(\d+(?:\.\d+)?)/);
|
||||
return {
|
||||
scores,
|
||||
overall: overallMatch ? Number(overallMatch[1]) : undefined,
|
||||
improvements:
|
||||
improvements.length > 0
|
||||
? improvements
|
||||
: ['(could not parse improvements from malformed JSON)'],
|
||||
_repaired: true,
|
||||
};
|
||||
}
|
||||
|
||||
function shape(parsed: unknown): ParsedModelResult {
|
||||
if (!parsed || typeof parsed !== 'object') {
|
||||
throw new Error('parseModelJSON: parsed value is not an object');
|
||||
}
|
||||
const p = parsed as Record<string, unknown>;
|
||||
const scoresRaw = (p.scores as Record<string, unknown>) ?? {};
|
||||
const scores: Record<string, ParsedScore> = {};
|
||||
for (const [dim, v] of Object.entries(scoresRaw)) {
|
||||
if (typeof v === 'number') {
|
||||
scores[dim] = { score: v };
|
||||
} else if (v && typeof v === 'object') {
|
||||
const vv = v as Record<string, unknown>;
|
||||
const score = typeof vv.score === 'number' ? vv.score : Number(vv.score);
|
||||
if (!Number.isFinite(score)) continue;
|
||||
const feedback = typeof vv.feedback === 'string' ? vv.feedback : undefined;
|
||||
scores[dim] = { score, feedback };
|
||||
}
|
||||
}
|
||||
|
||||
const improvements = Array.isArray(p.improvements)
|
||||
? (p.improvements as unknown[]).filter((x): x is string => typeof x === 'string')
|
||||
: [];
|
||||
|
||||
const overall = typeof p.overall === 'number' ? p.overall : undefined;
|
||||
|
||||
if (Object.keys(scores).length === 0) {
|
||||
throw new Error('parseModelJSON: parsed object has no usable scores');
|
||||
}
|
||||
|
||||
return { scores, overall, improvements };
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
/**
|
||||
* cross-modal-eval/receipt-name — bind a receipt to a specific skill version.
|
||||
*
|
||||
* Receipt filenames embed a SHA-8 of the SKILL.md content, so the audit can
|
||||
* tell whether the receipt corresponds to the *current* version of the skill
|
||||
* (T10=A). Filename pattern:
|
||||
*
|
||||
* <skill-slug>-<sha8>.json
|
||||
*
|
||||
* findReceiptForSkill returns one of:
|
||||
* - { status: 'found', path } — receipt matches current SKILL.md
|
||||
* - { status: 'stale', latestPath, sha } — receipt(s) exist for older versions
|
||||
* - { status: 'missing' } — no receipt for this skill
|
||||
*
|
||||
* Pure functions — no fs writes (the writer is in receipt-write.ts). The
|
||||
* skillify-check audit and the runner share these helpers so naming stays in
|
||||
* one place.
|
||||
*/
|
||||
|
||||
import { createHash } from 'crypto';
|
||||
import { existsSync, readdirSync, readFileSync, statSync } from 'fs';
|
||||
import { basename, join } from 'path';
|
||||
|
||||
export type ReceiptStatus =
|
||||
| { status: 'found'; path: string; sha: string }
|
||||
| { status: 'stale'; latestPath: string; latestSha: string; currentSha: string }
|
||||
| { status: 'missing'; currentSha: string };
|
||||
|
||||
/**
|
||||
* SHA-256 of skill content, truncated to 8 hex chars. 16M-receipt collision
|
||||
* space per slug is more than enough; the receipts are owned by one user.
|
||||
*/
|
||||
export function sha8(content: string): string {
|
||||
return createHash('sha256').update(content, 'utf8').digest('hex').slice(0, 8);
|
||||
}
|
||||
|
||||
/**
|
||||
* Generate the canonical receipt filename for a (slug, content) pair.
|
||||
* Returned as a bare filename (no directory), so the caller controls layout.
|
||||
*/
|
||||
export function receiptName(slug: string, content: string): string {
|
||||
if (!slug || typeof slug !== 'string') throw new Error('receiptName: slug required');
|
||||
if (!/^[a-z0-9][a-z0-9_-]*$/i.test(slug)) {
|
||||
throw new Error(`receiptName: slug must be alphanumeric/dash/underscore; got: ${slug}`);
|
||||
}
|
||||
return `${slug}-${sha8(content)}.json`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the SKILL.md at `skillPath` (or return null when missing) and look in
|
||||
* `receiptDir` for any receipt matching the slug embedded in skillPath.
|
||||
*/
|
||||
export function findReceiptForSkill(skillMdPath: string, receiptDir: string): ReceiptStatus {
|
||||
if (!existsSync(skillMdPath)) {
|
||||
return { status: 'missing', currentSha: '' };
|
||||
}
|
||||
|
||||
const slug = inferSlugFromSkillPath(skillMdPath);
|
||||
const content = readFileSync(skillMdPath, 'utf-8');
|
||||
const currentSha = sha8(content);
|
||||
const expectedName = `${slug}-${currentSha}.json`;
|
||||
const expectedPath = join(receiptDir, expectedName);
|
||||
|
||||
if (existsSync(expectedPath)) {
|
||||
return { status: 'found', path: expectedPath, sha: currentSha };
|
||||
}
|
||||
|
||||
if (!existsSync(receiptDir)) {
|
||||
return { status: 'missing', currentSha };
|
||||
}
|
||||
|
||||
// Look for stale receipts (same slug, different sha).
|
||||
const prefix = `${slug}-`;
|
||||
const matches: Array<{ path: string; sha: string; mtime: number }> = [];
|
||||
for (const entry of readdirSync(receiptDir)) {
|
||||
if (!entry.startsWith(prefix) || !entry.endsWith('.json')) continue;
|
||||
const sha = entry.slice(prefix.length, -'.json'.length);
|
||||
if (sha === currentSha) continue;
|
||||
if (!/^[0-9a-f]{8}$/i.test(sha)) continue;
|
||||
const path = join(receiptDir, entry);
|
||||
try {
|
||||
const mtime = statSync(path).mtimeMs;
|
||||
matches.push({ path, sha, mtime });
|
||||
} catch {
|
||||
// Skip files we can't stat — they'll be missing-by-effect.
|
||||
}
|
||||
}
|
||||
if (matches.length === 0) return { status: 'missing', currentSha };
|
||||
|
||||
matches.sort((a, b) => b.mtime - a.mtime);
|
||||
const latest = matches[0]!;
|
||||
return {
|
||||
status: 'stale',
|
||||
latestPath: latest.path,
|
||||
latestSha: latest.sha,
|
||||
currentSha,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Pull the slug out of a SKILL.md path. We accept:
|
||||
* - skills/<slug>/SKILL.md
|
||||
* - <skills-root>/<slug>/SKILL.md
|
||||
* - <slug>/SKILL.md (relative)
|
||||
* The slug is the immediate parent directory name.
|
||||
*/
|
||||
export function inferSlugFromSkillPath(skillMdPath: string): string {
|
||||
const parts = skillMdPath.replace(/\\/g, '/').split('/');
|
||||
const last = parts[parts.length - 1];
|
||||
if (last !== 'SKILL.md') {
|
||||
throw new Error(
|
||||
`inferSlugFromSkillPath: expected path ending in SKILL.md; got: ${skillMdPath}`,
|
||||
);
|
||||
}
|
||||
const parent = parts[parts.length - 2];
|
||||
if (!parent) {
|
||||
throw new Error(
|
||||
`inferSlugFromSkillPath: cannot infer slug — no parent directory in: ${skillMdPath}`,
|
||||
);
|
||||
}
|
||||
return parent;
|
||||
}
|
||||
|
||||
export function describeReceiptStatus(slug: string, status: ReceiptStatus): string {
|
||||
switch (status.status) {
|
||||
case 'found':
|
||||
return `cross-modal eval receipt found for ${slug} (sha ${status.sha}; matches current SKILL.md)`;
|
||||
case 'stale':
|
||||
return (
|
||||
`cross-modal eval receipt for ${slug} exists for an older SKILL.md ` +
|
||||
`(receipt sha ${status.latestSha}, current sha ${status.currentSha}). ` +
|
||||
`Re-run \`gbrain eval cross-modal\` against the current skill output.`
|
||||
);
|
||||
case 'missing':
|
||||
return `no cross-modal eval receipt for ${slug} yet — run \`gbrain eval cross-modal\` to add one`;
|
||||
}
|
||||
}
|
||||
|
||||
/** For tests + tools: pull all receipts for a slug, ordered newest first. */
|
||||
export function listReceiptsForSlug(slug: string, receiptDir: string): string[] {
|
||||
if (!existsSync(receiptDir)) return [];
|
||||
const prefix = `${slug}-`;
|
||||
const out: Array<{ path: string; mtime: number }> = [];
|
||||
for (const entry of readdirSync(receiptDir)) {
|
||||
if (!entry.startsWith(prefix) || !entry.endsWith('.json')) continue;
|
||||
const path = join(receiptDir, entry);
|
||||
try {
|
||||
out.push({ path, mtime: statSync(path).mtimeMs });
|
||||
} catch {
|
||||
// Skip unreadable.
|
||||
}
|
||||
}
|
||||
out.sort((a, b) => b.mtime - a.mtime);
|
||||
return out.map(o => o.path);
|
||||
}
|
||||
|
||||
/** Used by skillify-check to fall back to basename matching when needed. */
|
||||
export function isReceiptFile(path: string): boolean {
|
||||
const name = basename(path);
|
||||
return /^[a-z0-9][a-z0-9_-]*-[0-9a-f]{8}\.json$/i.test(name);
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
/**
|
||||
* cross-modal-eval/receipt-write — auto-mkdir receipt writer.
|
||||
*
|
||||
* `gbrainPath()` from `src/core/config.ts` does NOT auto-mkdir (Codex T5
|
||||
* correction). Every receipt write needs an explicit `mkdirSync({recursive})`
|
||||
* ahead of the write so first-run users don't get `ENOENT: no such file or
|
||||
* directory` from a fresh `~/.gbrain/`.
|
||||
*/
|
||||
|
||||
import { mkdirSync, writeFileSync } from 'fs';
|
||||
import { dirname } from 'path';
|
||||
|
||||
export function writeReceipt(path: string, content: string | object): void {
|
||||
const body = typeof content === 'string' ? content : JSON.stringify(content, null, 2);
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
writeFileSync(path, body, 'utf-8');
|
||||
}
|
||||
@@ -0,0 +1,357 @@
|
||||
/**
|
||||
* cross-modal-eval/runner — orchestrate one or more eval cycles.
|
||||
*
|
||||
* Each cycle: 3 different-provider models score the OUTPUT against the TASK
|
||||
* on a fixed dimension list. `Promise.allSettled` so a single-provider 5xx
|
||||
* doesn't kill the cycle (T4=A — bare allSettled, no rate-leases for the
|
||||
* CLI path; future minion-integration TODO recovers cross-process
|
||||
* concurrency control).
|
||||
*
|
||||
* Pass / FAIL / INCONCLUSIVE verdict per `aggregate()`. The receipt schema
|
||||
* (schema_version: 1) is stable: timestamps, model strings, raw scores, and
|
||||
* dim rolls. Receipt filename binds skill slug + content sha-8 (T10=A) so
|
||||
* `gbrain skillify check` can tell whether a receipt is current or stale.
|
||||
*/
|
||||
|
||||
import { join } from 'path';
|
||||
|
||||
import { chat as gwChat } from '../ai/gateway.ts';
|
||||
import type { ChatMessage } from '../ai/gateway.ts';
|
||||
import { aggregate } from './aggregate.ts';
|
||||
import type { AggregateResult, SlotResult } from './aggregate.ts';
|
||||
import { parseModelJSON } from './json-repair.ts';
|
||||
import { receiptName, sha8 } from './receipt-name.ts';
|
||||
import { writeReceipt } from './receipt-write.ts';
|
||||
|
||||
export const RECEIPT_SCHEMA_VERSION = 1;
|
||||
|
||||
/** Default dimensions match the v1.1.0 SKILL.md. */
|
||||
export const DEFAULT_DIMENSIONS: string[] = [
|
||||
'GOAL_ACHIEVEMENT — Does the output actually accomplish what the task asked for?',
|
||||
'DEPTH — Is the output substantive, or surface-level / thin?',
|
||||
'SOURCING — Are claims backed by evidence, links, or citations?',
|
||||
'SPECIFICITY — Are there concrete details, data, quotes, examples?',
|
||||
'USEFULNESS — Would the intended audience find this valuable?',
|
||||
];
|
||||
|
||||
/**
|
||||
* Default 3-provider slot configuration. Implementer should refresh the
|
||||
* model strings alongside model-family bumps in CLAUDE.md.
|
||||
*
|
||||
* The model strings here resolve through `src/core/ai/recipes/`. Each slot
|
||||
* uses a distinct family so blind spots don't correlate. Override via
|
||||
* `--slot-a-model`, `--slot-b-model`, `--slot-c-model` on the CLI.
|
||||
*/
|
||||
export const DEFAULT_SLOTS: SlotConfig[] = [
|
||||
{ id: 'A', model: 'openai:gpt-4o' },
|
||||
{ id: 'B', model: 'anthropic:claude-opus-4-7' },
|
||||
{ id: 'C', model: 'google:gemini-1.5-pro' },
|
||||
];
|
||||
|
||||
export interface SlotConfig {
|
||||
id: string;
|
||||
/** "<provider>:<modelId>" string consumed by gateway.ts:resolveChatProvider. */
|
||||
model: string;
|
||||
}
|
||||
|
||||
export interface RunEvalOpts {
|
||||
task: string;
|
||||
output: string;
|
||||
/** Optional skill slug for receipt naming (T10). Falls back to a content sha. */
|
||||
slug?: string;
|
||||
/** Override default dimensions list. */
|
||||
dimensions?: string[];
|
||||
/** Override default 3 slots. */
|
||||
slots?: SlotConfig[];
|
||||
/** 1-3. CLI defaults to 3 in TTY, 1 in non-TTY (T11=B). */
|
||||
cycles?: number;
|
||||
/** Where receipts are written. CLI defaults to gbrainPath('eval-receipts'). */
|
||||
receiptDir: string;
|
||||
/** Per-call max output tokens (default 4000). */
|
||||
maxTokens?: number;
|
||||
/** Optional abort signal threaded into gateway calls. */
|
||||
abortSignal?: AbortSignal;
|
||||
/** Stderr progress callback (cycle 1/3, slot A done, etc.). */
|
||||
onProgress?: (event: ProgressEvent) => void;
|
||||
}
|
||||
|
||||
export type ProgressEvent =
|
||||
| { kind: 'cycle_start'; cycle: number; total: number }
|
||||
| { kind: 'slot_done'; cycle: number; slotId: string; modelId: string; ok: boolean; ms: number }
|
||||
| { kind: 'cycle_end'; cycle: number; verdict: 'pass' | 'fail' | 'inconclusive' };
|
||||
|
||||
export interface CycleReceipt {
|
||||
schema_version: 1;
|
||||
cycle: number;
|
||||
task: string;
|
||||
output_sha8: string;
|
||||
/** Slug used in receipt filename. */
|
||||
slug: string;
|
||||
/** Skill SHA-8 used in receipt filename — caller-supplied via skill_sha. */
|
||||
skill_sha8?: string;
|
||||
timestamp: string;
|
||||
dimensions: string[];
|
||||
slots: Array<{
|
||||
id: string;
|
||||
model: string;
|
||||
ok: boolean;
|
||||
error?: string;
|
||||
raw?: string;
|
||||
parsed?: unknown;
|
||||
}>;
|
||||
aggregate: AggregateResult;
|
||||
/** Path the receipt was written to. */
|
||||
receipt_path: string;
|
||||
}
|
||||
|
||||
export interface RunEvalResult {
|
||||
/** Last cycle's aggregate (the verdict that drives exit code). */
|
||||
finalAggregate: AggregateResult;
|
||||
/** Receipt for each cycle that ran. */
|
||||
cycles: CycleReceipt[];
|
||||
/** Path of the LAST cycle's receipt (the one binding the current sha). */
|
||||
finalReceiptPath: string;
|
||||
}
|
||||
|
||||
/** Run up to `cycles` cycles. Stops early on PASS. */
|
||||
export async function runEval(opts: RunEvalOpts): Promise<RunEvalResult> {
|
||||
const dimensions = opts.dimensions ?? DEFAULT_DIMENSIONS;
|
||||
const slots = opts.slots ?? DEFAULT_SLOTS;
|
||||
const cycles = clampCycles(opts.cycles);
|
||||
const slug = opts.slug ?? `eval-${sha8(opts.output).slice(0, 6)}`;
|
||||
|
||||
const cycleReceipts: CycleReceipt[] = [];
|
||||
let finalAggregate: AggregateResult | null = null;
|
||||
let finalReceiptPath = '';
|
||||
|
||||
for (let cycle = 1; cycle <= cycles; cycle++) {
|
||||
opts.onProgress?.({ kind: 'cycle_start', cycle, total: cycles });
|
||||
|
||||
const slotResults = await runOneCycle({
|
||||
task: opts.task,
|
||||
output: opts.output,
|
||||
dimensions,
|
||||
slots,
|
||||
maxTokens: opts.maxTokens ?? 4000,
|
||||
abortSignal: opts.abortSignal,
|
||||
cycle,
|
||||
onProgress: opts.onProgress,
|
||||
});
|
||||
|
||||
const agg = aggregate({ slots: slotResults });
|
||||
finalAggregate = agg;
|
||||
|
||||
// Receipt filename: <slug>-<sha8 of output>.json on cycle 1; subsequent
|
||||
// cycles append `.cycle<N>` so we don't clobber.
|
||||
const baseName = receiptName(slug, opts.output);
|
||||
const receiptFile =
|
||||
cycle === 1 ? baseName : baseName.replace(/\.json$/, `.cycle${cycle}.json`);
|
||||
const receiptPath = join(opts.receiptDir, receiptFile);
|
||||
|
||||
const receipt: CycleReceipt = {
|
||||
schema_version: RECEIPT_SCHEMA_VERSION,
|
||||
cycle,
|
||||
task: opts.task,
|
||||
output_sha8: sha8(opts.output),
|
||||
slug,
|
||||
timestamp: new Date().toISOString(),
|
||||
dimensions,
|
||||
slots: slotResults.map(s => ({
|
||||
id: s.modelId.split(':')[0]!.toUpperCase().slice(0, 1),
|
||||
model: s.modelId,
|
||||
ok: s.ok,
|
||||
error: s.ok ? undefined : s.error,
|
||||
raw: s.ok ? undefined : undefined, // raw is large; skip from receipt by default
|
||||
parsed: s.ok ? s.parsed : undefined,
|
||||
})),
|
||||
aggregate: agg,
|
||||
receipt_path: receiptPath,
|
||||
};
|
||||
|
||||
writeReceipt(receiptPath, receipt);
|
||||
cycleReceipts.push(receipt);
|
||||
finalReceiptPath = receiptPath;
|
||||
|
||||
opts.onProgress?.({ kind: 'cycle_end', cycle, verdict: agg.verdict });
|
||||
|
||||
if (agg.verdict === 'pass' || agg.verdict === 'inconclusive') break;
|
||||
}
|
||||
|
||||
if (!finalAggregate) {
|
||||
throw new Error('runEval: no cycles ran');
|
||||
}
|
||||
|
||||
return { finalAggregate, cycles: cycleReceipts, finalReceiptPath };
|
||||
}
|
||||
|
||||
interface OneCycleOpts {
|
||||
task: string;
|
||||
output: string;
|
||||
dimensions: string[];
|
||||
slots: SlotConfig[];
|
||||
maxTokens: number;
|
||||
abortSignal?: AbortSignal;
|
||||
cycle: number;
|
||||
onProgress?: (event: ProgressEvent) => void;
|
||||
}
|
||||
|
||||
async function runOneCycle(opts: OneCycleOpts): Promise<SlotResult[]> {
|
||||
const prompt = buildPrompt(opts.task, opts.dimensions, opts.output);
|
||||
|
||||
const tasks = opts.slots.map(slot => callSlot(slot, prompt, opts));
|
||||
const settled = await Promise.allSettled(tasks);
|
||||
|
||||
const slotResults: SlotResult[] = settled.map((s, idx) => {
|
||||
const slot = opts.slots[idx]!;
|
||||
if (s.status === 'fulfilled') return s.value;
|
||||
return { ok: false, modelId: slot.model, error: errorMessage(s.reason) };
|
||||
});
|
||||
|
||||
return slotResults;
|
||||
}
|
||||
|
||||
async function callSlot(
|
||||
slot: SlotConfig,
|
||||
prompt: string,
|
||||
opts: OneCycleOpts,
|
||||
): Promise<SlotResult> {
|
||||
const start = Date.now();
|
||||
try {
|
||||
const messages: ChatMessage[] = [
|
||||
{ role: 'user', content: prompt },
|
||||
];
|
||||
const result = await gwChat({
|
||||
model: slot.model,
|
||||
system: SYSTEM_PROMPT,
|
||||
messages,
|
||||
maxTokens: opts.maxTokens,
|
||||
abortSignal: opts.abortSignal,
|
||||
});
|
||||
|
||||
const parsed = parseModelJSON(result.text ?? '');
|
||||
const ms = Date.now() - start;
|
||||
opts.onProgress?.({
|
||||
kind: 'slot_done',
|
||||
cycle: opts.cycle,
|
||||
slotId: slot.id,
|
||||
modelId: slot.model,
|
||||
ok: true,
|
||||
ms,
|
||||
});
|
||||
return { ok: true, modelId: slot.model, parsed };
|
||||
} catch (err) {
|
||||
const ms = Date.now() - start;
|
||||
const msg = errorMessage(err);
|
||||
opts.onProgress?.({
|
||||
kind: 'slot_done',
|
||||
cycle: opts.cycle,
|
||||
slotId: slot.id,
|
||||
modelId: slot.model,
|
||||
ok: false,
|
||||
ms,
|
||||
});
|
||||
return { ok: false, modelId: slot.model, error: msg };
|
||||
}
|
||||
}
|
||||
|
||||
function buildPrompt(task: string, dimensions: string[], output: string): string {
|
||||
const dimList = dimensions.map((d, i) => `${i + 1}. ${d}`).join('\n');
|
||||
return [
|
||||
'You are a strict quality evaluator. Given a TASK and an OUTPUT, evaluate whether the output achieves the task goals.',
|
||||
'',
|
||||
'TASK:',
|
||||
task,
|
||||
'',
|
||||
`Score the OUTPUT 1-10 on each dimension:`,
|
||||
dimList,
|
||||
'',
|
||||
'Scoring calibration:',
|
||||
' 9-10: Exceptional — would impress a domain expert',
|
||||
' 7-8: Solid — accomplishes the goal, no major gaps',
|
||||
' 5-6: Mediocre — obvious weaknesses',
|
||||
' 3-4: Poor — missing important elements',
|
||||
' 1-2: Failed',
|
||||
'',
|
||||
'Then list exactly 10 specific, actionable improvements — concrete changes with examples, prioritized by impact.',
|
||||
'',
|
||||
'Respond in JSON only (no markdown fences):',
|
||||
'{',
|
||||
' "scores": {',
|
||||
' "dim_1_name": { "score": N, "feedback": "..." },',
|
||||
' ...',
|
||||
' },',
|
||||
' "overall": N,',
|
||||
' "improvements": ["1. ...", "2. ...", ... "10. ..."]',
|
||||
'}',
|
||||
'',
|
||||
'OUTPUT:',
|
||||
output,
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
const SYSTEM_PROMPT =
|
||||
'You are a strict quality evaluator. Reply with JSON only. Do not wrap in markdown fences. ' +
|
||||
'Each score must be an integer 1-10. Improvements must be concrete and actionable.';
|
||||
|
||||
function clampCycles(n: number | undefined): number {
|
||||
if (typeof n !== 'number' || !Number.isFinite(n)) return 1;
|
||||
if (n < 1) return 1;
|
||||
if (n > 3) return 3;
|
||||
return Math.floor(n);
|
||||
}
|
||||
|
||||
function errorMessage(err: unknown): string {
|
||||
if (err instanceof Error) return err.message;
|
||||
return String(err);
|
||||
}
|
||||
|
||||
/**
|
||||
* Cost estimation table. Used by the CLI to print a per-run upper-bound
|
||||
* before each cycle (T11=B). Source: gateway recipes' price_last_verified
|
||||
* fields. Prices drift; this is intentionally rough.
|
||||
*/
|
||||
export interface CostEstimate {
|
||||
perCycleUSD: number;
|
||||
perRunMaxUSD: number;
|
||||
perCallTokens: number;
|
||||
notes: string[];
|
||||
}
|
||||
|
||||
export function estimateCost(slots: SlotConfig[], cycles: number, maxTokens: number): CostEstimate {
|
||||
// Per-call cost = (input_tokens × input_price + output_tokens × output_price) / 1e6.
|
||||
// Without knowing prompt size, estimate input ~5k tokens (a SKILL.md + scoring rubric).
|
||||
const ESTIMATED_INPUT_TOKENS = 5000;
|
||||
const PRICING: Record<string, { in: number; out: number } | undefined> = {
|
||||
'openai:gpt-4o': { in: 2.5, out: 10.0 },
|
||||
'openai:gpt-4o-mini': { in: 0.15, out: 0.6 },
|
||||
'anthropic:claude-opus-4-7': { in: 15.0, out: 75.0 },
|
||||
'anthropic:claude-sonnet-4-6-20250929': { in: 3.0, out: 15.0 },
|
||||
'anthropic:claude-haiku-4-5-20251001': { in: 0.25, out: 1.25 },
|
||||
'google:gemini-1.5-pro': { in: 1.25, out: 5.0 },
|
||||
'google:gemini-2.0-flash': { in: 0.1, out: 0.4 },
|
||||
'together:meta-llama/Llama-3.3-70B-Instruct-Turbo': { in: 0.88, out: 0.88 },
|
||||
'deepseek:deepseek-chat': { in: 0.14, out: 0.28 },
|
||||
};
|
||||
|
||||
const notes: string[] = [];
|
||||
let perCycle = 0;
|
||||
for (const slot of slots) {
|
||||
const p = PRICING[slot.model];
|
||||
if (!p) {
|
||||
notes.push(`(${slot.model}): no pricing on file; cost estimate may be low`);
|
||||
continue;
|
||||
}
|
||||
const cost = (ESTIMATED_INPUT_TOKENS * p.in + maxTokens * p.out) / 1_000_000;
|
||||
perCycle += cost;
|
||||
}
|
||||
return {
|
||||
perCycleUSD: round2(perCycle),
|
||||
perRunMaxUSD: round2(perCycle * cycles),
|
||||
perCallTokens: ESTIMATED_INPUT_TOKENS + maxTokens,
|
||||
notes,
|
||||
};
|
||||
}
|
||||
|
||||
function round2(n: number): number {
|
||||
return Math.round(n * 100) / 100;
|
||||
}
|
||||
@@ -70,6 +70,40 @@ export function skillMdTemplate(v: ScaffoldVars): string {
|
||||
lines.push(
|
||||
`Run the deterministic script: \`bun scripts/${v.name}.mjs\` (or whatever your harness prefix is).`,
|
||||
);
|
||||
lines.push('');
|
||||
// 11-item contract (T7=C in plans/radiant-napping-lerdorf.md): the new
|
||||
// Phase 3 cross-modal eval is informational. The scaffold tells the
|
||||
// implementer where the gate lives without forcing it as a blocker.
|
||||
lines.push('## Phase 3: Cross-modal eval (informational)');
|
||||
lines.push('');
|
||||
lines.push(
|
||||
`Once the SKILL.md body and \`scripts/${v.name}.mjs\` are real, run the cross-modal`,
|
||||
);
|
||||
lines.push(
|
||||
'eval gate against the SKILL.md output before locking behavior in tests:',
|
||||
);
|
||||
lines.push('');
|
||||
lines.push('```bash');
|
||||
lines.push('gbrain eval cross-modal \\');
|
||||
lines.push(` --task "What this skill is supposed to accomplish" \\`);
|
||||
lines.push(` --output skills/${v.name}/SKILL.md`);
|
||||
lines.push('```');
|
||||
lines.push('');
|
||||
lines.push(
|
||||
'Three frontier models (different providers) score the output on 5 dimensions.',
|
||||
);
|
||||
lines.push(
|
||||
'Pass criteria: every dim mean >=7 AND no model scored any dim <5. Receipts',
|
||||
);
|
||||
lines.push(
|
||||
'land at `~/.gbrain/.gbrain/eval-receipts/<slug>-<sha8>.json` (sha-8 of SKILL.md',
|
||||
);
|
||||
lines.push(
|
||||
'content). `gbrain skillify check` surfaces the receipt status as informational.',
|
||||
);
|
||||
lines.push(
|
||||
'See `skills/skillify/SKILL.md` Phase 3 for the full 11-item checklist.',
|
||||
);
|
||||
return lines.join('\n') + '\n';
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
|
||||
import { aggregate } from '../src/core/cross-modal-eval/aggregate.ts';
|
||||
import type { SlotResult } from '../src/core/cross-modal-eval/aggregate.ts';
|
||||
|
||||
function ok(model: string, scores: Record<string, number>, improvements: string[] = []): SlotResult {
|
||||
return {
|
||||
ok: true,
|
||||
modelId: model,
|
||||
parsed: {
|
||||
scores: Object.fromEntries(Object.entries(scores).map(([k, v]) => [k, { score: v }])),
|
||||
improvements,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function err(model: string, message = 'fetch failed'): SlotResult {
|
||||
return { ok: false, modelId: model, error: message };
|
||||
}
|
||||
|
||||
describe('cross-modal-eval/aggregate', () => {
|
||||
test('all 3 succeeded, all dims >=7 -> PASS', () => {
|
||||
const out = aggregate({
|
||||
slots: [
|
||||
ok('openai:gpt-4o', { goal: 9, depth: 8 }),
|
||||
ok('anthropic:claude-opus-4-7', { goal: 8, depth: 7 }),
|
||||
ok('google:gemini-1.5-pro', { goal: 8, depth: 8 }),
|
||||
],
|
||||
});
|
||||
expect(out.verdict).toBe('pass');
|
||||
expect(out.successes).toBe(3);
|
||||
expect(out.failures).toBe(0);
|
||||
expect(out.dimensions.goal!.mean).toBeGreaterThanOrEqual(7);
|
||||
expect(out.dimensions.depth!.mean).toBeGreaterThanOrEqual(7);
|
||||
expect(out.dimensions.goal!.failReason).toBeUndefined();
|
||||
expect(out.overall).toBeGreaterThan(7);
|
||||
});
|
||||
|
||||
test('one dim mean below 7 -> FAIL with mean_below_7', () => {
|
||||
const out = aggregate({
|
||||
slots: [
|
||||
ok('openai:gpt-4o', { goal: 9, depth: 6 }),
|
||||
ok('anthropic:claude-opus-4-7', { goal: 8, depth: 6 }),
|
||||
ok('google:gemini-1.5-pro', { goal: 8, depth: 6 }),
|
||||
],
|
||||
});
|
||||
expect(out.verdict).toBe('fail');
|
||||
expect(out.dimensions.depth!.failReason).toBe('mean_below_7');
|
||||
expect(out.dimensions.goal!.failReason).toBeUndefined();
|
||||
expect(out.verdictMessage).toContain('depth');
|
||||
});
|
||||
|
||||
test('mean >=7 but one model scored <5 -> FAIL with min_below_5 (Q2 score-floor)', () => {
|
||||
// [9, 8, 4] — mean = 7.0 ✓, but min = 4 → must FAIL per spec.
|
||||
const out = aggregate({
|
||||
slots: [
|
||||
ok('openai:gpt-4o', { goal: 9 }),
|
||||
ok('anthropic:claude-opus-4-7', { goal: 8 }),
|
||||
ok('google:gemini-1.5-pro', { goal: 4 }),
|
||||
],
|
||||
});
|
||||
expect(out.verdict).toBe('fail');
|
||||
expect(out.dimensions.goal!.mean).toBeCloseTo(7.0, 1);
|
||||
expect(out.dimensions.goal!.min).toBe(4);
|
||||
expect(out.dimensions.goal!.failReason).toBe('min_below_5');
|
||||
});
|
||||
|
||||
test('2 of 3 succeeded -> verdict is computable', () => {
|
||||
const out = aggregate({
|
||||
slots: [
|
||||
ok('openai:gpt-4o', { goal: 8 }),
|
||||
ok('anthropic:claude-opus-4-7', { goal: 9 }),
|
||||
err('google:gemini-1.5-pro', 'rate limited'),
|
||||
],
|
||||
});
|
||||
expect(out.verdict).toBe('pass');
|
||||
expect(out.successes).toBe(2);
|
||||
expect(out.failures).toBe(1);
|
||||
expect(out.errors).toEqual([{ modelId: 'google:gemini-1.5-pro', error: 'rate limited' }]);
|
||||
});
|
||||
|
||||
test('1 of 3 succeeded -> INCONCLUSIVE (Q3=A)', () => {
|
||||
const out = aggregate({
|
||||
slots: [
|
||||
ok('openai:gpt-4o', { goal: 9 }),
|
||||
err('anthropic:claude-opus-4-7', 'auth failed'),
|
||||
err('google:gemini-1.5-pro', 'rate limited'),
|
||||
],
|
||||
});
|
||||
expect(out.verdict).toBe('inconclusive');
|
||||
expect(out.successes).toBe(1);
|
||||
expect(out.overall).toBeUndefined();
|
||||
expect(out.verdictMessage).toContain('INCONCLUSIVE');
|
||||
});
|
||||
|
||||
test('0 of 3 succeeded -> INCONCLUSIVE (regression guard for empty-array .every() === true)', () => {
|
||||
const out = aggregate({
|
||||
slots: [
|
||||
err('openai:gpt-4o'),
|
||||
err('anthropic:claude-opus-4-7'),
|
||||
err('google:gemini-1.5-pro'),
|
||||
],
|
||||
});
|
||||
// The original v1 .mjs script returned PASS here because Object.values({}).every(...) === true.
|
||||
// The fix: aggregate must require >=2 successes BEFORE evaluating dim means.
|
||||
expect(out.verdict).toBe('inconclusive');
|
||||
expect(out.successes).toBe(0);
|
||||
expect(out.failures).toBe(3);
|
||||
expect(out.errors).toHaveLength(3);
|
||||
});
|
||||
|
||||
test('improvement dedup by 40-char prefix (case-insensitive)', () => {
|
||||
// Dedup is `first-40-chars(lowercased, whitespace-collapsed)`. To trigger
|
||||
// dedup, the first 40 chars must match — including any leading numeric
|
||||
// prefix. Two entries below match across slots after lowercasing.
|
||||
const out = aggregate({
|
||||
slots: [
|
||||
ok('a:m', { goal: 8 }, [
|
||||
'Add concrete examples to the introduction section',
|
||||
'Tighten the closing paragraph',
|
||||
]),
|
||||
ok('b:m', { goal: 8 }, [
|
||||
'add concrete examples to the introduction SECTION', // dup of first by 40-char prefix
|
||||
'Add citations',
|
||||
]),
|
||||
],
|
||||
});
|
||||
// Three uniques after dedup: "Add concrete examples...", "Tighten...", "Add citations".
|
||||
expect(out.topImprovements).toHaveLength(3);
|
||||
expect(out.topImprovements[0]).toContain('Add concrete examples');
|
||||
});
|
||||
|
||||
test('top improvements capped at 10', () => {
|
||||
const many = Array.from({ length: 30 }, (_, i) => `${i}. unique improvement number ${i}`);
|
||||
const out = aggregate({
|
||||
slots: [
|
||||
ok('a:m', { goal: 8 }, many.slice(0, 15)),
|
||||
ok('b:m', { goal: 8 }, many.slice(15)),
|
||||
],
|
||||
});
|
||||
expect(out.topImprovements.length).toBeLessThanOrEqual(10);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,195 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { mkdtempSync, mkdirSync, readdirSync, rmSync, writeFileSync, readFileSync, existsSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
|
||||
import {
|
||||
describeReceiptStatus,
|
||||
findReceiptForSkill,
|
||||
inferSlugFromSkillPath,
|
||||
isReceiptFile,
|
||||
listReceiptsForSlug,
|
||||
receiptName,
|
||||
sha8,
|
||||
} from '../src/core/cross-modal-eval/receipt-name.ts';
|
||||
import { writeReceipt } from '../src/core/cross-modal-eval/receipt-write.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
function makeTempDir(): string {
|
||||
return mkdtempSync(join(tmpdir(), 'gbrain-cme-cli-'));
|
||||
}
|
||||
|
||||
describe('cross-modal-eval CLI helpers', () => {
|
||||
test('sha8 is deterministic and 8 hex chars', () => {
|
||||
const a = sha8('hello world');
|
||||
const b = sha8('hello world');
|
||||
expect(a).toBe(b);
|
||||
expect(a).toMatch(/^[0-9a-f]{8}$/);
|
||||
expect(sha8('different')).not.toBe(a);
|
||||
});
|
||||
|
||||
test('receiptName: <slug>-<sha8>.json shape', () => {
|
||||
const name = receiptName('my-skill', '# SKILL.md content');
|
||||
expect(name).toMatch(/^my-skill-[0-9a-f]{8}\.json$/);
|
||||
const expected = `my-skill-${sha8('# SKILL.md content')}.json`;
|
||||
expect(name).toBe(expected);
|
||||
});
|
||||
|
||||
test('receiptName rejects invalid slug', () => {
|
||||
expect(() => receiptName('Bad Slug', 'x')).toThrow();
|
||||
expect(() => receiptName('', 'x')).toThrow();
|
||||
expect(() => receiptName('../escape', 'x')).toThrow();
|
||||
});
|
||||
|
||||
test('inferSlugFromSkillPath: pulls parent directory name', () => {
|
||||
expect(inferSlugFromSkillPath('skills/my-skill/SKILL.md')).toBe('my-skill');
|
||||
expect(inferSlugFromSkillPath('/abs/path/skills/foo-bar/SKILL.md')).toBe('foo-bar');
|
||||
expect(() => inferSlugFromSkillPath('skills/my-skill/notes.md')).toThrow();
|
||||
expect(() => inferSlugFromSkillPath('SKILL.md')).toThrow();
|
||||
});
|
||||
|
||||
test('findReceiptForSkill: missing skill -> missing', () => {
|
||||
const dir = makeTempDir();
|
||||
try {
|
||||
const result = findReceiptForSkill(join(dir, 'skills', 'nope', 'SKILL.md'), join(dir, 'receipts'));
|
||||
expect(result.status).toBe('missing');
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('findReceiptForSkill: found when sha matches', () => {
|
||||
const dir = makeTempDir();
|
||||
try {
|
||||
const skillDir = join(dir, 'skills', 'demo');
|
||||
mkdirSync(skillDir, { recursive: true });
|
||||
const skillPath = join(skillDir, 'SKILL.md');
|
||||
const content = '# demo skill\n';
|
||||
writeFileSync(skillPath, content);
|
||||
|
||||
const receiptDir = join(dir, 'receipts');
|
||||
mkdirSync(receiptDir, { recursive: true });
|
||||
const expected = receiptName('demo', content);
|
||||
writeFileSync(join(receiptDir, expected), '{"schema_version":1}');
|
||||
|
||||
const status = findReceiptForSkill(skillPath, receiptDir);
|
||||
expect(status.status).toBe('found');
|
||||
if (status.status === 'found') {
|
||||
expect(status.path.endsWith(expected)).toBe(true);
|
||||
expect(status.sha).toBe(sha8(content));
|
||||
}
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('findReceiptForSkill: stale when sha mismatches', () => {
|
||||
const dir = makeTempDir();
|
||||
try {
|
||||
const skillDir = join(dir, 'skills', 'demo');
|
||||
mkdirSync(skillDir, { recursive: true });
|
||||
const skillPath = join(skillDir, 'SKILL.md');
|
||||
writeFileSync(skillPath, '# updated content\n');
|
||||
|
||||
const receiptDir = join(dir, 'receipts');
|
||||
mkdirSync(receiptDir, { recursive: true });
|
||||
const oldSha = sha8('# original content\n');
|
||||
writeFileSync(join(receiptDir, `demo-${oldSha}.json`), '{"schema_version":1}');
|
||||
|
||||
const status = findReceiptForSkill(skillPath, receiptDir);
|
||||
expect(status.status).toBe('stale');
|
||||
if (status.status === 'stale') {
|
||||
expect(status.latestSha).toBe(oldSha);
|
||||
expect(status.currentSha).toBe(sha8('# updated content\n'));
|
||||
}
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('describeReceiptStatus: human-readable string for each status', () => {
|
||||
const found = describeReceiptStatus('demo', { status: 'found', path: '/r/demo-aabbccdd.json', sha: 'aabbccdd' });
|
||||
expect(found).toContain('found');
|
||||
expect(found).toContain('demo');
|
||||
|
||||
const stale = describeReceiptStatus('demo', {
|
||||
status: 'stale',
|
||||
latestPath: '/r/demo-old.json',
|
||||
latestSha: 'old-sha',
|
||||
currentSha: 'new-sha',
|
||||
});
|
||||
expect(stale).toContain('older');
|
||||
expect(stale).toContain('Re-run');
|
||||
|
||||
const missing = describeReceiptStatus('demo', { status: 'missing', currentSha: 'x' });
|
||||
expect(missing).toContain('no cross-modal eval receipt');
|
||||
});
|
||||
|
||||
test('listReceiptsForSlug returns newest first', () => {
|
||||
const dir = makeTempDir();
|
||||
try {
|
||||
mkdirSync(dir, { recursive: true });
|
||||
writeFileSync(join(dir, 'demo-11111111.json'), '{}');
|
||||
// Sleep tiny so mtimes differ on filesystems with low resolution.
|
||||
const start = Date.now();
|
||||
while (Date.now() - start < 10) {
|
||||
// tight spin
|
||||
}
|
||||
writeFileSync(join(dir, 'demo-22222222.json'), '{}');
|
||||
const list = listReceiptsForSlug('demo', dir);
|
||||
expect(list).toHaveLength(2);
|
||||
expect(list[0]!.endsWith('22222222.json')).toBe(true);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('isReceiptFile recognizes pattern', () => {
|
||||
expect(isReceiptFile('demo-aabbccdd.json')).toBe(true);
|
||||
expect(isReceiptFile('/some/dir/my-skill-12345678.json')).toBe(true);
|
||||
expect(isReceiptFile('demo.json')).toBe(false);
|
||||
expect(isReceiptFile('demo-toolong.json')).toBe(false);
|
||||
expect(isReceiptFile('not a receipt')).toBe(false);
|
||||
});
|
||||
|
||||
test('writeReceipt auto-mkdirs parent (T5 correction)', () => {
|
||||
const dir = makeTempDir();
|
||||
try {
|
||||
const target = join(dir, 'nested', 'deeper', 'demo-aabbccdd.json');
|
||||
expect(existsSync(target)).toBe(false);
|
||||
writeReceipt(target, { hello: 'world' });
|
||||
expect(existsSync(target)).toBe(true);
|
||||
const body = JSON.parse(readFileSync(target, 'utf-8'));
|
||||
expect(body).toEqual({ hello: 'world' });
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('writeReceipt accepts pre-stringified content', () => {
|
||||
const dir = makeTempDir();
|
||||
try {
|
||||
const target = join(dir, 'demo-aabbccdd.json');
|
||||
writeReceipt(target, '{"already":"json"}');
|
||||
const body = readFileSync(target, 'utf-8');
|
||||
expect(body).toBe('{"already":"json"}');
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('GBRAIN_HOME isolates receipts under <home>/.gbrain/eval-receipts', async () => {
|
||||
const dir = makeTempDir();
|
||||
try {
|
||||
await withEnv({ GBRAIN_HOME: dir }, async () => {
|
||||
const { gbrainPath } = await import('../src/core/config.ts');
|
||||
const path = gbrainPath('eval-receipts');
|
||||
expect(path.startsWith(dir)).toBe(true);
|
||||
expect(path.endsWith('eval-receipts')).toBe(true);
|
||||
expect(path.includes('.gbrain')).toBe(true);
|
||||
});
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,88 @@
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
|
||||
import { parseModelJSON } from '../src/core/cross-modal-eval/json-repair.ts';
|
||||
|
||||
describe('cross-modal-eval/json-repair', () => {
|
||||
test('parses clean JSON', () => {
|
||||
const raw = JSON.stringify({
|
||||
scores: {
|
||||
goal: { score: 9, feedback: 'on point' },
|
||||
depth: { score: 8 },
|
||||
},
|
||||
overall: 8.5,
|
||||
improvements: ['1. tighten the intro'],
|
||||
});
|
||||
const out = parseModelJSON(raw);
|
||||
expect(out.scores.goal!.score).toBe(9);
|
||||
expect(out.scores.depth!.score).toBe(8);
|
||||
expect(out.overall).toBe(8.5);
|
||||
expect(out.improvements).toEqual(['1. tighten the intro']);
|
||||
});
|
||||
|
||||
test('strips ```json markdown fences', () => {
|
||||
const raw = '```json\n{"scores": {"goal": {"score": 7}}, "improvements": []}\n```';
|
||||
const out = parseModelJSON(raw);
|
||||
expect(out.scores.goal!.score).toBe(7);
|
||||
});
|
||||
|
||||
test('strips bare ``` fences too', () => {
|
||||
const raw = '```\n{"scores": {"goal": {"score": 7}}, "improvements": []}\n```';
|
||||
const out = parseModelJSON(raw);
|
||||
expect(out.scores.goal!.score).toBe(7);
|
||||
});
|
||||
|
||||
test('repairs trailing commas before } and ]', () => {
|
||||
const raw = '{"scores": {"goal": {"score": 7,},}, "improvements": ["1. tighten",],}';
|
||||
const out = parseModelJSON(raw);
|
||||
expect(out.scores.goal!.score).toBe(7);
|
||||
expect(out.improvements).toEqual(['1. tighten']);
|
||||
});
|
||||
|
||||
test('repairs single-quote string delimiters', () => {
|
||||
const raw = "{'scores': {'goal': {'score': 7}}, 'improvements': ['1. tighten']}";
|
||||
const out = parseModelJSON(raw);
|
||||
expect(out.scores.goal!.score).toBe(7);
|
||||
});
|
||||
|
||||
test('repairs embedded newlines inside strings', () => {
|
||||
const raw = '{"scores": {"goal": {"score": 7, "feedback": "line one\nline two"}}, "improvements": []}';
|
||||
const out = parseModelJSON(raw);
|
||||
expect(out.scores.goal!.score).toBe(7);
|
||||
expect(out.scores.goal!.feedback).toContain('line one');
|
||||
});
|
||||
|
||||
test('nuclear option: reconstructs scores from mismatched-brace input', () => {
|
||||
// Outer object intentionally unclosed; strategies 1-3 fail, nuclear regex
|
||||
// walks the dim:{score:N} pattern at the top level.
|
||||
const raw =
|
||||
'{ "goal": { "score": 8, "feedback": "good" }, "depth": { "score": 7 }, ' +
|
||||
'"overall": 7.5, ' +
|
||||
'"improvements": ["1. add concrete examples", "2. tighten the intro"] ';
|
||||
const out = parseModelJSON(raw);
|
||||
expect(out._repaired).toBe(true);
|
||||
expect(out.scores.goal!.score).toBe(8);
|
||||
expect(out.scores.depth!.score).toBe(7);
|
||||
expect(out.improvements.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('nuclear option: throws when zero scores recoverable (no fabrication)', () => {
|
||||
const raw = 'this is not even close to JSON, just prose with random {"wonky" } shapes';
|
||||
expect(() => parseModelJSON(raw)).toThrow();
|
||||
});
|
||||
|
||||
test('throws on empty input', () => {
|
||||
expect(() => parseModelJSON('')).toThrow();
|
||||
expect(() => parseModelJSON(' ')).toThrow();
|
||||
});
|
||||
|
||||
test('throws when no { ... } object substring exists', () => {
|
||||
expect(() => parseModelJSON('just a plain string with no braces at all')).toThrow();
|
||||
});
|
||||
|
||||
test('numeric-shorthand scores are accepted ({"dim": 7})', () => {
|
||||
const raw = '{"scores": {"goal": 7, "depth": 8}, "improvements": ["1. x"]}';
|
||||
const out = parseModelJSON(raw);
|
||||
expect(out.scores.goal!.score).toBe(7);
|
||||
expect(out.scores.depth!.score).toBe(8);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,198 @@
|
||||
/**
|
||||
* E2E for `gbrain eval cross-modal` runner via mocked gateway chat().
|
||||
*
|
||||
* Lives under test/e2e/ so the test-isolation lint (R2 — mock.module quarantine)
|
||||
* does not require a *.serial.test.ts rename: test/e2e/* is exempt from the
|
||||
* lint, and `scripts/run-e2e.sh` already runs one file per Bun process so
|
||||
* `mock.module` leaks are contained.
|
||||
*
|
||||
* Verifies the verdict / exit-code contract end-to-end:
|
||||
* PASS (verdict='pass') when every dim mean >=7 and no model <5
|
||||
* FAIL (verdict='fail') when any dim breaches mean OR floor
|
||||
* INCONCLUSIVE (verdict='inconclusive') when <2/3 model calls succeed
|
||||
*/
|
||||
|
||||
import { afterEach, beforeEach, describe, expect, mock, test } from 'bun:test';
|
||||
import { mkdtempSync, readdirSync, readFileSync, rmSync } from 'fs';
|
||||
import { tmpdir } from 'os';
|
||||
import { join } from 'path';
|
||||
|
||||
import { configureGateway } from '../../src/core/ai/gateway.ts';
|
||||
|
||||
let tempDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tempDir = mkdtempSync(join(tmpdir(), 'gbrain-cme-e2e-'));
|
||||
// Configure the gateway so our mock can pretend providers are available.
|
||||
configureGateway({
|
||||
embedding_model: 'openai:text-embedding-3-large',
|
||||
embedding_dimensions: 1536,
|
||||
expansion_model: 'anthropic:claude-haiku-4-5-20251001',
|
||||
chat_model: 'anthropic:claude-sonnet-4-6-20250929',
|
||||
base_urls: undefined,
|
||||
env: {
|
||||
OPENAI_API_KEY: 'sk-test',
|
||||
ANTHROPIC_API_KEY: 'sk-ant-test',
|
||||
GOOGLE_GENERATIVE_AI_API_KEY: 'sk-google-test',
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tempDir, { recursive: true, force: true });
|
||||
mock.restore();
|
||||
});
|
||||
|
||||
function makeChatStub(scoresBySlot: Record<string, number[]>) {
|
||||
let callIdx = 0;
|
||||
const order = ['openai:gpt-4o', 'anthropic:claude-opus-4-7', 'google:gemini-1.5-pro'];
|
||||
return mock(async (opts: { model?: string }) => {
|
||||
const model = opts.model ?? '';
|
||||
callIdx++;
|
||||
const slotIdx = order.indexOf(model);
|
||||
const scores = scoresBySlot[model];
|
||||
if (!scores) {
|
||||
throw new Error(`mock: no scores configured for model ${model}`);
|
||||
}
|
||||
const goal = scores[0]!;
|
||||
const depth = scores[1]!;
|
||||
return {
|
||||
text: JSON.stringify({
|
||||
scores: { goal: { score: goal }, depth: { score: depth } },
|
||||
overall: (goal + depth) / 2,
|
||||
improvements: [`${slotIdx + 1}. tighten the intro`],
|
||||
}),
|
||||
blocks: [],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 100, output_tokens: 50, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model,
|
||||
providerId: model.split(':')[0]!,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
describe('gbrain eval cross-modal — runner verdict contract', () => {
|
||||
test('PASS: 3 happy responses, all dims >=7', async () => {
|
||||
const chatStub = makeChatStub({
|
||||
'openai:gpt-4o': [9, 8],
|
||||
'anthropic:claude-opus-4-7': [8, 7],
|
||||
'google:gemini-1.5-pro': [8, 8],
|
||||
});
|
||||
mock.module('../../src/core/ai/gateway.ts', () => ({
|
||||
chat: chatStub,
|
||||
configureGateway,
|
||||
isAvailable: () => true,
|
||||
}));
|
||||
|
||||
const { runEval } = await import('../../src/core/cross-modal-eval/runner.ts');
|
||||
const result = await runEval({
|
||||
task: 'sample task',
|
||||
output: 'sample output content',
|
||||
slug: 'demo',
|
||||
receiptDir: tempDir,
|
||||
cycles: 1,
|
||||
});
|
||||
|
||||
expect(result.finalAggregate.verdict).toBe('pass');
|
||||
expect(result.cycles).toHaveLength(1);
|
||||
const files = readdirSync(tempDir);
|
||||
expect(files.length).toBeGreaterThan(0);
|
||||
expect(files[0]!.startsWith('demo-')).toBe(true);
|
||||
const receipt = JSON.parse(readFileSync(join(tempDir, files[0]!), 'utf-8'));
|
||||
expect(receipt.schema_version).toBe(1);
|
||||
expect(receipt.aggregate.verdict).toBe('pass');
|
||||
});
|
||||
|
||||
test('FAIL: one dim mean below 7', async () => {
|
||||
const chatStub = makeChatStub({
|
||||
'openai:gpt-4o': [9, 6],
|
||||
'anthropic:claude-opus-4-7': [8, 6],
|
||||
'google:gemini-1.5-pro': [8, 6],
|
||||
});
|
||||
mock.module('../../src/core/ai/gateway.ts', () => ({
|
||||
chat: chatStub,
|
||||
configureGateway,
|
||||
isAvailable: () => true,
|
||||
}));
|
||||
|
||||
const { runEval } = await import('../../src/core/cross-modal-eval/runner.ts');
|
||||
const result = await runEval({
|
||||
task: 'sample task',
|
||||
output: 'sample output content',
|
||||
slug: 'demo',
|
||||
receiptDir: tempDir,
|
||||
cycles: 1,
|
||||
});
|
||||
|
||||
expect(result.finalAggregate.verdict).toBe('fail');
|
||||
expect(result.finalAggregate.dimensions.depth!.failReason).toBe('mean_below_7');
|
||||
});
|
||||
|
||||
test('FAIL: min-score floor caught when one model scores <5 (Q2)', async () => {
|
||||
const chatStub = makeChatStub({
|
||||
'openai:gpt-4o': [9, 8],
|
||||
'anthropic:claude-opus-4-7': [8, 8],
|
||||
'google:gemini-1.5-pro': [4, 8], // goal=4 trips the floor
|
||||
});
|
||||
mock.module('../../src/core/ai/gateway.ts', () => ({
|
||||
chat: chatStub,
|
||||
configureGateway,
|
||||
isAvailable: () => true,
|
||||
}));
|
||||
|
||||
const { runEval } = await import('../../src/core/cross-modal-eval/runner.ts');
|
||||
const result = await runEval({
|
||||
task: 'sample task',
|
||||
output: 'sample output content',
|
||||
slug: 'demo',
|
||||
receiptDir: tempDir,
|
||||
cycles: 1,
|
||||
});
|
||||
|
||||
expect(result.finalAggregate.verdict).toBe('fail');
|
||||
expect(result.finalAggregate.dimensions.goal!.failReason).toBe('min_below_5');
|
||||
});
|
||||
|
||||
test('INCONCLUSIVE: 2 of 3 mock 5xx -> exit 2 contract (Q3)', async () => {
|
||||
const chatStub = mock(async (opts: { model?: string }) => {
|
||||
if (opts.model === 'openai:gpt-4o') {
|
||||
return {
|
||||
text: JSON.stringify({
|
||||
scores: { goal: { score: 8 } },
|
||||
improvements: ['1. ok'],
|
||||
}),
|
||||
blocks: [],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: opts.model,
|
||||
providerId: 'openai',
|
||||
};
|
||||
}
|
||||
throw new Error(`mock: forced 5xx for ${opts.model}`);
|
||||
});
|
||||
mock.module('../../src/core/ai/gateway.ts', () => ({
|
||||
chat: chatStub,
|
||||
configureGateway,
|
||||
isAvailable: () => true,
|
||||
}));
|
||||
|
||||
const { runEval } = await import('../../src/core/cross-modal-eval/runner.ts');
|
||||
const result = await runEval({
|
||||
task: 'sample task',
|
||||
output: 'sample output content',
|
||||
slug: 'demo',
|
||||
receiptDir: tempDir,
|
||||
cycles: 1,
|
||||
});
|
||||
|
||||
expect(result.finalAggregate.verdict).toBe('inconclusive');
|
||||
expect(result.finalAggregate.successes).toBe(1);
|
||||
expect(result.finalAggregate.failures).toBe(2);
|
||||
// Receipt is still written even on INCONCLUSIVE — forensics path.
|
||||
const files = readdirSync(tempDir);
|
||||
expect(files.length).toBe(1);
|
||||
const receipt = JSON.parse(readFileSync(join(tempDir, files[0]!), 'utf-8'));
|
||||
expect(receipt.aggregate.verdict).toBe('inconclusive');
|
||||
expect(receipt.aggregate.errors).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
@@ -418,3 +418,45 @@ describe('applyScaffold', () => {
|
||||
expect(agents).toContain('`skills/openclaw-demo/SKILL.md`');
|
||||
});
|
||||
});
|
||||
|
||||
describe('11-item scaffold contract (T9 + Phase 3 cross-modal eval)', () => {
|
||||
it('scaffolded SKILL.md teaches the cross-modal eval Phase 3', () => {
|
||||
// T9 (plans/radiant-napping-lerdorf.md): non-mutating verification of
|
||||
// the 11-item contract bump. Calls planScaffold + applyScaffold against
|
||||
// an in-memory tempdir, then asserts the produced SKILL.md mentions
|
||||
// cross-modal eval. No `gbrain skillify scaffold demo-eleven` shell-out
|
||||
// (which would mutate the real repo and require --description).
|
||||
const { root, skillsDir } = scratchRepo();
|
||||
const plan = planScaffold({
|
||||
skillsDir,
|
||||
vars: {
|
||||
name: 'phase-three-demo',
|
||||
description: 'demo skill that verifies Phase 3 lands in scaffold output',
|
||||
triggers: ['phase three demo'],
|
||||
writesTo: [],
|
||||
writesPages: false,
|
||||
mutating: false,
|
||||
},
|
||||
});
|
||||
applyScaffold(plan);
|
||||
|
||||
const skillMdPath = join(skillsDir, 'phase-three-demo', 'SKILL.md');
|
||||
const body = readFileSync(skillMdPath, 'utf-8');
|
||||
|
||||
expect(body).toContain('## Phase 3: Cross-modal eval');
|
||||
expect(body).toContain('gbrain eval cross-modal');
|
||||
expect(body).toContain('skills/phase-three-demo/SKILL.md');
|
||||
// Receipts naming convention is documented in the scaffold so the
|
||||
// implementer knows where to look.
|
||||
expect(body).toContain('eval-receipts');
|
||||
expect(body).toContain('<sha8>');
|
||||
// Pass criterion is documented in the scaffold (Q2 floor + Q3 floor).
|
||||
expect(body).toMatch(/dim mean\s*>=\s*7/i);
|
||||
expect(body).toMatch(/no model scored any dim\s*<\s*5/i);
|
||||
// Resolver row is appended once and only once — the verification step
|
||||
// does not need any cleanup beyond the tempdir afterEach drops.
|
||||
const resolverContent = readFileSync(join(skillsDir, 'RESOLVER.md'), 'utf-8');
|
||||
const occurrences = resolverContent.split('skills/phase-three-demo/SKILL.md').length - 1;
|
||||
expect(occurrences).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user