Files
gbrain/src/commands/skillify.ts
T
a1a2671c21 v0.28.4 feat(skillpack): enhance skillify with cross-modal eval quality gate (#674)
* feat(skillpack): enhance skillify with cross-modal eval quality gate

Updates skillify from v1.0.0 to v2.0.0 with the key innovation:
cross-modal evaluation runs BEFORE tests (step 3) to establish
quality, then tests lock in the proven-good behavior.

Key changes:
- 11-item checklist (was 10) - adds cross-modal eval as step 3
- Cross-modal eval uses 3 models to score output on 5 dimensions
- Quality gate: all dimensions ≥ 7 average before proceeding to tests
- Prevents locking in mediocrity through tests-first approach
- References cross-modal-review skill for eval pipeline
- Updated all gbrain-specific paths (bun test, scripts/*.ts)
- Maintains compatibility with gbrain check-resolvable workflow

The meta-skill for turning raw features into properly-skilled,
tested, resolvable capabilities. Cross-modal eval ensures output
quality before tests cement the behavior.

* feat: skillify hardened via 2 cross-modal eval cycles (8.1/10)

Applied top improvements from GPT-5.5 + Opus 4-7 + DeepSeek V4 Pro:
- Named 3 frontier models explicitly with provider table
- Inlined eval prompt template with CONTEXT param + scoring calibration
- Defined aggregation math: mean >= 7 AND no single dim < 5
- Added eval receipt JSON schema
- Structured 3-cycle fix loop with before/after delta tracking
- Added worked example (summarize-pr, end-to-end)
- Added cost guardrails (skip < 200 tokens, max 9 API calls)
- Added representative input selection rule
- Added SKILL.md frontmatter template (copy-paste ready)
- Added Phase 0 decision gate (is this worth skillifying?)

Also includes cross-modal-eval runner recipe with robust JSON
parsing for LLMs that return malformed JSON (3-tier repair).

* chore(recipes): remove cross-modal-eval.mjs

Superseded by `gbrain eval cross-modal` (next commit). The .mjs script
was the original PR's hand-rolled provider stack; the replacement reuses
src/core/ai/gateway.ts so config/auth/model-aliasing comes from the
canonical recipe registry instead of a parallel stack.

No code references the .mjs (it was invoked by skill prose only), so
this delete is independently safe to bisect through.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): cross-modal-eval core module + unit tests

Pure-logic foundation for the new `gbrain eval cross-modal` command
(wired in the next commit). All five modules are self-contained — no
CLI surface, no I/O outside the receipt writer's mkdirSync. Imported
from src/core/ai/gateway.ts at runtime via gwChat (no config impact
at load time).

Modules:
  - json-repair.ts:    parseModelJSON 4-strategy fallback chain.
                       Adversarial nuclear-option throws rather than
                       fabricating scores (Q6 + Q3 in plan).
  - aggregate.ts:      verdict logic. PASS = (>=2 successes) AND
                       (every dim mean >= 7) AND (every dim min
                       across models >= 5). INCONCLUSIVE when <2/3
                       models returned parseable scores — closes the
                       v1 .mjs `Object.values({}).every(...) === true`
                       empty-array silent-PASS bug (Q2 + Q3).
  - receipt-name.ts:   receipt filename binds (slug, sha8 of SKILL.md)
                       so `gbrain skillify check` can detect stale
                       audits (T10 in plan).
  - receipt-write.ts:  thin wrapper over writeFileSync that auto-mkdirs
                       the parent directory. Standalone module because
                       gbrainPath() does NOT auto-mkdir (T5 plan
                       correction — Codex caught this).
  - runner.ts:         orchestrator. Promise.allSettled across 3 slots
                       per cycle; up to 3 cycles; stops early on PASS
                       or INCONCLUSIVE. Default slots: openai:gpt-4o /
                       anthropic:claude-opus-4-7 / google:gemini-1.5-pro.
                       estimateCost() exports a small per-model
                       pricing table (drifts; refresh alongside
                       model-family bumps).

Tests (32 cases total, all green):
  - json-repair.test.ts:  10 cases (clean JSON, fences, trailing
                          commas, single quotes, embedded newlines,
                          mismatched braces, nuclear-option success
                          + adversarial throws, empty input,
                          numeric-shorthand scores).
  - aggregate.test.ts:    8 cases pinning Q2/Q3/dedup. The 0-of-3
                          INCONCLUSIVE case is the regression guard
                          for the v1 silent-PASS bug.
  - cli.test.ts:          12 cases on receipt-name / receipt-write /
                          GBRAIN_HOME isolation. Uses withEnv()
                          helper for env mutation (R1 isolation rule).

Verifies bisect-clean: typecheck passes, all 32 unit cases green.
The runner.ts import of gateway.chat() is dead until commit 3 wires
the CLI surface.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): wire `gbrain eval cross-modal` CLI subcommand

User-facing surface for the multi-model quality gate. Three different-
provider frontier models score the OUTPUT against the TASK on a 5-dim
rubric. Verdict drives exit code: 0 PASS, 1 FAIL, 2 INCONCLUSIVE
(<2/3 models returned parseable scores per Q3 in plan).

Wiring touches three files:

  - src/commands/eval-cross-modal.ts (new, ~290 lines)
    CLI handler. Self-configures the AI gateway from loadConfig() +
    process.env so it works without `gbrain init` (the cli.ts no-DB
    branch bypasses connectEngine()). Defaults: cycles=3 in TTY,
    cycles=1 in non-TTY (T11 partial cost guardrail — limits scripted
    bulk spend; full --budget-usd hard cap is a v0.27.x TODO). Prints
    estimated max-cost-per-cycle to stderr before each run. Uses
    gbrainPath('eval-receipts') for receipt directory.

  - src/cli.ts (no-DB dispatch branch, 5-line addition)
    Special-cases `eval cross-modal` BEFORE the existing
    handleCliOnly path that requires connectEngine(). Mirrors the
    `dream` no-DB pattern but doesn't even attempt the connect — the
    command never touches the DB. New users can run the gate before
    `gbrain init` (T3 in plan).

  - src/commands/eval.ts (sub-subcommand dispatch)
    Adds `cross-modal` alongside `export`/`prune`/`replay`. The
    cli.ts branch takes precedence in the user-facing path; this
    branch only fires when callers re-enter runEvalCommand with an
    existing engine. Engine is intentionally unused — the handler
    self-routes.

  - test/e2e/cross-modal-eval.test.ts (new, 4 cases)
    Mocked-fetch E2E. Lives at test/e2e/* (NOT *.serial.test.ts) per
    plan T8: test/e2e/* is exempt from the test-isolation lint and
    already runs serially via scripts/run-e2e.sh, so the
    mock.module() call doesn't need a quarantine rename. Cases:
    PASS / FAIL (mean<7) / FAIL (min<5 — Q2 floor) / INCONCLUSIVE
    (2 mock 5xx — Q3 contract).

The runner from commit 2 now has live callers. typecheck passes;
the 4 E2E cases all green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(skillify): add informational 11th item (cross-modal eval)

Promotes the skillify contract from 10 to 11 items. The 11th item
(cross-modal eval) is `required:false` per T7 in the plan — a
missing or stale receipt surfaces in the audit output but does not
fail the gate. Existing skills keep their current required-score;
the bump is additive, not breaking.

Changes:

  - src/commands/skillify.ts
    Header jsdoc updated 10-item -> 11-item. No code-flow changes.

  - src/commands/skillify-check.ts (the per-skill audit; not
    src/commands/skillpack-check.ts which is a different command —
    plan T6 corrected the conflation in the original plan)
    New informational item at position 11. Reuses
    findReceiptForSkill() helper from
    src/core/cross-modal-eval/receipt-name.ts to detect:
      * found  — receipt matches current SKILL.md sha-8
      * stale  — receipt exists for an older SKILL.md
      * missing — no receipt yet
    Audit output cases pass through to existing pretty/JSON formats.

  - src/core/skillify/templates.ts
    Scaffolded SKILL.md now includes a "Phase 3: Cross-modal eval
    (informational)" section with copy-paste `gbrain eval cross-modal`
    invocation, pass criteria, and receipt-naming convention. Helps
    new skill authors discover the gate.

  - test/skillify-scaffold.test.ts
    New T9 case verifies the scaffold emits the Phase 3 section,
    points at the correct command, documents the receipt path, and
    appends exactly one resolver row. Replaces the original plan's
    `gbrain skillify scaffold demo-eleven` shell verification (which
    Codex caught as invalid + repo-mutating).

Verifies: typecheck passes; scaffold test 19/19 (was 18, +1 T9 case).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: skillify v1.1.0 + cross-modal-eval references

Documentation catches up with the new behavior shipped in commits 1-4.

  - skills/skillify/SKILL.md (1.0.0 -> 1.1.0)
    Full rewrite. Frontmatter version is additive (T7 in plan); the
    11th item is informational, not breaking. Phase 3 now points at
    `gbrain eval cross-modal` with copy-paste invocation, default
    slot table, pass criteria, receipt-naming convention, cycles +
    cost guardrails (T11 partial cap), provider configuration via
    the AI gateway, and the cycle-1/2/3 fix loop. Adds Output Format
    section (skills-conformance.test.ts requires it). Drops the
    original `(or lib/cross-modal-eval.ts)` parenthetical (Q5 plan
    correction — that path never existed).

  - skills/cross-modal-review/SKILL.md
    Adds 4-line Relationship section pointing at `gbrain eval
    cross-modal` (D3 plan reciprocal). Distinguishes the manual
    second-opinion gate (this skill) from the automated multi-model
    score-and-iterate gate (the new command).

  - CLAUDE.md
    Key Files entries for src/commands/eval-cross-modal.ts and the
    five new src/core/cross-modal-eval/* modules. Commands list
    gains the `gbrain eval cross-modal` entry under v0.27.x. Notes
    the non-TTY default 1-cycle behavior + the gbrainPath('eval-
    receipts') resolution.

  - TODOS.md
    Four v0.27.x follow-ups filed under a new "cross-modal-eval"
    section: full --budget-usd cap (T11 follow-up), subagent
    integration (recovers cross-process rate-leases T4 deferred),
    skill adoption telemetry (revisit T7=C with data after 30 days),
    docs/cross-modal-eval.md user guide.

  - llms-full.txt
    Regenerated via `bun run build:llms` to match the CLAUDE.md
    edits — sync guard at test/build-llms.test.ts requires this.

Verifies: typecheck passes; skills-conformance 199/199 green;
build-llms 7/7 green; full unit fast loop 3861/3861 green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.28.4)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Garry Tan <garrytan@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-06 20:39:56 -07:00

314 lines
10 KiB
TypeScript

/**
* gbrain skillify <scaffold|check> — W4 CLI namespace.
*
* `scaffold`: creates 5 stub files for a new skill. Mechanical only.
* `check`: 11-item audit of an existing skill (item 11, cross-modal
* eval, is informational; T7=C in plans/radiant-napping-lerdorf.md).
* Promoted from `scripts/skillify-check.ts` (D-CX-2). The
* legacy script remains as a thin shim that invokes this
* subcommand.
*
* The markdown skill at `skills/skillify/SKILL.md` orchestrates the
* full 11-step loop (essay's "skillify it!"): scaffold → fill in the
* body → run cross-modal eval → run check → run check-resolvable →
* run tests → commit. The CLI primitives do the mechanical steps;
* the skill carries the judgment steps.
*/
import { isAbsolute, resolve as resolvePath } from 'path';
import {
applyScaffold,
planScaffold,
SkillifyScaffoldError,
SKILL_NAME_PATTERN,
type ScaffoldPlan,
} from '../core/skillify/generator.ts';
import { autoDetectSkillsDir } from '../core/repo-root.ts';
import { RESOLVER_FILENAMES_LABEL } from '../core/resolver-filenames.ts';
// Re-exports for tests.
export { planScaffold, applyScaffold, SkillifyScaffoldError, SKILL_NAME_PATTERN };
// ---------------------------------------------------------------------------
// Top-level dispatcher
// ---------------------------------------------------------------------------
const HELP_TOP = `gbrain skillify <subcommand> [options]
Subcommands:
scaffold <name> Create SKILL.md, script, routing-eval, test stubs
and append a resolver row. Mechanical; no LLM.
check [path] Run the 10-item skillify audit on a target path
(or --recent). Wraps the legacy scripts/skillify-check.ts
(D-CX-2: subcommand namespace).
Run \`gbrain skillify <subcommand> --help\` for per-subcommand options.
`;
export async function runSkillify(args: string[]): Promise<void> {
const sub = args[0];
const rest = args.slice(1);
if (!sub || sub === '--help' || sub === '-h') {
console.log(HELP_TOP);
process.exit(0);
}
if (sub === 'scaffold') {
await runSkillifyScaffold(rest);
return;
}
if (sub === 'check') {
await runSkillifyCheck(rest);
return;
}
console.error(`Unknown subcommand: ${sub}\n`);
console.error(HELP_TOP);
process.exit(2);
}
// ---------------------------------------------------------------------------
// `gbrain skillify scaffold`
// ---------------------------------------------------------------------------
interface ScaffoldFlags {
help: boolean;
json: boolean;
dryRun: boolean;
force: boolean;
name: string | null;
description: string | null;
triggers: string[];
writesTo: string[];
writesPages: boolean;
mutating: boolean;
skillsDir: string | null;
}
const HELP_SCAFFOLD = `gbrain skillify scaffold <name> [options]
Create 5 scaffold files for a new skill:
1. skills/<name>/SKILL.md frontmatter + body template
2. skills/<name>/scripts/<name>.mjs deterministic-code stub
3. skills/<name>/routing-eval.jsonl routing fixture seed
4. test/<name>.test.ts vitest skeleton
5. (append) RESOLVER.md or AGENTS.md trigger row under "## Uncategorized"
All generated files carry the SKILLIFY_STUB sentinel until replaced.
\`gbrain check-resolvable --strict\` fails if any skill still has the
sentinel in its committed script.
Options:
--description "..." one-liner for SKILL.md frontmatter (required)
--triggers "p1,p2,p3" trigger phrases (comma-separated; defaults to TBD)
--writes-to "d1,d2" brain dirs this skill will write to
--writes-pages mark the skill as a brain-page writer
--mutating mark the skill as mutating: true
--force overwrite existing stubs (not resolver rows)
--dry-run print the plan; no writes
--json machine-readable plan envelope
--skills-dir PATH override auto-detected skills/
--help show this message
Idempotency: re-running without --force errors on any existing file.
With --force, scaffold files are regenerated BUT resolver rows are
never duplicated (D-CX-7 contract).
`;
function parseScaffoldFlags(argv: string[]): ScaffoldFlags {
const f: ScaffoldFlags = {
help: false,
json: false,
dryRun: false,
force: false,
name: null,
description: null,
triggers: [],
writesTo: [],
writesPages: false,
mutating: false,
skillsDir: null,
};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--help' || a === '-h') f.help = true;
else if (a === '--json') f.json = true;
else if (a === '--dry-run') f.dryRun = true;
else if (a === '--force') f.force = true;
else if (a === '--writes-pages') f.writesPages = true;
else if (a === '--mutating') f.mutating = true;
else if (a === '--description') {
f.description = argv[i + 1] ?? null;
i++;
} else if (a?.startsWith('--description=')) {
f.description = a.slice('--description='.length) || null;
} else if (a === '--triggers') {
const v = argv[i + 1] ?? '';
f.triggers = splitList(v);
i++;
} else if (a?.startsWith('--triggers=')) {
f.triggers = splitList(a.slice('--triggers='.length));
} else if (a === '--writes-to') {
const v = argv[i + 1] ?? '';
f.writesTo = splitList(v);
i++;
} else if (a?.startsWith('--writes-to=')) {
f.writesTo = splitList(a.slice('--writes-to='.length));
} else if (a === '--skills-dir') {
f.skillsDir = argv[i + 1] ?? null;
i++;
} else if (a?.startsWith('--skills-dir=')) {
f.skillsDir = a.slice('--skills-dir='.length) || null;
} else if (a && !a.startsWith('--') && !f.name) {
f.name = a;
}
}
return f;
}
function splitList(v: string): string[] {
return v
.split(',')
.map(s => s.trim())
.filter(Boolean);
}
export async function runSkillifyScaffold(args: string[]): Promise<void> {
const flags = parseScaffoldFlags(args);
if (flags.help) {
console.log(HELP_SCAFFOLD);
process.exit(0);
}
if (!flags.name) {
console.error('Error: skill name is required.\n');
console.error(HELP_SCAFFOLD);
process.exit(2);
}
if (!flags.description) {
console.error('Error: --description is required.\n');
console.error(HELP_SCAFFOLD);
process.exit(2);
}
// Resolve skills directory.
let skillsDir: string | null = null;
if (flags.skillsDir) {
skillsDir = isAbsolute(flags.skillsDir)
? flags.skillsDir
: resolvePath(process.cwd(), flags.skillsDir);
} else {
const detected = autoDetectSkillsDir();
skillsDir = detected.dir;
}
if (!skillsDir) {
console.error(
'Error: could not auto-detect skills/. Pass --skills-dir or set $OPENCLAW_WORKSPACE.',
);
process.exit(2);
}
let plan: ScaffoldPlan;
try {
plan = planScaffold({
skillsDir,
force: flags.force,
vars: {
name: flags.name,
description: flags.description,
triggers: flags.triggers,
writesTo: flags.writesTo,
writesPages: flags.writesPages,
mutating: flags.mutating,
},
});
} catch (err) {
if (err instanceof SkillifyScaffoldError) {
if (flags.json) {
console.log(JSON.stringify({ ok: false, error: err.code, message: err.message }, null, 2));
} else {
console.error(`skillify scaffold: ${err.message}`);
}
process.exit(1);
}
throw err;
}
if (!plan.resolverFile) {
const msg = `${RESOLVER_FILENAMES_LABEL} not found in ${skillsDir} or its parent. Create one before scaffolding skills.`;
if (flags.json) {
console.log(JSON.stringify({ ok: false, error: 'no_resolver', message: msg }, null, 2));
} else {
console.error(msg);
}
process.exit(2);
}
if (flags.dryRun) {
if (flags.json) {
console.log(
JSON.stringify(
{
ok: true,
dryRun: true,
files: plan.files.map(f => ({ path: f.path, kind: f.kind })),
resolverFile: plan.resolverFile,
resolverAppendBytes: plan.resolverAppend?.length ?? 0,
},
null,
2,
),
);
} else {
console.log(`skillify scaffold --dry-run (${plan.files.length} files):`);
for (const f of plan.files) console.log(` [${f.kind}] ${f.path}`);
if (plan.resolverAppend !== null) {
console.log(` [append] ${plan.resolverFile} (+${plan.resolverAppend.length} bytes)`);
} else {
console.log(` [skip] ${plan.resolverFile} (row already present — idempotent)`);
}
}
process.exit(0);
}
applyScaffold(plan);
if (flags.json) {
console.log(
JSON.stringify(
{
ok: true,
dryRun: false,
files: plan.files.map(f => ({ path: f.path, kind: f.kind })),
resolverFile: plan.resolverFile,
resolverAppended: plan.resolverAppend !== null,
},
null,
2,
),
);
} else {
console.log(`skillify scaffold: wrote ${plan.files.length} files.`);
for (const f of plan.files) console.log(` [${f.kind}] ${f.path}`);
if (plan.resolverAppend !== null) {
console.log(` [append] ${plan.resolverFile}`);
}
console.log('\nNext:');
console.log(` 1. Replace SKILLIFY_STUB sentinels in the generated files.`);
console.log(` 2. bun test test/${flags.name}.test.ts`);
console.log(` 3. gbrain skillify check skills/${flags.name}/scripts/${flags.name}.mjs`);
console.log(` 4. gbrain check-resolvable`);
}
}
// ---------------------------------------------------------------------------
// `gbrain skillify check` — delegates to scripts/skillify-check.ts via same
// internal helpers. Current design shells out to the script (kept as the
// single source of truth for the check logic); a future release may inline
// it further.
// ---------------------------------------------------------------------------
async function runSkillifyCheck(args: string[]): Promise<void> {
// Late-import to avoid pulling the helpers at module init.
const { runSkillifyCheckInline } = await import('./skillify-check.ts');
await runSkillifyCheckInline(args);
}