mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
* feat(skillpack): enhance skillify with cross-modal eval quality gate Updates skillify from v1.0.0 to v2.0.0 with the key innovation: cross-modal evaluation runs BEFORE tests (step 3) to establish quality, then tests lock in the proven-good behavior. Key changes: - 11-item checklist (was 10) - adds cross-modal eval as step 3 - Cross-modal eval uses 3 models to score output on 5 dimensions - Quality gate: all dimensions ≥ 7 average before proceeding to tests - Prevents locking in mediocrity through tests-first approach - References cross-modal-review skill for eval pipeline - Updated all gbrain-specific paths (bun test, scripts/*.ts) - Maintains compatibility with gbrain check-resolvable workflow The meta-skill for turning raw features into properly-skilled, tested, resolvable capabilities. Cross-modal eval ensures output quality before tests cement the behavior. * feat: skillify hardened via 2 cross-modal eval cycles (8.1/10) Applied top improvements from GPT-5.5 + Opus 4-7 + DeepSeek V4 Pro: - Named 3 frontier models explicitly with provider table - Inlined eval prompt template with CONTEXT param + scoring calibration - Defined aggregation math: mean >= 7 AND no single dim < 5 - Added eval receipt JSON schema - Structured 3-cycle fix loop with before/after delta tracking - Added worked example (summarize-pr, end-to-end) - Added cost guardrails (skip < 200 tokens, max 9 API calls) - Added representative input selection rule - Added SKILL.md frontmatter template (copy-paste ready) - Added Phase 0 decision gate (is this worth skillifying?) Also includes cross-modal-eval runner recipe with robust JSON parsing for LLMs that return malformed JSON (3-tier repair). * chore(recipes): remove cross-modal-eval.mjs Superseded by `gbrain eval cross-modal` (next commit). The .mjs script was the original PR's hand-rolled provider stack; the replacement reuses src/core/ai/gateway.ts so config/auth/model-aliasing comes from the canonical recipe registry instead of a parallel stack. No code references the .mjs (it was invoked by skill prose only), so this delete is independently safe to bisect through. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): cross-modal-eval core module + unit tests Pure-logic foundation for the new `gbrain eval cross-modal` command (wired in the next commit). All five modules are self-contained — no CLI surface, no I/O outside the receipt writer's mkdirSync. Imported from src/core/ai/gateway.ts at runtime via gwChat (no config impact at load time). Modules: - json-repair.ts: parseModelJSON 4-strategy fallback chain. Adversarial nuclear-option throws rather than fabricating scores (Q6 + Q3 in plan). - aggregate.ts: verdict logic. PASS = (>=2 successes) AND (every dim mean >= 7) AND (every dim min across models >= 5). INCONCLUSIVE when <2/3 models returned parseable scores — closes the v1 .mjs `Object.values({}).every(...) === true` empty-array silent-PASS bug (Q2 + Q3). - receipt-name.ts: receipt filename binds (slug, sha8 of SKILL.md) so `gbrain skillify check` can detect stale audits (T10 in plan). - receipt-write.ts: thin wrapper over writeFileSync that auto-mkdirs the parent directory. Standalone module because gbrainPath() does NOT auto-mkdir (T5 plan correction — Codex caught this). - runner.ts: orchestrator. Promise.allSettled across 3 slots per cycle; up to 3 cycles; stops early on PASS or INCONCLUSIVE. Default slots: openai:gpt-4o / anthropic:claude-opus-4-7 / google:gemini-1.5-pro. estimateCost() exports a small per-model pricing table (drifts; refresh alongside model-family bumps). Tests (32 cases total, all green): - json-repair.test.ts: 10 cases (clean JSON, fences, trailing commas, single quotes, embedded newlines, mismatched braces, nuclear-option success + adversarial throws, empty input, numeric-shorthand scores). - aggregate.test.ts: 8 cases pinning Q2/Q3/dedup. The 0-of-3 INCONCLUSIVE case is the regression guard for the v1 silent-PASS bug. - cli.test.ts: 12 cases on receipt-name / receipt-write / GBRAIN_HOME isolation. Uses withEnv() helper for env mutation (R1 isolation rule). Verifies bisect-clean: typecheck passes, all 32 unit cases green. The runner.ts import of gateway.chat() is dead until commit 3 wires the CLI surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): wire `gbrain eval cross-modal` CLI subcommand User-facing surface for the multi-model quality gate. Three different- provider frontier models score the OUTPUT against the TASK on a 5-dim rubric. Verdict drives exit code: 0 PASS, 1 FAIL, 2 INCONCLUSIVE (<2/3 models returned parseable scores per Q3 in plan). Wiring touches three files: - src/commands/eval-cross-modal.ts (new, ~290 lines) CLI handler. Self-configures the AI gateway from loadConfig() + process.env so it works without `gbrain init` (the cli.ts no-DB branch bypasses connectEngine()). Defaults: cycles=3 in TTY, cycles=1 in non-TTY (T11 partial cost guardrail — limits scripted bulk spend; full --budget-usd hard cap is a v0.27.x TODO). Prints estimated max-cost-per-cycle to stderr before each run. Uses gbrainPath('eval-receipts') for receipt directory. - src/cli.ts (no-DB dispatch branch, 5-line addition) Special-cases `eval cross-modal` BEFORE the existing handleCliOnly path that requires connectEngine(). Mirrors the `dream` no-DB pattern but doesn't even attempt the connect — the command never touches the DB. New users can run the gate before `gbrain init` (T3 in plan). - src/commands/eval.ts (sub-subcommand dispatch) Adds `cross-modal` alongside `export`/`prune`/`replay`. The cli.ts branch takes precedence in the user-facing path; this branch only fires when callers re-enter runEvalCommand with an existing engine. Engine is intentionally unused — the handler self-routes. - test/e2e/cross-modal-eval.test.ts (new, 4 cases) Mocked-fetch E2E. Lives at test/e2e/* (NOT *.serial.test.ts) per plan T8: test/e2e/* is exempt from the test-isolation lint and already runs serially via scripts/run-e2e.sh, so the mock.module() call doesn't need a quarantine rename. Cases: PASS / FAIL (mean<7) / FAIL (min<5 — Q2 floor) / INCONCLUSIVE (2 mock 5xx — Q3 contract). The runner from commit 2 now has live callers. typecheck passes; the 4 E2E cases all green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(skillify): add informational 11th item (cross-modal eval) Promotes the skillify contract from 10 to 11 items. The 11th item (cross-modal eval) is `required:false` per T7 in the plan — a missing or stale receipt surfaces in the audit output but does not fail the gate. Existing skills keep their current required-score; the bump is additive, not breaking. Changes: - src/commands/skillify.ts Header jsdoc updated 10-item -> 11-item. No code-flow changes. - src/commands/skillify-check.ts (the per-skill audit; not src/commands/skillpack-check.ts which is a different command — plan T6 corrected the conflation in the original plan) New informational item at position 11. Reuses findReceiptForSkill() helper from src/core/cross-modal-eval/receipt-name.ts to detect: * found — receipt matches current SKILL.md sha-8 * stale — receipt exists for an older SKILL.md * missing — no receipt yet Audit output cases pass through to existing pretty/JSON formats. - src/core/skillify/templates.ts Scaffolded SKILL.md now includes a "Phase 3: Cross-modal eval (informational)" section with copy-paste `gbrain eval cross-modal` invocation, pass criteria, and receipt-naming convention. Helps new skill authors discover the gate. - test/skillify-scaffold.test.ts New T9 case verifies the scaffold emits the Phase 3 section, points at the correct command, documents the receipt path, and appends exactly one resolver row. Replaces the original plan's `gbrain skillify scaffold demo-eleven` shell verification (which Codex caught as invalid + repo-mutating). Verifies: typecheck passes; scaffold test 19/19 (was 18, +1 T9 case). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: skillify v1.1.0 + cross-modal-eval references Documentation catches up with the new behavior shipped in commits 1-4. - skills/skillify/SKILL.md (1.0.0 -> 1.1.0) Full rewrite. Frontmatter version is additive (T7 in plan); the 11th item is informational, not breaking. Phase 3 now points at `gbrain eval cross-modal` with copy-paste invocation, default slot table, pass criteria, receipt-naming convention, cycles + cost guardrails (T11 partial cap), provider configuration via the AI gateway, and the cycle-1/2/3 fix loop. Adds Output Format section (skills-conformance.test.ts requires it). Drops the original `(or lib/cross-modal-eval.ts)` parenthetical (Q5 plan correction — that path never existed). - skills/cross-modal-review/SKILL.md Adds 4-line Relationship section pointing at `gbrain eval cross-modal` (D3 plan reciprocal). Distinguishes the manual second-opinion gate (this skill) from the automated multi-model score-and-iterate gate (the new command). - CLAUDE.md Key Files entries for src/commands/eval-cross-modal.ts and the five new src/core/cross-modal-eval/* modules. Commands list gains the `gbrain eval cross-modal` entry under v0.27.x. Notes the non-TTY default 1-cycle behavior + the gbrainPath('eval- receipts') resolution. - TODOS.md Four v0.27.x follow-ups filed under a new "cross-modal-eval" section: full --budget-usd cap (T11 follow-up), subagent integration (recovers cross-process rate-leases T4 deferred), skill adoption telemetry (revisit T7=C with data after 30 days), docs/cross-modal-eval.md user guide. - llms-full.txt Regenerated via `bun run build:llms` to match the CLAUDE.md edits — sync guard at test/build-llms.test.ts requires this. Verifies: typecheck passes; skills-conformance 199/199 green; build-llms 7/7 green; full unit fast loop 3861/3861 green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.28.4) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Garry Tan <garrytan@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
314 lines
10 KiB
TypeScript
314 lines
10 KiB
TypeScript
/**
|
|
* gbrain skillify <scaffold|check> — W4 CLI namespace.
|
|
*
|
|
* `scaffold`: creates 5 stub files for a new skill. Mechanical only.
|
|
* `check`: 11-item audit of an existing skill (item 11, cross-modal
|
|
* eval, is informational; T7=C in plans/radiant-napping-lerdorf.md).
|
|
* Promoted from `scripts/skillify-check.ts` (D-CX-2). The
|
|
* legacy script remains as a thin shim that invokes this
|
|
* subcommand.
|
|
*
|
|
* The markdown skill at `skills/skillify/SKILL.md` orchestrates the
|
|
* full 11-step loop (essay's "skillify it!"): scaffold → fill in the
|
|
* body → run cross-modal eval → run check → run check-resolvable →
|
|
* run tests → commit. The CLI primitives do the mechanical steps;
|
|
* the skill carries the judgment steps.
|
|
*/
|
|
|
|
import { isAbsolute, resolve as resolvePath } from 'path';
|
|
|
|
import {
|
|
applyScaffold,
|
|
planScaffold,
|
|
SkillifyScaffoldError,
|
|
SKILL_NAME_PATTERN,
|
|
type ScaffoldPlan,
|
|
} from '../core/skillify/generator.ts';
|
|
import { autoDetectSkillsDir } from '../core/repo-root.ts';
|
|
import { RESOLVER_FILENAMES_LABEL } from '../core/resolver-filenames.ts';
|
|
|
|
// Re-exports for tests.
|
|
export { planScaffold, applyScaffold, SkillifyScaffoldError, SKILL_NAME_PATTERN };
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Top-level dispatcher
|
|
// ---------------------------------------------------------------------------
|
|
|
|
const HELP_TOP = `gbrain skillify <subcommand> [options]
|
|
|
|
Subcommands:
|
|
scaffold <name> Create SKILL.md, script, routing-eval, test stubs
|
|
and append a resolver row. Mechanical; no LLM.
|
|
check [path] Run the 10-item skillify audit on a target path
|
|
(or --recent). Wraps the legacy scripts/skillify-check.ts
|
|
(D-CX-2: subcommand namespace).
|
|
|
|
Run \`gbrain skillify <subcommand> --help\` for per-subcommand options.
|
|
`;
|
|
|
|
export async function runSkillify(args: string[]): Promise<void> {
|
|
const sub = args[0];
|
|
const rest = args.slice(1);
|
|
if (!sub || sub === '--help' || sub === '-h') {
|
|
console.log(HELP_TOP);
|
|
process.exit(0);
|
|
}
|
|
if (sub === 'scaffold') {
|
|
await runSkillifyScaffold(rest);
|
|
return;
|
|
}
|
|
if (sub === 'check') {
|
|
await runSkillifyCheck(rest);
|
|
return;
|
|
}
|
|
console.error(`Unknown subcommand: ${sub}\n`);
|
|
console.error(HELP_TOP);
|
|
process.exit(2);
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// `gbrain skillify scaffold`
|
|
// ---------------------------------------------------------------------------
|
|
|
|
interface ScaffoldFlags {
|
|
help: boolean;
|
|
json: boolean;
|
|
dryRun: boolean;
|
|
force: boolean;
|
|
name: string | null;
|
|
description: string | null;
|
|
triggers: string[];
|
|
writesTo: string[];
|
|
writesPages: boolean;
|
|
mutating: boolean;
|
|
skillsDir: string | null;
|
|
}
|
|
|
|
const HELP_SCAFFOLD = `gbrain skillify scaffold <name> [options]
|
|
|
|
Create 5 scaffold files for a new skill:
|
|
1. skills/<name>/SKILL.md frontmatter + body template
|
|
2. skills/<name>/scripts/<name>.mjs deterministic-code stub
|
|
3. skills/<name>/routing-eval.jsonl routing fixture seed
|
|
4. test/<name>.test.ts vitest skeleton
|
|
5. (append) RESOLVER.md or AGENTS.md trigger row under "## Uncategorized"
|
|
|
|
All generated files carry the SKILLIFY_STUB sentinel until replaced.
|
|
\`gbrain check-resolvable --strict\` fails if any skill still has the
|
|
sentinel in its committed script.
|
|
|
|
Options:
|
|
--description "..." one-liner for SKILL.md frontmatter (required)
|
|
--triggers "p1,p2,p3" trigger phrases (comma-separated; defaults to TBD)
|
|
--writes-to "d1,d2" brain dirs this skill will write to
|
|
--writes-pages mark the skill as a brain-page writer
|
|
--mutating mark the skill as mutating: true
|
|
--force overwrite existing stubs (not resolver rows)
|
|
--dry-run print the plan; no writes
|
|
--json machine-readable plan envelope
|
|
--skills-dir PATH override auto-detected skills/
|
|
--help show this message
|
|
|
|
Idempotency: re-running without --force errors on any existing file.
|
|
With --force, scaffold files are regenerated BUT resolver rows are
|
|
never duplicated (D-CX-7 contract).
|
|
`;
|
|
|
|
function parseScaffoldFlags(argv: string[]): ScaffoldFlags {
|
|
const f: ScaffoldFlags = {
|
|
help: false,
|
|
json: false,
|
|
dryRun: false,
|
|
force: false,
|
|
name: null,
|
|
description: null,
|
|
triggers: [],
|
|
writesTo: [],
|
|
writesPages: false,
|
|
mutating: false,
|
|
skillsDir: null,
|
|
};
|
|
for (let i = 0; i < argv.length; i++) {
|
|
const a = argv[i];
|
|
if (a === '--help' || a === '-h') f.help = true;
|
|
else if (a === '--json') f.json = true;
|
|
else if (a === '--dry-run') f.dryRun = true;
|
|
else if (a === '--force') f.force = true;
|
|
else if (a === '--writes-pages') f.writesPages = true;
|
|
else if (a === '--mutating') f.mutating = true;
|
|
else if (a === '--description') {
|
|
f.description = argv[i + 1] ?? null;
|
|
i++;
|
|
} else if (a?.startsWith('--description=')) {
|
|
f.description = a.slice('--description='.length) || null;
|
|
} else if (a === '--triggers') {
|
|
const v = argv[i + 1] ?? '';
|
|
f.triggers = splitList(v);
|
|
i++;
|
|
} else if (a?.startsWith('--triggers=')) {
|
|
f.triggers = splitList(a.slice('--triggers='.length));
|
|
} else if (a === '--writes-to') {
|
|
const v = argv[i + 1] ?? '';
|
|
f.writesTo = splitList(v);
|
|
i++;
|
|
} else if (a?.startsWith('--writes-to=')) {
|
|
f.writesTo = splitList(a.slice('--writes-to='.length));
|
|
} else if (a === '--skills-dir') {
|
|
f.skillsDir = argv[i + 1] ?? null;
|
|
i++;
|
|
} else if (a?.startsWith('--skills-dir=')) {
|
|
f.skillsDir = a.slice('--skills-dir='.length) || null;
|
|
} else if (a && !a.startsWith('--') && !f.name) {
|
|
f.name = a;
|
|
}
|
|
}
|
|
return f;
|
|
}
|
|
|
|
function splitList(v: string): string[] {
|
|
return v
|
|
.split(',')
|
|
.map(s => s.trim())
|
|
.filter(Boolean);
|
|
}
|
|
|
|
export async function runSkillifyScaffold(args: string[]): Promise<void> {
|
|
const flags = parseScaffoldFlags(args);
|
|
if (flags.help) {
|
|
console.log(HELP_SCAFFOLD);
|
|
process.exit(0);
|
|
}
|
|
if (!flags.name) {
|
|
console.error('Error: skill name is required.\n');
|
|
console.error(HELP_SCAFFOLD);
|
|
process.exit(2);
|
|
}
|
|
if (!flags.description) {
|
|
console.error('Error: --description is required.\n');
|
|
console.error(HELP_SCAFFOLD);
|
|
process.exit(2);
|
|
}
|
|
|
|
// Resolve skills directory.
|
|
let skillsDir: string | null = null;
|
|
if (flags.skillsDir) {
|
|
skillsDir = isAbsolute(flags.skillsDir)
|
|
? flags.skillsDir
|
|
: resolvePath(process.cwd(), flags.skillsDir);
|
|
} else {
|
|
const detected = autoDetectSkillsDir();
|
|
skillsDir = detected.dir;
|
|
}
|
|
if (!skillsDir) {
|
|
console.error(
|
|
'Error: could not auto-detect skills/. Pass --skills-dir or set $OPENCLAW_WORKSPACE.',
|
|
);
|
|
process.exit(2);
|
|
}
|
|
|
|
let plan: ScaffoldPlan;
|
|
try {
|
|
plan = planScaffold({
|
|
skillsDir,
|
|
force: flags.force,
|
|
vars: {
|
|
name: flags.name,
|
|
description: flags.description,
|
|
triggers: flags.triggers,
|
|
writesTo: flags.writesTo,
|
|
writesPages: flags.writesPages,
|
|
mutating: flags.mutating,
|
|
},
|
|
});
|
|
} catch (err) {
|
|
if (err instanceof SkillifyScaffoldError) {
|
|
if (flags.json) {
|
|
console.log(JSON.stringify({ ok: false, error: err.code, message: err.message }, null, 2));
|
|
} else {
|
|
console.error(`skillify scaffold: ${err.message}`);
|
|
}
|
|
process.exit(1);
|
|
}
|
|
throw err;
|
|
}
|
|
|
|
if (!plan.resolverFile) {
|
|
const msg = `${RESOLVER_FILENAMES_LABEL} not found in ${skillsDir} or its parent. Create one before scaffolding skills.`;
|
|
if (flags.json) {
|
|
console.log(JSON.stringify({ ok: false, error: 'no_resolver', message: msg }, null, 2));
|
|
} else {
|
|
console.error(msg);
|
|
}
|
|
process.exit(2);
|
|
}
|
|
|
|
if (flags.dryRun) {
|
|
if (flags.json) {
|
|
console.log(
|
|
JSON.stringify(
|
|
{
|
|
ok: true,
|
|
dryRun: true,
|
|
files: plan.files.map(f => ({ path: f.path, kind: f.kind })),
|
|
resolverFile: plan.resolverFile,
|
|
resolverAppendBytes: plan.resolverAppend?.length ?? 0,
|
|
},
|
|
null,
|
|
2,
|
|
),
|
|
);
|
|
} else {
|
|
console.log(`skillify scaffold --dry-run (${plan.files.length} files):`);
|
|
for (const f of plan.files) console.log(` [${f.kind}] ${f.path}`);
|
|
if (plan.resolverAppend !== null) {
|
|
console.log(` [append] ${plan.resolverFile} (+${plan.resolverAppend.length} bytes)`);
|
|
} else {
|
|
console.log(` [skip] ${plan.resolverFile} (row already present — idempotent)`);
|
|
}
|
|
}
|
|
process.exit(0);
|
|
}
|
|
|
|
applyScaffold(plan);
|
|
|
|
if (flags.json) {
|
|
console.log(
|
|
JSON.stringify(
|
|
{
|
|
ok: true,
|
|
dryRun: false,
|
|
files: plan.files.map(f => ({ path: f.path, kind: f.kind })),
|
|
resolverFile: plan.resolverFile,
|
|
resolverAppended: plan.resolverAppend !== null,
|
|
},
|
|
null,
|
|
2,
|
|
),
|
|
);
|
|
} else {
|
|
console.log(`skillify scaffold: wrote ${plan.files.length} files.`);
|
|
for (const f of plan.files) console.log(` [${f.kind}] ${f.path}`);
|
|
if (plan.resolverAppend !== null) {
|
|
console.log(` [append] ${plan.resolverFile}`);
|
|
}
|
|
console.log('\nNext:');
|
|
console.log(` 1. Replace SKILLIFY_STUB sentinels in the generated files.`);
|
|
console.log(` 2. bun test test/${flags.name}.test.ts`);
|
|
console.log(` 3. gbrain skillify check skills/${flags.name}/scripts/${flags.name}.mjs`);
|
|
console.log(` 4. gbrain check-resolvable`);
|
|
}
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// `gbrain skillify check` — delegates to scripts/skillify-check.ts via same
|
|
// internal helpers. Current design shells out to the script (kept as the
|
|
// single source of truth for the check logic); a future release may inline
|
|
// it further.
|
|
// ---------------------------------------------------------------------------
|
|
|
|
async function runSkillifyCheck(args: string[]): Promise<void> {
|
|
// Late-import to avoid pulling the helpers at module init.
|
|
const { runSkillifyCheckInline } = await import('./skillify-check.ts');
|
|
await runSkillifyCheckInline(args);
|
|
}
|