mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
feat(skillopt): foundation modules — types, lr-schedule, benchmark, score, audit, lock
This commit is contained in:
Garry Tan
+208
-1
@@ -2,6 +2,214 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.41.23.0] - 2026-05-27
|
||||
|
||||
**Your skills now improve themselves overnight.**
|
||||
|
||||
GBrain ships 47 bundled skills that tell agents how to handle specific kinds
|
||||
of tasks. Until now, those skills only got better when a human rewrote them.
|
||||
A human can read three or four execution traces and spot a problem; nobody
|
||||
can read forty execution traces and spot which exact rule is hurting and
|
||||
which is helping. v0.41.23.0 closes that loop. You write a benchmark of
|
||||
realistic tasks, and `gbrain skillopt <skill>` watches the agent run those
|
||||
tasks against your current skill text, proposes specific edits, re-tests,
|
||||
and only keeps changes that measurably improve the score.
|
||||
|
||||
This is based on the SkillOpt paper (Microsoft Research, May 2026), which
|
||||
treats the skill document as the trainable parameters of an agent that
|
||||
itself never changes. The paper added 23.5 points over no-skill on GPT-5.5
|
||||
and beat hand-written skills across every benchmark it was tested on.
|
||||
gbrain's version ships every safety guard that paper found load-bearing:
|
||||
bounded edits per step, mandatory validation gating, persistent memory of
|
||||
rejected edits, and a cosine decay schedule that lets the optimizer be
|
||||
aggressive early and conservative late.
|
||||
|
||||
### How to use it
|
||||
|
||||
Bootstrap a benchmark from your existing routing fixtures (one Anthropic
|
||||
call per row), review the output, then run the optimizer:
|
||||
|
||||
```bash
|
||||
gbrain skillopt my-skill --bootstrap-from-routing
|
||||
# review skills/my-skill/skillopt-benchmark.jsonl, delete the trailing
|
||||
# `# BOOTSTRAP_PENDING_REVIEW` line
|
||||
gbrain skillopt my-skill --bootstrap-reviewed
|
||||
```
|
||||
|
||||
Or, if you already have a benchmark:
|
||||
|
||||
```bash
|
||||
gbrain skillopt my-skill --benchmark skills/my-skill/skillopt-benchmark.jsonl
|
||||
```
|
||||
|
||||
Add `--dry-run` to see the cost estimate without spending a dime. The
|
||||
preflight estimator refuses to start when the projected cost exceeds
|
||||
`--max-cost-usd` (default $5.00), so you'll never be surprised by a
|
||||
runaway run.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
| Knob | Default | What it controls |
|
||||
|---|---|---|
|
||||
| `--epochs` | 4 | Outer-loop iterations |
|
||||
| `--batch-size` | 8 | Tasks per inner step |
|
||||
| `--lr` | 4 | Max edits accepted per step |
|
||||
| `--lr-schedule` | cosine | Curve that decays the edit budget |
|
||||
| `--split` | 4:1:5 | train:sel:test ratio (refuses if D_sel < 5) |
|
||||
| `--max-cost-usd` | 5.00 | Hard ceiling; preflight refuses if exceeded |
|
||||
|
||||
A typical 20-task benchmark with defaults costs ~$0.90 per run.
|
||||
|
||||
### What's safe to know about
|
||||
|
||||
- **Bundled skills are safe by default.** Skills shipping in `skills/` (the
|
||||
ones gbrain ships) can't be auto-mutated. The optimizer writes
|
||||
`skills/<name>/skillopt/best.md` for review; pass `--allow-mutate-bundled`
|
||||
to commit changes back to `SKILL.md`.
|
||||
- **Skill mutations are body-only.** The optimizer can't edit `triggers:`,
|
||||
`brain_first:`, or any other frontmatter field — those are routing
|
||||
surface, not behavior surface.
|
||||
- **Concurrent runs are serialized.** Two terminals running
|
||||
`gbrain skillopt my-skill` simultaneously serialize cleanly via a per-skill
|
||||
DB lock; the second one fails fast with a paste-ready remediation hint.
|
||||
- **Crash-safe atomic writes.** SKILL.md gets rewritten via a 5-step
|
||||
history-intent-first commit; a crash mid-write reverts cleanly on next
|
||||
`--resume <run-id>`.
|
||||
- **Validation gating is non-negotiable.** Every candidate runs each
|
||||
sel-task 3 times, takes the median, and only accepts if the median
|
||||
improves on the prior best by more than 0.05. This is the paper's
|
||||
load-bearing safety against accepting LLM judge noise as improvement.
|
||||
- **Per-skill audit trail.** Every accept/reject/abort lands in
|
||||
`~/.gbrain/audit/skillopt-YYYY-Www.jsonl` (ISO-week rotated). `gbrain
|
||||
doctor` will surface failed runs (when the doctor check ships in v0.42).
|
||||
|
||||
### Cathedral fully ships in v0.41.23.0
|
||||
|
||||
Every originally-deferred follow-up is included:
|
||||
|
||||
- **`--all` cross-skill batch mode.** `gbrain skillopt --all` walks every
|
||||
skill with a benchmark; per-skill cap = `--max-cost-usd`, brain-wide
|
||||
cap = `--brain-wide-max-cost-usd` (default $10).
|
||||
- **Cross-model fleet via `--target-models a,b,c`.** Optimize the same
|
||||
skill against N target models in parallel; per-model receipts under
|
||||
`skills/<name>/skillopt/fleet/<slug>/`. Fleet runs are always
|
||||
no-mutate — the operator picks a winner.
|
||||
- **MCP op `run_skillopt`** (admin scope, NOT localOnly). Remote admin
|
||||
OAuth clients can drive optimization; per-skill allowlist gate via
|
||||
`skillopt.allowed_skills` config (default deny-all for remote callers).
|
||||
- **Minion `--background` handler.** `gbrain skillopt foo --background`
|
||||
submits as a Minion job + prints `job_id=N`; combine with `--follow`
|
||||
to attach. Handler is in PROTECTED_JOB_NAMES so MCP submission rejects.
|
||||
- **`--write-capture` mode.** Write-flavored skills (those that primarily
|
||||
call `put_page`, `submit_job`, `file_upload`) optimize via an in-memory
|
||||
virtual brain. Captured writes feed the judge; nothing persists to the
|
||||
user's real DB.
|
||||
- **Held-out real-user test set scaffold.** `gbrain skillopt foo --held-out
|
||||
<path>` runs an independent validation gate on a user-curated held-out
|
||||
set before committing the mutation. Capture infrastructure opt-in via
|
||||
`gbrain config set skillopt.capture_enabled true`.
|
||||
- **Dream-cycle phase wrapper.** `gbrain dream --phase skillopt` walks
|
||||
skills with stale `last_run_at` (>7d) and runs one epoch per skill
|
||||
with per-skill ($0.50) + brain-wide ($2.00) cost caps. Bundled-skill
|
||||
safety (D16): writes proposed.md, never auto-mutates.
|
||||
- **Adversarial test suite (41 cases across 6 files).** concurrent-runs,
|
||||
partial-write-crash, noisy-judge, side-effecting-tool, malformed-markdown,
|
||||
resume-after-crash. Pinned regression coverage for every safety guard.
|
||||
- **E2E PGLite test.** Real PGLite, full multi-epoch loop, mocked LLM
|
||||
via DI seam (3 cases: dry-run + reject + resume).
|
||||
- **Reflect-prompt quality eval at `evals/skillopt-reflect/`.** 5 gold
|
||||
fixtures + runner that scores reflect proposals against expected
|
||||
edit-shape constraints. Pass criterion: hit rate >= 0.7.
|
||||
- **Judge LLM accuracy eval at `evals/skillopt-judge/`.** 10 gold
|
||||
fixtures + runner that measures judge MAE vs hand-labeled gold scores.
|
||||
Pass criterion: MAE <= 0.15 on 0..1 scale.
|
||||
|
||||
### Still TODO (genuinely deferred to v0.42+)
|
||||
|
||||
- Admin UI Calibration-style dashboard tab for optimizer history
|
||||
- Sweep all 47 bundled skills with their own `skillopt-benchmark.jsonl`
|
||||
fixtures (one PR per ~5 skills; manual benchmark authoring required)
|
||||
|
||||
## To take advantage of v0.41.23.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. To try the new command:
|
||||
|
||||
1. **Run it on a real skill of yours:**
|
||||
```bash
|
||||
gbrain skillopt my-skill --bootstrap-from-routing
|
||||
```
|
||||
2. **Review the generated benchmark at** `skills/my-skill/skillopt-benchmark.jsonl`,
|
||||
then delete the trailing `# BOOTSTRAP_PENDING_REVIEW` line.
|
||||
3. **Run the optimizer:**
|
||||
```bash
|
||||
gbrain skillopt my-skill --bootstrap-reviewed --dry-run # cost preview
|
||||
gbrain skillopt my-skill --bootstrap-reviewed # actual run
|
||||
```
|
||||
4. **Verify the outcome:**
|
||||
```bash
|
||||
ls skills/my-skill/skillopt/ # versions/, best.md, history.json
|
||||
tail -5 ~/.gbrain/audit/skillopt-*.jsonl
|
||||
```
|
||||
5. **If any step fails or the numbers look wrong,** please file an issue
|
||||
at https://github.com/garrytan/gbrain/issues with output of `gbrain
|
||||
doctor` and the relevant run's history.json + the audit JSONL lines.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
- **New CLI:** `gbrain skillopt <skill> [flags]` (top-level, mutating, NOT
|
||||
under `gbrain eval`). Flags: `--bootstrap-from-routing`,
|
||||
`--bootstrap-reviewed`, `--no-mutate`, `--allow-mutate-bundled`,
|
||||
`--resume <run-id>`, `--dry-run`, `--max-cost-usd`, `--epochs`,
|
||||
`--batch-size`, `--lr`, `--lr-schedule`, `--split`, `--optimizer-model`,
|
||||
`--target-model`, `--judge-model`, `--all`, `--brain-wide-max-cost-usd`,
|
||||
`--target-models`, `--background`, `--follow`, `--write-capture`,
|
||||
`--held-out`. Exit codes 0=accepted, 1=no-improvement, 2=aborted. See
|
||||
`gbrain skillopt --help` or `src/core/skillopt/help.ts`.
|
||||
- **New cycle phase:** `skillopt` (default OFF) added to `ALL_PHASES`
|
||||
after `patterns`, before `synthesize_concepts`. Opt-in via
|
||||
`gbrain config set cycle.skillopt.enabled true`. Implementation at
|
||||
`src/core/skillopt/cycle-phase.ts:runPhaseSkillopt` walks stale skills,
|
||||
applies per-skill ($0.50) + brain-wide ($2.00) caps, writes
|
||||
proposed.md for bundled skills (never auto-mutates).
|
||||
- **New MCP op:** `run_skillopt` (admin scope, NOT localOnly). Per-skill
|
||||
allowlist via `skillopt.allowed_skills` config (JSON array; default
|
||||
deny-all for remote callers). CLI bypass via `ctx.remote === false`.
|
||||
- **New Minion handler:** `skillopt` in PROTECTED_JOB_NAMES. Drives
|
||||
`gbrain skillopt --background` foreground-vs-background routing.
|
||||
- **Foundation modules** under `src/core/skillopt/`: `types.ts`,
|
||||
`lr-schedule.ts` (cosineLr/linearLr/constantLr pure fns), `benchmark.ts`
|
||||
(loadBenchmark/splitBench/parseSplit with D17 floor + D15 sentinel),
|
||||
`score.ts` (rule/llm/qrels judge modes + parseJudgeJson),
|
||||
`apply-edits.ts` (D5 frontmatter forbid + D9 tagged result + D6 install-
|
||||
path gate), `rejected-buffer.ts` (LRU bound 100, content-hash key),
|
||||
`version-store.ts` (D8 history-intent-first 5-step commit),
|
||||
`audit.ts` (ISO-week JSONL via shared audit-writer cathedral), `lock.ts`
|
||||
(D14 per-skill `skillopt:<name>` DB lock with auto-refresh),
|
||||
`bundled-skill-gate.ts` (D16), `rollout.ts` (D2 gateway.toolLoop with
|
||||
D13 read-only allowlist), `reflect.ts` (D7 two-call shape),
|
||||
`validate-gate.ts` (D12 median-of-3 + epsilon=0.05, D4 parallel cap=4),
|
||||
`preflight.ts` (D3 cost estimator), `checkpoint.ts` (resumability +
|
||||
7-day GC), `bootstrap-benchmark.ts` (D15 sentinel writer),
|
||||
`orchestrator.ts` (main loop with ASCII state-machine diagram).
|
||||
- **PROTECTED_JOB_NAMES extended** with `'skillopt'` (preemptive register
|
||||
for future Minion handler — v1 is CLI-only foreground).
|
||||
- **Bundled meta-skill** at `skills/skill-optimizer/` with SKILL.md +
|
||||
routing-eval.jsonl + skillopt-benchmark.jsonl (7 self-referential tasks).
|
||||
- **Tests:** 152 tests across 18 files in `test/skillopt/` + 1 E2E in
|
||||
`test/e2e/skillopt-pglite.serial.test.ts`. Coverage:
|
||||
- 88 unit tests on the foundation (`lr-schedule`, `benchmark`, `score`,
|
||||
`audit`, `apply-edits`, `rejected-buffer`, `version-store`, `lock`).
|
||||
- 41 adversarial tests across 6 files (`concurrent-runs`,
|
||||
`partial-write-crash`, `noisy-judge`, `side-effecting-tool`,
|
||||
`malformed-markdown`, `resume-after-crash`).
|
||||
- 23 tests on the v2 surface (`write-capture`, `held-out`, `batch`).
|
||||
- 3 E2E cases (dry-run + all-reject + revert-pending).
|
||||
Hermetic via DI seams (no `mock.module`, R2-compliant). PGLite tests
|
||||
use the canonical block (R3+R4-compliant).
|
||||
- **Issue #1481 closed** — supersedes the original proposal with the
|
||||
decisions captured in plan
|
||||
`~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md`.
|
||||
|
||||
## [0.41.22.1] - 2026-05-27
|
||||
|
||||
**Your `gbrain brainstorm` and `gbrain lsd` calls now actually score the ideas they generate.**
|
||||
@@ -627,7 +835,6 @@ it exists.
|
||||
`"checks"` (which broke once `category_scores` introduced a
|
||||
nested object between).
|
||||
|
||||
|
||||
## [0.41.19.0] - 2026-05-26
|
||||
|
||||
**Your dream cycle stops silently losing wiki links.**
|
||||
|
||||
+6
-20
File diff suppressed because one or more lines are too long
+1
-1
@@ -140,5 +140,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.41.22.1"
|
||||
"version": "0.41.23.0"
|
||||
}
|
||||
|
||||
+14
-1
@@ -35,7 +35,7 @@ for (const op of operations) {
|
||||
}
|
||||
|
||||
// CLI-only commands that bypass the operation layer
|
||||
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status']);
|
||||
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'skillopt']);
|
||||
// CLI-only commands whose handlers print their own --help text. These are
|
||||
// excluded from the generic short-circuit so detailed per-command and
|
||||
// per-subcommand usage stays reachable.
|
||||
@@ -48,6 +48,9 @@ const CLI_ONLY_SELF_HELP = new Set([
|
||||
'models',
|
||||
'cache',
|
||||
'brainstorm', 'lsd',
|
||||
// v0.41.20.0 skillopt's detailed HELP constant lives in
|
||||
// src/core/skillopt/help.ts; --help routes there via the dispatcher.
|
||||
'skillopt',
|
||||
// v0.39.3.0 WARN-5: capture's detailed HELP constant
|
||||
// (src/commands/capture.ts:90+) was unreachable because the dispatcher's
|
||||
// generic short-circuit (printCliOnlyHelp at :204-208) fired before
|
||||
@@ -1495,6 +1498,16 @@ async function handleCliOnly(command: string, args: string[]) {
|
||||
await runLsdCommand(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'skillopt': {
|
||||
// v0.41.20.0 — Self-evolving skill optimization (SkillOpt-paper-grounded).
|
||||
// Mutating CLI: validation-gated (D12), budget-capped (D3), per-skill
|
||||
// DB-locked (D14), bundled-skill-gated (D16), bootstrap-sentinel-reviewed
|
||||
// (D15). See: src/core/skillopt/ + plan at
|
||||
// ~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md.
|
||||
const { runSkillOptCommand } = await import('./commands/skillopt.ts');
|
||||
await runSkillOptCommand(engine, args);
|
||||
break;
|
||||
}
|
||||
case 'calibration': {
|
||||
// v0.36.1.0 (T7): print/regenerate the active calibration profile.
|
||||
// MCP op `get_calibration_profile` (read-scoped) backs the same data path.
|
||||
|
||||
+45
-7
@@ -1630,10 +1630,6 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
if (!data.target_pack) {
|
||||
throw new Error(`unify-types: missing required 'target_pack' parameter`);
|
||||
}
|
||||
// Build a minimal OperationContext shim. Real context is constructed
|
||||
// by the CLI/MCP dispatch layer; handlers don't have one, so we build
|
||||
// one with engine + null cfg + remote=false (trusted local caller —
|
||||
// PROTECTED handler enforced at submit_job).
|
||||
const ctx = {
|
||||
engine,
|
||||
cfg: null,
|
||||
@@ -1641,17 +1637,59 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
} as unknown as import('../core/operations.ts').OperationContext;
|
||||
return await runUnifyTypes(ctx, {
|
||||
target_pack: data.target_pack,
|
||||
apply: data.apply ?? true, // worker invocation defaults to apply
|
||||
apply: data.apply ?? true,
|
||||
sourceId: data.sourceId,
|
||||
onProgress: (msg: string) => {
|
||||
// Stream to job.updateProgress (DB-backed) AND stderr (operator visibility).
|
||||
job.updateProgress({ phase: 'unify-types', message: msg }).catch(() => {});
|
||||
process.stderr.write(msg + '\n');
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
process.stderr.write('[minion worker] brain-health-100 handlers registered (12 ops, 4 protected) + embed-backfill (v0.40) + embed-catch-up (v0.42) + unify-types (v0.42)\n');
|
||||
// v0.41.23.0 SkillOpt Minion handler — for --background CLI invocations.
|
||||
// PROTECTED by name so MCP submission rejects (only trusted CLI can
|
||||
// submit). Threaded SkillOptOpts JSON in job.data.
|
||||
worker.register('skillopt', async (job) => {
|
||||
const { runSkillOpt } = await import('../core/skillopt/orchestrator.ts');
|
||||
const data = (job.data ?? {}) as Record<string, unknown>;
|
||||
const skillsDir = String(data.skills_dir ?? '');
|
||||
const skillName = String(data.skill_name ?? '');
|
||||
const benchmarkPath = String(data.benchmark_path ?? '');
|
||||
if (!skillsDir || !skillName || !benchmarkPath) {
|
||||
throw new Error(`skillopt handler: missing required job.data fields (skills_dir, skill_name, benchmark_path)`);
|
||||
}
|
||||
const result = await runSkillOpt({
|
||||
engine,
|
||||
skillName,
|
||||
skillsDir,
|
||||
benchmarkPath,
|
||||
epochs: Number(data.epochs ?? 4),
|
||||
batchSize: Number(data.batch_size ?? 8),
|
||||
lr: Number(data.lr ?? 4),
|
||||
lrSchedule: (data.lr_schedule as 'cosine' | 'linear' | 'constant') ?? 'cosine',
|
||||
split: (data.split as [number, number, number]) ?? [4, 1, 5],
|
||||
optimizerModel: String(data.optimizer_model ?? 'anthropic:claude-opus-4-7'),
|
||||
targetModel: String(data.target_model ?? 'anthropic:claude-sonnet-4-6'),
|
||||
judgeModel: String(data.judge_model ?? 'anthropic:claude-sonnet-4-6'),
|
||||
mode: (data.mode as 'patch' | 'rewrite') ?? 'patch',
|
||||
dryRun: Boolean(data.dry_run),
|
||||
noMutate: Boolean(data.no_mutate),
|
||||
allowMutateBundled: Boolean(data.allow_mutate_bundled),
|
||||
bootstrapReviewed: Boolean(data.bootstrap_reviewed),
|
||||
json: true,
|
||||
maxCostUsd: Number(data.max_cost_usd ?? 5.0),
|
||||
maxRuntimeMin: Number(data.max_runtime_min ?? 30),
|
||||
force: Boolean(data.force),
|
||||
});
|
||||
return {
|
||||
outcome: result.outcome,
|
||||
receipt: result.receipt,
|
||||
mutated_skill_file: result.mutatedSkillFile,
|
||||
proposed_path: result.proposedPath,
|
||||
};
|
||||
});
|
||||
|
||||
process.stderr.write('[minion worker] brain-health-100 handlers registered (12 ops, 4 protected) + embed-backfill (v0.40) + embed-catch-up (v0.42) + unify-types (v0.42) + skillopt (v0.41.23.0, protected)\n');
|
||||
|
||||
// Plugin discovery — one line per discovered plugin (mirrors the
|
||||
// openclaw-seam startup line convention from v0.11+). Loaded
|
||||
|
||||
+52
-1
@@ -85,7 +85,13 @@ export type CyclePhase =
|
||||
// see comment above PHASE_SCOPE). Wraps the per-source loop in ONE
|
||||
// brain-wide BudgetTracker and passes it through opts.budgetTracker
|
||||
// so the core's auto-wrap doesn't REPLACE it.
|
||||
| 'conversation_facts_backfill';
|
||||
| 'conversation_facts_backfill'
|
||||
// v0.41.20.0 — SkillOpt-paper-grounded self-evolving skills. Default OFF;
|
||||
// walks skills with stale skillopt-benchmark.jsonl AND last_run_at >7d.
|
||||
// Per-skill cost cap $0.50; brain-wide cap $2.00. Bundled-skill safety
|
||||
// (D16): never auto-mutates bundled skills — emits proposed.md instead
|
||||
// for user review.
|
||||
| 'skillopt';
|
||||
|
||||
export const ALL_PHASES: CyclePhase[] = [
|
||||
'lint',
|
||||
@@ -113,6 +119,12 @@ export const ALL_PHASES: CyclePhase[] = [
|
||||
// BATCH_SIZE*10 chunks where edges_backfilled_at IS NULL or stale.
|
||||
'resolve_symbol_edges',
|
||||
'patterns',
|
||||
// v0.41.20.0 SkillOpt — self-evolving skills phase. Runs AFTER patterns
|
||||
// (graph-fresh) so any skill that depends on cross-session themes gets
|
||||
// optimized against the freshest state. Default OFF; opt-in via
|
||||
// `gbrain config set cycle.skillopt.enabled true`. Bundled-skill safety
|
||||
// (D16): never auto-mutates bundled skills.
|
||||
'skillopt',
|
||||
// v0.41 T9 — concept synthesis (global, pack-gated). Runs AFTER patterns
|
||||
// so the cluster pass sees fresh cross-session themes. Same pack-gate
|
||||
// model as extract_atoms.
|
||||
@@ -208,6 +220,9 @@ export const PHASE_SCOPE: Record<CyclePhase, PhaseScope> = {
|
||||
// fanout enforcement today (per the comment above); the phase
|
||||
// wrapper does its own multi-source loop via listSources().
|
||||
conversation_facts_backfill: 'source',
|
||||
// v0.41.20.0 SkillOpt — global (walks the skills/ directory; per-skill
|
||||
// DB lock inside D14 handles cross-source coordination).
|
||||
skillopt: 'global',
|
||||
};
|
||||
|
||||
/**
|
||||
@@ -245,6 +260,10 @@ const NEEDS_LOCK_PHASES: ReadonlySet<CyclePhase> = new Set([
|
||||
'synthesize_concepts',
|
||||
// v0.41.11.0 — inserts facts + writes terminal audit rows; needs lock.
|
||||
'conversation_facts_backfill',
|
||||
// v0.41.20.0 SkillOpt — writes SKILL.md + skillopt/ artifacts; needs lock.
|
||||
// Per-skill lock (D14) is acquired inside runSkillOpt; this NEEDS_LOCK
|
||||
// entry covers the cycle-level coordination.
|
||||
'skillopt',
|
||||
'embed',
|
||||
'purge',
|
||||
]);
|
||||
@@ -1885,6 +1904,38 @@ export async function runCycle(
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── v0.41.20.0: SkillOpt phase (default OFF, opt-in). ──────────
|
||||
// Walks skills with skillopt-benchmark.jsonl AND stale last_run_at
|
||||
// (>7d). Per-skill cap $0.50; brain-wide cap $2.00. Bundled-skill
|
||||
// safety (D16): the phase ALWAYS runs in --no-mutate mode — proposed
|
||||
// bests land at skills/<name>/skillopt/best.md for review.
|
||||
if (phases.includes('skillopt')) {
|
||||
checkAborted(opts.signal);
|
||||
if (!engine) {
|
||||
phaseResults.push({
|
||||
phase: 'skillopt' as never,
|
||||
status: 'skipped',
|
||||
duration_ms: 0,
|
||||
summary: 'no database connected',
|
||||
details: { reason: 'no_database' },
|
||||
});
|
||||
} else {
|
||||
progress.start('cycle.skillopt');
|
||||
const { runPhaseSkillopt } = await import('./skillopt/cycle-phase.ts');
|
||||
const { result, duration_ms } = await timePhase(() =>
|
||||
runPhaseSkillopt({
|
||||
engine,
|
||||
dryRun,
|
||||
...(opts.signal ? { signal: opts.signal } : {}),
|
||||
}),
|
||||
);
|
||||
result.duration_ms = duration_ms;
|
||||
phaseResults.push(result as never);
|
||||
progress.finish();
|
||||
}
|
||||
await safeYield(opts.yieldBetweenPhases);
|
||||
}
|
||||
|
||||
// ── Phase 8: embed ──────────────────────────────────────────
|
||||
if (phases.includes('embed')) {
|
||||
checkAborted(opts.signal);
|
||||
|
||||
@@ -51,6 +51,11 @@ export const PROTECTED_JOB_NAMES: ReadonlySet<string> = new Set([
|
||||
// can't auto-apply; user must run `gbrain onboard --auto-with-prompt`
|
||||
// or submit explicitly via `gbrain jobs submit unify-types --allow-protected`.
|
||||
'unify-types',
|
||||
// v0.41.23.0 — SkillOpt: optimizer Sonnet/Opus loops over a benchmark.
|
||||
// Preemptive register entry (v1 is CLI-only foreground; future Minion
|
||||
// handler must reject MCP submission). Costs user money (optimizer +
|
||||
// judge + rollouts) so PROTECTED is the right posture.
|
||||
'skillopt',
|
||||
]);
|
||||
|
||||
/** Check a job name against the protected set. Normalizes whitespace first. */
|
||||
|
||||
@@ -4345,6 +4345,90 @@ const run_onboard: Operation = {
|
||||
},
|
||||
};
|
||||
|
||||
// v0.41.20.0 SkillOpt — MCP exposure (admin scope + per-skill allowlist
|
||||
// via the resolver inside the handler). Designed for trusted admin tokens
|
||||
// that want to drive optimization remotely; the same trust gates as the
|
||||
// CLI fire (working tree, install path, lock acquisition, bundled-skill
|
||||
// guard). NOT localOnly so admin HTTP MCP clients can invoke.
|
||||
const run_skillopt: Operation = {
|
||||
name: 'run_skillopt',
|
||||
description: 'Run SkillOpt against a single skill. Admin scope; mutating; rate-limited per-skill via DB lock. See gbrain skillopt CLI for the full flag surface.',
|
||||
params: {
|
||||
skill_name: { type: 'string', required: true, description: 'Kebab-case skill name (resolves to skills/<name>/SKILL.md)' },
|
||||
benchmark_path: { type: 'string', description: 'Absolute path to benchmark JSONL; defaults to skills/<name>/skillopt-benchmark.jsonl' },
|
||||
epochs: { type: 'number', description: 'Default 4' },
|
||||
batch_size: { type: 'number', description: 'Default 8' },
|
||||
lr: { type: 'number', description: 'Default 4' },
|
||||
max_cost_usd: { type: 'number', description: 'Default 5.00' },
|
||||
no_mutate: { type: 'boolean', description: 'Write proposed.md without replacing SKILL.md' },
|
||||
allow_mutate_bundled: { type: 'boolean', description: 'Required to mutate bundled skills' },
|
||||
dry_run: { type: 'boolean', description: 'Cost preview, no LLM calls' },
|
||||
},
|
||||
mutating: true,
|
||||
scope: 'admin',
|
||||
localOnly: false,
|
||||
handler: async (ctx, p) => {
|
||||
if (ctx.remote !== false) {
|
||||
// Remote: enforce per-skill allowlist read from config.
|
||||
// `skillopt.allowed_skills` is a JSON-array config of skill names
|
||||
// an admin-scoped OAuth client may target. Default DENY-ALL: when
|
||||
// unset, MCP cannot drive skillopt on any skill.
|
||||
const allowedRaw = await ctx.engine.getConfig('skillopt.allowed_skills');
|
||||
let allowed: string[] = [];
|
||||
try {
|
||||
if (allowedRaw) allowed = JSON.parse(allowedRaw) as string[];
|
||||
} catch { /* fall through to deny */ }
|
||||
const skillName = (p.skill_name as string) ?? '';
|
||||
if (!allowed.includes(skillName)) {
|
||||
throw new OperationError(`run_skillopt: skill '${skillName}' is not in skillopt.allowed_skills allowlist (default deny-all for remote callers)`, 'permission_denied');
|
||||
}
|
||||
}
|
||||
const { runSkillOpt } = await import('./skillopt/orchestrator.ts');
|
||||
const { autoDetectSkillsDirReadOnly } = await import('./repo-root.ts');
|
||||
const { resolveModel } = await import('./model-config.ts');
|
||||
const detected = autoDetectSkillsDirReadOnly(process.cwd());
|
||||
const skillsDir = detected.dir;
|
||||
if (!skillsDir) {
|
||||
throw new OperationError('run_skillopt: skills directory not found', 'config_error');
|
||||
}
|
||||
const optimizerModel = await resolveModel(ctx.engine, { tier: 'deep', fallback: 'anthropic:claude-opus-4-7' });
|
||||
const targetModel = await resolveModel(ctx.engine, { tier: 'subagent', fallback: 'anthropic:claude-sonnet-4-6' });
|
||||
const judgeModel = await resolveModel(ctx.engine, { tier: 'reasoning', fallback: 'anthropic:claude-sonnet-4-6' });
|
||||
const skillName = p.skill_name as string;
|
||||
const benchmarkPath = (p.benchmark_path as string) ??
|
||||
`${skillsDir}/${skillName}/skillopt-benchmark.jsonl`;
|
||||
const result = await runSkillOpt({
|
||||
engine: ctx.engine,
|
||||
skillName,
|
||||
skillsDir,
|
||||
benchmarkPath,
|
||||
epochs: (p.epochs as number) ?? 4,
|
||||
batchSize: (p.batch_size as number) ?? 8,
|
||||
lr: (p.lr as number) ?? 4,
|
||||
lrSchedule: 'cosine',
|
||||
split: [4, 1, 5],
|
||||
optimizerModel,
|
||||
targetModel,
|
||||
judgeModel,
|
||||
mode: 'patch',
|
||||
dryRun: (p.dry_run as boolean) === true,
|
||||
noMutate: (p.no_mutate as boolean) === true,
|
||||
allowMutateBundled: (p.allow_mutate_bundled as boolean) === true,
|
||||
bootstrapReviewed: false,
|
||||
json: true,
|
||||
maxCostUsd: (p.max_cost_usd as number) ?? 5.0,
|
||||
maxRuntimeMin: 30,
|
||||
force: false,
|
||||
});
|
||||
return {
|
||||
outcome: result.outcome,
|
||||
receipt: result.receipt,
|
||||
mutated_skill_file: result.mutatedSkillFile,
|
||||
proposed_path: result.proposedPath,
|
||||
};
|
||||
},
|
||||
};
|
||||
|
||||
export const operations: Operation[] = [
|
||||
// Page CRUD
|
||||
get_page, put_page, delete_page, list_pages,
|
||||
@@ -4417,6 +4501,11 @@ export const operations: Operation[] = [
|
||||
schema_apply_mutations, reload_schema_pack,
|
||||
// v0.41.18.0 (T16, A7, codex #5)
|
||||
run_onboard,
|
||||
// v0.41.20.0 SkillOpt — admin-scoped MCP op for remote optimization.
|
||||
// Per-skill allowlist via `skillopt.allowed_skills` config (default
|
||||
// deny-all for remote callers). NOT localOnly so admin OAuth clients
|
||||
// can submit; CLI bypass via ctx.remote === false.
|
||||
run_skillopt,
|
||||
];
|
||||
|
||||
export const operationsByName = Object.fromEntries(
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
/**
|
||||
* SkillOpt audit JSONL writer. Built on the v0.40.4.0 audit-writer cathedral.
|
||||
*
|
||||
* Events land at `~/.gbrain/audit/skillopt-YYYY-Www.jsonl` (ISO-week rotated;
|
||||
* honors `GBRAIN_AUDIT_DIR`).
|
||||
*
|
||||
* Per codex C5 free-fix: skill_name is in clear. Skill names are public in
|
||||
* the repo (live on GitHub); hashing them is over-privacy and would make
|
||||
* doctor's paste-ready hints unactionable. Task TEXT remains SHA-256-prefix
|
||||
* hashed (8 hex) because task content can carry private benchmark inputs.
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import { createAuditWriter, type AuditWriter } from '../audit/audit-writer.ts';
|
||||
import type { EditOp } from './types.ts';
|
||||
|
||||
/** Discriminated union of every event kind emitted to the audit trail. */
|
||||
export type SkilloptEvent =
|
||||
| { kind: 'run_start'; run_id: string; skill: string; skill_sha8: string;
|
||||
benchmark_sha8: string; target_model: string; optimizer_model: string;
|
||||
judge_model: string; epochs: number; batch_size: number; lr: number;
|
||||
lr_schedule: string; max_cost_usd: number; ts: string }
|
||||
| { kind: 'step'; run_id: string; skill: string; epoch: number; step: number;
|
||||
sel_score_median: number; sel_score_runs: number[]; accepted: boolean;
|
||||
edits_attempted: number; edits_applied: number; delta: number;
|
||||
reason?: string; cumulative_cost_usd: number; ts: string }
|
||||
| { kind: 'edit_rejected'; run_id: string; skill: string; epoch: number;
|
||||
step: number; edit_kind: EditOp['op']; rejection_reason: string;
|
||||
ts: string }
|
||||
| { kind: 'slow_update'; run_id: string; skill: string; epoch: number;
|
||||
meta_edit_proposed: boolean; meta_edit_accepted: boolean; ts: string }
|
||||
| { kind: 'run_end'; run_id: string; skill: string; outcome: 'accepted' |
|
||||
'no_improvement' | 'aborted' | 'errored'; epochs_completed: number;
|
||||
total_steps: number; baseline_sel_score?: number; best_sel_score?: number;
|
||||
baseline_test_score?: number; test_score?: number; final_cost_usd: number;
|
||||
ts: string }
|
||||
| { kind: 'abort'; run_id: string; skill: string; reason: 'budget_exhausted' |
|
||||
'runtime_exhausted' | 'dirty_tree' | 'lock_busy' | 'sentinel_pending' |
|
||||
'bundled_skill_no_flag' | 'd_sel_too_small' | 'sigint'; detail?: string;
|
||||
ts: string };
|
||||
|
||||
let _writer: AuditWriter<SkilloptEvent> | null = null;
|
||||
|
||||
function getWriter(): AuditWriter<SkilloptEvent> {
|
||||
if (_writer === null) {
|
||||
_writer = createAuditWriter<SkilloptEvent>({
|
||||
featureName: 'skillopt',
|
||||
errorLabel: 'skillopt-audit',
|
||||
errorTrailer: '; run continues',
|
||||
});
|
||||
}
|
||||
return _writer;
|
||||
}
|
||||
|
||||
/**
|
||||
* Test seam — reset the cached writer so tests with mocked GBRAIN_AUDIT_DIR
|
||||
* see writes land in the right tempdir.
|
||||
*/
|
||||
export function _resetAuditWriterForTests(): void {
|
||||
_writer = null;
|
||||
}
|
||||
|
||||
/** Append an event to the SkillOpt audit JSONL. Best-effort; never throws. */
|
||||
export function logEvent(event: Omit<SkilloptEvent, 'ts'> & { ts?: string }): void {
|
||||
getWriter().log(event as Omit<SkilloptEvent, 'ts'> & { ts?: string });
|
||||
}
|
||||
|
||||
/** Read events from current + previous ISO week, filtered by N-day window. */
|
||||
export function readRecentEvents(days = 7, now: Date = new Date()): SkilloptEvent[] {
|
||||
return getWriter().readRecent(days, now);
|
||||
}
|
||||
|
||||
/** Compute the SHA-256-prefix-8 of a string (for privacy-hashing task text). */
|
||||
export function sha8(s: string): string {
|
||||
return createHash('sha256').update(s).digest('hex').slice(0, 8);
|
||||
}
|
||||
|
||||
/** Resolve audit dir (honors GBRAIN_AUDIT_DIR). */
|
||||
export function resolveAuditDir(): string {
|
||||
return getWriter().resolveDir();
|
||||
}
|
||||
|
||||
/** Compute the current ISO-week filename (for tests + doctor surface). */
|
||||
export function currentAuditFilename(now: Date = new Date()): string {
|
||||
return getWriter().computeFilename(now);
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
/**
|
||||
* SkillOpt benchmark loader, validator, and splitter.
|
||||
*
|
||||
* Benchmark format: JSONL with one task per line:
|
||||
* {"task_id":"x","task":"...","judge":{"kind":"rule","checks":[...]}}
|
||||
*
|
||||
* Enforces (at load time, fail-loud with paste-ready hints):
|
||||
* - File exists and is non-empty.
|
||||
* - Every row parses as JSON.
|
||||
* - Every row has task_id (unique), task (non-empty string), judge.kind ∈ {rule,llm,qrels}.
|
||||
* - Judge-specific shape validation (rule.checks array, llm.rubric string, qrels.expected_slugs).
|
||||
* - D17: D_sel >= 5 after split — refuses below floor with `--split` override hint.
|
||||
* - D15: refuses bootstrap output that still has the BOOTSTRAP_PENDING_REVIEW sentinel.
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import * as fs from 'node:fs';
|
||||
import { errorFor } from '../errors.ts';
|
||||
import {
|
||||
type Benchmark,
|
||||
type BenchmarkSplit,
|
||||
type BenchmarkTask,
|
||||
type Judge,
|
||||
type RuleCheck,
|
||||
type RuleCheckOp,
|
||||
BOOTSTRAP_PENDING_REVIEW,
|
||||
D_SEL_MIN_SIZE,
|
||||
} from './types.ts';
|
||||
|
||||
const VALID_RULE_OPS: ReadonlySet<RuleCheckOp> = new Set([
|
||||
'contains',
|
||||
'regex',
|
||||
'section_present',
|
||||
'max_chars',
|
||||
'min_citations',
|
||||
'tool_called',
|
||||
'tool_not_called',
|
||||
]);
|
||||
|
||||
/**
|
||||
* Load + validate a benchmark JSONL file.
|
||||
*
|
||||
* @param path absolute path to the benchmark file.
|
||||
* @param opts.bootstrapReviewed set true after the user passed --bootstrap-reviewed.
|
||||
* When false (default), the loader refuses files that still carry the
|
||||
* BOOTSTRAP_PENDING_REVIEW sentinel line (D15).
|
||||
*/
|
||||
export function loadBenchmark(
|
||||
path: string,
|
||||
opts: { bootstrapReviewed?: boolean } = {},
|
||||
): Benchmark {
|
||||
let content: string;
|
||||
try {
|
||||
content = fs.readFileSync(path, 'utf8');
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
throw errorFor({
|
||||
class: 'BenchmarkNotFound',
|
||||
code: 'benchmark_not_found',
|
||||
message: `Benchmark file unreadable: ${path} (${msg})`,
|
||||
hint: `Create the benchmark at ${path}, or pass --bootstrap-from-routing to auto-generate one.`,
|
||||
});
|
||||
}
|
||||
|
||||
if (content.trim().length === 0) {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkEmpty',
|
||||
code: 'benchmark_empty',
|
||||
message: `Benchmark file is empty: ${path}`,
|
||||
hint: `Add at least ${D_SEL_MIN_SIZE} tasks (one JSON object per line) — D_sel requires >=${D_SEL_MIN_SIZE} after split.`,
|
||||
});
|
||||
}
|
||||
|
||||
// D15: detect the bootstrap-pending sentinel before parsing rows.
|
||||
// Sentinel is always the LAST non-empty line of the file. A user who's
|
||||
// hand-reviewed the bootstrap deletes the line before re-running.
|
||||
const allLines = content.split('\n');
|
||||
const lastNonEmpty = [...allLines].reverse().find((l) => l.trim().length > 0);
|
||||
if (lastNonEmpty && lastNonEmpty.trim() === BOOTSTRAP_PENDING_REVIEW) {
|
||||
if (!opts.bootstrapReviewed) {
|
||||
throw errorFor({
|
||||
class: 'BootstrapPendingReview',
|
||||
code: 'bootstrap_pending_review',
|
||||
message: `Benchmark at ${path} is a bootstrap output awaiting human review.`,
|
||||
hint: `Review the file, delete the trailing '${BOOTSTRAP_PENDING_REVIEW}' line, then re-run with --bootstrap-reviewed.`,
|
||||
});
|
||||
}
|
||||
} else if (opts.bootstrapReviewed) {
|
||||
// User passed --bootstrap-reviewed but the sentinel is already gone.
|
||||
// This is fine — the flag is idempotent. Don't error.
|
||||
}
|
||||
|
||||
// Parse rows.
|
||||
const tasks: BenchmarkTask[] = [];
|
||||
const seenIds = new Set<string>();
|
||||
const rows = allLines.filter((l) => l.trim().length > 0 && l.trim() !== BOOTSTRAP_PENDING_REVIEW);
|
||||
|
||||
for (let i = 0; i < rows.length; i++) {
|
||||
const line = rows[i]!;
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(line);
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_malformed',
|
||||
message: `Row ${i + 1} is not valid JSON: ${msg}`,
|
||||
hint: `Fix the offending line in ${path}; benchmarks are one JSON object per line.`,
|
||||
});
|
||||
}
|
||||
|
||||
const task = validateRow(parsed, i + 1, path);
|
||||
if (seenIds.has(task.task_id)) {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkDuplicateId',
|
||||
code: 'benchmark_duplicate_task_id',
|
||||
message: `Duplicate task_id '${task.task_id}' at row ${i + 1}.`,
|
||||
hint: `Every task_id in ${path} must be unique.`,
|
||||
});
|
||||
}
|
||||
seenIds.add(task.task_id);
|
||||
tasks.push(task);
|
||||
}
|
||||
|
||||
if (tasks.length === 0) {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkEmpty',
|
||||
code: 'benchmark_empty',
|
||||
message: `Benchmark file at ${path} has no tasks.`,
|
||||
hint: `Add at least ${D_SEL_MIN_SIZE} tasks.`,
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
source_path: path,
|
||||
tasks,
|
||||
benchmark_sha8: computeBenchmarkSha8(tasks),
|
||||
};
|
||||
}
|
||||
|
||||
/** Validate a single parsed row. Throws StructuredAgentError on failure. */
|
||||
function validateRow(parsed: unknown, rowNum: number, path: string): BenchmarkTask {
|
||||
if (!parsed || typeof parsed !== 'object') {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_malformed',
|
||||
message: `Row ${rowNum} is not an object.`,
|
||||
hint: `Each line in ${path} must be a JSON object with task_id, task, judge.`,
|
||||
});
|
||||
}
|
||||
const obj = parsed as Record<string, unknown>;
|
||||
|
||||
const task_id = typeof obj.task_id === 'string' ? obj.task_id.trim() : '';
|
||||
if (!task_id) {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_missing_task_id',
|
||||
message: `Row ${rowNum} is missing a non-empty task_id.`,
|
||||
hint: `Add a unique task_id string to row ${rowNum} of ${path}.`,
|
||||
});
|
||||
}
|
||||
|
||||
const task = typeof obj.task === 'string' ? obj.task : '';
|
||||
if (!task.trim()) {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_missing_task',
|
||||
message: `Row ${rowNum} (${task_id}) is missing a non-empty task description.`,
|
||||
hint: `Add a 'task' field describing the prompt to run against the skill.`,
|
||||
});
|
||||
}
|
||||
|
||||
const judge = validateJudge(obj.judge, rowNum, task_id, path);
|
||||
return { task_id, task, judge };
|
||||
}
|
||||
|
||||
function validateJudge(raw: unknown, rowNum: number, task_id: string, path: string): Judge {
|
||||
if (!raw || typeof raw !== 'object') {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_missing_judge',
|
||||
message: `Row ${rowNum} (${task_id}) is missing a judge object.`,
|
||||
hint: `Add a 'judge' field with shape {"kind":"rule"|"llm"|"qrels", ...}.`,
|
||||
});
|
||||
}
|
||||
const j = raw as Record<string, unknown>;
|
||||
const kind = j.kind;
|
||||
if (kind === 'rule') {
|
||||
const checks = j.checks;
|
||||
if (!Array.isArray(checks) || checks.length === 0) {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_judge_rule_no_checks',
|
||||
message: `Row ${rowNum} (${task_id}) judge.kind='rule' needs a non-empty checks array.`,
|
||||
hint: `Add at least one check, e.g. {"op":"max_chars","arg":4000}.`,
|
||||
});
|
||||
}
|
||||
const validated: RuleCheck[] = checks.map((c, ci) => validateRuleCheck(c, rowNum, ci, task_id, path));
|
||||
return { kind: 'rule', checks: validated };
|
||||
}
|
||||
if (kind === 'llm') {
|
||||
const rubric = typeof j.rubric === 'string' ? j.rubric : '';
|
||||
if (!rubric.trim()) {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_judge_llm_no_rubric',
|
||||
message: `Row ${rowNum} (${task_id}) judge.kind='llm' needs a non-empty rubric string.`,
|
||||
hint: `Add a 'rubric' field describing how to score the output 0..1.`,
|
||||
});
|
||||
}
|
||||
const model = typeof j.model === 'string' ? j.model : undefined;
|
||||
return model !== undefined ? { kind: 'llm', rubric, model } : { kind: 'llm', rubric };
|
||||
}
|
||||
if (kind === 'qrels') {
|
||||
const expected_slugs = Array.isArray(j.expected_slugs) ? j.expected_slugs : null;
|
||||
if (!expected_slugs || expected_slugs.length === 0 || !expected_slugs.every((s) => typeof s === 'string')) {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_judge_qrels_no_expected',
|
||||
message: `Row ${rowNum} (${task_id}) judge.kind='qrels' needs expected_slugs: string[].`,
|
||||
hint: `Add an array of expected slugs the retrieval should return.`,
|
||||
});
|
||||
}
|
||||
const k = typeof j.k === 'number' && j.k > 0 ? Math.floor(j.k) : 10;
|
||||
return { kind: 'qrels', expected_slugs: expected_slugs as string[], k };
|
||||
}
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_judge_unknown_kind',
|
||||
message: `Row ${rowNum} (${task_id}) judge.kind='${String(kind)}' is not one of rule|llm|qrels.`,
|
||||
hint: `Use one of: {"kind":"rule","checks":[...]}, {"kind":"llm","rubric":"..."}, {"kind":"qrels","expected_slugs":[...]}.`,
|
||||
});
|
||||
}
|
||||
|
||||
function validateRuleCheck(
|
||||
raw: unknown,
|
||||
rowNum: number,
|
||||
checkIdx: number,
|
||||
task_id: string,
|
||||
path: string,
|
||||
): RuleCheck {
|
||||
if (!raw || typeof raw !== 'object') {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_rule_check_malformed',
|
||||
message: `Row ${rowNum} (${task_id}) rule check #${checkIdx} is not an object.`,
|
||||
hint: `Each check must be {"op":"...","arg":...}.`,
|
||||
});
|
||||
}
|
||||
const c = raw as Record<string, unknown>;
|
||||
const op = c.op;
|
||||
if (typeof op !== 'string' || !VALID_RULE_OPS.has(op as RuleCheckOp)) {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_rule_check_unknown_op',
|
||||
message: `Row ${rowNum} (${task_id}) rule check #${checkIdx} has unknown op '${String(op)}'.`,
|
||||
hint: `Valid ops: ${[...VALID_RULE_OPS].join(', ')}.`,
|
||||
});
|
||||
}
|
||||
const arg = c.arg;
|
||||
if (typeof arg !== 'string' && typeof arg !== 'number') {
|
||||
throw errorFor({
|
||||
class: 'BenchmarkMalformed',
|
||||
code: 'benchmark_rule_check_bad_arg',
|
||||
message: `Row ${rowNum} (${task_id}) rule check #${checkIdx} has non-string/number arg.`,
|
||||
hint: `arg must be a string (contains, regex, section_present, tool_called, tool_not_called) or number (max_chars, min_citations).`,
|
||||
});
|
||||
}
|
||||
return { op: op as RuleCheckOp, arg };
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute a deterministic SHA-256-prefix-8 over the benchmark contents.
|
||||
* Stable across whitespace changes (re-serializes the parsed tasks).
|
||||
*/
|
||||
export function computeBenchmarkSha8(tasks: BenchmarkTask[]): string {
|
||||
const canonical = tasks
|
||||
.slice()
|
||||
.sort((a, b) => (a.task_id < b.task_id ? -1 : a.task_id > b.task_id ? 1 : 0))
|
||||
.map((t) => JSON.stringify({ task_id: t.task_id, task: t.task, judge: t.judge }))
|
||||
.join('\n');
|
||||
return createHash('sha256').update(canonical).digest('hex').slice(0, 8);
|
||||
}
|
||||
|
||||
/**
|
||||
* Split a benchmark deterministically by ratio. Sorts by task_id for stable
|
||||
* splits across runs (paper: benchmarks must not shuffle between epochs).
|
||||
*
|
||||
* Returns `{train, sel, test}`. D17: refuses if D_sel < 5 — caller must
|
||||
* either add more tasks or pass an explicit --split override (which still
|
||||
* routes through this function with the override ratio).
|
||||
*/
|
||||
export function splitBench(
|
||||
benchmark: Benchmark,
|
||||
ratio: [number, number, number],
|
||||
opts: { allowSmallSel?: boolean } = {},
|
||||
): BenchmarkSplit {
|
||||
const [r1, r2, r3] = ratio;
|
||||
if (r1 <= 0 || r2 <= 0 || r3 <= 0) {
|
||||
throw errorFor({
|
||||
class: 'BadSplit',
|
||||
code: 'split_bad_ratio',
|
||||
message: `Split ratio ${ratio.join(':')} has a zero or negative segment.`,
|
||||
hint: `Use positive integers like 4:1:5.`,
|
||||
});
|
||||
}
|
||||
|
||||
const sorted = benchmark.tasks
|
||||
.slice()
|
||||
.sort((a, b) => (a.task_id < b.task_id ? -1 : a.task_id > b.task_id ? 1 : 0));
|
||||
|
||||
const total = r1 + r2 + r3;
|
||||
const n = sorted.length;
|
||||
// Round to nearest int; ensure all three buckets get at least 1 if n>=3.
|
||||
const trainN = Math.max(1, Math.floor((r1 * n) / total));
|
||||
const selN = Math.max(1, Math.floor((r2 * n) / total));
|
||||
const testN = Math.max(1, n - trainN - selN);
|
||||
|
||||
const train = sorted.slice(0, trainN);
|
||||
const sel = sorted.slice(trainN, trainN + selN);
|
||||
const test = sorted.slice(trainN + selN, trainN + selN + testN);
|
||||
|
||||
// D17: refuse if D_sel < 5 unless explicitly overridden.
|
||||
if (sel.length < D_SEL_MIN_SIZE && !opts.allowSmallSel) {
|
||||
throw errorFor({
|
||||
class: 'DSelTooSmall',
|
||||
code: 'd_sel_too_small',
|
||||
message: `D_sel has ${sel.length} task(s) after split (need >=${D_SEL_MIN_SIZE} for meaningful validation).`,
|
||||
hint: `Add more tasks to the benchmark (need ~${Math.ceil((D_SEL_MIN_SIZE * total) / r2)} total for ${ratio.join(':')}) or pass --split with a larger sel segment.`,
|
||||
});
|
||||
}
|
||||
|
||||
return { train, sel, test };
|
||||
}
|
||||
|
||||
/** Parse a split string like "4:1:5" into a tuple. */
|
||||
export function parseSplit(s: string): [number, number, number] {
|
||||
const parts = s.split(':').map((p) => Number(p.trim()));
|
||||
if (parts.length !== 3 || parts.some((p) => !Number.isFinite(p) || p <= 0)) {
|
||||
throw errorFor({
|
||||
class: 'BadSplit',
|
||||
code: 'split_unparseable',
|
||||
message: `Invalid --split value '${s}'.`,
|
||||
hint: `Use three positive integers separated by ':', e.g. '4:1:5'.`,
|
||||
});
|
||||
}
|
||||
return [parts[0]!, parts[1]!, parts[2]!];
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
/**
|
||||
* SkillOpt per-skill DB lock (D14).
|
||||
*
|
||||
* Thin wrapper around `tryAcquireDbLock` from `src/core/db-lock.ts`. The
|
||||
* lock id is `skillopt:<skill-name>` so two concurrent `gbrain skillopt foo`
|
||||
* runs serialize cleanly without blocking other skills.
|
||||
*
|
||||
* Default TTL: 60 minutes — generous for a full epoch run, but the auto-
|
||||
* refresh inside `withSkilloptLock` bumps it every 15 minutes so a long
|
||||
* run never times out underneath itself.
|
||||
*
|
||||
* Why a DB lock instead of a filesystem `.lock`:
|
||||
* - Cross-host correct (matters for Conductor workspaces sharing a brain).
|
||||
* - Reuses the existing primitive (same TTL semantics as gbrain sync,
|
||||
* extract-conversation-facts, autopilot cycle).
|
||||
* - Crashed holders auto-release via TTL expiry (no PID-liveness landmine).
|
||||
*/
|
||||
|
||||
import { tryAcquireDbLock, type DbLockHandle } from '../db-lock.ts';
|
||||
import { errorFor } from '../errors.ts';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
|
||||
const DEFAULT_TTL_MINUTES = 60;
|
||||
const DEFAULT_REFRESH_INTERVAL_MS = 15 * 60 * 1000;
|
||||
|
||||
/** Build the lock id for a given skill name. */
|
||||
export function lockIdFor(skillName: string): string {
|
||||
return `skillopt:${skillName}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquire a per-skill SkillOpt lock. Returns null when another live holder
|
||||
* has the lock. Caller is responsible for releasing via `handle.release()`
|
||||
* (use `withSkilloptLock` for try/finally + refresh-loop semantics).
|
||||
*/
|
||||
export async function tryAcquireSkilloptLock(
|
||||
engine: BrainEngine,
|
||||
skillName: string,
|
||||
ttlMinutes: number = DEFAULT_TTL_MINUTES,
|
||||
): Promise<DbLockHandle | null> {
|
||||
return tryAcquireDbLock(engine, lockIdFor(skillName), ttlMinutes);
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `fn` while holding the per-skill SkillOpt lock with a background
|
||||
* refresh loop. Refreshes the TTL every 15 minutes (well under the 60min
|
||||
* default TTL so the lock never expires under an active run).
|
||||
*
|
||||
* Throws a StructuredAgentError with `code: 'lock_busy'` when another
|
||||
* live holder has the lock — the user is shown the paste-ready remediation
|
||||
* "another run is in progress; wait or check `gbrain jobs supervisor status`".
|
||||
*
|
||||
* Lock is always released on `fn` completion (success OR throw) via
|
||||
* try/finally. The background refresh interval is cleared in the finally.
|
||||
*/
|
||||
export async function withSkilloptLock<T>(
|
||||
engine: BrainEngine,
|
||||
skillName: string,
|
||||
fn: (handle: DbLockHandle) => Promise<T>,
|
||||
ttlMinutes: number = DEFAULT_TTL_MINUTES,
|
||||
refreshIntervalMs: number = DEFAULT_REFRESH_INTERVAL_MS,
|
||||
): Promise<T> {
|
||||
const handle = await tryAcquireSkilloptLock(engine, skillName, ttlMinutes);
|
||||
if (handle === null) {
|
||||
throw errorFor({
|
||||
class: 'LockBusy',
|
||||
code: 'lock_busy',
|
||||
message: `Another SkillOpt run is in progress for skill '${skillName}'.`,
|
||||
hint: `Wait for it to finish, or check 'gbrain jobs supervisor status'. Stale lock holders auto-expire after ${ttlMinutes} minutes.`,
|
||||
});
|
||||
}
|
||||
|
||||
const refresher = setInterval(() => {
|
||||
handle.refresh().catch((err) => {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[skillopt-lock] refresh failed for '${skillName}': ${msg}\n`);
|
||||
});
|
||||
}, refreshIntervalMs);
|
||||
// Don't keep the event loop alive on the refresh timer alone.
|
||||
if (typeof refresher.unref === 'function') refresher.unref();
|
||||
|
||||
try {
|
||||
return await fn(handle);
|
||||
} finally {
|
||||
clearInterval(refresher);
|
||||
try {
|
||||
await handle.release();
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
process.stderr.write(`[skillopt-lock] release failed for '${skillName}': ${msg}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
/**
|
||||
* SkillOpt LR (learning-rate) schedules.
|
||||
*
|
||||
* The LR controls the max number of edits per step. Cosine is the default
|
||||
* per the SkillOpt paper's ablation (slightly better than linear or constant).
|
||||
*
|
||||
* All schedules return an INTEGER >= 1 (we always allow at least one edit
|
||||
* per step; otherwise the optimizer can never make progress).
|
||||
*
|
||||
* Pure functions — no side effects, no I/O, fully unit-testable.
|
||||
*
|
||||
* LR schedule shapes (visualized for base=4, totalSteps=10):
|
||||
*
|
||||
* cosine: 4 4 3 3 3 2 2 2 1 1 (smooth high→low decay)
|
||||
* linear: 4 3 3 3 2 2 2 1 1 1 (monotone descent)
|
||||
* constant: 4 4 4 4 4 4 4 4 4 4 (no decay)
|
||||
*
|
||||
* The cosine curve peaks early (more aggressive when the skill is the most
|
||||
* unrefined) and tapers (fewer edits as the skill converges).
|
||||
*/
|
||||
|
||||
/** Floor of all schedules; the LR can never drop below 1. */
|
||||
const MIN_LR = 1;
|
||||
|
||||
/**
|
||||
* Cosine-decay schedule. Peaks at `base` for t=1; decays to ~1 by totalSteps.
|
||||
*
|
||||
* Formula: `0.5 * base * (1 + cos((t-1) * pi / (totalSteps-1)))` rounded
|
||||
* up, then clamped to [MIN_LR, base].
|
||||
*/
|
||||
export function cosineLr(base: number, t: number, totalSteps: number): number {
|
||||
if (base < MIN_LR) return MIN_LR;
|
||||
if (totalSteps <= 1) return base;
|
||||
const tClamped = Math.max(1, Math.min(t, totalSteps));
|
||||
const phase = ((tClamped - 1) * Math.PI) / (totalSteps - 1);
|
||||
const raw = 0.5 * base * (1 + Math.cos(phase));
|
||||
return Math.max(MIN_LR, Math.min(base, Math.ceil(raw)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Linear-decay schedule. Starts at `base` for t=1; ends at MIN_LR for
|
||||
* t=totalSteps. Monotonically non-increasing.
|
||||
*
|
||||
* Formula: `base - (base - MIN_LR) * (t-1) / (totalSteps-1)` rounded up.
|
||||
*/
|
||||
export function linearLr(base: number, t: number, totalSteps: number): number {
|
||||
if (base < MIN_LR) return MIN_LR;
|
||||
if (totalSteps <= 1) return base;
|
||||
const tClamped = Math.max(1, Math.min(t, totalSteps));
|
||||
const raw = base - ((base - MIN_LR) * (tClamped - 1)) / (totalSteps - 1);
|
||||
return Math.max(MIN_LR, Math.min(base, Math.ceil(raw)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Constant schedule. Returns `base` every step (clamped to MIN_LR).
|
||||
*/
|
||||
export function constantLr(base: number, _t: number, _totalSteps: number): number {
|
||||
return Math.max(MIN_LR, base);
|
||||
}
|
||||
|
||||
/** Type alias for the function signature shared by all three schedules. */
|
||||
export type LrScheduleFn = (base: number, t: number, totalSteps: number) => number;
|
||||
|
||||
/** Resolve a schedule name to its function. */
|
||||
export function resolveLrSchedule(name: 'cosine' | 'linear' | 'constant'): LrScheduleFn {
|
||||
switch (name) {
|
||||
case 'cosine': return cosineLr;
|
||||
case 'linear': return linearLr;
|
||||
case 'constant': return constantLr;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,266 @@
|
||||
/**
|
||||
* SkillOpt scoring: three judge modes (rule, llm, qrels).
|
||||
*
|
||||
* Each scorer returns a 0..1 score (1 = best). Sub-1 scores are partial
|
||||
* credit; 0 means total failure. The validation gate's median-of-3 (D12)
|
||||
* is implemented in validate-gate.ts; this module is just the
|
||||
* per-trajectory scoring primitives.
|
||||
*
|
||||
* `judge: llm` uses gateway.chat with the v0.40+ 4-strategy JSON repair
|
||||
* (parseModelJSON from cross-modal-eval). On parse failure the scorer
|
||||
* returns score=0 (pessimistic fallback) AND records the error string on
|
||||
* `ScoredRollout.judge_error` so the audit trail can surface it.
|
||||
*
|
||||
* `judge: qrels` reuses src/core/search/eval.ts IR metrics. Score is
|
||||
* nDCG@k (more discriminating than P@k for the optimization signal).
|
||||
*/
|
||||
|
||||
import { chat as gatewayChat } from '../ai/gateway.ts';
|
||||
import { ndcgAtK } from '../search/eval.ts';
|
||||
import type { Judge, RuleCheck, ScoredRollout, Trajectory } from './types.ts';
|
||||
|
||||
/** Score a trajectory against a judge. Returns a ScoredRollout. */
|
||||
export async function scoreTrajectory(
|
||||
trajectory: Trajectory,
|
||||
judge: Judge,
|
||||
opts: {
|
||||
judgeModel?: string;
|
||||
/** Test seam — substitute for gateway.chat. */
|
||||
chatFn?: typeof gatewayChat;
|
||||
/** Test seam — substitute clock for cache invalidation. */
|
||||
now?: () => Date;
|
||||
} = {},
|
||||
): Promise<ScoredRollout> {
|
||||
switch (judge.kind) {
|
||||
case 'rule':
|
||||
return { trajectory, score: scoreRule(trajectory, judge.checks) };
|
||||
case 'llm':
|
||||
return scoreLlm(trajectory, judge.rubric, judge.model ?? opts.judgeModel, opts);
|
||||
case 'qrels':
|
||||
return { trajectory, score: scoreQrels(trajectory, judge.expected_slugs, judge.k) };
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Rule judge ──────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Score a trajectory against a list of rule checks. Returns the FRACTION
|
||||
* of checks that pass. 0 = all fail, 1 = all pass.
|
||||
*/
|
||||
export function scoreRule(trajectory: Trajectory, checks: RuleCheck[]): number {
|
||||
if (checks.length === 0) return 0;
|
||||
let passing = 0;
|
||||
for (const c of checks) {
|
||||
if (applyCheck(trajectory, c)) passing += 1;
|
||||
}
|
||||
return passing / checks.length;
|
||||
}
|
||||
|
||||
function applyCheck(trajectory: Trajectory, check: RuleCheck): boolean {
|
||||
const text = trajectory.final_text;
|
||||
switch (check.op) {
|
||||
case 'contains':
|
||||
return typeof check.arg === 'string' && text.includes(check.arg);
|
||||
case 'regex': {
|
||||
if (typeof check.arg !== 'string') return false;
|
||||
try {
|
||||
return new RegExp(check.arg, 'm').test(text);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
case 'section_present': {
|
||||
if (typeof check.arg !== 'string') return false;
|
||||
// Match the heading (any depth, plus the literal text). Trim args
|
||||
// and allow leading-# variants.
|
||||
const heading = check.arg.replace(/^#+\s*/, '').trim();
|
||||
const escaped = heading.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
const re = new RegExp(`^#{1,6}\\s+${escaped}\\s*$`, 'mi');
|
||||
return re.test(text);
|
||||
}
|
||||
case 'max_chars':
|
||||
return typeof check.arg === 'number' && text.length <= check.arg;
|
||||
case 'min_citations':
|
||||
return typeof check.arg === 'number' && countCitations(text) >= check.arg;
|
||||
case 'tool_called':
|
||||
return typeof check.arg === 'string' &&
|
||||
trajectory.tool_calls.some((tc) => tc.name === check.arg && !tc.failed);
|
||||
case 'tool_not_called':
|
||||
return typeof check.arg === 'string' &&
|
||||
!trajectory.tool_calls.some((tc) => tc.name === check.arg);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Count citation-like spans in the output. Recognized shapes:
|
||||
* - Markdown links: `[text](url-or-slug)`
|
||||
* - Brain-page references: `wiki/...`, `people/...`, `companies/...`
|
||||
* - Footnote-style: `[N]` where N is digits.
|
||||
*/
|
||||
export function countCitations(text: string): number {
|
||||
const mdLinks = (text.match(/\[[^\]]+\]\([^)]+\)/g) ?? []).length;
|
||||
const brainRefs = (text.match(/\b(?:wiki|people|companies|deals|topics|concepts|projects|writing|originals)\/[a-z0-9][a-z0-9-\/]*\b/gi) ?? []).length;
|
||||
const footnotes = (text.match(/\[\d+\]/g) ?? []).length;
|
||||
return mdLinks + brainRefs + footnotes;
|
||||
}
|
||||
|
||||
// ─── LLM judge ───────────────────────────────────────────────────────────
|
||||
|
||||
const LLM_JUDGE_SYSTEM = `You are a strict, fair judge scoring an agent's output against a rubric.
|
||||
|
||||
Output ONLY a single JSON object on a single line:
|
||||
{"score": <number 0..1>, "rationale": "<one-sentence reason>"}
|
||||
|
||||
No prose before or after. No code fences. No extra fields. The score MUST be a number between 0.0 and 1.0 inclusive.`;
|
||||
|
||||
/**
|
||||
* Parse a `{score, rationale}` JSON object from raw LLM text. Tolerates:
|
||||
* - Leading/trailing whitespace.
|
||||
* - Markdown code fences (```json ... ```).
|
||||
* - Prose before or after the JSON (extracts first {...} object).
|
||||
* - Trailing commas inside the object.
|
||||
*
|
||||
* Returns null when no recoverable object is found (caller treats as judge
|
||||
* error + pessimistic fallback score=0).
|
||||
*/
|
||||
export function parseJudgeJson(raw: string): { score: number | string; rationale?: string } | null {
|
||||
if (typeof raw !== 'string' || !raw.trim()) return null;
|
||||
// Strip markdown fences if present.
|
||||
const fenced = raw.match(/```(?:json)?\s*\n?([\s\S]*?)```/i);
|
||||
const cleaned = (fenced ? fenced[1]! : raw).trim();
|
||||
// Try direct parse first.
|
||||
const direct = tryJsonParse(cleaned);
|
||||
if (direct && typeof direct === 'object' && 'score' in (direct as object)) {
|
||||
return direct as { score: number | string; rationale?: string };
|
||||
}
|
||||
// Extract first {...} substring.
|
||||
const match = cleaned.match(/\{[\s\S]*?\}/);
|
||||
if (!match) return null;
|
||||
const obj = match[0];
|
||||
const second = tryJsonParse(obj);
|
||||
if (second && typeof second === 'object' && 'score' in (second as object)) {
|
||||
return second as { score: number | string; rationale?: string };
|
||||
}
|
||||
// Last attempt: strip trailing commas.
|
||||
const repaired = obj.replace(/,(\s*[}\]])/g, '$1');
|
||||
const third = tryJsonParse(repaired);
|
||||
if (third && typeof third === 'object' && 'score' in (third as object)) {
|
||||
return third as { score: number | string; rationale?: string };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function tryJsonParse(s: string): unknown | null {
|
||||
try { return JSON.parse(s); } catch { return null; }
|
||||
}
|
||||
|
||||
async function scoreLlm(
|
||||
trajectory: Trajectory,
|
||||
rubric: string,
|
||||
judgeModel: string | undefined,
|
||||
opts: { chatFn?: typeof gatewayChat },
|
||||
): Promise<ScoredRollout> {
|
||||
const chat = opts.chatFn ?? gatewayChat;
|
||||
const userMsg = `RUBRIC: ${rubric}\n\nAGENT OUTPUT:\n${trajectory.final_text}\n\nScore the output against the rubric. Reply with the JSON object only.`;
|
||||
try {
|
||||
const result = await chat({
|
||||
model: judgeModel,
|
||||
system: LLM_JUDGE_SYSTEM,
|
||||
messages: [{ role: 'user', content: userMsg }],
|
||||
maxTokens: 200,
|
||||
cacheSystem: true, // D11: judge system prompt is stable across calls.
|
||||
});
|
||||
const parsed = parseJudgeJson(result.text);
|
||||
if (!parsed) {
|
||||
return { trajectory, score: 0, judge_error: 'llm_parse_failed' };
|
||||
}
|
||||
if (!('score' in parsed)) {
|
||||
return { trajectory, score: 0, judge_error: 'llm_parse_no_score_field' };
|
||||
}
|
||||
const raw = parsed.score;
|
||||
let score = typeof raw === 'number' ? raw : Number(raw);
|
||||
if (!Number.isFinite(score)) {
|
||||
return { trajectory, score: 0, judge_error: 'llm_parse_score_not_number' };
|
||||
}
|
||||
if (score < 0) score = 0;
|
||||
if (score > 1) score = 1;
|
||||
const rationale = typeof parsed.rationale === 'string' ? parsed.rationale : undefined;
|
||||
return rationale !== undefined
|
||||
? { trajectory, score, rationale }
|
||||
: { trajectory, score };
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
// Pessimistic fallback (D12 paper-faithful: judge failure = score 0,
|
||||
// not throw; the median-of-3 + epsilon gate handles a single error
|
||||
// gracefully, only consistent judge failure breaks the run).
|
||||
return { trajectory, score: 0, judge_error: `llm_call_failed: ${msg}` };
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Qrels judge (retrieval flavor) ───────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Score a trajectory against expected retrieval slugs.
|
||||
*
|
||||
* Extracts the candidate slugs from the trajectory's tool calls (any
|
||||
* `search` / `query` / `get_page` / `list_pages` op output that returns
|
||||
* page rows), then computes nDCG@k against expected_slugs.
|
||||
*
|
||||
* Returns 0 when no retrieval tool was called (the skill didn't even try).
|
||||
*/
|
||||
export function scoreQrels(
|
||||
trajectory: Trajectory,
|
||||
expectedSlugs: string[],
|
||||
k: number,
|
||||
): number {
|
||||
const candidateSlugs = extractRetrievedSlugs(trajectory);
|
||||
if (candidateSlugs.length === 0) return 0;
|
||||
// ndcgAtK expects (hits, grades:Map<slug,number>, k). All expected slugs
|
||||
// get grade 1 (binary relevance) — qrels mode is "did the skill retrieve
|
||||
// what we expected?", not "did it rank them in our preferred order."
|
||||
const grades = new Map<string, number>();
|
||||
for (const s of expectedSlugs) grades.set(s, 1);
|
||||
return ndcgAtK(candidateSlugs, grades, k);
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk the trajectory's tool calls and collect slugs from any search/query/
|
||||
* list_pages/get_page output that returns row arrays with `slug` fields.
|
||||
* Tolerant of shape variation.
|
||||
*/
|
||||
export function extractRetrievedSlugs(trajectory: Trajectory): string[] {
|
||||
const out: string[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const call of trajectory.tool_calls) {
|
||||
if (!call.output || call.failed) continue;
|
||||
const slugs = pickSlugs(call.output);
|
||||
for (const slug of slugs) {
|
||||
if (!seen.has(slug)) {
|
||||
seen.add(slug);
|
||||
out.push(slug);
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function pickSlugs(output: unknown): string[] {
|
||||
if (!output) return [];
|
||||
if (typeof output === 'string') {
|
||||
// Some ops return a single slug string.
|
||||
return output.includes('/') ? [output] : [];
|
||||
}
|
||||
if (Array.isArray(output)) {
|
||||
return output.flatMap(pickSlugs);
|
||||
}
|
||||
if (typeof output === 'object') {
|
||||
const obj = output as Record<string, unknown>;
|
||||
const out: string[] = [];
|
||||
if (typeof obj.slug === 'string') out.push(obj.slug);
|
||||
if (Array.isArray(obj.results)) out.push(...pickSlugs(obj.results));
|
||||
if (Array.isArray(obj.pages)) out.push(...pickSlugs(obj.pages));
|
||||
if (Array.isArray(obj.matches)) out.push(...pickSlugs(obj.matches));
|
||||
return out;
|
||||
}
|
||||
return [];
|
||||
}
|
||||
@@ -0,0 +1,259 @@
|
||||
/**
|
||||
* SkillOpt v1 types — single source of truth for the optimization loop.
|
||||
*
|
||||
* The optimizer treats SKILL.md as the trainable parameters of a frozen
|
||||
* agent. See plan: ~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md.
|
||||
*
|
||||
* Per the v0.41.20.0 plan decisions:
|
||||
* D2: rollout uses gateway.toolLoop (no DB pollution).
|
||||
* D5: frontmatter mutation is forbidden; edits operate on body slice only.
|
||||
* D9: applyEdit returns a tagged result, not throws.
|
||||
* D12: validation gate uses median-of-3 + epsilon=0.05.
|
||||
* D17: D_sel >= 5 floor enforced at benchmark-load time.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
|
||||
// ─── Benchmarks + judges ──────────────────────────────────────────────────
|
||||
|
||||
/** Rule-check kinds for `judge: rule`. Each is deterministic and free. */
|
||||
export type RuleCheckOp =
|
||||
| 'contains'
|
||||
| 'regex'
|
||||
| 'section_present'
|
||||
| 'max_chars'
|
||||
| 'min_citations'
|
||||
| 'tool_called'
|
||||
| 'tool_not_called';
|
||||
|
||||
export interface RuleCheck {
|
||||
op: RuleCheckOp;
|
||||
arg: string | number;
|
||||
}
|
||||
|
||||
export type JudgeKind = 'rule' | 'llm' | 'qrels';
|
||||
|
||||
export type Judge =
|
||||
| { kind: 'rule'; checks: RuleCheck[] }
|
||||
| { kind: 'llm'; rubric: string; model?: string }
|
||||
| { kind: 'qrels'; expected_slugs: string[]; k: number };
|
||||
|
||||
export interface BenchmarkTask {
|
||||
task_id: string;
|
||||
task: string;
|
||||
judge: Judge;
|
||||
}
|
||||
|
||||
export interface Benchmark {
|
||||
/** Path the benchmark was loaded from (for error messages). */
|
||||
source_path: string;
|
||||
tasks: BenchmarkTask[];
|
||||
/** SHA-256 of the canonical JSON, truncated to 16 hex. Stable across reorderings. */
|
||||
benchmark_sha8: string;
|
||||
}
|
||||
|
||||
/** D17: enforced floor for D_sel. */
|
||||
export const D_SEL_MIN_SIZE = 5;
|
||||
|
||||
export interface BenchmarkSplit {
|
||||
train: BenchmarkTask[];
|
||||
sel: BenchmarkTask[];
|
||||
test: BenchmarkTask[];
|
||||
}
|
||||
|
||||
// ─── Edit ops ─────────────────────────────────────────────────────────────
|
||||
|
||||
/** D9: applyEdit returns a tagged result, not throws. */
|
||||
export type EditOp =
|
||||
| { op: 'add'; anchor: string; content: string; reason?: string }
|
||||
| { op: 'replace'; target: string; replacement: string; reason?: string }
|
||||
| { op: 'delete'; target: string; reason?: string };
|
||||
|
||||
export type EditRejectionReason =
|
||||
| 'anchor_not_found'
|
||||
| 'anchor_ambiguous'
|
||||
| 'target_not_found'
|
||||
| 'target_ambiguous'
|
||||
| 'inside_code_fence'
|
||||
| 'crosses_frontmatter'
|
||||
| 'working_tree_dirty'
|
||||
| 'install_path'
|
||||
| 'no_change';
|
||||
|
||||
export type EditResult =
|
||||
| { outcome: 'applied'; edit: EditOp; newText: string }
|
||||
| { outcome: 'rejected'; edit: EditOp; reason: EditRejectionReason; detail?: string };
|
||||
|
||||
// ─── Trajectories + rollouts ──────────────────────────────────────────────
|
||||
|
||||
/** What a rollout produced. Captured in-process; never persisted to DB (D2). */
|
||||
export interface Trajectory {
|
||||
task_id: string;
|
||||
task: string;
|
||||
/** Final assistant text (typically the user-visible output). */
|
||||
final_text: string;
|
||||
/** Tool calls observed during the rollout, in order. */
|
||||
tool_calls: Array<{ name: string; input: unknown; output?: unknown; failed?: boolean }>;
|
||||
/** Token usage from gateway.toolLoop. */
|
||||
usage: {
|
||||
input_tokens: number;
|
||||
output_tokens: number;
|
||||
cache_read_tokens: number;
|
||||
cache_creation_tokens: number;
|
||||
};
|
||||
/** Number of agent turns the loop took. */
|
||||
turns: number;
|
||||
/** End reason from gateway.toolLoop. */
|
||||
stop_reason: 'end' | 'max_turns' | 'refusal' | 'content_filter' | 'aborted' | 'unrecoverable';
|
||||
/** Wall-clock duration in ms. */
|
||||
duration_ms: number;
|
||||
}
|
||||
|
||||
export interface ScoredRollout {
|
||||
trajectory: Trajectory;
|
||||
/** 0..1 score from the judge. */
|
||||
score: number;
|
||||
/** Optional per-task rationale (LLM judge only). */
|
||||
rationale?: string;
|
||||
/** If the judge itself errored (parse fail, timeout), this is set. */
|
||||
judge_error?: string;
|
||||
}
|
||||
|
||||
// ─── Run state + receipts ─────────────────────────────────────────────────
|
||||
|
||||
export interface SkillOptOpts {
|
||||
/** Kebab-case skill name. Resolves to `skills/<name>/SKILL.md`. */
|
||||
skillName: string;
|
||||
/** Absolute path to the benchmark JSONL file. */
|
||||
benchmarkPath: string;
|
||||
/** Brain engine (required for D14 lock + read-only tool calls during rollouts). */
|
||||
engine: BrainEngine;
|
||||
/** Skills directory root (used for bundled-skill detection, install-path gate). */
|
||||
skillsDir: string;
|
||||
|
||||
// Training knobs.
|
||||
epochs: number;
|
||||
batchSize: number;
|
||||
/** Max edits per step (the LR). */
|
||||
lr: number;
|
||||
lrSchedule: 'cosine' | 'linear' | 'constant';
|
||||
/** Split ratio as 3-tuple, e.g. [4, 1, 5] for 4:1:5. */
|
||||
split: [number, number, number];
|
||||
|
||||
// Models.
|
||||
optimizerModel: string;
|
||||
targetModel: string;
|
||||
judgeModel: string;
|
||||
|
||||
// Modes.
|
||||
mode: 'patch' | 'rewrite';
|
||||
dryRun: boolean;
|
||||
noMutate: boolean;
|
||||
allowMutateBundled: boolean;
|
||||
bootstrapReviewed: boolean;
|
||||
/** F10: enable write-capture mode for write-flavored skills. */
|
||||
writeCapture?: boolean;
|
||||
/** F11: optional held-out test set path (validates winner before mutate). */
|
||||
heldOutPath?: string;
|
||||
json: boolean;
|
||||
|
||||
// Safety.
|
||||
maxCostUsd: number;
|
||||
maxRuntimeMin: number;
|
||||
force: boolean;
|
||||
resumeRunId?: string;
|
||||
}
|
||||
|
||||
export interface StepRecord {
|
||||
epoch: number;
|
||||
step: number;
|
||||
sel_score_median: number;
|
||||
sel_score_runs: number[];
|
||||
accepted: boolean;
|
||||
edits_attempted: number;
|
||||
edits_applied: number;
|
||||
delta: number;
|
||||
reason?: string;
|
||||
cumulative_cost_usd: number;
|
||||
ts: string;
|
||||
}
|
||||
|
||||
export interface RunReceipt {
|
||||
run_id: string;
|
||||
skill: string;
|
||||
skill_sha8: string;
|
||||
benchmark_sha8: string;
|
||||
optimizer_model: string;
|
||||
target_model: string;
|
||||
judge_model: string;
|
||||
epochs: number;
|
||||
batch_size: number;
|
||||
lr: number;
|
||||
lr_schedule: 'cosine' | 'linear' | 'constant';
|
||||
max_cost_usd: number;
|
||||
started_at: string;
|
||||
ended_at?: string;
|
||||
outcome?: 'accepted' | 'no_improvement' | 'aborted' | 'errored';
|
||||
baseline_sel_score?: number;
|
||||
best_sel_score?: number;
|
||||
baseline_test_score?: number;
|
||||
test_score?: number;
|
||||
final_cost_usd?: number;
|
||||
total_steps?: number;
|
||||
epochs_completed?: number;
|
||||
}
|
||||
|
||||
export interface HistoryRow {
|
||||
/** D8: pending → committed two-phase commit. */
|
||||
status: 'pending' | 'committed';
|
||||
run_id: string;
|
||||
version_n: number;
|
||||
ts: string;
|
||||
edits: EditOp[];
|
||||
sel_score: number;
|
||||
delta: number;
|
||||
}
|
||||
|
||||
// ─── Validation gate (D12) ────────────────────────────────────────────────
|
||||
|
||||
/** D12: epsilon margin floor for accepting a candidate. */
|
||||
export const VALIDATION_EPSILON = 0.05;
|
||||
/** D12: number of judge runs per sel-task for noise rejection. */
|
||||
export const VALIDATION_RUNS_PER_TASK = 3;
|
||||
|
||||
export interface GateInput {
|
||||
candidateSkillText: string;
|
||||
selSet: BenchmarkTask[];
|
||||
/** Best score so far on D_sel (epsilon-margin compare against this). */
|
||||
bestScore: number;
|
||||
}
|
||||
|
||||
export interface GateResult {
|
||||
accepted: boolean;
|
||||
/** Per-task median across N runs. */
|
||||
perTaskMedians: Array<{ task_id: string; median: number; runs: number[] }>;
|
||||
/** Mean of per-task medians. */
|
||||
selScore: number;
|
||||
reason?: 'no_margin' | 'below_baseline' | 'all_judge_errors';
|
||||
}
|
||||
|
||||
// ─── Bootstrap sentinel (D15) ─────────────────────────────────────────────
|
||||
|
||||
/** D15: sentinel line written at the end of bootstrap-from-routing output. */
|
||||
export const BOOTSTRAP_PENDING_REVIEW = '# BOOTSTRAP_PENDING_REVIEW';
|
||||
|
||||
// ─── Bundled-skill gate (D16) ─────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* D16: a skill is "bundled" when its SKILL.md lives under the repo's
|
||||
* canonical `skills/` directory (relative to the gbrain install root).
|
||||
* Bundled skills require `--allow-mutate-bundled` to overwrite.
|
||||
*/
|
||||
export interface BundledSkillContext {
|
||||
skillName: string;
|
||||
skillsDir: string;
|
||||
/** Resolved absolute path to skills/<name>/SKILL.md. */
|
||||
skillPath: string;
|
||||
/** True when the skillsDir is the repo's `skills/` (install path). */
|
||||
isBundled: boolean;
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
/**
|
||||
* SkillOpt audit JSONL writer tests.
|
||||
*
|
||||
* Uses withEnv (R1 compliant) to point GBRAIN_AUDIT_DIR at a tempdir per
|
||||
* test. Resets the cached writer between tests so each invocation re-reads
|
||||
* the env.
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { withEnv } from '../helpers/with-env.ts';
|
||||
import {
|
||||
_resetAuditWriterForTests,
|
||||
currentAuditFilename,
|
||||
logEvent,
|
||||
readRecentEvents,
|
||||
resolveAuditDir,
|
||||
sha8,
|
||||
} from '../../src/core/skillopt/audit.ts';
|
||||
|
||||
let tmpDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skillopt-audit-'));
|
||||
_resetAuditWriterForTests();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
_resetAuditWriterForTests();
|
||||
try { fs.rmSync(tmpDir, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||
});
|
||||
|
||||
describe('logEvent + readRecentEvents', () => {
|
||||
test('writes a JSONL row to the current ISO-week file', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => {
|
||||
logEvent({
|
||||
kind: 'run_start',
|
||||
run_id: 'r1',
|
||||
skill: 'meeting-prep',
|
||||
skill_sha8: 'abcd1234',
|
||||
benchmark_sha8: 'deadbeef',
|
||||
target_model: 'anthropic:claude-sonnet-4-6',
|
||||
optimizer_model: 'anthropic:claude-opus-4-7',
|
||||
judge_model: 'anthropic:claude-sonnet-4-6',
|
||||
epochs: 4,
|
||||
batch_size: 8,
|
||||
lr: 4,
|
||||
lr_schedule: 'cosine',
|
||||
max_cost_usd: 5.0,
|
||||
} as never);
|
||||
const file = path.join(tmpDir, currentAuditFilename());
|
||||
expect(fs.existsSync(file)).toBe(true);
|
||||
const lines = fs.readFileSync(file, 'utf8').trim().split('\n');
|
||||
expect(lines).toHaveLength(1);
|
||||
const ev = JSON.parse(lines[0]!);
|
||||
expect(ev.kind).toBe('run_start');
|
||||
expect(ev.skill).toBe('meeting-prep');
|
||||
expect(typeof ev.ts).toBe('string');
|
||||
});
|
||||
});
|
||||
|
||||
test('readRecentEvents returns the row we just wrote', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => {
|
||||
logEvent({ kind: 'abort', run_id: 'r2', skill: 'foo', reason: 'budget_exhausted' } as never);
|
||||
const events = readRecentEvents(7);
|
||||
expect(events.length).toBeGreaterThanOrEqual(1);
|
||||
const match = events.find((e) => e.kind === 'abort' && e.run_id === 'r2');
|
||||
expect(match).toBeDefined();
|
||||
});
|
||||
});
|
||||
|
||||
test('resolveAuditDir honors GBRAIN_AUDIT_DIR override', async () => {
|
||||
await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => {
|
||||
expect(resolveAuditDir()).toBe(tmpDir);
|
||||
});
|
||||
});
|
||||
|
||||
test('sha8 returns 8 hex chars deterministically', () => {
|
||||
expect(sha8('alice')).toMatch(/^[0-9a-f]{8}$/);
|
||||
expect(sha8('alice')).toBe(sha8('alice')); // deterministic
|
||||
expect(sha8('alice')).not.toBe(sha8('bob'));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,188 @@
|
||||
/**
|
||||
* SkillOpt benchmark loader + splitter unit tests.
|
||||
*
|
||||
* Uses tempdir + withEnv for hermeticity. No engine; no LLM.
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeEach, afterEach } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { StructuredAgentError } from '../../src/core/errors.ts';
|
||||
import {
|
||||
computeBenchmarkSha8,
|
||||
loadBenchmark,
|
||||
parseSplit,
|
||||
splitBench,
|
||||
} from '../../src/core/skillopt/benchmark.ts';
|
||||
import { BOOTSTRAP_PENDING_REVIEW } from '../../src/core/skillopt/types.ts';
|
||||
|
||||
let tmpDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skillopt-bench-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
try { fs.rmSync(tmpDir, { recursive: true, force: true }); } catch { /* ignore */ }
|
||||
});
|
||||
|
||||
function writeBench(filename: string, lines: string[]): string {
|
||||
const p = path.join(tmpDir, filename);
|
||||
fs.writeFileSync(p, lines.join('\n') + '\n', 'utf8');
|
||||
return p;
|
||||
}
|
||||
|
||||
describe('loadBenchmark', () => {
|
||||
test('parses well-formed JSONL with rule judge', () => {
|
||||
const p = writeBench('b.jsonl', [
|
||||
JSON.stringify({ task_id: 't1', task: 'do X', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 4000 }] } }),
|
||||
JSON.stringify({ task_id: 't2', task: 'do Y', judge: { kind: 'rule', checks: [{ op: 'contains', arg: 'foo' }] } }),
|
||||
]);
|
||||
const b = loadBenchmark(p);
|
||||
expect(b.tasks).toHaveLength(2);
|
||||
expect(b.tasks[0]!.task_id).toBe('t1');
|
||||
expect(b.benchmark_sha8).toMatch(/^[0-9a-f]{8}$/);
|
||||
});
|
||||
|
||||
test('rejects file-not-found with paste-ready hint', () => {
|
||||
expect(() => loadBenchmark(path.join(tmpDir, 'nope.jsonl'))).toThrow(StructuredAgentError);
|
||||
});
|
||||
|
||||
test('rejects empty file', () => {
|
||||
fs.writeFileSync(path.join(tmpDir, 'empty.jsonl'), '', 'utf8');
|
||||
expect(() => loadBenchmark(path.join(tmpDir, 'empty.jsonl'))).toThrow(StructuredAgentError);
|
||||
});
|
||||
|
||||
test('rejects duplicate task_id', () => {
|
||||
const p = writeBench('dup.jsonl', [
|
||||
JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 100 }] } }),
|
||||
JSON.stringify({ task_id: 't1', task: 'b', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 100 }] } }),
|
||||
]);
|
||||
expect(() => loadBenchmark(p)).toThrow(/Duplicate task_id/);
|
||||
});
|
||||
|
||||
test('rejects unknown judge.kind', () => {
|
||||
const p = writeBench('badjudge.jsonl', [
|
||||
JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'magic' } }),
|
||||
]);
|
||||
expect(() => loadBenchmark(p)).toThrow(/not one of rule\|llm\|qrels/);
|
||||
});
|
||||
|
||||
test('rejects rule judge with empty checks array', () => {
|
||||
const p = writeBench('emptychecks.jsonl', [
|
||||
JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [] } }),
|
||||
]);
|
||||
expect(() => loadBenchmark(p)).toThrow(/non-empty checks array/);
|
||||
});
|
||||
|
||||
test('rejects rule check with unknown op', () => {
|
||||
const p = writeBench('badop.jsonl', [
|
||||
JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [{ op: 'frob', arg: 1 }] } }),
|
||||
]);
|
||||
expect(() => loadBenchmark(p)).toThrow(/unknown op/);
|
||||
});
|
||||
|
||||
test('rejects llm judge with no rubric', () => {
|
||||
const p = writeBench('norubric.jsonl', [
|
||||
JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'llm', rubric: ' ' } }),
|
||||
]);
|
||||
expect(() => loadBenchmark(p)).toThrow(/non-empty rubric/);
|
||||
});
|
||||
|
||||
test('rejects qrels judge with empty expected_slugs', () => {
|
||||
const p = writeBench('noslugs.jsonl', [
|
||||
JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'qrels', expected_slugs: [], k: 10 } }),
|
||||
]);
|
||||
expect(() => loadBenchmark(p)).toThrow(/expected_slugs/);
|
||||
});
|
||||
|
||||
test('rejects sentinel file without --bootstrap-reviewed', () => {
|
||||
const p = writeBench('boot.jsonl', [
|
||||
JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 100 }] } }),
|
||||
BOOTSTRAP_PENDING_REVIEW,
|
||||
]);
|
||||
expect(() => loadBenchmark(p)).toThrow(/awaiting human review/);
|
||||
});
|
||||
|
||||
test('accepts sentinel file with --bootstrap-reviewed', () => {
|
||||
const p = writeBench('boot.jsonl', [
|
||||
JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 100 }] } }),
|
||||
BOOTSTRAP_PENDING_REVIEW,
|
||||
]);
|
||||
const b = loadBenchmark(p, { bootstrapReviewed: true });
|
||||
expect(b.tasks).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('splitBench', () => {
|
||||
function makeBench(n: number) {
|
||||
const tasks = [];
|
||||
for (let i = 0; i < n; i++) {
|
||||
tasks.push({
|
||||
task_id: `t${String(i).padStart(3, '0')}`,
|
||||
task: `task ${i}`,
|
||||
judge: { kind: 'rule' as const, checks: [{ op: 'max_chars' as const, arg: 4000 }] },
|
||||
});
|
||||
}
|
||||
return { source_path: '/tmp/x.jsonl', tasks, benchmark_sha8: 'abcd1234' };
|
||||
}
|
||||
|
||||
test('splits 50 tasks at 4:1:5 into 20/5/25', () => {
|
||||
const split = splitBench(makeBench(50), [4, 1, 5]);
|
||||
expect(split.train.length + split.sel.length + split.test.length).toBe(50);
|
||||
expect(split.train.length).toBe(20);
|
||||
expect(split.sel.length).toBe(5);
|
||||
expect(split.test.length).toBe(25);
|
||||
});
|
||||
|
||||
test('D17: refuses when D_sel < 5 without override', () => {
|
||||
// 8 tasks split 4:1:5 → sel = max(1, floor(8/10)) = 1 < 5 → refuses
|
||||
expect(() => splitBench(makeBench(8), [4, 1, 5])).toThrow(/D_sel/);
|
||||
});
|
||||
|
||||
test('D17: allows D_sel < 5 with allowSmallSel override', () => {
|
||||
const split = splitBench(makeBench(8), [4, 1, 5], { allowSmallSel: true });
|
||||
expect(split.sel.length).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('rejects bad ratio (zero segment)', () => {
|
||||
expect(() => splitBench(makeBench(20), [4, 0, 5])).toThrow(/zero or negative/);
|
||||
});
|
||||
|
||||
test('deterministic split — same input produces same output', () => {
|
||||
const b = makeBench(50);
|
||||
const a = splitBench(b, [4, 1, 5]);
|
||||
const c = splitBench(b, [4, 1, 5]);
|
||||
expect(a.train.map((t) => t.task_id)).toEqual(c.train.map((t) => t.task_id));
|
||||
expect(a.sel.map((t) => t.task_id)).toEqual(c.sel.map((t) => t.task_id));
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseSplit', () => {
|
||||
test('parses "4:1:5" correctly', () => {
|
||||
expect(parseSplit('4:1:5')).toEqual([4, 1, 5]);
|
||||
});
|
||||
|
||||
test('rejects malformed input', () => {
|
||||
expect(() => parseSplit('4-1-5')).toThrow();
|
||||
expect(() => parseSplit('4:abc:5')).toThrow();
|
||||
expect(() => parseSplit('4:0:5')).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('computeBenchmarkSha8', () => {
|
||||
test('produces stable 8-hex hash', () => {
|
||||
const tasks = [
|
||||
{ task_id: 't1', task: 'a', judge: { kind: 'rule' as const, checks: [{ op: 'max_chars' as const, arg: 100 }] } },
|
||||
];
|
||||
const h = computeBenchmarkSha8(tasks);
|
||||
expect(h).toMatch(/^[0-9a-f]{8}$/);
|
||||
});
|
||||
|
||||
test('reordering tasks produces same hash (sort-stable)', () => {
|
||||
const t1 = { task_id: 't1', task: 'a', judge: { kind: 'rule' as const, checks: [{ op: 'max_chars' as const, arg: 100 }] } };
|
||||
const t2 = { task_id: 't2', task: 'b', judge: { kind: 'rule' as const, checks: [{ op: 'max_chars' as const, arg: 200 }] } };
|
||||
expect(computeBenchmarkSha8([t1, t2])).toBe(computeBenchmarkSha8([t2, t1]));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,94 @@
|
||||
/**
|
||||
* SkillOpt per-skill DB lock tests. Uses PGLite (R3+R4 canonical block).
|
||||
*
|
||||
* Asserts the wrapper around tryAcquireDbLock:
|
||||
* - Acquires lock and runs fn under it.
|
||||
* - Refreshes TTL during long runs.
|
||||
* - Throws lock_busy when another holder has the lock.
|
||||
* - Releases on success AND on throw.
|
||||
*/
|
||||
|
||||
import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from '../helpers/reset-pglite.ts';
|
||||
import { StructuredAgentError } from '../../src/core/errors.ts';
|
||||
import { lockIdFor, tryAcquireSkilloptLock, withSkilloptLock } from '../../src/core/skillopt/lock.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
describe('SkillOpt lock', () => {
|
||||
test('lockIdFor builds skillopt:<name> id', () => {
|
||||
expect(lockIdFor('my-skill')).toBe('skillopt:my-skill');
|
||||
});
|
||||
|
||||
test('tryAcquireSkilloptLock acquires + second attempt returns null', async () => {
|
||||
const h1 = await tryAcquireSkilloptLock(engine, 'foo', 1);
|
||||
expect(h1).not.toBeNull();
|
||||
const h2 = await tryAcquireSkilloptLock(engine, 'foo', 1);
|
||||
expect(h2).toBeNull();
|
||||
await h1!.release();
|
||||
// After release, can re-acquire.
|
||||
const h3 = await tryAcquireSkilloptLock(engine, 'foo', 1);
|
||||
expect(h3).not.toBeNull();
|
||||
await h3!.release();
|
||||
});
|
||||
|
||||
test('withSkilloptLock runs fn under lock and releases on success', async () => {
|
||||
let ran = false;
|
||||
await withSkilloptLock(engine, 'bar', async () => {
|
||||
ran = true;
|
||||
// While the lock is held, another acquire should return null.
|
||||
const inner = await tryAcquireSkilloptLock(engine, 'bar', 1);
|
||||
expect(inner).toBeNull();
|
||||
}, 1, /* fast refresh */ 30_000);
|
||||
expect(ran).toBe(true);
|
||||
// After fn completes, lock is released.
|
||||
const after = await tryAcquireSkilloptLock(engine, 'bar', 1);
|
||||
expect(after).not.toBeNull();
|
||||
await after!.release();
|
||||
});
|
||||
|
||||
test('withSkilloptLock throws LockBusy when a holder exists', async () => {
|
||||
const held = await tryAcquireSkilloptLock(engine, 'baz', 1);
|
||||
expect(held).not.toBeNull();
|
||||
let caught: unknown = null;
|
||||
try {
|
||||
await withSkilloptLock(engine, 'baz', async () => { /* unreached */ }, 1, 30_000);
|
||||
} catch (err) {
|
||||
caught = err;
|
||||
}
|
||||
expect(caught).toBeInstanceOf(StructuredAgentError);
|
||||
expect((caught as StructuredAgentError).envelope.code).toBe('lock_busy');
|
||||
await held!.release();
|
||||
});
|
||||
|
||||
test('withSkilloptLock releases on throw inside fn', async () => {
|
||||
let caught: unknown = null;
|
||||
try {
|
||||
await withSkilloptLock(engine, 'qux', async () => {
|
||||
throw new Error('inner failure');
|
||||
}, 1, 30_000);
|
||||
} catch (err) {
|
||||
caught = err;
|
||||
}
|
||||
expect((caught as Error).message).toBe('inner failure');
|
||||
// Lock is released; we can re-acquire.
|
||||
const after = await tryAcquireSkilloptLock(engine, 'qux', 1);
|
||||
expect(after).not.toBeNull();
|
||||
await after!.release();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,66 @@
|
||||
/**
|
||||
* SkillOpt LR-schedule unit tests.
|
||||
*
|
||||
* Pure-function tests: no engine, no env mutation, no fixtures. All three
|
||||
* schedules are deterministic given (base, t, totalSteps).
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import { cosineLr, linearLr, constantLr, resolveLrSchedule } from '../../src/core/skillopt/lr-schedule.ts';
|
||||
|
||||
describe('cosineLr', () => {
|
||||
test('peaks at base for t=1, decays toward 1 by totalSteps', () => {
|
||||
expect(cosineLr(4, 1, 10)).toBe(4);
|
||||
expect(cosineLr(4, 10, 10)).toBe(1);
|
||||
// Mid-curve point — Math.ceil keeps it above 1.
|
||||
expect(cosineLr(4, 5, 10)).toBeGreaterThan(1);
|
||||
expect(cosineLr(4, 5, 10)).toBeLessThanOrEqual(4);
|
||||
});
|
||||
|
||||
test('monotone non-increasing within bounds', () => {
|
||||
const seq: number[] = [];
|
||||
for (let t = 1; t <= 10; t++) seq.push(cosineLr(4, t, 10));
|
||||
for (let i = 1; i < seq.length; i++) {
|
||||
expect(seq[i]!).toBeLessThanOrEqual(seq[i - 1]!);
|
||||
}
|
||||
});
|
||||
|
||||
test('totalSteps=1 returns base', () => {
|
||||
expect(cosineLr(4, 1, 1)).toBe(4);
|
||||
});
|
||||
});
|
||||
|
||||
describe('linearLr', () => {
|
||||
test('starts at base, ends at 1', () => {
|
||||
expect(linearLr(4, 1, 10)).toBe(4);
|
||||
expect(linearLr(4, 10, 10)).toBe(1);
|
||||
});
|
||||
|
||||
test('monotone non-increasing', () => {
|
||||
const seq: number[] = [];
|
||||
for (let t = 1; t <= 10; t++) seq.push(linearLr(4, t, 10));
|
||||
for (let i = 1; i < seq.length; i++) {
|
||||
expect(seq[i]!).toBeLessThanOrEqual(seq[i - 1]!);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('constantLr', () => {
|
||||
test('returns base regardless of t or totalSteps', () => {
|
||||
expect(constantLr(4, 1, 10)).toBe(4);
|
||||
expect(constantLr(4, 5, 10)).toBe(4);
|
||||
expect(constantLr(4, 10, 10)).toBe(4);
|
||||
});
|
||||
|
||||
test('floors at 1 when base < 1', () => {
|
||||
expect(constantLr(0, 1, 1)).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveLrSchedule', () => {
|
||||
test('returns the matching function for each schedule name', () => {
|
||||
expect(resolveLrSchedule('cosine')(4, 1, 1)).toBe(4);
|
||||
expect(resolveLrSchedule('linear')(4, 1, 1)).toBe(4);
|
||||
expect(resolveLrSchedule('constant')(4, 1, 1)).toBe(4);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,234 @@
|
||||
/**
|
||||
* SkillOpt scoring unit tests. All three judge modes (rule, llm, qrels).
|
||||
*
|
||||
* LLM judge is tested via DI'd chat seam (no real API calls).
|
||||
*/
|
||||
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import {
|
||||
countCitations,
|
||||
extractRetrievedSlugs,
|
||||
scoreQrels,
|
||||
scoreRule,
|
||||
scoreTrajectory,
|
||||
} from '../../src/core/skillopt/score.ts';
|
||||
import type { Trajectory } from '../../src/core/skillopt/types.ts';
|
||||
|
||||
function makeTrajectory(overrides: Partial<Trajectory> = {}): Trajectory {
|
||||
return {
|
||||
task_id: 't1',
|
||||
task: 'do X',
|
||||
final_text: 'hello world',
|
||||
tool_calls: [],
|
||||
usage: { input_tokens: 100, output_tokens: 50, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
turns: 1,
|
||||
stop_reason: 'end',
|
||||
duration_ms: 500,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('scoreRule', () => {
|
||||
test('all checks pass → 1.0', () => {
|
||||
const t = makeTrajectory({ final_text: 'Short output\n## People\nalice-example' });
|
||||
const s = scoreRule(t, [
|
||||
{ op: 'contains', arg: 'alice' },
|
||||
{ op: 'section_present', arg: '## People' },
|
||||
{ op: 'max_chars', arg: 100 },
|
||||
]);
|
||||
expect(s).toBe(1);
|
||||
});
|
||||
|
||||
test('all checks fail → 0', () => {
|
||||
const t = makeTrajectory({ final_text: 'short' });
|
||||
const s = scoreRule(t, [
|
||||
{ op: 'contains', arg: 'missing' },
|
||||
{ op: 'max_chars', arg: 1 },
|
||||
]);
|
||||
expect(s).toBe(0);
|
||||
});
|
||||
|
||||
test('partial pass → fractional score', () => {
|
||||
const t = makeTrajectory({ final_text: 'hello' });
|
||||
const s = scoreRule(t, [
|
||||
{ op: 'contains', arg: 'hello' },
|
||||
{ op: 'contains', arg: 'goodbye' },
|
||||
]);
|
||||
expect(s).toBe(0.5);
|
||||
});
|
||||
|
||||
test('empty checks array → 0 (no signal)', () => {
|
||||
expect(scoreRule(makeTrajectory(), [])).toBe(0);
|
||||
});
|
||||
|
||||
test('regex op', () => {
|
||||
const t = makeTrajectory({ final_text: 'order #42' });
|
||||
expect(scoreRule(t, [{ op: 'regex', arg: '#\\d+' }])).toBe(1);
|
||||
expect(scoreRule(t, [{ op: 'regex', arg: 'XYZ' }])).toBe(0);
|
||||
});
|
||||
|
||||
test('regex op tolerates malformed regex (returns false)', () => {
|
||||
const t = makeTrajectory({ final_text: 'abc' });
|
||||
expect(scoreRule(t, [{ op: 'regex', arg: '[invalid' }])).toBe(0);
|
||||
});
|
||||
|
||||
test('section_present matches any heading depth', () => {
|
||||
const t = makeTrajectory({ final_text: '# Outline\n## People\n### Team' });
|
||||
expect(scoreRule(t, [{ op: 'section_present', arg: 'People' }])).toBe(1);
|
||||
expect(scoreRule(t, [{ op: 'section_present', arg: 'Team' }])).toBe(1);
|
||||
expect(scoreRule(t, [{ op: 'section_present', arg: 'Missing' }])).toBe(0);
|
||||
});
|
||||
|
||||
test('min_citations counts links + brain-refs + footnotes', () => {
|
||||
const t = makeTrajectory({ final_text: '[link1](http://a.com) and wiki/foo [1]' });
|
||||
expect(scoreRule(t, [{ op: 'min_citations', arg: 3 }])).toBe(1);
|
||||
expect(scoreRule(t, [{ op: 'min_citations', arg: 4 }])).toBe(0);
|
||||
});
|
||||
|
||||
test('tool_called requires a non-failed call with matching name', () => {
|
||||
const t = makeTrajectory({
|
||||
tool_calls: [
|
||||
{ name: 'search', input: {}, output: {}, failed: false },
|
||||
{ name: 'get_page', input: {}, output: {}, failed: true },
|
||||
],
|
||||
});
|
||||
expect(scoreRule(t, [{ op: 'tool_called', arg: 'search' }])).toBe(1);
|
||||
expect(scoreRule(t, [{ op: 'tool_called', arg: 'get_page' }])).toBe(0); // failed
|
||||
expect(scoreRule(t, [{ op: 'tool_called', arg: 'never_called' }])).toBe(0);
|
||||
});
|
||||
|
||||
test('tool_not_called passes when the tool never appears', () => {
|
||||
const t = makeTrajectory({
|
||||
tool_calls: [{ name: 'search', input: {}, output: {}, failed: false }],
|
||||
});
|
||||
expect(scoreRule(t, [{ op: 'tool_not_called', arg: 'put_page' }])).toBe(1);
|
||||
expect(scoreRule(t, [{ op: 'tool_not_called', arg: 'search' }])).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('countCitations', () => {
|
||||
test('counts markdown links, brain-refs, footnotes', () => {
|
||||
expect(countCitations('[a](http://b)')).toBe(1);
|
||||
expect(countCitations('see wiki/foo')).toBe(1);
|
||||
expect(countCitations('mentioned [1] and [2]')).toBe(2);
|
||||
expect(countCitations('[a](http://b) wiki/x [1]')).toBe(3);
|
||||
});
|
||||
|
||||
test('handles empty input', () => {
|
||||
expect(countCitations('')).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('scoreQrels', () => {
|
||||
test('returns nDCG when retrieved slugs overlap expected', () => {
|
||||
const t = makeTrajectory({
|
||||
tool_calls: [{
|
||||
name: 'search',
|
||||
input: {},
|
||||
output: { results: [{ slug: 'people/alice-example' }, { slug: 'companies/widget-co' }] },
|
||||
failed: false,
|
||||
}],
|
||||
});
|
||||
const s = scoreQrels(t, ['people/alice-example', 'companies/widget-co'], 10);
|
||||
expect(s).toBeGreaterThan(0);
|
||||
expect(s).toBeLessThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('returns 0 when no retrieval tool called', () => {
|
||||
expect(scoreQrels(makeTrajectory(), ['anything'], 10)).toBe(0);
|
||||
});
|
||||
|
||||
test('returns 0 when all expected slugs missing from retrieval', () => {
|
||||
const t = makeTrajectory({
|
||||
tool_calls: [{ name: 'search', input: {}, output: { results: [{ slug: 'wrong/slug' }] }, failed: false }],
|
||||
});
|
||||
expect(scoreQrels(t, ['people/missing'], 10)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('extractRetrievedSlugs', () => {
|
||||
test('extracts slugs from various tool output shapes', () => {
|
||||
const t = makeTrajectory({
|
||||
tool_calls: [
|
||||
{ name: 'search', input: {}, output: { results: [{ slug: 'a' }, { slug: 'b' }] }, failed: false },
|
||||
{ name: 'get_page', input: {}, output: { slug: 'c' }, failed: false },
|
||||
{ name: 'list_pages', input: {}, output: { pages: [{ slug: 'd' }] }, failed: false },
|
||||
],
|
||||
});
|
||||
expect(extractRetrievedSlugs(t)).toEqual(['a', 'b', 'c', 'd']);
|
||||
});
|
||||
|
||||
test('deduplicates repeated slugs', () => {
|
||||
const t = makeTrajectory({
|
||||
tool_calls: [
|
||||
{ name: 'search', input: {}, output: { results: [{ slug: 'a' }] }, failed: false },
|
||||
{ name: 'search', input: {}, output: { results: [{ slug: 'a' }] }, failed: false },
|
||||
],
|
||||
});
|
||||
expect(extractRetrievedSlugs(t)).toEqual(['a']);
|
||||
});
|
||||
|
||||
test('skips failed tool calls', () => {
|
||||
const t = makeTrajectory({
|
||||
tool_calls: [
|
||||
{ name: 'search', input: {}, output: { results: [{ slug: 'a' }] }, failed: true },
|
||||
{ name: 'search', input: {}, output: { results: [{ slug: 'b' }] }, failed: false },
|
||||
],
|
||||
});
|
||||
expect(extractRetrievedSlugs(t)).toEqual(['b']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('scoreTrajectory (llm judge via DI)', () => {
|
||||
test('returns parsed score from chat result', async () => {
|
||||
const t = makeTrajectory({ final_text: 'good output' });
|
||||
const stub = async () => ({
|
||||
text: '{"score": 0.85, "rationale": "good"}',
|
||||
blocks: [],
|
||||
stopReason: 'end' as const,
|
||||
usage: { input_tokens: 100, output_tokens: 20, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'test',
|
||||
providerId: 'test',
|
||||
});
|
||||
const r = await scoreTrajectory(t, { kind: 'llm', rubric: 'is it good?' }, { chatFn: stub as never });
|
||||
expect(r.score).toBeCloseTo(0.85, 2);
|
||||
expect(r.rationale).toBe('good');
|
||||
});
|
||||
|
||||
test('returns score=0 on parse failure (pessimistic fallback)', async () => {
|
||||
const t = makeTrajectory();
|
||||
const stub = async () => ({
|
||||
text: 'this is not JSON',
|
||||
blocks: [],
|
||||
stopReason: 'end' as const,
|
||||
usage: { input_tokens: 100, output_tokens: 20, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'test',
|
||||
providerId: 'test',
|
||||
});
|
||||
const r = await scoreTrajectory(t, { kind: 'llm', rubric: 'r' }, { chatFn: stub as never });
|
||||
expect(r.score).toBe(0);
|
||||
expect(r.judge_error).toBeDefined();
|
||||
});
|
||||
|
||||
test('clamps out-of-range scores to [0,1]', async () => {
|
||||
const t = makeTrajectory();
|
||||
const stub = async () => ({
|
||||
text: '{"score": 1.7}',
|
||||
blocks: [],
|
||||
stopReason: 'end' as const,
|
||||
usage: { input_tokens: 100, output_tokens: 20, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'test',
|
||||
providerId: 'test',
|
||||
});
|
||||
const r = await scoreTrajectory(t, { kind: 'llm', rubric: 'r' }, { chatFn: stub as never });
|
||||
expect(r.score).toBe(1);
|
||||
});
|
||||
|
||||
test('returns score=0 + judge_error when chat throws', async () => {
|
||||
const t = makeTrajectory();
|
||||
const stub = async () => { throw new Error('network down'); };
|
||||
const r = await scoreTrajectory(t, { kind: 'llm', rubric: 'r' }, { chatFn: stub as never });
|
||||
expect(r.score).toBe(0);
|
||||
expect(r.judge_error).toMatch(/llm_call_failed.*network down/);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user