mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.32.3.0 skill: functional-area-resolver — pattern for compressing routing tables
Headline: compress a 25KB AGENTS.md down to 13KB without losing routing accuracy. Pattern proven across Opus 4.7, Sonnet 4.6, and Haiku 4.5 — beats the verbose baseline by +13 to +17pp at 48% the size. Empirical (training, n=20, 3 seeds, lenient): baseline 25KB: Opus 81.7% | Sonnet 86.7% | Haiku 73.3% functional-areas 13KB: Opus 98.3% | Sonnet 100% | Haiku 88.3% resolver-of-resolvers 10KB: Opus 63.3% | Sonnet 41.7% | Haiku 65.0% The (dispatcher for: ...) clause is the load-bearing signal. Strip it (the resolver-of-resolvers variant) and Sonnet collapses to 41.7% — the failure case the pattern's authors predicted, now observed. Files in this release: - VERSION + package.json bumped to 0.32.3.0 (4-segment per CLAUDE.md). - CHANGELOG.md: full empirical story, cross-model table, three prior-art citations (AnyTool, RAG-MCP, Anthropic Agent Skills progressive disclosure). - TODOS.md: nine v0.33.x follow-ups (dogfood on gbrain's own RESOLVER.md, CLI promotion to gbrain routing-eval --ab-compare, held-out corpus growth, cross-vendor Gemini+GPT verification, per-row description length sweep, structural compression to ~10KB, hierarchical area-of-areas, embedding pre-router, adversarial fixtures, prompt-design ablation doc). - llms-full.txt regenerated. Bisect-friendly history on this branch:502d447eskill: rename + content rewrite + routing-eval.jsonl472cc686evals: A/B harness + variants + fixtures + tests (no receipts)243e013eevals: cross-model baseline receipts (Opus + Sonnet + Haiku)ecab180bskill: wire-up to RESOLVER.md + manifest.json + openclaw.plugin.json THIS: v0.32.3.0 release marker Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
ecab180bb5
commit
8870c64e86
@@ -2,6 +2,86 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.32.3.0] - 2026-05-11
|
||||
|
||||
**Compress a 25KB AGENTS.md down to 13KB without losing routing accuracy.**
|
||||
**Pattern proven across Opus 4.7, Sonnet 4.6, and Haiku 4.5 — beats the verbose baseline by +13 to +17pp at 48% the size.**
|
||||
|
||||
Downstream agent forks (OpenClaw and friends) grow their AGENTS.md / RESOLVER.md routing files as they add skills. At ~200+ skills these files hit 25-30KB and eat the agent's context budget. v0.32.3.0 ships `functional-area-resolver`, a skill documenting a two-layer dispatch pattern: replace one row per skill with one entry per functional area, with each area listing its sub-skills in a `(dispatcher for: ...)` clause. The LLM reads one area entry and routes to the correct sub-skill.
|
||||
|
||||
This skill is the *static-prompt analog* of hierarchical agent routing — a 2024-2025 research direction (AnyTool, RAG-MCP, Anthropic Agent Skills progressive disclosure). The published hierarchical schemes resolve at runtime via a second LLM call. This one inlines the hierarchy into a single-LLM-pass dispatcher list. Our contribution: showing that single-pass dispatch holds up empirically across three model tiers.
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
Training corpus (n=20 fixtures × 3 seeds, LENIENT scoring — predicted slug shares dispatcher area with expected):
|
||||
|
||||
| Variant | Opus 4.7 | Sonnet 4.6 | Haiku 4.5 | Size |
|
||||
|---|---|---|---|---|
|
||||
| baseline (270 bullet rows) | 81.7% | 86.7% | 73.3% | 25KB |
|
||||
| **functional-areas** (this pattern) | **98.3%** | **100%** | **88.3%** | **13KB** |
|
||||
| resolver-of-resolvers (compression WITHOUT dispatcher clause) | 63.3% | **41.7%** | 65.0% | 10KB |
|
||||
|
||||
Three findings:
|
||||
|
||||
1. **Functional-areas beats the verbose baseline on training across all three models** (+13 to +17pp) at 48% the size. Held-out (n=5, lenient) saturates at 100% for both baseline and functional-areas across all three models.
|
||||
2. **The `(dispatcher for: ...)` clause is the load-bearing signal.** resolver-of-resolvers strips it and collapses to 41.7% on Sonnet — exactly the failure case the pattern's authors predicted, now observed.
|
||||
3. **Strict scoring under-counts.** A prompt that tells the LLM "drill into the dispatcher list" causes the model to predict more-specific sub-skills (`gmail` instead of `executive-assistant`). Strict scoring marks that as failure; lenient (same-area) scoring counts it correct. Lenient matches production agent behavior — an agent that lands in `gmail` for an email intent succeeds either way.
|
||||
|
||||
Receipts at `evals/functional-area-resolver/baseline-runs/2026-05-11-{opus-4-7,sonnet-4-6,haiku-4-5}.jsonl`. Reproduce with `cd evals/functional-area-resolver && node harness.mjs --model {opus|sonnet|haiku}`. The receipt format binds (model, prompt_template_hash, fixtures_hash, harness_sha, ts) so future contributors can verify whether published numbers still reproduce.
|
||||
|
||||
### What this means for downstream agents
|
||||
|
||||
If you maintain an agent fork with >150 skills and AGENTS.md hitting 25KB+:
|
||||
|
||||
1. Apply the functional-area compression pattern to your AGENTS.md (read `skills/functional-area-resolver/SKILL.md` for the procedure).
|
||||
2. Update your agent's harness prompt to handle the `(dispatcher for: ...)` clause — this is load-bearing. Without it, compression collapses routing accuracy to ~30-60%. Reference prompt in `evals/functional-area-resolver/harness-runner.ts:PROMPT_TEMPLATE`.
|
||||
3. Run `gbrain routing-eval` after compression to verify structural routing still passes. Run the harness for an end-to-end LLM check at ~$0.30-1.70 per model.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
**New skill: `functional-area-resolver`** — renamed from the original `compress-agents-md` PR. The pattern is general (any routing table) so the skill name describes the contribution, not the file. Triggers broadened to cover both RESOLVER.md and AGENTS.md phrasings ("compress my resolver", "AGENTS.md too large", "RESOLVER.md too big", "functional area dispatcher", "shrink routing table"). SKILL.md adds preconditions (refuse to compress if file <12KB or working tree dirty), a multi-file routing precedence subsection (v0.31.7 merge of `skills/RESOLVER.md` + `../AGENTS.md`), and a MANDATORY verification step that gates on >=95% accuracy via the new harness.
|
||||
|
||||
**A/B eval surface at `evals/functional-area-resolver/`** — lives OUTSIDE `skills/` deliberately so the skillpack bundler doesn't ship eval infrastructure to every downstream install. Three real production resolver variants extracted from a private deployment's AGENTS.md at git commits `93848ff3b^` (baseline, 25KB) and `93848ff3b` (functional-areas, 13KB), with owner PII scrubbed. `resolver-of-resolvers.md` derived mechanically from functional-areas by stripping `(dispatcher for: ...)` clauses (the ablation case). 20-fixture training corpus + 5-fixture held-out blind corpus, n=3 seeded repeats per call, t-distribution 95% CIs.
|
||||
|
||||
**TypeScript harness (`evals/functional-area-resolver/harness-runner.ts`)** — routed through gbrain's gateway (`src/core/ai/gateway.ts:chat()`) with self-configuration. Supports `--model {opus|sonnet|haiku}` for cross-model eval, `--variants-dir` + `--variants` for description-length sweeps, strict + lenient scoring (both columns in every output row), cost-estimate prompt before each run, missing-binary fallback. Receipt header binds (model, prompt_template_hash, fixtures_hash, harness_sha, ts) — re-runs are auditable against the harness state. 45 unit tests in `harness-runner.test.ts`. Companion `rescore.mjs` re-scores existing JSONL with lenient tolerance for zero API cost.
|
||||
|
||||
**Three baseline receipts committed** in `evals/functional-area-resolver/baseline-runs/`: one per model (Opus 4.7, Sonnet 4.6, Haiku 4.5). Each contains the full 225-row run (3 variants × 25 fixtures × 3 seeds) with both strict and lenient scoring. ~$3 total API spend across the cross-model sweep.
|
||||
|
||||
**Strict + lenient scoring** — every output row carries `correct` (strict, slug match) and `correct_lenient` (predicted shares dispatcher area with expected). Strict scoring under-counts: when the LLM correctly drills into a sub-skill listed in the dispatcher clause (e.g. predicts `gmail` for an email intent when the fixture wrote `executive-assistant`), strict marks it wrong. Lenient reflects production behavior where any sub-skill in the right area resolves correctly.
|
||||
|
||||
**Bundle wire-up** — added `functional-area-resolver` to `skills/manifest.json`, `skills/RESOLVER.md` Operational section (adjacent to `skillify`), and `openclaw.plugin.json` skills array. Plugin manifest version bumped from `0.25.1` to `0.32.3.0` so install receipts stop being stale.
|
||||
|
||||
**Routing fixtures** — `skills/functional-area-resolver/routing-eval.jsonl` has 8 positive fixtures (5 original + 3 covering broadened triggers) plus 4 adversarial negative fixtures targeting `skillify`, `skill-creator`, `book-mirror`, `concept-synthesis` to prove the broadened triggers don't over-capture adjacent meta-skills. `gbrain routing-eval` reports 70/70 passes (100% structural accuracy).
|
||||
|
||||
**Prior-art citations** — SKILL.md now cites AnyTool ([arXiv:2402.04253](https://arxiv.org/abs/2402.04253), the hierarchical-routing-helps argument), RAG-MCP ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1), the 49.2% token-reduction prior result), and Anthropic Agent Skills progressive disclosure (the structural design pattern). The skill is positioned as the static-prompt analog of the runtime hierarchical schemes in the 2024-2025 literature.
|
||||
|
||||
**Nine v0.33.x follow-up TODOs filed** — dogfood gbrain's own RESOLVER.md, CLI promotion to `gbrain routing-eval --ab-compare`, held-out corpus growth to >=20, cross-vendor (Gemini + GPT) verification, per-row description length sweep, structural compression to ~10KB, hierarchical area-of-areas, embedding-based pre-router, adversarial fixtures, run-1 vs run-2 prompt-design ablation methodology doc. See `TODOS.md` for context.
|
||||
|
||||
## To take advantage of v0.32.3.0
|
||||
|
||||
`gbrain upgrade` does not auto-surface new skills in existing installs — skills are installed via `gbrain skillpack install`. After upgrading the binary:
|
||||
|
||||
1. **Surface the new skill in your install:**
|
||||
```bash
|
||||
gbrain skillpack install --update
|
||||
```
|
||||
This adds `functional-area-resolver` to your managed skills block. The skill is discoverable on next agent invocation.
|
||||
|
||||
2. **(Optional) Verify the skill is reachable:**
|
||||
```bash
|
||||
gbrain check-resolvable --json | grep functional-area-resolver
|
||||
gbrain routing-eval # All routing fixtures, including the new ones, must pass
|
||||
```
|
||||
|
||||
3. **(Maintainer-only) Re-baseline the eval table** — only relevant if you're contributing to gbrain itself:
|
||||
```bash
|
||||
cd evals/functional-area-resolver
|
||||
node harness.mjs # ~$1.70 at Opus 4.7 pricing; needs ANTHROPIC_API_KEY
|
||||
```
|
||||
Move the resulting JSONL to `baseline-runs/<date>-opus-4-7.jsonl` and update the SKILL.md table with the new CI numbers.
|
||||
|
||||
4. **If `gbrain doctor` warns about anything after upgrade**, file an issue at https://github.com/garrytan/gbrain/issues with the doctor output. v0.32.3.0 adds no schema migrations, so the upgrade should be invisible to existing brains.
|
||||
|
||||
## [0.32.0] - 2026-05-10
|
||||
|
||||
**5 new embedding providers + the discoverability fix that closes the 17-PR dupe cluster.**
|
||||
|
||||
@@ -1,5 +1,27 @@
|
||||
# TODOS
|
||||
|
||||
## functional-area-resolver follow-ups (v0.32.3.0)
|
||||
|
||||
- [ ] **v0.33.x: Dogfood `functional-area-resolver` on gbrain's own `skills/RESOLVER.md`** when it crosses ~12KB (currently 8KB). Apply the pattern to the Operational section first (largest). Filed during v0.32.3.0 CEO review.
|
||||
|
||||
- [ ] **v0.33.x: Promote `evals/functional-area-resolver/harness.mjs` to a first-class CLI command** `gbrain routing-eval --ab-compare <variant-dir>`. Removes the one-off harness as maintenance debt; gives every pattern-skill a way to ship its eval. Replaces the placeholder `--llm` flag in `src/core/routing-eval.ts:17-20`. Filed during v0.32.3.0 CEO review.
|
||||
|
||||
- [ ] **v0.33.x: Expand held-out corpus to >=20 fixtures.** The current n=5 saturates at 100% across most cells and can't distinguish "100%" from "95% with one nondeterministic miss." Author independently (don't see variants while authoring). Filed during v0.32.3.0 boil-the-ocean push after codex outside-voice review.
|
||||
|
||||
- [ ] **v0.33.x: Cross-vendor model verification.** Run the harness on Gemini 2.5 Pro and GPT-4o/5 in addition to the three Anthropic models we already covered. Compression gains may not transfer across vendor families (the `(dispatcher for: ...)` clause is interpreted differently by different prompt-tuned models). Wire through the existing gbrain gateway (recipes already exist for both vendors).
|
||||
|
||||
- [ ] **v0.33.x: Per-row description length sweep.** Anthropic's Agent Skills median is ~80 tokens of frontmatter per skill ([Anthropic engineering blog](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)). Sweep functional-areas at {20, 40, 80, 160} tokens per dispatcher row, eval each. Novel published contribution — no public data exists. ~$5 in API spend. Filed during v0.32.3.0 web research.
|
||||
|
||||
- [ ] **v0.33.x: Structural compression of functional-areas (`(dispatcher for: ...)` → `dispatcher: [...]` YAML form, trim verbose triggers, separate hard gates to sibling file).** Target 13KB → 9-10KB without accuracy regression. Requires another full re-baseline run (~$3 across 3 models) to confirm no regression.
|
||||
|
||||
- [ ] **v0.33.x: Hierarchical compression (area-of-areas).** Two-level: top-level mega-areas (knowledge / ops / comms) pointing to functional-area files loaded lazily. Predicted 13KB → 4-6KB. Risks resolver-of-resolvers-style collapse on the top-level layer. Worth an A/B but its own piece of work. Cross-reference AnyTool ([arXiv:2402.04253](https://arxiv.org/abs/2402.04253)) which formalizes this hierarchy at runtime.
|
||||
|
||||
- [ ] **v0.33.x: Embedding-based area pre-router.** RAG-MCP shape ([arXiv:2505.03275](https://arxiv.org/html/2505.03275v1)) — cheap embedding model picks the area; only that area's sub-skills get sent to the LLM. Dramatic per-call payload reduction (~80%). Significant new code surface but big production cost win. Wire through the existing gateway's voyage or openai embedding recipes.
|
||||
|
||||
- [ ] **v0.33.x: Adversarial-intent fixtures.** Intents specifically designed to test dispatcher-vs-subskill behavior on edge cases ("I want to do something brain-related" without specifying what). Targets the prompt-design failure mode (run-1 collapse) that our current 25 fixtures don't surface. ~10-15 fixtures, authored without looking at variant content.
|
||||
|
||||
- [ ] **v0.33.x: Run-2 vs Run-1 prompt-design ablation.** Document the difference between the naive classifier prompt (run-1, every variant 30-60% training) and the dispatcher-aware prompt (run-2+, functional-areas 88-100% training) as a reproducible result. This is the strongest empirical finding from v0.32.3.0 and deserves its own callout in SKILL.md or a sibling METHODOLOGY.md.
|
||||
|
||||
## Embedding-provider follow-ups (v0.32.0)
|
||||
|
||||
- [ ] **v0.32.x: Vertex AI ADC embedding provider (#729 originally).** lucha0404
|
||||
|
||||
@@ -1745,6 +1745,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef
|
||||
| Save or load reports | `skills/reports/SKILL.md` |
|
||||
| "Create a skill", "improve this skill" | `skills/skill-creator/SKILL.md` |
|
||||
| "Skillify this", "is this a skill?", "make this proper" | `skills/skillify/SKILL.md` |
|
||||
| "Compress my resolver", "AGENTS.md too large", "RESOLVER.md too big", "functional area dispatcher", "shrink routing table" | `skills/functional-area-resolver/SKILL.md` |
|
||||
| "Is gbrain healthy?", morning health check, skillpack-check | `skills/skillpack-check/SKILL.md` |
|
||||
| Post-restart health + auto-fix, "did the container restart break anything", smoke test | `skills/smoke-test/SKILL.md` |
|
||||
| Cross-modal review, second opinion | `skills/cross-modal-review/SKILL.md` |
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.32.0",
|
||||
"version": "0.32.3.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
Reference in New Issue
Block a user