mirror of
https://github.com/open-jarvis/OpenJarvis.git
synced 2026-07-27 21:05:34 +00:00
feat: add skills system with import, learning loop, and benchmark harness (#230)
This commit is contained in:
@@ -0,0 +1,78 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to OpenJarvis are documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
||||
|
||||
---
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
#### Skills System (Plans 1, 2A, 2B)
|
||||
|
||||
- **Skills core** — every skill is a tool. Skills appear in a system prompt catalog, agents invoke them on demand, content (pipeline results, markdown instructions, or both) gets injected into context.
|
||||
- `SkillManifest` + `SkillStep` types with tags, depends, invocation flags, markdown content
|
||||
- `SkillManager` — discovery, precedence resolution, catalog XML generation, tool wrapping
|
||||
- `SkillTool(BaseTool)` — auto-extracts parameters from step argument templates
|
||||
- `SkillExecutor` — sequential pipeline execution with sub-skill delegation
|
||||
- Dependency graph with cycle detection, max depth enforcement, capability unions
|
||||
- Security: four trust tiers (bundled/indexed/unreviewed/workspace), capability-gated enforcement
|
||||
- Skill index module for git-backed registry search
|
||||
|
||||
- **agentskills.io spec adoption** — canonical `SKILL.md` format with YAML frontmatter following the [agentskills.io](https://agentskills.io/specification) open standard.
|
||||
- `SkillParser` with strict spec validation + tolerant field mapping via `FIELD_MAPPING` table
|
||||
- `ToolTranslator` for external tool name translation (Bash -> shell_exec, Read -> file_read, etc.)
|
||||
- Source resolvers: `HermesResolver`, `OpenClawResolver`, `GitHubResolver`
|
||||
- `SkillImporter` with provenance tracking (`.source` metadata files), optional script import
|
||||
- Sourced subdirectory layout (`~/.openjarvis/skills/<source>/<name>/`)
|
||||
|
||||
- **Skills learning loop** — trace tagging, pattern discovery, DSPy/GEPA optimization.
|
||||
- Trace metadata tagging: `skill`, `skill_source`, `skill_kind` flow through ToolExecutor -> TraceCollector -> TraceStep
|
||||
- `SkillDiscovery` wired into `SkillManager.discover_from_traces()` with kebab name normalization
|
||||
- `SkillOptimizer` — per-skill DSPy/GEPA wrapper that buckets traces and writes sidecar overlays
|
||||
- `SkillOverlay` — sidecar storage at `~/.openjarvis/learning/skills/<name>/optimized.toml`
|
||||
- `SkillManager._load_overlays()` applies optimized descriptions + few-shot examples at discovery time
|
||||
- `LearningOrchestrator._maybe_optimize_skills()` — opt-in auto-trigger
|
||||
|
||||
- **Skills benchmark harness** — 4-condition PinchBench evaluation.
|
||||
- I3 fix: `skill_few_shot_examples` wired through SystemBuilder -> `_run_agent` -> `ToolUsingAgent` -> `native_react.REACT_SYSTEM_PROMPT`
|
||||
- `SkillBenchmarkRunner` — 4-condition x N-seed x M-task sweep with markdown report
|
||||
- `JarvisAgentBackend` accepts `skills_enabled` and `overlay_dir` kwargs
|
||||
- Conditions: `no_skills`, `skills_on`, `skills_optimized_dspy`, `skills_optimized_gepa`
|
||||
|
||||
- **CLI commands:**
|
||||
- `jarvis skill list` / `info` / `run` / `install` / `sync` / `sources` / `update` / `remove` / `search`
|
||||
- `jarvis skill discover` — mine traces for recurring tool patterns
|
||||
- `jarvis skill show-overlay` — inspect optimization output
|
||||
- `jarvis optimize skills` — run DSPy/GEPA per-skill optimization
|
||||
- `jarvis bench skills` — run the PinchBench skills benchmark
|
||||
|
||||
- **Agent prompt improvement:**
|
||||
- `native_react.REACT_SYSTEM_PROMPT` now includes "Using Skills" guidance that teaches agents to distinguish executable vs. instructional skill responses
|
||||
- `{skill_examples}` placeholder for optimized few-shot example injection
|
||||
|
||||
- **Configuration:**
|
||||
- `[skills]` section: `enabled`, `skills_dir`, `active`, `auto_discover`, `auto_sync`, `max_depth`, `sandbox_dangerous`
|
||||
- `[[skills.sources]]` section: `source`, `url`, `filter`, `auto_update`
|
||||
- `[learning.skills]` section: `auto_optimize`, `optimizer`, `min_traces_per_skill`, `optimization_interval_seconds`, `overlay_dir`
|
||||
- `SkillSourceConfig` and `SkillsLearningConfig` dataclasses
|
||||
|
||||
- **Documentation:**
|
||||
- `docs/user-guide/skills.md` — comprehensive user guide
|
||||
- `docs/architecture/skills.md` — technical deep-dive
|
||||
- `docs/tutorials/skills-workflow.md` — end-to-end tutorial
|
||||
- `docs/getting-started/configuration.md` — expanded with skills config sections
|
||||
- `CLAUDE.md` — updated architecture section
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Trace metadata flow** — `ToolResult.metadata` now propagates through `TOOL_CALL_END` event to `TraceStep.metadata` (was silently dropped at the event-bus boundary)
|
||||
- **TaintSet JSON serialization** — `ToolExecutor._json_safe_metadata()` filters non-JSON-serializable values (like `TaintSet`) from event payloads before they reach `TraceStore`
|
||||
- **Non-dict YAML frontmatter** — source resolvers handle `yaml.safe_load()` returning a string instead of a dict (discovered on real OpenClaw imports)
|
||||
- **OpenClaw category/name queries** — `jarvis skill install openclaw:owner/slug` now correctly splits into category + name match
|
||||
- **SkillDiscovery trace compatibility** — `_extract_tool_sequence` reads from `step.input["tool"]` (the actual `TraceStep` format), not the nonexistent `step.tool_name` attribute
|
||||
- **LearningOrchestrator skill trigger** — `_maybe_optimize_skills` runs BEFORE the SFT-data short-circuit (skills are tagged via trace metadata, not mined as SFT pairs)
|
||||
- **PinchBenchScorer constructor** — `SkillBenchmarkRunner` constructs `PinchBenchScorer(judge_backend, model)` instead of no-args
|
||||
- **EvalRunner results access** — reads per-task data from `eval_runner.results` property, not nonexistent `summary.results`
|
||||
@@ -107,6 +107,29 @@ jarvis memory index ./docs/ # index your documents
|
||||
jarvis ask "Summarize all emails about Project X"
|
||||
```
|
||||
|
||||
### Skills
|
||||
|
||||
Skills teach agents how to better use tools and improve their reasoning. Every skill is a tool — agents discover them from a catalog and invoke them on demand.
|
||||
|
||||
```bash
|
||||
# Install skills from public sources
|
||||
jarvis skill install hermes:arxiv
|
||||
jarvis skill sync hermes --category research
|
||||
|
||||
# Use skills with any agent
|
||||
jarvis ask "Use the code-explainer skill to explain this Python code: for i in range(5): print(i*2)"
|
||||
|
||||
# Optimize skills from your trace history
|
||||
jarvis optimize skills --policy dspy
|
||||
|
||||
# Benchmark the impact
|
||||
jarvis bench skills --max-samples 5 --seeds 42
|
||||
```
|
||||
|
||||
Import from [Hermes Agent](https://github.com/NousResearch/hermes-agent) (~150 skills), [OpenClaw](https://github.com/openclaw/skills) (~13,700 community skills), or any GitHub repo. Skills follow the [agentskills.io](https://agentskills.io/specification) open standard.
|
||||
|
||||
See the [Skills User Guide](https://open-jarvis.github.io/OpenJarvis/user-guide/skills/) and [Skills Tutorial](https://open-jarvis.github.io/OpenJarvis/tutorials/skills-workflow/) for details.
|
||||
|
||||
### Built-in Agents
|
||||
|
||||
| Agent | Type | What it does |
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
---
|
||||
title: Skills Architecture
|
||||
description: Technical deep-dive into the skills system design, components, and integration patterns
|
||||
---
|
||||
|
||||
# Skills Architecture
|
||||
|
||||
Skills are a **cross-cutting orchestration layer** that sits across the five existing primitives (Intelligence, Engine, Agents, Memory/Tools, Learning). They connect tools, agents, memory, and learning into reusable workflows without replacing or subsumming any primitive.
|
||||
|
||||
## System Design
|
||||
|
||||
```
|
||||
┌──────────────────────┐
|
||||
│ SystemBuilder │
|
||||
│ .build() │
|
||||
└──────────┬───────────┘
|
||||
│
|
||||
┌──────────▼───────────┐
|
||||
│ SkillManager │
|
||||
│ • discover() │
|
||||
│ • get_skill_tools()│
|
||||
│ • get_catalog_xml()│
|
||||
└──────────┬───────────┘
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
│ │ │
|
||||
┌─────────▼──────┐ ┌──────▼──────┐ ┌───────▼───────┐
|
||||
│ SkillTool │ │ Catalog │ │ Overlay │
|
||||
│ (BaseTool) │ │ XML │ │ Loader │
|
||||
│ → agent tools │ │ → sys. │ │ → optimized │
|
||||
│ list │ │ prompt │ │ desc + │
|
||||
│ │ │ │ │ few-shot │
|
||||
└────────────────┘ └─────────────┘ └───────────────┘
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
### SkillManifest (`skills/types.py`)
|
||||
|
||||
The canonical data structure for a loaded skill:
|
||||
|
||||
```python
|
||||
@dataclass(slots=True)
|
||||
class SkillManifest:
|
||||
name: str
|
||||
version: str = "0.1.0"
|
||||
description: str = ""
|
||||
author: str = ""
|
||||
steps: List[SkillStep] = field(default_factory=list)
|
||||
required_capabilities: List[str] = field(default_factory=list)
|
||||
signature: str = ""
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
tags: List[str] = field(default_factory=list)
|
||||
depends: List[str] = field(default_factory=list)
|
||||
user_invocable: bool = True
|
||||
disable_model_invocation: bool = False
|
||||
markdown_content: str = ""
|
||||
```
|
||||
|
||||
### SkillManager (`skills/manager.py`)
|
||||
|
||||
The central coordinator. Created by `SystemBuilder.build()` during system composition.
|
||||
|
||||
**Lifecycle:**
|
||||
1. `discover(paths)` — scans skill directories in precedence order, loads manifests, validates the dependency graph, applies optimization overlays
|
||||
2. `get_skill_tools()` — wraps each discovered skill as a `SkillTool(BaseTool)`, wires sub-skill resolver callbacks
|
||||
3. `get_catalog_xml()` — generates the lightweight `<available_skills>` XML for system prompt injection
|
||||
4. `get_few_shot_examples()` — returns formatted few-shot strings from optimization overlays
|
||||
|
||||
### SkillTool (`skills/tool_adapter.py`)
|
||||
|
||||
Adapter that makes any skill look like a regular `BaseTool` to agents:
|
||||
|
||||
- `spec` property derives `ToolSpec` from the manifest — auto-extracts input parameters from step argument templates
|
||||
- `execute(**params)` runs the pipeline (if steps exist), returns markdown content (if SKILL.md exists), or both
|
||||
- `_build_result_metadata()` tags every invocation with `skill`, `skill_source`, `skill_kind` for downstream trace analysis
|
||||
|
||||
### SkillParser (`skills/parser.py`)
|
||||
|
||||
Two-pass parser for agentskills.io-compatible SKILL.md frontmatter:
|
||||
|
||||
1. **Strict pass** — validates required fields (`name`, `description`), length limits, kebab-case naming rules
|
||||
2. **Tolerant pass** — maps non-spec vendor fields to canonical locations via `FIELD_MAPPING` table. Unmapped fields are logged and preserved in `metadata.openjarvis.original_frontmatter`
|
||||
|
||||
The mapping table is data, not code paths. Adding support for new vendor fields means adding entries — no logic changes.
|
||||
|
||||
### SkillExecutor (`skills/executor.py`)
|
||||
|
||||
Sequential pipeline executor:
|
||||
|
||||
- Steps with `tool_name` → delegate to `ToolExecutor.execute()`
|
||||
- Steps with `skill_name` → delegate to a resolver callback (set by SkillManager)
|
||||
- Template rendering: `{placeholder}` syntax resolved from a shared context dict
|
||||
- `output_key` stores each step's result for downstream steps
|
||||
- Publishes `SKILL_EXECUTE_START` / `SKILL_EXECUTE_END` events on the EventBus
|
||||
|
||||
### Source Resolvers (`skills/sources/`)
|
||||
|
||||
One resolver per import source, all implementing `SourceResolver` ABC:
|
||||
|
||||
| Resolver | Repo layout | Special handling |
|
||||
|----------|-------------|------------------|
|
||||
| `HermesResolver` | `skills/<category>/<skill>/` | Skips `DESCRIPTION.md`, reads Hermes vendor metadata |
|
||||
| `OpenClawResolver` | `skills/<owner>/<skill>/` | Reads `_meta.json` sidecars |
|
||||
| `GitHubResolver` | Recursive walk for `SKILL.md` | Generic — accepts any repo URL |
|
||||
|
||||
### SkillImporter (`skills/importer.py`)
|
||||
|
||||
Takes a `ResolvedSkill` from a source resolver and installs it on disk:
|
||||
|
||||
1. Parse source SKILL.md through `SkillParser`
|
||||
2. Translate tool references (`Bash` → `shell_exec`, `Read` → `file_read`, etc.)
|
||||
3. Compatibility check (platform, missing tools)
|
||||
4. Copy SKILL.md + references/assets/templates (scripts gated by `--with-scripts`)
|
||||
5. Write `.source` provenance file with commit SHA, translated tools, timestamps
|
||||
|
||||
### SkillOverlay (`skills/overlay.py`)
|
||||
|
||||
Sidecar storage for optimization output at `~/.openjarvis/learning/skills/<name>/optimized.toml`:
|
||||
|
||||
```toml
|
||||
[optimized]
|
||||
skill_name = "research-and-summarize"
|
||||
optimizer = "dspy"
|
||||
optimized_at = "2026-04-08T14:30:00Z"
|
||||
trace_count = 47
|
||||
description = "An optimized description"
|
||||
|
||||
[[optimized.few_shot]]
|
||||
input = "transformer attention mechanisms"
|
||||
output = "## Recent Advances..."
|
||||
```
|
||||
|
||||
The overlay is the **contract** between the optimizer and the SkillManager. Both sides agree on the schema; either can be swapped independently.
|
||||
|
||||
### SkillOptimizer (`learning/agents/skill_optimizer.py`)
|
||||
|
||||
Per-skill wrapper around DSPy/GEPA:
|
||||
|
||||
1. Buckets traces by `metadata.skill` (from the C1 trace tagging)
|
||||
2. Skips skills below `min_traces_per_skill` threshold
|
||||
3. Calls `_run_dspy()` or `_run_gepa()` per qualifying skill
|
||||
4. Writes overlay TOML files
|
||||
|
||||
## Integration Points
|
||||
|
||||
### SystemBuilder Wiring
|
||||
|
||||
`SystemBuilder.build()` handles skill integration:
|
||||
|
||||
```python
|
||||
# 1. Create SkillManager
|
||||
skill_manager = SkillManager(bus, capability_policy=...)
|
||||
|
||||
# 2. Discover skills from disk
|
||||
skill_manager.discover(paths=[workspace_skills, user_skills])
|
||||
|
||||
# 3. Wrap as tools and merge into tool list
|
||||
skill_tools = skill_manager.get_skill_tools(tool_executor=...)
|
||||
tool_list.extend(skill_tools)
|
||||
|
||||
# 4. Capture few-shot examples for agents
|
||||
system._skill_few_shot_examples = skill_manager.get_few_shot_examples()
|
||||
```
|
||||
|
||||
### Trace Metadata Flow
|
||||
|
||||
When an agent invokes a `SkillTool`:
|
||||
|
||||
```
|
||||
SkillTool.execute()
|
||||
→ ToolResult(metadata={"skill": name, "skill_source": src, "skill_kind": kind})
|
||||
→ ToolExecutor._json_safe_metadata() filters non-serializable values
|
||||
→ TOOL_CALL_END event with metadata
|
||||
→ TraceCollector._on_tool_end() → TraceStep(metadata=...)
|
||||
→ TraceStore saves to SQLite (metadata as JSON)
|
||||
→ SkillOptimizer._bucket_traces_by_skill() reads metadata.skill
|
||||
```
|
||||
|
||||
### Agent Few-Shot Injection
|
||||
|
||||
Optimized few-shot examples flow through:
|
||||
|
||||
```
|
||||
SkillManager.get_few_shot_examples()
|
||||
→ system._skill_few_shot_examples (stashed on JarvisSystem)
|
||||
→ _run_agent() → agent_kwargs["skill_few_shot_examples"]
|
||||
→ ToolUsingAgent._skill_few_shot_examples
|
||||
→ native_react.run() → REACT_SYSTEM_PROMPT.format(skill_examples=...)
|
||||
```
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
Skills can compose other skills. At discovery time, SkillManager validates:
|
||||
|
||||
1. **Cycle detection** — Kahn's algorithm for topological sort
|
||||
2. **Max depth enforcement** — configurable (default 5)
|
||||
3. **Capability union** — parent must declare all transitive child capabilities
|
||||
|
||||
## File Layout
|
||||
|
||||
```
|
||||
src/openjarvis/skills/
|
||||
├── __init__.py # Public exports
|
||||
├── types.py # SkillManifest, SkillStep
|
||||
├── manager.py # SkillManager
|
||||
├── executor.py # SkillExecutor + sub-skill delegation
|
||||
├── loader.py # TOML + Markdown + directory loading
|
||||
├── tool_adapter.py # SkillTool(BaseTool) wrapper
|
||||
├── parser.py # Strict + tolerant agentskills.io parser
|
||||
├── tool_translator.py # External tool name translation
|
||||
├── importer.py # Install from resolved sources
|
||||
├── overlay.py # Optimization sidecar storage
|
||||
├── dependency.py # Graph validation
|
||||
├── security.py # Trust tiers, capability validation
|
||||
├── index.py # Git-backed skill index
|
||||
└── sources/
|
||||
├── base.py # SourceResolver ABC
|
||||
├── hermes.py # HermesResolver
|
||||
├── openclaw.py # OpenClawResolver
|
||||
└── github.py # GitHubResolver
|
||||
```
|
||||
@@ -432,6 +432,81 @@ db_path = "~/.openjarvis/traces.db"
|
||||
|
||||
---
|
||||
|
||||
### `[skills]` — Skills System
|
||||
|
||||
Controls the skills system — reusable compositions of tools and agent instructions. Skills teach agents how to better use tools and improve their reasoning. See the [Skills User Guide](../user-guide/skills.md) for full documentation.
|
||||
|
||||
```toml
|
||||
[skills]
|
||||
enabled = true
|
||||
skills_dir = "~/.openjarvis/skills/"
|
||||
active = "*"
|
||||
auto_discover = true
|
||||
auto_sync = false
|
||||
max_depth = 5
|
||||
sandbox_dangerous = true
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `enabled` | bool | `true` | Whether to enable the skills system. When disabled, no skills are loaded or exposed to agents. |
|
||||
| `skills_dir` | string | `~/.openjarvis/skills/` | Directory where skills are installed. |
|
||||
| `active` | string | `"*"` | Comma-separated list of skill names to activate, or `"*"` for all discovered skills. |
|
||||
| `auto_discover` | bool | `true` | Whether to scan `skills_dir` for skills on startup. |
|
||||
| `auto_sync` | bool | `false` | Whether to pull from configured sources on session start (checks freshness every 24h). |
|
||||
| `max_depth` | int | `5` | Maximum sub-skill nesting depth for composed skills. |
|
||||
| `sandbox_dangerous` | bool | `true` | Whether to warn about skills with dangerous capabilities (`shell:execute`, `network:listen`, `filesystem:write`). |
|
||||
|
||||
#### `[[skills.sources]]` — Skill Import Sources
|
||||
|
||||
Configure one or more skill sources for automatic import. Each `[[skills.sources]]` entry defines a source to pull from.
|
||||
|
||||
```toml
|
||||
[[skills.sources]]
|
||||
source = "hermes"
|
||||
filter = { category = ["research", "coding", "productivity"] }
|
||||
auto_update = true
|
||||
|
||||
[[skills.sources]]
|
||||
source = "openclaw"
|
||||
filter = { search = "web3|crypto" }
|
||||
|
||||
[[skills.sources]]
|
||||
source = "github"
|
||||
url = "https://github.com/myorg/internal-skills"
|
||||
auto_update = true
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `source` | string | `""` | Source type: `"hermes"`, `"openclaw"`, or `"github"`. |
|
||||
| `url` | string | `""` | Repository URL. Required when `source = "github"`. |
|
||||
| `filter` | table | `{}` | Filter criteria. Supported keys: `category` (list of strings), `search` (regex string). |
|
||||
| `auto_update` | bool | `false` | Whether to pull latest commits when syncing this source. |
|
||||
|
||||
#### `[learning.skills]` — Skills Learning Loop
|
||||
|
||||
Controls the automatic optimization of skill descriptions and few-shot examples from trace data. Requires `[traces] enabled = true` to collect the traces that the optimizer analyzes.
|
||||
|
||||
```toml
|
||||
[learning.skills]
|
||||
auto_optimize = false
|
||||
optimizer = "dspy"
|
||||
min_traces_per_skill = 20
|
||||
optimization_interval_seconds = 86400
|
||||
overlay_dir = "~/.openjarvis/learning/skills/"
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `auto_optimize` | bool | `false` | Whether to run skill optimization automatically after each learning cycle. |
|
||||
| `optimizer` | string | `"dspy"` | Optimization policy: `"dspy"` (bootstrap few-shot) or `"gepa"` (evolutionary). |
|
||||
| `min_traces_per_skill` | int | `20` | Minimum trace count for a skill to be eligible for optimization. |
|
||||
| `optimization_interval_seconds` | int | `86400` | Run optimization at most once per this interval (default: once per day). |
|
||||
| `overlay_dir` | string | `~/.openjarvis/learning/skills/` | Where optimized skill overlays are stored. |
|
||||
|
||||
---
|
||||
|
||||
### `[channel]` — Channel Messaging
|
||||
|
||||
Controls the channel messaging bridge for multi-platform communication. Each supported platform has its own nested sub-section.
|
||||
|
||||
@@ -44,6 +44,14 @@ Hands-on guides that walk through building real applications with OpenJarvis. Ea
|
||||
|
||||
[:octicons-arrow-right-24: Get started](code-companion.md)
|
||||
|
||||
- :material-puzzle:{ .lg .middle } **Skills Workflow**
|
||||
|
||||
---
|
||||
|
||||
Install skills from Hermes Agent, use them with a local agent, discover patterns from traces, optimize with DSPy, and benchmark the impact — the complete skills lifecycle.
|
||||
|
||||
[:octicons-arrow-right-24: Get started](skills-workflow.md)
|
||||
|
||||
</div>
|
||||
|
||||
## What You Will Learn
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
---
|
||||
title: Skills Workflow
|
||||
description: End-to-end tutorial — install skills, use them with an agent, discover patterns from traces, and optimize with DSPy
|
||||
---
|
||||
|
||||
# Skills Workflow Tutorial
|
||||
|
||||
This tutorial walks through the complete skills lifecycle: installing skills from public sources, using them with a local agent, discovering patterns from trace history, and optimizing skill descriptions with DSPy. By the end you will have a working skills setup that improves over time.
|
||||
|
||||
!!! note "Before you begin"
|
||||
This tutorial assumes OpenJarvis is installed with Ollama running and a model available (e.g., `qwen3.5:9b`). If you have not completed setup yet, start with the [Quick Start guide](../getting-started/quickstart.md).
|
||||
|
||||
## Step 1: Install Skills from Hermes Agent
|
||||
|
||||
OpenJarvis can import skills from the [Hermes Agent](https://github.com/NousResearch/hermes-agent) skill library maintained by NousResearch. Let's install a few useful ones.
|
||||
|
||||
```bash
|
||||
# Install individual skills
|
||||
jarvis skill install hermes:arxiv
|
||||
jarvis skill install hermes:github-pr-workflow
|
||||
|
||||
# Or bulk install an entire category
|
||||
jarvis skill sync hermes --category research
|
||||
```
|
||||
|
||||
The first install clones the Hermes repo to `~/.openjarvis/skill-cache/hermes/` (one-time, ~5s). Subsequent installs reuse the cache.
|
||||
|
||||
Verify what's installed:
|
||||
|
||||
```bash
|
||||
jarvis skill list
|
||||
```
|
||||
|
||||
You should see a table with each skill's name, description, version, and tags.
|
||||
|
||||
## Step 2: Inspect an Installed Skill
|
||||
|
||||
Let's look at what the `arxiv` skill contains:
|
||||
|
||||
```bash
|
||||
jarvis skill info arxiv
|
||||
```
|
||||
|
||||
This shows the skill's metadata — author, description, tags, capabilities, whether it has structured steps or markdown instructions, and its invocation flags.
|
||||
|
||||
You can also inspect the raw SKILL.md:
|
||||
|
||||
```bash
|
||||
cat ~/.openjarvis/skills/hermes/arxiv/SKILL.md | head -40
|
||||
```
|
||||
|
||||
The `.source` file records provenance:
|
||||
|
||||
```bash
|
||||
cat ~/.openjarvis/skills/hermes/arxiv/.source
|
||||
```
|
||||
|
||||
This shows the source (`hermes:arxiv`), the git commit it was imported from, which tool names were translated (e.g., `Edit→file_edit`), and the install timestamp.
|
||||
|
||||
## Step 3: Use Skills with an Agent
|
||||
|
||||
Now let's ask the agent a question that should trigger skill usage:
|
||||
|
||||
```bash
|
||||
jarvis ask "Use the code-explainer skill to explain this Python code: for i in range(5): print(i*2)" \
|
||||
--engine ollama --model qwen3.5:9b
|
||||
```
|
||||
|
||||
The agent will:
|
||||
1. See the skill catalog in its system prompt
|
||||
2. Decide to invoke `skill_code-explainer`
|
||||
3. Receive the markdown instructions from the skill
|
||||
4. Follow the 5-step pattern to explain the code
|
||||
|
||||
Try a pipeline skill too:
|
||||
|
||||
```bash
|
||||
jarvis ask "Use the math-solver skill to compute 17 * 23" \
|
||||
--engine ollama --model qwen3.5:9b
|
||||
```
|
||||
|
||||
This time the agent invokes `skill_math-solver`, which executes a deterministic pipeline (calling the `calculator` tool internally) and returns the computed result directly.
|
||||
|
||||
## Step 4: Create Your Own Skill
|
||||
|
||||
Create a new skill directory:
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.openjarvis/skills/my-reviewer
|
||||
```
|
||||
|
||||
Write a SKILL.md:
|
||||
|
||||
```bash
|
||||
cat > ~/.openjarvis/skills/my-reviewer/SKILL.md << 'EOF'
|
||||
---
|
||||
name: my-reviewer
|
||||
description: Review code changes with a security-first approach
|
||||
license: MIT
|
||||
metadata:
|
||||
openjarvis:
|
||||
version: "0.1.0"
|
||||
author: me
|
||||
tags: [coding, review, security]
|
||||
---
|
||||
|
||||
When asked to review code, follow this approach:
|
||||
|
||||
1. **Security scan first** — check for injection vulnerabilities, hardcoded secrets, unsafe deserialization
|
||||
2. **Correctness** — verify logic, edge cases, error handling
|
||||
3. **Style** — naming, structure, consistency with surrounding code
|
||||
4. **Summary** — one paragraph with the verdict: approve, request changes, or block
|
||||
|
||||
Always start with security. If you find a security issue, flag it as BLOCKING regardless of other concerns.
|
||||
EOF
|
||||
```
|
||||
|
||||
Verify it's discovered:
|
||||
|
||||
```bash
|
||||
jarvis skill list
|
||||
```
|
||||
|
||||
You should see `my-reviewer` in the table. Try it:
|
||||
|
||||
```bash
|
||||
jarvis ask "Use the my-reviewer skill to review this function: def login(user, pwd): return db.query(f'SELECT * FROM users WHERE name={user} AND pass={pwd}')" \
|
||||
--engine ollama --model qwen3.5:9b
|
||||
```
|
||||
|
||||
The agent should follow the security-first approach and flag the SQL injection vulnerability.
|
||||
|
||||
## Step 5: Generate Traces
|
||||
|
||||
For the learning loop to work, we need traces. Run several queries that use skills:
|
||||
|
||||
```bash
|
||||
# Generate a few traces
|
||||
jarvis ask "Use math-solver to compute 100 / 7"
|
||||
jarvis ask "Use code-explainer to explain: lambda x: x**2"
|
||||
jarvis ask "Use my-reviewer to review: def add(a,b): return a+b"
|
||||
jarvis ask "Use math-solver to compute 2**10"
|
||||
jarvis ask "Use code-explainer to explain: [x for x in range(10) if x % 2 == 0]"
|
||||
```
|
||||
|
||||
Each query produces a trace in `~/.openjarvis/traces.db` with skill metadata tags (`skill`, `skill_source`, `skill_kind`).
|
||||
|
||||
## Step 6: Discover Patterns from Traces
|
||||
|
||||
Mine the trace store for recurring tool sequences:
|
||||
|
||||
```bash
|
||||
# Preview without writing
|
||||
jarvis skill discover --dry-run --min-frequency 2
|
||||
|
||||
# Write discovered patterns as skill manifests
|
||||
jarvis skill discover --min-frequency 2
|
||||
```
|
||||
|
||||
Discovered skills land in `~/.openjarvis/skills/discovered/` and automatically appear in `jarvis skill list` on the next session.
|
||||
|
||||
## Step 7: Optimize Skills with DSPy
|
||||
|
||||
Once you have enough traces (at least 3-5 per skill), run the optimizer:
|
||||
|
||||
```bash
|
||||
# Preview what would be optimized
|
||||
jarvis optimize skills --dry-run
|
||||
|
||||
# Run DSPy optimization
|
||||
jarvis optimize skills --policy dspy --min-traces 3
|
||||
```
|
||||
|
||||
This produces overlay files at `~/.openjarvis/learning/skills/<skill-name>/optimized.toml` with improved descriptions and few-shot examples extracted from your best traces.
|
||||
|
||||
Inspect what was produced:
|
||||
|
||||
```bash
|
||||
jarvis skill show-overlay math-solver
|
||||
jarvis skill show-overlay code-explainer
|
||||
```
|
||||
|
||||
The next time you run a query, the agent sees the optimized descriptions and few-shot examples in its system prompt.
|
||||
|
||||
## Step 8: Benchmark the Impact
|
||||
|
||||
Run a quick benchmark to see if skills + optimization actually help:
|
||||
|
||||
```bash
|
||||
# Smoke test: 4 conditions × 1 seed × 5 tasks
|
||||
jarvis bench skills --max-samples 5 --seeds 42
|
||||
```
|
||||
|
||||
This runs the PinchBench benchmark in four conditions (no skills, skills on, DSPy-optimized, GEPA-optimized) and produces a markdown report at `docs/superpowers/results/`.
|
||||
|
||||
## Step 9: Configure Auto-Import and Auto-Optimization
|
||||
|
||||
For a hands-off experience, add this to `~/.openjarvis/config.toml`:
|
||||
|
||||
```toml
|
||||
[skills]
|
||||
enabled = true
|
||||
auto_sync = true
|
||||
|
||||
[[skills.sources]]
|
||||
source = "hermes"
|
||||
filter = { category = ["research", "coding"] }
|
||||
auto_update = true
|
||||
|
||||
[learning.skills]
|
||||
auto_optimize = true
|
||||
optimizer = "dspy"
|
||||
min_traces_per_skill = 20
|
||||
```
|
||||
|
||||
Now skills are automatically synced from Hermes on session start, and the optimizer runs after each learning cycle when enough traces accumulate.
|
||||
|
||||
## What You Learned
|
||||
|
||||
| Concept | What you did |
|
||||
|---------|-------------|
|
||||
| **Installing skills** | `jarvis skill install hermes:arxiv` — imported from public sources |
|
||||
| **Using skills** | `jarvis ask "Use the code-explainer skill..."` — agent invokes skills as tools |
|
||||
| **Creating skills** | Wrote a `SKILL.md` with YAML frontmatter and markdown instructions |
|
||||
| **Generating traces** | Ran skill-using queries to populate the trace store |
|
||||
| **Discovering patterns** | `jarvis skill discover` — mined traces for recurring tool sequences |
|
||||
| **Optimizing skills** | `jarvis optimize skills --policy dspy` — improved descriptions + few-shot examples |
|
||||
| **Benchmarking** | `jarvis bench skills` — measured the impact across 4 conditions |
|
||||
| **Auto configuration** | Added `[skills]` and `[learning.skills]` config sections |
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Browse the [full skills user guide](../user-guide/skills.md) for all CLI commands and configuration options
|
||||
- Read the [skills architecture](../architecture/skills.md) for the technical deep-dive
|
||||
- Explore the [Hermes Agent skill library](https://github.com/NousResearch/hermes-agent/tree/main/skills) for more skills to install
|
||||
- Try [OpenClaw skills](https://github.com/openclaw/skills) for community-contributed skills
|
||||
@@ -0,0 +1,405 @@
|
||||
---
|
||||
title: Skills
|
||||
description: Reusable compositions of tools and agent instructions — discover, import, optimize, and share
|
||||
search.boost: 2.0
|
||||
---
|
||||
|
||||
# Skills
|
||||
|
||||
Skills teach agents **how to better use tools and improve their reasoning**. They are reusable compositions of tools, sub-skills, and agent instructions that can be shared via public registries.
|
||||
|
||||
Every skill is a tool. Skills appear in a lightweight catalog in the agent's system prompt, and when the agent invokes one, its content (pipeline results, markdown instructions, or both) gets injected into the conversation context.
|
||||
|
||||
## Overview
|
||||
|
||||
| Concept | Description |
|
||||
|---------|-------------|
|
||||
| **Skill** | A directory containing `skill.toml` (structured pipeline), `SKILL.md` (markdown instructions), or both |
|
||||
| **SkillManager** | Central coordinator for discovery, resolution, catalog generation, and tool wrapping |
|
||||
| **SkillTool** | Adapter that wraps any skill as a `BaseTool` so agents can invoke it |
|
||||
| **Overlay** | Sidecar file at `~/.openjarvis/learning/skills/` storing optimized descriptions and few-shot examples |
|
||||
| **Source** | A resolver for importing skills from Hermes Agent, OpenClaw, or any GitHub repo |
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# List installed skills
|
||||
jarvis skill list
|
||||
|
||||
# Install a skill from Hermes Agent
|
||||
jarvis skill install hermes:apple-notes
|
||||
|
||||
# Bulk install a category
|
||||
jarvis skill sync hermes --category research
|
||||
|
||||
# Run a skill directly
|
||||
jarvis skill run math-solver -a expression="41 + 82"
|
||||
|
||||
# See skill details
|
||||
jarvis skill info research-and-summarize
|
||||
```
|
||||
|
||||
## Skill Definition Format
|
||||
|
||||
A skill is a directory containing a `skill.toml`, a `SKILL.md`, or both.
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
research-and-summarize/
|
||||
├── SKILL.md # Markdown instructions (loaded on invocation)
|
||||
├── skill.toml # Structured pipeline steps
|
||||
├── templates/ # Optional Jinja2 templates
|
||||
├── scripts/ # Optional executable helpers
|
||||
├── references/ # Optional detailed docs
|
||||
├── assets/ # Optional static resources
|
||||
└── examples/ # Optional usage examples
|
||||
```
|
||||
|
||||
### skill.toml (Structured Pipeline)
|
||||
|
||||
Pipeline skills define a sequence of tool calls that execute deterministically:
|
||||
|
||||
```toml
|
||||
[skill]
|
||||
name = "research-and-summarize"
|
||||
version = "0.1.0"
|
||||
description = "Search the web and produce a structured summary"
|
||||
author = "openjarvis"
|
||||
tags = ["research", "summarization"]
|
||||
required_capabilities = ["network:fetch"]
|
||||
depends = ["summarize"]
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "web_search"
|
||||
arguments_template = '{"query": "{query}"}'
|
||||
output_key = "search_results"
|
||||
|
||||
[[skill.steps]]
|
||||
skill_name = "summarize"
|
||||
arguments_template = '{"text": "{search_results}"}'
|
||||
output_key = "summary"
|
||||
```
|
||||
|
||||
Steps can call tools (`tool_name`) or other skills (`skill_name`). Template placeholders like `{query}` become the skill's input parameters. Output keys chain between steps.
|
||||
|
||||
### SKILL.md (Instructional Content)
|
||||
|
||||
Instructional skills provide markdown guidance that agents follow using their other tools:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: code-explainer
|
||||
description: Explain code in plain language with examples
|
||||
license: MIT
|
||||
metadata:
|
||||
openjarvis:
|
||||
version: "0.1.0"
|
||||
author: openjarvis
|
||||
tags: [coding, explanation]
|
||||
---
|
||||
|
||||
When asked to explain code, follow this approach:
|
||||
|
||||
1. Identify the programming language
|
||||
2. Break the code into logical sections
|
||||
3. Explain each section in plain language
|
||||
4. Highlight any patterns, idioms, or potential issues
|
||||
5. Provide a one-sentence summary at the end
|
||||
```
|
||||
|
||||
The YAML frontmatter follows the [agentskills.io](https://agentskills.io/specification) open standard. Required fields: `name`, `description`. Optional: `license`, `compatibility`, `metadata`, `allowed-tools`.
|
||||
|
||||
### What Happens on Invocation
|
||||
|
||||
| Skill has | On invocation |
|
||||
|-----------|---------------|
|
||||
| `skill.toml` steps only | Execute the pipeline, return results |
|
||||
| `SKILL.md` only | Return the markdown instructions — agent follows them in subsequent turns |
|
||||
| Both | Execute pipeline steps AND return the markdown guidance alongside results |
|
||||
|
||||
## Installing Skills
|
||||
|
||||
### From Hermes Agent
|
||||
|
||||
```bash
|
||||
# Single skill
|
||||
jarvis skill install hermes:apple-notes
|
||||
|
||||
# Bulk install by category
|
||||
jarvis skill sync hermes --category research
|
||||
jarvis skill sync hermes --category coding
|
||||
jarvis skill sync hermes # everything (~150 skills)
|
||||
```
|
||||
|
||||
### From OpenClaw
|
||||
|
||||
```bash
|
||||
# Single skill (owner/slug format)
|
||||
jarvis skill install openclaw:0xv4l3nt1n3/etherscan
|
||||
|
||||
# Bulk install with search filter
|
||||
jarvis skill sync openclaw --search "web3|crypto"
|
||||
```
|
||||
|
||||
### From Any GitHub Repo
|
||||
|
||||
```bash
|
||||
jarvis skill install github:user/repo/path/to/skill --url https://github.com/user/repo
|
||||
```
|
||||
|
||||
### Config-Driven Auto Import
|
||||
|
||||
Add sources to `~/.openjarvis/config.toml` for automatic syncing:
|
||||
|
||||
```toml
|
||||
[skills]
|
||||
enabled = true
|
||||
auto_sync = true
|
||||
|
||||
[[skills.sources]]
|
||||
source = "hermes"
|
||||
filter = { category = ["research", "coding", "productivity"] }
|
||||
auto_update = true
|
||||
|
||||
[[skills.sources]]
|
||||
source = "openclaw"
|
||||
filter = { search = "web3|crypto" }
|
||||
```
|
||||
|
||||
When `auto_sync = true`, the SkillManager checks source freshness on each session start and pulls updates in the background.
|
||||
|
||||
### Managing Sources
|
||||
|
||||
```bash
|
||||
# List configured sources
|
||||
jarvis skill sources
|
||||
|
||||
# Update all configured sources
|
||||
jarvis skill update
|
||||
```
|
||||
|
||||
## How Agents Use Skills
|
||||
|
||||
### Skill Catalog in the System Prompt
|
||||
|
||||
All available skills appear as a lightweight XML catalog in the agent's system prompt:
|
||||
|
||||
```xml
|
||||
<available_skills>
|
||||
<skill name="research-and-summarize" description="Search the web and produce a structured summary" />
|
||||
<skill name="code-explainer" description="Explain code in plain language with examples" />
|
||||
<skill name="math-solver" description="Solve a math problem step by step using the calculator" />
|
||||
</available_skills>
|
||||
```
|
||||
|
||||
The agent reads this catalog and decides when to invoke a skill based on the user's request.
|
||||
|
||||
### Invocation Control
|
||||
|
||||
Per-skill flags control visibility:
|
||||
|
||||
```toml
|
||||
[skill]
|
||||
user_invocable = true # expose as CLI command (default: true)
|
||||
disable_model_invocation = false # hide from agent catalog (default: false)
|
||||
```
|
||||
|
||||
| `user_invocable` | `disable_model_invocation` | CLI command? | Agent discovers? |
|
||||
|---|---|---|---|
|
||||
| true (default) | false (default) | Yes | Yes |
|
||||
| true | true | Yes | No |
|
||||
| false | false | No | Yes |
|
||||
| false | true | No | No (dormant) |
|
||||
|
||||
### Pipeline vs. Instructional Skills
|
||||
|
||||
Agents handle both skill types correctly:
|
||||
|
||||
- **Pipeline skills** (with `skill.toml` steps) execute deterministically and return computed results. The agent uses the result directly in its answer.
|
||||
- **Instructional skills** (with `SKILL.md` only) return markdown text describing HOW to accomplish a task. The agent reads the instructions and follows them using its other tools (web_search, shell_exec, calculator, etc.).
|
||||
|
||||
## Skill Discovery from Traces
|
||||
|
||||
OpenJarvis can automatically mine your trace history for recurring tool sequences and surface them as candidate skills:
|
||||
|
||||
```bash
|
||||
# Preview discovered patterns without writing
|
||||
jarvis skill discover --dry-run --min-frequency 3
|
||||
|
||||
# Write discovered skills to ~/.openjarvis/skills/discovered/
|
||||
jarvis skill discover
|
||||
```
|
||||
|
||||
Discovered skills land in `~/.openjarvis/skills/discovered/` and automatically appear in `jarvis skill list` on the next session.
|
||||
|
||||
## Skill Optimization
|
||||
|
||||
### Optimizing with DSPy or GEPA
|
||||
|
||||
The skills learning loop uses your trace history to optimize skill descriptions and extract few-shot examples:
|
||||
|
||||
```bash
|
||||
# Preview what would be optimized
|
||||
jarvis optimize skills --dry-run
|
||||
|
||||
# Run DSPy optimization
|
||||
jarvis optimize skills --policy dspy --min-traces 3
|
||||
|
||||
# Run GEPA evolutionary optimization
|
||||
jarvis optimize skills --policy gepa --min-traces 3
|
||||
|
||||
# Inspect what optimization produced
|
||||
jarvis skill show-overlay research-and-summarize
|
||||
```
|
||||
|
||||
Optimization results are stored as sidecar overlays at `~/.openjarvis/learning/skills/<skill-name>/optimized.toml`. They override the skill's description and add few-shot examples to the agent's system prompt. The original skill files are never modified.
|
||||
|
||||
### Auto-Optimization
|
||||
|
||||
Enable automatic optimization in config:
|
||||
|
||||
```toml
|
||||
[learning.skills]
|
||||
auto_optimize = false # set to true to enable
|
||||
optimizer = "dspy" # "dspy" or "gepa"
|
||||
min_traces_per_skill = 20
|
||||
```
|
||||
|
||||
When enabled, the `LearningOrchestrator` runs skill optimization after each learning cycle.
|
||||
|
||||
## Benchmarking Skills
|
||||
|
||||
Measure whether skills improve agent performance:
|
||||
|
||||
```bash
|
||||
# Full sweep: 4 conditions × 3 seeds
|
||||
jarvis bench skills
|
||||
|
||||
# Smoke test: 4 conditions × 1 seed × 5 tasks
|
||||
jarvis bench skills --max-samples 5 --seeds 42
|
||||
|
||||
# Single condition
|
||||
jarvis bench skills --condition skills_optimized_dspy
|
||||
```
|
||||
|
||||
The four benchmark conditions are:
|
||||
|
||||
| Condition | What it tests |
|
||||
|---|---|
|
||||
| `no_skills` | Skills disabled (control) |
|
||||
| `skills_on` | Skills enabled, no optimization |
|
||||
| `skills_optimized_dspy` | DSPy-optimized overlays |
|
||||
| `skills_optimized_gepa` | GEPA-optimized overlays |
|
||||
|
||||
Results are written to `docs/superpowers/results/pinchbench-skills-eval-{date}.md` with a summary table, per-task breakdown, deltas, and skill invocation counts.
|
||||
|
||||
## Security & Trust
|
||||
|
||||
### Trust Tiers
|
||||
|
||||
| Tier | Source | Verification | Runtime |
|
||||
|------|--------|-------------|---------|
|
||||
| **Bundled** | Ships with OpenJarvis | Implicit trust | Full access within declared capabilities |
|
||||
| **Indexed** | In official skill index, signed | SHA256 + Ed25519 | Capability-gated |
|
||||
| **Unreviewed** | Arbitrary GitHub URL | SHA256 only | Capability-gated + sandbox warning |
|
||||
| **Workspace** | Local `./skills/` directory | None (user code) | Trusted |
|
||||
|
||||
### Capability Enforcement
|
||||
|
||||
Skills declare required capabilities. At runtime, the SkillExecutor checks that each tool call falls within the skill's declared capabilities:
|
||||
|
||||
- `network:fetch` — outbound HTTP requests
|
||||
- `filesystem:read` / `filesystem:write` — file access
|
||||
- `shell:execute` — run shell commands (dangerous)
|
||||
- `memory:read` / `memory:write` — memory backend access
|
||||
- `engine:inference` — LLM calls
|
||||
|
||||
Skills declaring dangerous capabilities (`shell:execute`, `network:listen`, `filesystem:write`) trigger install-time warnings and sandbox recommendations.
|
||||
|
||||
### Scripts
|
||||
|
||||
Imported skills may include `scripts/` directories with executable code. These are **skipped by default** for security. Use `--with-scripts` to opt in:
|
||||
|
||||
```bash
|
||||
jarvis skill install hermes:arxiv --with-scripts
|
||||
```
|
||||
|
||||
## Skill Composition
|
||||
|
||||
Skills can invoke other skills as sub-steps:
|
||||
|
||||
```toml
|
||||
[[skill.steps]]
|
||||
skill_name = "summarize"
|
||||
arguments_template = '{"text": "{search_results}"}'
|
||||
output_key = "summary"
|
||||
```
|
||||
|
||||
The SkillManager builds a dependency graph at discovery time and validates:
|
||||
|
||||
1. **No cycles** — `A → B → C → A` is rejected with a clear error
|
||||
2. **Max depth** — default 5 levels (configurable)
|
||||
3. **Capability unions** — parent must declare all capabilities its children need
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
### `[skills]` Section
|
||||
|
||||
```toml
|
||||
[skills]
|
||||
enabled = true # enable/disable the skill system
|
||||
skills_dir = "~/.openjarvis/skills/" # where skills are installed
|
||||
active = "*" # which skills to activate ("*" = all)
|
||||
auto_discover = true # scan skills_dir on startup
|
||||
auto_sync = false # pull from configured sources on startup
|
||||
max_depth = 5 # max sub-skill nesting depth
|
||||
sandbox_dangerous = true # warn about dangerous capabilities
|
||||
```
|
||||
|
||||
### `[[skills.sources]]` Section
|
||||
|
||||
```toml
|
||||
[[skills.sources]]
|
||||
source = "hermes" # "hermes", "openclaw", or "github"
|
||||
url = "" # required when source = "github"
|
||||
filter = { category = ["research", "coding"] }
|
||||
auto_update = true # pull latest on sync
|
||||
```
|
||||
|
||||
### `[learning.skills]` Section
|
||||
|
||||
```toml
|
||||
[learning.skills]
|
||||
auto_optimize = false # opt-in automatic optimization
|
||||
optimizer = "dspy" # "dspy" or "gepa"
|
||||
min_traces_per_skill = 20 # minimum traces before optimizing
|
||||
optimization_interval_seconds = 86400 # at most once per day
|
||||
overlay_dir = "~/.openjarvis/learning/skills/"
|
||||
```
|
||||
|
||||
## Name Precedence
|
||||
|
||||
When the same skill name exists in multiple locations, closest scope wins:
|
||||
|
||||
1. **Workspace** `./skills/` (highest priority)
|
||||
2. **User** `~/.openjarvis/skills/`
|
||||
3. **Bundled** (shipped with OpenJarvis)
|
||||
|
||||
## CLI Reference
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `jarvis skill list` | List installed skills |
|
||||
| `jarvis skill info <name>` | Show detailed skill information |
|
||||
| `jarvis skill run <name> [-a key=value]` | Execute a skill directly |
|
||||
| `jarvis skill install <source>:<name>` | Install from Hermes, OpenClaw, or GitHub |
|
||||
| `jarvis skill sync [<source>] [--category C]` | Bulk install + update from sources |
|
||||
| `jarvis skill sources` | List configured skill sources |
|
||||
| `jarvis skill update` | Pull latest from configured sources |
|
||||
| `jarvis skill remove <name>` | Remove an installed skill |
|
||||
| `jarvis skill search <query>` | Search the skill index |
|
||||
| `jarvis skill discover [--dry-run]` | Mine traces for recurring tool patterns |
|
||||
| `jarvis skill show-overlay <name>` | Inspect optimization output for a skill |
|
||||
| `jarvis optimize skills [--policy dspy\|gepa]` | Optimize skill descriptions + few-shot examples |
|
||||
| `jarvis bench skills [--condition C]` | Run the PinchBench skills benchmark |
|
||||
@@ -306,6 +306,7 @@ class ToolUsingAgent(BaseAgent):
|
||||
agent_id: Optional[str] = None,
|
||||
interactive: bool = False,
|
||||
confirm_callback: Optional[Any] = None,
|
||||
skill_few_shot_examples: Optional[List[str]] = None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine,
|
||||
@@ -317,6 +318,9 @@ class ToolUsingAgent(BaseAgent):
|
||||
from openjarvis.tools._stubs import ToolExecutor
|
||||
|
||||
self._tools = tools or []
|
||||
# Plan 2B I3: store optimized few-shot examples for agents to inject
|
||||
# into their own system prompt templates as appropriate.
|
||||
self._skill_few_shot_examples = list(skill_few_shot_examples or [])
|
||||
_aid = agent_id or getattr(self, "agent_id", "")
|
||||
self._executor = ToolExecutor(
|
||||
self._tools,
|
||||
|
||||
@@ -28,7 +28,25 @@ Action Input: <json arguments>
|
||||
Thought: <your reasoning>
|
||||
Final Answer: <your answer>
|
||||
|
||||
{tool_descriptions}"""
|
||||
# Using Skills
|
||||
|
||||
Tools whose names begin with `skill_` are SKILLS. When you call a skill tool,
|
||||
the response can take one of two forms:
|
||||
|
||||
- **Computed result**: The skill ran a deterministic pipeline and returned a
|
||||
value (number, string, JSON, etc.). Use the value directly in your answer.
|
||||
|
||||
- **Procedural instructions**: The skill returned markdown text describing
|
||||
HOW to accomplish a task. Recognize this when the response starts with
|
||||
`#` headings, contains bullet lists, or uses phrases like "When asked
|
||||
to...", "First...", "Steps:". When you receive instructions:
|
||||
1. READ the instructions carefully — they are your playbook
|
||||
2. FOLLOW the steps using your OTHER tools (e.g. calculator, web_search,
|
||||
shell_exec, file_read) — not the same skill
|
||||
3. DO NOT call the same skill again — you already have its instructions
|
||||
4. Synthesize a Final Answer from what you learned
|
||||
|
||||
{skill_examples}{tool_descriptions}"""
|
||||
|
||||
|
||||
@AgentRegistry.register("native_react")
|
||||
@@ -52,6 +70,7 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
max_tokens: Optional[int] = None,
|
||||
interactive: bool = False,
|
||||
confirm_callback=None,
|
||||
skill_few_shot_examples: Optional[List[str]] = None,
|
||||
) -> None:
|
||||
super().__init__(
|
||||
engine,
|
||||
@@ -63,6 +82,7 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
max_tokens=max_tokens,
|
||||
interactive=interactive,
|
||||
confirm_callback=confirm_callback,
|
||||
skill_few_shot_examples=skill_few_shot_examples,
|
||||
)
|
||||
|
||||
def _parse_response(self, text: str) -> dict:
|
||||
@@ -111,7 +131,22 @@ class NativeReActAgent(ToolUsingAgent):
|
||||
|
||||
# Build system prompt with rich tool descriptions
|
||||
tool_desc = build_tool_descriptions(self._tools)
|
||||
system_prompt = REACT_SYSTEM_PROMPT.format(tool_descriptions=tool_desc)
|
||||
# Plan 2B I3: render the optimized few-shot examples as a section
|
||||
# that the model sees BEFORE the tool descriptions. When no
|
||||
# examples are present, this is an empty string and the rendered
|
||||
# prompt is unchanged.
|
||||
if self._skill_few_shot_examples:
|
||||
skill_examples_block = (
|
||||
"## Skill Examples\n\n"
|
||||
+ "\n\n".join(self._skill_few_shot_examples)
|
||||
+ "\n\n"
|
||||
)
|
||||
else:
|
||||
skill_examples_block = ""
|
||||
system_prompt = REACT_SYSTEM_PROMPT.format(
|
||||
tool_descriptions=tool_desc,
|
||||
skill_examples=skill_examples_block,
|
||||
)
|
||||
|
||||
messages = self._build_messages(input, context, system_prompt=system_prompt)
|
||||
|
||||
|
||||
@@ -358,3 +358,130 @@ def run(
|
||||
energy_monitor.close()
|
||||
except Exception as exc:
|
||||
logger.debug("Energy monitor cleanup failed: %s", exc)
|
||||
|
||||
|
||||
@bench.command("skills")
|
||||
@click.option(
|
||||
"--condition",
|
||||
"-c",
|
||||
type=click.Choice(
|
||||
[
|
||||
"all",
|
||||
"no_skills",
|
||||
"skills_on",
|
||||
"skills_optimized_dspy",
|
||||
"skills_optimized_gepa",
|
||||
]
|
||||
),
|
||||
default="all",
|
||||
show_default=True,
|
||||
help="Which condition(s) to run.",
|
||||
)
|
||||
@click.option(
|
||||
"--seeds",
|
||||
"-s",
|
||||
multiple=True,
|
||||
type=int,
|
||||
default=(42, 43, 44),
|
||||
show_default=True,
|
||||
help="Random seeds to run per condition.",
|
||||
)
|
||||
@click.option(
|
||||
"--max-samples",
|
||||
"-n",
|
||||
default=None,
|
||||
type=int,
|
||||
help="Limit number of PinchBench tasks (default: full benchmark).",
|
||||
)
|
||||
@click.option(
|
||||
"--model",
|
||||
"-m",
|
||||
default="qwen3.5:9b",
|
||||
show_default=True,
|
||||
help="Model identifier.",
|
||||
)
|
||||
@click.option(
|
||||
"--engine",
|
||||
"-e",
|
||||
default="ollama",
|
||||
show_default=True,
|
||||
help="Engine backend.",
|
||||
)
|
||||
@click.option(
|
||||
"--output-dir",
|
||||
"-o",
|
||||
default="docs/superpowers/results/",
|
||||
show_default=True,
|
||||
help="Where the markdown report lands.",
|
||||
)
|
||||
def skills(
|
||||
condition: str,
|
||||
seeds: tuple,
|
||||
max_samples: int,
|
||||
model: str,
|
||||
engine: str,
|
||||
output_dir: str,
|
||||
) -> None:
|
||||
"""Run the PinchBench skills benchmark across one or all conditions."""
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
SkillBenchmarkConfig,
|
||||
SkillBenchmarkRunner,
|
||||
)
|
||||
|
||||
console = Console()
|
||||
|
||||
cfg = SkillBenchmarkConfig(
|
||||
model=model,
|
||||
engine=engine,
|
||||
seeds=list(seeds),
|
||||
max_samples=max_samples,
|
||||
output_dir=Path(output_dir),
|
||||
)
|
||||
runner = SkillBenchmarkRunner(cfg)
|
||||
|
||||
if condition == "all":
|
||||
comparison = runner.run_all_conditions()
|
||||
|
||||
# Print summary table
|
||||
table = Table(title="PinchBench Skills Evaluation")
|
||||
table.add_column("Condition", style="cyan")
|
||||
table.add_column("Mean pass rate")
|
||||
table.add_column("Stddev")
|
||||
table.add_column("Tokens")
|
||||
table.add_column("Runtime (s)")
|
||||
for cond_name, result in comparison.results.items():
|
||||
table.add_row(
|
||||
cond_name,
|
||||
f"{result.mean_pass_rate:.3f}",
|
||||
f"{result.stddev_pass_rate:.3f}",
|
||||
str(result.total_tokens),
|
||||
f"{result.total_runtime_seconds:.1f}",
|
||||
)
|
||||
console.print(table)
|
||||
|
||||
if comparison.deltas:
|
||||
console.print("\n[bold]Deltas:[/bold]")
|
||||
for name, value in comparison.deltas.items():
|
||||
sign = "+" if value >= 0 else ""
|
||||
console.print(f" {name}: {sign}{value:.3f}")
|
||||
|
||||
report_path = runner.write_report(comparison)
|
||||
console.print(f"\n[green]Report written:[/green] {report_path}")
|
||||
else:
|
||||
result = runner.run_condition(condition)
|
||||
table = Table(title=f"PinchBench Skills — {condition}")
|
||||
table.add_column("Field", style="cyan")
|
||||
table.add_column("Value")
|
||||
table.add_row("Condition", result.condition)
|
||||
table.add_row("Mean pass rate", f"{result.mean_pass_rate:.3f}")
|
||||
table.add_row("Stddev", f"{result.stddev_pass_rate:.3f}")
|
||||
table.add_row(
|
||||
"Per-seed",
|
||||
", ".join(f"{s}={r:.3f}" for s, r in result.per_seed_pass_rate.items()),
|
||||
)
|
||||
table.add_row("Total tokens", str(result.total_tokens))
|
||||
table.add_row("Runtime (s)", f"{result.total_runtime_seconds:.1f}")
|
||||
console.print(table)
|
||||
|
||||
@@ -11,6 +11,18 @@ from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
|
||||
def _get_trace_store():
|
||||
"""Return a TraceStore from user config, or None on failure."""
|
||||
try:
|
||||
from openjarvis.core.config import load_config
|
||||
from openjarvis.traces.store import TraceStore
|
||||
|
||||
cfg = load_config()
|
||||
return TraceStore(cfg.traces.db_path)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
@click.group("optimize")
|
||||
def optimize_group() -> None:
|
||||
"""LLM-driven configuration optimization."""
|
||||
@@ -417,4 +429,85 @@ def optimize_personal(action: str, workflow: str, trials: int) -> None:
|
||||
)
|
||||
|
||||
|
||||
@optimize_group.command("skills")
|
||||
@click.option(
|
||||
"--policy",
|
||||
"-p",
|
||||
type=click.Choice(["dspy", "gepa"]),
|
||||
default="dspy",
|
||||
show_default=True,
|
||||
help="Optimization policy to use.",
|
||||
)
|
||||
@click.option(
|
||||
"--min-traces",
|
||||
"-n",
|
||||
default=20,
|
||||
show_default=True,
|
||||
type=int,
|
||||
help="Minimum traces required per skill.",
|
||||
)
|
||||
@click.option(
|
||||
"--dry-run",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help="Show planned work without invoking the optimizer LM.",
|
||||
)
|
||||
def skills(policy: str, min_traces: int, dry_run: bool) -> None:
|
||||
"""Optimize per-skill descriptions and few-shot examples from traces."""
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.learning.agents.skill_optimizer import SkillOptimizer
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
console = Console()
|
||||
store = _get_trace_store()
|
||||
if store is None:
|
||||
console.print("[red]No trace store found. Enable tracing first.[/red]")
|
||||
raise SystemExit(1)
|
||||
|
||||
if dry_run:
|
||||
# Just bucket and report counts
|
||||
traces = store.list_traces(limit=10000)
|
||||
opt = SkillOptimizer(min_traces_per_skill=min_traces, optimizer=policy)
|
||||
buckets = opt._bucket_traces_by_skill(traces)
|
||||
if not buckets:
|
||||
console.print("[dim]No skill-tagged traces found.[/dim]")
|
||||
return
|
||||
table = Table(title="Optimization plan (dry run)")
|
||||
table.add_column("Skill", style="cyan")
|
||||
table.add_column("Trace count")
|
||||
table.add_column("Action")
|
||||
for name, bucket in buckets.items():
|
||||
action = (
|
||||
"would optimize"
|
||||
if len(bucket) >= min_traces
|
||||
else f"skip (< {min_traces} traces)"
|
||||
)
|
||||
table.add_row(name, str(len(bucket)), action)
|
||||
console.print(table)
|
||||
return
|
||||
|
||||
# Real run
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover()
|
||||
optimizer = SkillOptimizer(min_traces_per_skill=min_traces, optimizer=policy)
|
||||
results = optimizer.optimize(store, mgr)
|
||||
|
||||
if not results:
|
||||
console.print("[dim]No skill-tagged traces found.[/dim]")
|
||||
return
|
||||
|
||||
table = Table(title=f"Skill optimization ({policy})")
|
||||
table.add_column("Skill", style="cyan")
|
||||
table.add_column("Status")
|
||||
table.add_column("Traces")
|
||||
table.add_column("Overlay path")
|
||||
for name, res in results.items():
|
||||
path_str = str(res.overlay_path) if res.overlay_path else "—"
|
||||
table.add_row(name, res.status, str(res.trace_count), path_str)
|
||||
console.print(table)
|
||||
|
||||
|
||||
__all__ = ["optimize_group"]
|
||||
|
||||
+498
-48
@@ -1,75 +1,525 @@
|
||||
"""``jarvis skill`` — skill management commands."""
|
||||
"""CLI commands for skill management."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
import click
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
from openjarvis.core.config import load_config
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
|
||||
def _get_trace_store():
|
||||
"""Return a TraceStore instance from the user config (or None)."""
|
||||
try:
|
||||
from openjarvis.core.config import load_config
|
||||
from openjarvis.traces.store import TraceStore
|
||||
|
||||
cfg = load_config()
|
||||
return TraceStore(cfg.traces.db_path)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _get_discovered_dir() -> Path:
|
||||
"""Return the directory where discovered skill manifests are written."""
|
||||
return Path("~/.openjarvis/skills/discovered/").expanduser()
|
||||
|
||||
|
||||
def _get_overlay_dir() -> Path:
|
||||
"""Return the directory where optimization overlays are stored."""
|
||||
return Path("~/.openjarvis/learning/skills/").expanduser()
|
||||
|
||||
|
||||
def _get_skill_paths() -> List[Path]:
|
||||
paths: List[Path] = []
|
||||
workspace = Path("./skills")
|
||||
if workspace.exists():
|
||||
paths.append(workspace)
|
||||
user_dir = Path("~/.openjarvis/skills/").expanduser()
|
||||
paths.append(user_dir)
|
||||
return paths
|
||||
|
||||
|
||||
def _get_manager() -> SkillManager:
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=_get_skill_paths())
|
||||
return mgr
|
||||
|
||||
|
||||
@click.group()
|
||||
def skill() -> None:
|
||||
"""Manage skills — list, install, remove."""
|
||||
def skill():
|
||||
"""Manage reusable skills."""
|
||||
|
||||
|
||||
@skill.command("list")
|
||||
def list_skills() -> None:
|
||||
def list_skills():
|
||||
"""List installed skills."""
|
||||
console = Console(stderr=True)
|
||||
try:
|
||||
from openjarvis.core.registry import SkillRegistry
|
||||
|
||||
keys = sorted(SkillRegistry.keys())
|
||||
if not keys:
|
||||
console.print("[dim]No skills installed.[/dim]")
|
||||
return
|
||||
table = Table(title="Installed Skills")
|
||||
table.add_column("Name", style="cyan")
|
||||
table.add_column("Description", style="green")
|
||||
for key in keys:
|
||||
skill_cls = SkillRegistry.get(key)
|
||||
desc = ""
|
||||
if hasattr(skill_cls, "manifest"):
|
||||
m = skill_cls.manifest if not callable(skill_cls.manifest) else None
|
||||
if m and hasattr(m, "description"):
|
||||
desc = m.description[:60]
|
||||
table.add_row(key, desc)
|
||||
console.print(table)
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Error: {exc}[/red]")
|
||||
console = Console()
|
||||
mgr = _get_manager()
|
||||
names = mgr.skill_names()
|
||||
if not names:
|
||||
console.print("[dim]No skills installed.[/dim]")
|
||||
return
|
||||
table = Table(title="Installed Skills")
|
||||
table.add_column("Name", style="cyan")
|
||||
table.add_column("Description", max_width=50)
|
||||
table.add_column("Version")
|
||||
table.add_column("Tags")
|
||||
for name in sorted(names):
|
||||
m = mgr.resolve(name)
|
||||
tags = ", ".join(m.tags) if m.tags else ""
|
||||
desc = m.description[:50] + "..." if len(m.description) > 50 else m.description
|
||||
table.add_row(name, desc, m.version, tags)
|
||||
console.print(table)
|
||||
|
||||
|
||||
@skill.command()
|
||||
@skill.command("info")
|
||||
@click.argument("skill_name")
|
||||
def install(skill_name: str) -> None:
|
||||
"""Install a skill from the bundled library."""
|
||||
console = Console(stderr=True)
|
||||
console.print(f"[yellow]Installing skill: {skill_name}[/yellow]")
|
||||
# Skills are discovered from TOML files — point user to the right place
|
||||
def info(skill_name: str):
|
||||
"""Show detailed information about a skill."""
|
||||
console = Console()
|
||||
mgr = _get_manager()
|
||||
try:
|
||||
m = mgr.resolve(skill_name)
|
||||
except KeyError:
|
||||
console.print(f"[red]Skill '{skill_name}' not found.[/red]")
|
||||
raise SystemExit(1)
|
||||
console.print(f"[bold]{m.name}[/bold] v{m.version}")
|
||||
if m.author:
|
||||
console.print(f"Author: {m.author}")
|
||||
if m.description:
|
||||
console.print(f"Description: {m.description}")
|
||||
if m.tags:
|
||||
console.print(f"Tags: {', '.join(m.tags)}")
|
||||
if m.required_capabilities:
|
||||
console.print(f"Capabilities: {', '.join(m.required_capabilities)}")
|
||||
if m.depends:
|
||||
console.print(f"Dependencies: {', '.join(m.depends)}")
|
||||
if m.steps:
|
||||
console.print(f"Steps: {len(m.steps)}")
|
||||
if m.markdown_content:
|
||||
console.print("Has instructions: yes")
|
||||
console.print(f"User invocable: {m.user_invocable}")
|
||||
console.print(
|
||||
f"[dim]Place skill TOML file in ~/.openjarvis/skills/{skill_name}.toml[/dim]"
|
||||
f"Model invocation: {'disabled' if m.disable_model_invocation else 'enabled'}"
|
||||
)
|
||||
|
||||
|
||||
@skill.command()
|
||||
@skill.command("run")
|
||||
@click.argument("skill_name")
|
||||
def remove(skill_name: str) -> None:
|
||||
"""Remove an installed skill."""
|
||||
console = Console(stderr=True)
|
||||
console.print(f"[yellow]Removing skill: {skill_name}[/yellow]")
|
||||
console.print("[dim]Skill removal not yet implemented.[/dim]")
|
||||
@click.option("--arg", "-a", multiple=True, help="Arguments as key=value pairs.")
|
||||
def run(skill_name: str, arg: tuple):
|
||||
"""Execute a skill directly."""
|
||||
console = Console()
|
||||
mgr = _get_manager()
|
||||
context = {}
|
||||
for a in arg:
|
||||
if "=" in a:
|
||||
k, v = a.split("=", 1)
|
||||
context[k.strip()] = v.strip()
|
||||
try:
|
||||
result = mgr.execute(skill_name, context)
|
||||
except KeyError:
|
||||
console.print(f"[red]Skill '{skill_name}' not found.[/red]")
|
||||
raise SystemExit(1)
|
||||
if result.success:
|
||||
console.print("[green]Success[/green]")
|
||||
if result.step_results:
|
||||
console.print(result.step_results[-1].content)
|
||||
else:
|
||||
console.print("[red]Failed[/red]")
|
||||
if result.step_results:
|
||||
console.print(result.step_results[-1].content)
|
||||
|
||||
|
||||
@skill.command()
|
||||
@click.argument("query", default="")
|
||||
def search(query: str) -> None:
|
||||
"""Search for available skills."""
|
||||
console = Console(stderr=True)
|
||||
if not query:
|
||||
console.print("[dim]Provide a search query.[/dim]")
|
||||
def _parse_source_query(query: str) -> tuple[str, str]:
|
||||
"""Parse a ``<source>:<name>`` query into (source, name).
|
||||
|
||||
Raises ``click.BadParameter`` if the format is wrong.
|
||||
"""
|
||||
if ":" not in query:
|
||||
raise click.BadParameter(
|
||||
f"Expected '<source>:<name>' format (e.g. 'hermes:apple-notes'), "
|
||||
f"got: {query!r}"
|
||||
)
|
||||
source, _, name = query.partition(":")
|
||||
if not source or not name:
|
||||
raise click.BadParameter(
|
||||
f"Both source and name are required: got source={source!r}, name={name!r}"
|
||||
)
|
||||
return source, name
|
||||
|
||||
|
||||
def _get_resolver(source: str, url: str = ""):
|
||||
"""Return a resolver instance for the given source name."""
|
||||
if source == "hermes":
|
||||
from openjarvis.skills.sources.hermes import HermesResolver
|
||||
|
||||
return HermesResolver()
|
||||
if source == "openclaw":
|
||||
from openjarvis.skills.sources.openclaw import OpenClawResolver
|
||||
|
||||
return OpenClawResolver()
|
||||
if source == "github":
|
||||
if not url:
|
||||
raise click.BadParameter("github source requires --url")
|
||||
from pathlib import Path as _Path
|
||||
|
||||
from openjarvis.skills.sources.github import GitHubResolver
|
||||
|
||||
cache = _Path(
|
||||
"~/.openjarvis/skill-cache/github/" + url.rstrip("/").rsplit("/", 1)[-1]
|
||||
).expanduser()
|
||||
return GitHubResolver(cache_root=cache, repo_url=url)
|
||||
raise click.BadParameter(f"Unknown source: {source!r}")
|
||||
|
||||
|
||||
@skill.command("install")
|
||||
@click.argument("query")
|
||||
@click.option(
|
||||
"--with-scripts",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help="Import the skill's scripts/ directory (security-sensitive).",
|
||||
)
|
||||
@click.option(
|
||||
"--force", is_flag=True, default=False, help="Overwrite existing install."
|
||||
)
|
||||
@click.option(
|
||||
"--url",
|
||||
default="",
|
||||
help="Repo URL (required when source is 'github').",
|
||||
)
|
||||
def install(query: str, with_scripts: bool, force: bool, url: str):
|
||||
"""Install a skill from a source.
|
||||
|
||||
Example: ``jarvis skill install hermes:apple-notes``
|
||||
"""
|
||||
console = Console()
|
||||
source, name = _parse_source_query(query)
|
||||
|
||||
resolver = _get_resolver(source, url=url)
|
||||
try:
|
||||
resolver.sync()
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Failed to sync source {source}: {exc}[/red]")
|
||||
raise SystemExit(1)
|
||||
|
||||
# Support category/name queries (e.g. openclaw:owner/slug, hermes:apple/apple-notes)
|
||||
if "/" in name:
|
||||
category, _, skill_name = name.partition("/")
|
||||
matches = [
|
||||
s
|
||||
for s in resolver.list_skills()
|
||||
if s.name == skill_name and s.category == category
|
||||
]
|
||||
else:
|
||||
matches = [s for s in resolver.list_skills() if s.name == name]
|
||||
if not matches:
|
||||
console.print(f"[red]No skill named '{name}' found in source '{source}'[/red]")
|
||||
raise SystemExit(1)
|
||||
|
||||
from openjarvis.skills.importer import SkillImporter
|
||||
from openjarvis.skills.parser import SkillParser
|
||||
from openjarvis.skills.tool_translator import ToolTranslator
|
||||
|
||||
importer = SkillImporter(parser=SkillParser(), tool_translator=ToolTranslator())
|
||||
result = importer.import_skill(matches[0], with_scripts=with_scripts, force=force)
|
||||
|
||||
if result.success:
|
||||
if result.skipped:
|
||||
console.print("[yellow]Skill already installed[/yellow]")
|
||||
else:
|
||||
console.print(f"[green]Installed:[/green] {result.target_path}")
|
||||
if result.translated_tools:
|
||||
console.print(f" Translated tools: {', '.join(result.translated_tools)}")
|
||||
if result.untranslated_tools:
|
||||
console.print(
|
||||
f" [yellow]Untranslated tools:[/yellow] "
|
||||
f"{', '.join(result.untranslated_tools)}"
|
||||
)
|
||||
for warning in result.warnings:
|
||||
console.print(f" [yellow]{warning}[/yellow]")
|
||||
else:
|
||||
console.print(
|
||||
"[red]Install failed: "
|
||||
+ "; ".join(result.warnings or ["unknown error"])
|
||||
+ "[/red]"
|
||||
)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
@skill.command("sync")
|
||||
@click.argument("source", required=False)
|
||||
@click.option("--category", default="", help="Filter by category.")
|
||||
@click.option("--tag", default="", help="Filter by tag.")
|
||||
@click.option("--search", default="", help="Substring search across name+description.")
|
||||
@click.option(
|
||||
"--with-scripts",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help="Import scripts/ directories.",
|
||||
)
|
||||
@click.option("--force", is_flag=True, default=False, help="Re-import existing skills.")
|
||||
def sync(
|
||||
source: str,
|
||||
category: str,
|
||||
tag: str,
|
||||
search: str,
|
||||
with_scripts: bool,
|
||||
force: bool,
|
||||
):
|
||||
"""Bulk install + update from a source (or all configured sources)."""
|
||||
console = Console()
|
||||
|
||||
cfg = load_config()
|
||||
|
||||
# Determine which sources to sync
|
||||
source_configs: list = []
|
||||
if source:
|
||||
source_configs.append({"source": source, "filter": {}, "url": ""})
|
||||
else:
|
||||
for src_cfg in cfg.skills.sources:
|
||||
source_configs.append(
|
||||
{
|
||||
"source": src_cfg.source,
|
||||
"filter": dict(src_cfg.filter or {}),
|
||||
"url": src_cfg.url,
|
||||
}
|
||||
)
|
||||
|
||||
if not source_configs:
|
||||
console.print(
|
||||
"[yellow]No sources to sync. "
|
||||
"Add sources to [skills.sources] in config.toml "
|
||||
"or pass a source name.[/yellow]"
|
||||
)
|
||||
return
|
||||
console.print(f"[dim]Searching for skills matching '{query}'...[/dim]")
|
||||
console.print("[dim]Skill search not yet implemented.[/dim]")
|
||||
|
||||
from openjarvis.skills.importer import SkillImporter
|
||||
from openjarvis.skills.parser import SkillParser
|
||||
from openjarvis.skills.tool_translator import ToolTranslator
|
||||
|
||||
importer = SkillImporter(parser=SkillParser(), tool_translator=ToolTranslator())
|
||||
|
||||
total_installed = 0
|
||||
for src in source_configs:
|
||||
console.print(f"[cyan]Syncing {src['source']}...[/cyan]")
|
||||
try:
|
||||
resolver = _get_resolver(src["source"], url=src["url"])
|
||||
resolver.sync()
|
||||
except Exception as exc:
|
||||
console.print(f"[red]Failed to sync {src['source']}: {exc}[/red]")
|
||||
continue
|
||||
|
||||
skills_to_import = resolver.list_skills()
|
||||
|
||||
# Apply CLI filters
|
||||
if category:
|
||||
skills_to_import = [s for s in skills_to_import if s.category == category]
|
||||
if search:
|
||||
sl = search.lower()
|
||||
skills_to_import = [
|
||||
s
|
||||
for s in skills_to_import
|
||||
if sl in s.name.lower() or sl in s.description.lower()
|
||||
]
|
||||
|
||||
# Apply config filter (categories list)
|
||||
cfg_categories = src["filter"].get("category") or []
|
||||
if cfg_categories:
|
||||
skills_to_import = [
|
||||
s for s in skills_to_import if s.category in cfg_categories
|
||||
]
|
||||
|
||||
installed_count = 0
|
||||
for resolved in skills_to_import:
|
||||
r = importer.import_skill(resolved, with_scripts=with_scripts, force=force)
|
||||
if r.success and not r.skipped:
|
||||
installed_count += 1
|
||||
console.print(f" Imported {installed_count}/{len(skills_to_import)} skills")
|
||||
total_installed += installed_count
|
||||
|
||||
console.print(f"[green]Total installed: {total_installed}[/green]")
|
||||
|
||||
|
||||
@skill.command("sources")
|
||||
def sources():
|
||||
"""List configured skill sources."""
|
||||
console = Console()
|
||||
|
||||
cfg = load_config()
|
||||
|
||||
if not cfg.skills.sources:
|
||||
console.print(
|
||||
"[dim]No skill sources configured. "
|
||||
"Add entries to [skills.sources] in config.toml.[/dim]"
|
||||
)
|
||||
return
|
||||
|
||||
table = Table(title="Configured Skill Sources")
|
||||
table.add_column("Source", style="cyan")
|
||||
table.add_column("URL")
|
||||
table.add_column("Filter")
|
||||
table.add_column("Auto-update")
|
||||
for s in cfg.skills.sources:
|
||||
filt = ", ".join(f"{k}={v}" for k, v in (s.filter or {}).items()) or "—"
|
||||
table.add_row(
|
||||
s.source,
|
||||
s.url or "(default)",
|
||||
filt,
|
||||
"yes" if s.auto_update else "no",
|
||||
)
|
||||
console.print(table)
|
||||
|
||||
|
||||
@skill.command("remove")
|
||||
@click.argument("skill_name")
|
||||
def remove(skill_name: str):
|
||||
"""Remove an installed skill."""
|
||||
Console().print("[yellow]Skill removal not yet implemented.[/yellow]")
|
||||
|
||||
|
||||
@skill.command("search")
|
||||
@click.argument("query")
|
||||
def search(query: str):
|
||||
"""Search available skills."""
|
||||
Console().print(
|
||||
"[yellow]Skill search not yet implemented. "
|
||||
"Run 'jarvis skill update' first.[/yellow]"
|
||||
)
|
||||
|
||||
|
||||
@skill.command("update")
|
||||
def update():
|
||||
"""Pull latest commits for all installed skill sources."""
|
||||
console = Console()
|
||||
|
||||
cfg = load_config()
|
||||
if not cfg.skills.sources:
|
||||
console.print("[dim]No sources configured.[/dim]")
|
||||
return
|
||||
|
||||
for src in cfg.skills.sources:
|
||||
console.print(f"[cyan]Updating {src.source}...[/cyan]")
|
||||
try:
|
||||
resolver = _get_resolver(src.source, url=src.url)
|
||||
resolver.sync()
|
||||
console.print(" [green]OK[/green]")
|
||||
except Exception as exc:
|
||||
console.print(f" [red]Failed: {exc}[/red]")
|
||||
|
||||
|
||||
@skill.command("discover")
|
||||
@click.option(
|
||||
"--min-frequency",
|
||||
"-f",
|
||||
default=3,
|
||||
show_default=True,
|
||||
type=int,
|
||||
help="Minimum recurrence count to surface a tool sequence as a skill.",
|
||||
)
|
||||
@click.option(
|
||||
"--min-outcome",
|
||||
"-o",
|
||||
default=0.5,
|
||||
show_default=True,
|
||||
type=float,
|
||||
help="Minimum average outcome score (0.0-1.0) to qualify.",
|
||||
)
|
||||
@click.option(
|
||||
"--dry-run",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help="Print discovered patterns without writing manifests.",
|
||||
)
|
||||
def discover(min_frequency: int, min_outcome: float, dry_run: bool) -> None:
|
||||
"""Mine the trace store for recurring tool sequences and write them as
|
||||
discovered skill manifests under ~/.openjarvis/skills/discovered/."""
|
||||
console = Console()
|
||||
store = _get_trace_store()
|
||||
if store is None:
|
||||
console.print(
|
||||
"[red]No trace store found. "
|
||||
"Enable tracing in config (traces.enabled = true) and run "
|
||||
"some queries first.[/red]"
|
||||
)
|
||||
raise SystemExit(1)
|
||||
|
||||
output_dir = _get_discovered_dir()
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
|
||||
if dry_run:
|
||||
# Use a temporary directory so nothing is persisted
|
||||
import tempfile
|
||||
|
||||
tmp = Path(tempfile.mkdtemp(prefix="openjarvis-discover-dryrun-"))
|
||||
try:
|
||||
written = mgr.discover_from_traces(
|
||||
store,
|
||||
min_frequency=min_frequency,
|
||||
min_outcome=min_outcome,
|
||||
output_dir=tmp,
|
||||
)
|
||||
finally:
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
else:
|
||||
written = mgr.discover_from_traces(
|
||||
store,
|
||||
min_frequency=min_frequency,
|
||||
min_outcome=min_outcome,
|
||||
output_dir=output_dir,
|
||||
)
|
||||
|
||||
if not written:
|
||||
console.print(
|
||||
"[dim]No recurring tool sequences found above the threshold.[/dim]"
|
||||
)
|
||||
return
|
||||
|
||||
table = Table(title="Discovered Skills")
|
||||
table.add_column("Name", style="cyan")
|
||||
table.add_column("Path")
|
||||
for item in written:
|
||||
table.add_row(item["name"], item["path"])
|
||||
console.print(table)
|
||||
if dry_run:
|
||||
console.print("[yellow]--dry-run: no files were written.[/yellow]")
|
||||
|
||||
|
||||
@skill.command("show-overlay")
|
||||
@click.argument("skill_name")
|
||||
def show_overlay(skill_name: str) -> None:
|
||||
"""Show the optimization overlay for a skill, if one exists."""
|
||||
console = Console()
|
||||
from openjarvis.skills.overlay import SkillOverlayLoader
|
||||
|
||||
loader = SkillOverlayLoader(_get_overlay_dir())
|
||||
overlay = loader.load(skill_name)
|
||||
if overlay is None:
|
||||
console.print(f"[red]No overlay found for skill '{skill_name}'.[/red]")
|
||||
raise SystemExit(1)
|
||||
|
||||
console.print(f"[bold]{overlay.skill_name}[/bold]")
|
||||
console.print(f"Optimizer: {overlay.optimizer}")
|
||||
console.print(f"Optimized at: {overlay.optimized_at}")
|
||||
console.print(f"Trace count: {overlay.trace_count}")
|
||||
console.print(f"Description: {overlay.description}")
|
||||
if overlay.few_shot:
|
||||
console.print(f"Few-shot examples ({len(overlay.few_shot)}):")
|
||||
for i, ex in enumerate(overlay.few_shot, start=1):
|
||||
inp = (ex.get("input", "") or "")[:100]
|
||||
out = (ex.get("output", "") or "")[:100]
|
||||
console.print(f" {i}. input={inp!r}")
|
||||
console.print(f" output={out!r}")
|
||||
|
||||
|
||||
__all__ = ["skill"]
|
||||
|
||||
@@ -653,6 +653,17 @@ class AgentLearningConfig:
|
||||
gepa: GEPAOptimizerConfig = field(default_factory=GEPAOptimizerConfig)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SkillsLearningConfig:
|
||||
"""Configuration for the skills learning loop (Plan 2A)."""
|
||||
|
||||
auto_optimize: bool = False # opt in via config
|
||||
optimizer: str = "dspy" # "dspy" or "gepa"
|
||||
min_traces_per_skill: int = 20
|
||||
optimization_interval_seconds: int = 86400
|
||||
overlay_dir: str = "~/.openjarvis/learning/skills/"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class MetricsConfig:
|
||||
"""Reward / optimization metric weights."""
|
||||
@@ -675,6 +686,7 @@ class LearningConfig:
|
||||
default_factory=IntelligenceLearningConfig,
|
||||
)
|
||||
agent: AgentLearningConfig = field(default_factory=AgentLearningConfig)
|
||||
skills: SkillsLearningConfig = field(default_factory=SkillsLearningConfig)
|
||||
metrics: MetricsConfig = field(default_factory=MetricsConfig)
|
||||
|
||||
# Training pipeline
|
||||
@@ -1260,13 +1272,31 @@ class CompressionConfig:
|
||||
strategy: str = "session_consolidation"
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SkillSourceConfig:
|
||||
"""Configuration for a single skill source (Hermes, OpenClaw, GitHub)."""
|
||||
|
||||
source: str = "" # "hermes", "openclaw", or "github"
|
||||
url: str = "" # required when source = "github"
|
||||
filter: Dict[str, Any] = field(default_factory=dict)
|
||||
auto_update: bool = False
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SkillsConfig:
|
||||
"""Configuration for agent-authored procedural skills."""
|
||||
|
||||
enabled: bool = True
|
||||
skills_dir: str = "~/.openjarvis/skills/"
|
||||
nudge_interval: int = 15
|
||||
active: str = "*"
|
||||
auto_discover: bool = True
|
||||
auto_sync: bool = False
|
||||
nudge_interval: int = 15
|
||||
index_repo: str = "https://github.com/openjarvis/skill-index.git"
|
||||
index_dir: str = "~/.openjarvis/skill-index/"
|
||||
max_depth: int = 5
|
||||
sandbox_dangerous: bool = True
|
||||
sources: List[SkillSourceConfig] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from openjarvis.evals.core.backend import InferenceBackend
|
||||
@@ -26,6 +27,8 @@ class JarvisAgentBackend(InferenceBackend):
|
||||
gpu_metrics: bool = False,
|
||||
model: Optional[str] = None,
|
||||
max_turns: Optional[int] = None,
|
||||
skills_enabled: bool = True,
|
||||
overlay_dir: Optional[Path] = None,
|
||||
) -> None:
|
||||
from openjarvis.system import SystemBuilder
|
||||
|
||||
@@ -52,6 +55,11 @@ class JarvisAgentBackend(InferenceBackend):
|
||||
# GAIA tasks before this was configurable per-eval).
|
||||
if max_turns is not None:
|
||||
builder._config.agent.max_turns = max_turns
|
||||
# Plan 2B: per-condition skill switches. Mutate the builder's
|
||||
# config directly so SystemBuilder picks them up at build time.
|
||||
builder._config.skills.enabled = skills_enabled
|
||||
if overlay_dir is not None:
|
||||
builder._config.learning.skills.overlay_dir = str(overlay_dir)
|
||||
self._system = builder.telemetry(telemetry).traces(True).build()
|
||||
|
||||
def generate(
|
||||
|
||||
@@ -0,0 +1,508 @@
|
||||
"""SkillBenchmarkRunner — orchestrate the 4-condition × N-seed × M-task
|
||||
PinchBench sweep that measures whether skills + DSPy/GEPA optimization
|
||||
improves agent performance.
|
||||
|
||||
Plan 2B implementation. See:
|
||||
docs/superpowers/specs/2026-04-08-skills-benchmark-evaluation-design.md
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import statistics
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
CONDITIONS = (
|
||||
"no_skills",
|
||||
"skills_on",
|
||||
"skills_optimized_dspy",
|
||||
"skills_optimized_gepa",
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config + result types
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SkillBenchmarkConfig:
|
||||
"""Configuration for a single SkillBenchmarkRunner sweep."""
|
||||
|
||||
benchmark: str = "pinchbench"
|
||||
model: str = "qwen3.5:9b"
|
||||
engine: str = "ollama"
|
||||
agent: str = "native_react"
|
||||
tools: List[str] = field(
|
||||
default_factory=lambda: [
|
||||
"calculator",
|
||||
"think",
|
||||
"shell_exec",
|
||||
"web_search",
|
||||
"file_read",
|
||||
"file_write",
|
||||
]
|
||||
)
|
||||
seeds: List[int] = field(default_factory=lambda: [42, 43, 44])
|
||||
max_samples: Optional[int] = None
|
||||
output_dir: Path = field(default_factory=lambda: Path("docs/superpowers/results/"))
|
||||
skills_dir: Path = field(
|
||||
default_factory=lambda: Path("~/.openjarvis/skills/").expanduser()
|
||||
)
|
||||
overlay_dir_dspy: Path = field(
|
||||
default_factory=lambda: Path("~/.openjarvis/learning/skills-dspy/").expanduser()
|
||||
)
|
||||
overlay_dir_gepa: Path = field(
|
||||
default_factory=lambda: Path("~/.openjarvis/learning/skills-gepa/").expanduser()
|
||||
)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ConditionResult:
|
||||
"""Aggregated result for a single condition across all seeds."""
|
||||
|
||||
condition: str
|
||||
seeds: List[int]
|
||||
per_seed_pass_rate: Dict[int, float]
|
||||
mean_pass_rate: float
|
||||
stddev_pass_rate: float
|
||||
per_task_results: Dict[str, List[bool]] # task_id → [pass_seed1, ...]
|
||||
skill_invocation_counts: Dict[str, int] # skill_name → total invocations
|
||||
total_tokens: int
|
||||
total_runtime_seconds: float
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ConditionComparison:
|
||||
"""Top-level result of a SkillBenchmarkRunner sweep."""
|
||||
|
||||
config: SkillBenchmarkConfig
|
||||
started_at: str
|
||||
ended_at: str
|
||||
results: Dict[str, ConditionResult]
|
||||
deltas: Dict[str, float]
|
||||
|
||||
|
||||
class SkillBenchmarkRunner:
|
||||
"""Orchestrates the 4-condition × N-seed × M-task PinchBench sweep.
|
||||
|
||||
Each condition is a different SystemBuilder configuration:
|
||||
- no_skills: cfg.skills.enabled = False
|
||||
- skills_on: enabled, overlay_dir = empty (no overlays load)
|
||||
- skills_optimized_dspy: enabled, overlay_dir = config.overlay_dir_dspy
|
||||
- skills_optimized_gepa: enabled, overlay_dir = config.overlay_dir_gepa
|
||||
|
||||
Per-seed runs share the same backend instantiation but pass a fresh
|
||||
seed to the EvalRunner.
|
||||
"""
|
||||
|
||||
def __init__(self, config: SkillBenchmarkConfig) -> None:
|
||||
self._config = config
|
||||
# An "empty" overlay dir for the skills_on condition. We point at
|
||||
# a known-empty subdirectory under the output dir so SkillManager
|
||||
# finds zero overlays even if the user happens to have populated
|
||||
# the default ~/.openjarvis/learning/skills/ tree.
|
||||
self._empty_overlay_dir = (
|
||||
Path(self._config.output_dir).expanduser() / "_skills_on_empty_overlays"
|
||||
)
|
||||
self._empty_overlay_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Per-condition backend construction
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _backend_kwargs_for_condition(self, condition: str) -> Dict[str, Any]:
|
||||
"""Return the kwargs to pass to JarvisAgentBackend for *condition*.
|
||||
|
||||
Pure function — no side effects, no SystemBuilder construction.
|
||||
Tested in isolation so we can verify the per-condition switches
|
||||
without invoking the engine.
|
||||
"""
|
||||
if condition == "no_skills":
|
||||
return {
|
||||
"skills_enabled": False,
|
||||
"overlay_dir": None,
|
||||
}
|
||||
if condition == "skills_on":
|
||||
return {
|
||||
"skills_enabled": True,
|
||||
"overlay_dir": self._empty_overlay_dir,
|
||||
}
|
||||
if condition == "skills_optimized_dspy":
|
||||
return {
|
||||
"skills_enabled": True,
|
||||
"overlay_dir": self._config.overlay_dir_dspy,
|
||||
}
|
||||
if condition == "skills_optimized_gepa":
|
||||
return {
|
||||
"skills_enabled": True,
|
||||
"overlay_dir": self._config.overlay_dir_gepa,
|
||||
}
|
||||
raise ValueError(
|
||||
f"Unknown condition '{condition}'. Expected one of: "
|
||||
f"{', '.join(CONDITIONS)}"
|
||||
)
|
||||
|
||||
def _build_backend_for_condition(self, condition: str) -> Any:
|
||||
"""Construct a JarvisAgentBackend for *condition*.
|
||||
|
||||
Separate from `_backend_kwargs_for_condition` so the kwarg logic
|
||||
can be tested without instantiating an engine.
|
||||
"""
|
||||
from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
|
||||
|
||||
kw = self._backend_kwargs_for_condition(condition)
|
||||
return JarvisAgentBackend(
|
||||
engine_key=self._config.engine,
|
||||
agent_name=self._config.agent,
|
||||
tools=list(self._config.tools),
|
||||
telemetry=False,
|
||||
model=self._config.model,
|
||||
skills_enabled=kw["skills_enabled"],
|
||||
overlay_dir=kw["overlay_dir"],
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Per-seed and per-condition runs
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _run_single_seed(
|
||||
self,
|
||||
condition: str,
|
||||
seed: int,
|
||||
) -> Dict[str, Any]:
|
||||
"""Run the benchmark once for one (condition, seed) pair.
|
||||
|
||||
Returns a raw dict with per-task results, pass rate, skill
|
||||
invocation counts, total tokens, and total runtime.
|
||||
|
||||
This method is the integration boundary with the existing
|
||||
EvalRunner / PinchBenchDataset. It is intentionally a thin
|
||||
shim so tests can monkeypatch it without instantiating an
|
||||
engine or running real benchmark tasks.
|
||||
"""
|
||||
from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
|
||||
from openjarvis.evals.core.runner import EvalRunner
|
||||
from openjarvis.evals.core.types import RunConfig
|
||||
from openjarvis.evals.datasets.pinchbench import PinchBenchDataset
|
||||
from openjarvis.evals.scorers.pinchbench import PinchBenchScorer
|
||||
|
||||
backend = self._build_backend_for_condition(condition)
|
||||
|
||||
dataset = PinchBenchDataset(path=None)
|
||||
dataset.load(
|
||||
max_samples=self._config.max_samples,
|
||||
split=None,
|
||||
seed=seed,
|
||||
)
|
||||
|
||||
# PinchBenchScorer is an LLM-as-judge scorer; it needs a judge
|
||||
# backend. We reuse the same engine the agent uses (typically
|
||||
# a local Ollama model) so the headline run is fully local.
|
||||
try:
|
||||
judge_backend = JarvisDirectBackend(
|
||||
engine_key=self._config.engine,
|
||||
)
|
||||
except RuntimeError as exc:
|
||||
LOGGER.warning(
|
||||
"Judge backend unavailable, automated checks only: %s",
|
||||
exc,
|
||||
)
|
||||
judge_backend = None
|
||||
scorer = PinchBenchScorer(judge_backend, self._config.model)
|
||||
|
||||
runner_cfg = RunConfig(
|
||||
benchmark=self._config.benchmark,
|
||||
backend="jarvis-agent",
|
||||
model=self._config.model,
|
||||
max_workers=1,
|
||||
episode_mode=False,
|
||||
seed=seed,
|
||||
)
|
||||
|
||||
eval_runner = EvalRunner(
|
||||
config=runner_cfg,
|
||||
dataset=dataset,
|
||||
backend=backend,
|
||||
scorer=scorer,
|
||||
)
|
||||
|
||||
t0 = time.monotonic()
|
||||
summary = eval_runner.run()
|
||||
elapsed = time.monotonic() - t0
|
||||
|
||||
# Aggregate per-task results. RunSummary doesn't carry the
|
||||
# per-record list — that lives on the runner instance via the
|
||||
# `results` property (which returns List[EvalResult]).
|
||||
per_task: Dict[str, bool] = {}
|
||||
for r in eval_runner.results:
|
||||
rid = getattr(r, "record_id", "") or ""
|
||||
ok = bool(getattr(r, "is_correct", False))
|
||||
if rid:
|
||||
per_task[rid] = ok
|
||||
|
||||
# Pass rate. Prefer the runner-computed accuracy when we have
|
||||
# records; fall back to summary.accuracy otherwise.
|
||||
if per_task:
|
||||
pass_rate = sum(1 for v in per_task.values() if v) / len(per_task)
|
||||
else:
|
||||
pass_rate = float(getattr(summary, "accuracy", 0.0) or 0.0)
|
||||
|
||||
# Skill invocation counts from the trace store (if any)
|
||||
skill_invocations = self._extract_skill_invocations(backend)
|
||||
|
||||
# Token totals. RunSummary tracks input/output tokens separately.
|
||||
total_input = int(getattr(summary, "total_input_tokens", 0) or 0)
|
||||
total_output = int(getattr(summary, "total_output_tokens", 0) or 0)
|
||||
total_tokens = total_input + total_output
|
||||
|
||||
return {
|
||||
"pass_rate": pass_rate,
|
||||
"per_task": per_task,
|
||||
"skill_invocations": skill_invocations,
|
||||
"total_tokens": total_tokens,
|
||||
"total_runtime_seconds": elapsed,
|
||||
}
|
||||
|
||||
def _extract_skill_invocations(self, backend: Any) -> Dict[str, int]:
|
||||
"""Count per-skill invocations in the trace store from this run.
|
||||
|
||||
Walks all traces in the backend's system trace store and counts
|
||||
TraceStep.metadata.skill values. Returns an empty dict on any
|
||||
failure (the count is informational, not load-bearing).
|
||||
"""
|
||||
try:
|
||||
system = getattr(backend, "_system", None)
|
||||
if system is None:
|
||||
return {}
|
||||
store = getattr(system, "trace_store", None)
|
||||
if store is None:
|
||||
return {}
|
||||
traces = store.list_traces(limit=10000)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
counts: Dict[str, int] = {}
|
||||
for trace in traces:
|
||||
steps = getattr(trace, "steps", None) or []
|
||||
for step in steps:
|
||||
meta = getattr(step, "metadata", None) or {}
|
||||
if isinstance(meta, dict):
|
||||
name = meta.get("skill")
|
||||
if name:
|
||||
counts[name] = counts.get(name, 0) + 1
|
||||
return counts
|
||||
|
||||
def run_condition(self, condition: str) -> ConditionResult:
|
||||
"""Run the benchmark for *condition* across all configured seeds.
|
||||
|
||||
Returns a ConditionResult with mean ± stddev pass rate and the
|
||||
per-task / per-skill aggregations.
|
||||
"""
|
||||
if condition not in CONDITIONS:
|
||||
raise ValueError(
|
||||
f"Unknown condition '{condition}'. Expected one of: "
|
||||
f"{', '.join(CONDITIONS)}"
|
||||
)
|
||||
|
||||
per_seed_pass_rate: Dict[int, float] = {}
|
||||
per_task_results: Dict[str, List[bool]] = {}
|
||||
skill_counts: Dict[str, int] = {}
|
||||
total_tokens = 0
|
||||
total_runtime = 0.0
|
||||
|
||||
for seed in self._config.seeds:
|
||||
seed_data = self._run_single_seed(condition, seed)
|
||||
per_seed_pass_rate[seed] = float(seed_data["pass_rate"])
|
||||
|
||||
for task_id, ok in seed_data["per_task"].items():
|
||||
per_task_results.setdefault(task_id, []).append(bool(ok))
|
||||
|
||||
for skill_name, count in seed_data["skill_invocations"].items():
|
||||
skill_counts[skill_name] = skill_counts.get(skill_name, 0) + int(count)
|
||||
|
||||
total_tokens += int(seed_data["total_tokens"])
|
||||
total_runtime += float(seed_data["total_runtime_seconds"])
|
||||
|
||||
rates = list(per_seed_pass_rate.values())
|
||||
mean_rate = statistics.fmean(rates) if rates else 0.0
|
||||
stddev_rate = statistics.stdev(rates) if len(rates) > 1 else 0.0
|
||||
|
||||
return ConditionResult(
|
||||
condition=condition,
|
||||
seeds=list(self._config.seeds),
|
||||
per_seed_pass_rate=per_seed_pass_rate,
|
||||
mean_pass_rate=mean_rate,
|
||||
stddev_pass_rate=stddev_rate,
|
||||
per_task_results=per_task_results,
|
||||
skill_invocation_counts=skill_counts,
|
||||
total_tokens=total_tokens,
|
||||
total_runtime_seconds=total_runtime,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Sweep + report
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def run_all_conditions(self) -> ConditionComparison:
|
||||
"""Run all 4 conditions × all seeds.
|
||||
|
||||
Returns a ConditionComparison with per-condition results and the
|
||||
computed deltas (skills_on - no_skills, etc.).
|
||||
"""
|
||||
started_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
results: Dict[str, ConditionResult] = {}
|
||||
for condition in CONDITIONS:
|
||||
LOGGER.info("Running condition: %s", condition)
|
||||
results[condition] = self.run_condition(condition)
|
||||
|
||||
ended_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
# Compute the headline deltas
|
||||
deltas: Dict[str, float] = {}
|
||||
if "no_skills" in results and "skills_on" in results:
|
||||
deltas["skills_on - no_skills"] = (
|
||||
results["skills_on"].mean_pass_rate
|
||||
- results["no_skills"].mean_pass_rate
|
||||
)
|
||||
if "skills_on" in results and "skills_optimized_dspy" in results:
|
||||
deltas["skills_optimized_dspy - skills_on"] = (
|
||||
results["skills_optimized_dspy"].mean_pass_rate
|
||||
- results["skills_on"].mean_pass_rate
|
||||
)
|
||||
if "skills_on" in results and "skills_optimized_gepa" in results:
|
||||
deltas["skills_optimized_gepa - skills_on"] = (
|
||||
results["skills_optimized_gepa"].mean_pass_rate
|
||||
- results["skills_on"].mean_pass_rate
|
||||
)
|
||||
|
||||
return ConditionComparison(
|
||||
config=self._config,
|
||||
started_at=started_at,
|
||||
ended_at=ended_at,
|
||||
results=results,
|
||||
deltas=deltas,
|
||||
)
|
||||
|
||||
def write_report(self, comparison: ConditionComparison) -> Path:
|
||||
"""Write a markdown report to output_dir.
|
||||
|
||||
Filename: pinchbench-skills-eval-{YYYY-MM-DD}.md
|
||||
"""
|
||||
date_str = datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
||||
out_dir = Path(self._config.output_dir).expanduser()
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = out_dir / f"pinchbench-skills-eval-{date_str}.md"
|
||||
|
||||
lines: List[str] = []
|
||||
lines.append(f"# PinchBench Skills Evaluation — {date_str}")
|
||||
lines.append("")
|
||||
lines.append(f"**Started:** {comparison.started_at}")
|
||||
lines.append(f"**Ended:** {comparison.ended_at}")
|
||||
lines.append(f"**Model:** {comparison.config.model}")
|
||||
lines.append(f"**Engine:** {comparison.config.engine}")
|
||||
lines.append(f"**Agent:** {comparison.config.agent}")
|
||||
lines.append(f"**Seeds:** {', '.join(str(s) for s in comparison.config.seeds)}")
|
||||
max_samples = comparison.config.max_samples
|
||||
lines.append(
|
||||
f"**Max samples:** {max_samples if max_samples is not None else 'full'}"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# Summary table
|
||||
lines.append("## Summary")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
"| Condition | Mean pass rate | Stddev | Total tokens | Runtime (s) |"
|
||||
)
|
||||
lines.append("|---|---|---|---|---|")
|
||||
for condition in CONDITIONS:
|
||||
r = comparison.results.get(condition)
|
||||
if r is None:
|
||||
continue
|
||||
lines.append(
|
||||
f"| {condition} | {r.mean_pass_rate:.3f} | "
|
||||
f"{r.stddev_pass_rate:.3f} | {r.total_tokens} | "
|
||||
f"{r.total_runtime_seconds:.1f} |"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# Deltas
|
||||
if comparison.deltas:
|
||||
lines.append("## Deltas")
|
||||
lines.append("")
|
||||
for name, value in comparison.deltas.items():
|
||||
sign = "+" if value >= 0 else ""
|
||||
lines.append(f"- **{name}**: {sign}{value:.3f}")
|
||||
lines.append("")
|
||||
|
||||
# Per-task breakdown
|
||||
lines.append("## Per-task results")
|
||||
lines.append("")
|
||||
all_tasks: set[str] = set()
|
||||
for r in comparison.results.values():
|
||||
all_tasks.update(r.per_task_results.keys())
|
||||
if all_tasks:
|
||||
header = "| Task | " + " | ".join(CONDITIONS) + " |"
|
||||
sep = "|---|" + "|".join(["---"] * len(CONDITIONS)) + "|"
|
||||
lines.append(header)
|
||||
lines.append(sep)
|
||||
for task_id in sorted(all_tasks):
|
||||
row = [task_id]
|
||||
for condition in CONDITIONS:
|
||||
r = comparison.results.get(condition)
|
||||
passes = (
|
||||
r.per_task_results.get(task_id, []) if r is not None else []
|
||||
)
|
||||
if not passes:
|
||||
row.append("—")
|
||||
else:
|
||||
n_pass = sum(1 for v in passes if v)
|
||||
row.append(f"{n_pass}/{len(passes)}")
|
||||
lines.append("| " + " | ".join(row) + " |")
|
||||
lines.append("")
|
||||
|
||||
# Per-skill invocation counts
|
||||
lines.append("## Per-skill invocation counts")
|
||||
lines.append("")
|
||||
all_skills: set[str] = set()
|
||||
for r in comparison.results.values():
|
||||
all_skills.update(r.skill_invocation_counts.keys())
|
||||
if all_skills:
|
||||
header = "| Skill | " + " | ".join(CONDITIONS) + " |"
|
||||
sep = "|---|" + "|".join(["---"] * len(CONDITIONS)) + "|"
|
||||
lines.append(header)
|
||||
lines.append(sep)
|
||||
for skill_name in sorted(all_skills):
|
||||
row = [skill_name]
|
||||
for condition in CONDITIONS:
|
||||
r = comparison.results.get(condition)
|
||||
count = (
|
||||
r.skill_invocation_counts.get(skill_name, 0)
|
||||
if r is not None
|
||||
else 0
|
||||
)
|
||||
row.append(str(count))
|
||||
lines.append("| " + " | ".join(row) + " |")
|
||||
else:
|
||||
lines.append("(no skill invocations recorded)")
|
||||
lines.append("")
|
||||
|
||||
path.write_text("\n".join(lines), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
__all__ = [
|
||||
"CONDITIONS",
|
||||
"ConditionComparison",
|
||||
"ConditionResult",
|
||||
"SkillBenchmarkConfig",
|
||||
"SkillBenchmarkRunner",
|
||||
]
|
||||
@@ -130,15 +130,44 @@ class SkillDiscovery:
|
||||
"tool_name",
|
||||
getattr(step, "name", ""),
|
||||
)
|
||||
# Also check step.input dict for tool name (TraceStep format)
|
||||
if not name:
|
||||
step_input = getattr(step, "input", {}) or {}
|
||||
if isinstance(step_input, dict):
|
||||
name = step_input.get("tool", "") or ""
|
||||
if name:
|
||||
tools.append(name)
|
||||
return tools
|
||||
|
||||
def _extract_outcome(self, trace: Any) -> float:
|
||||
"""Extract outcome score from a trace."""
|
||||
"""Extract outcome score from a trace.
|
||||
|
||||
Handles both numeric outcome scores and string labels
|
||||
("success" / "failure"). Falls back to the ``feedback`` field when
|
||||
present, then defaults to 0.0.
|
||||
"""
|
||||
if isinstance(trace, dict):
|
||||
return float(trace.get("outcome", 0.0))
|
||||
return float(getattr(trace, "outcome", 0.0))
|
||||
raw = trace.get("outcome")
|
||||
if raw is None:
|
||||
return float(trace.get("feedback", 0.0) or 0.0)
|
||||
else:
|
||||
raw = getattr(trace, "outcome", None)
|
||||
if raw is None:
|
||||
return float(getattr(trace, "feedback", 0.0) or 0.0)
|
||||
|
||||
if isinstance(raw, (int, float)):
|
||||
return float(raw)
|
||||
if isinstance(raw, str):
|
||||
lraw = raw.lower()
|
||||
if lraw == "success":
|
||||
return 1.0
|
||||
if lraw == "failure":
|
||||
return 0.0
|
||||
try:
|
||||
return float(raw)
|
||||
except ValueError:
|
||||
return 0.0
|
||||
return 0.0
|
||||
|
||||
def _extract_query(self, trace: Any) -> str:
|
||||
"""Extract the original query from a trace."""
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
"""SkillOptimizer — per-skill DSPy/GEPA optimization wrapper (Plan 2A).
|
||||
|
||||
Buckets traces by skill name, runs the underlying optimizer on each skill's
|
||||
bucket, and writes the result as a sidecar overlay file in
|
||||
``~/.openjarvis/learning/skills/<skill-name>/optimized.toml``.
|
||||
|
||||
The actual DSPy/GEPA invocation is done in ``_run_dspy`` / ``_run_gepa``,
|
||||
which are isolated for easy mocking in tests. In Plan 2A these are
|
||||
deliberately minimal — they call the existing optimizer modules and extract
|
||||
the description + few-shot examples. Plan 2B will measure the impact via
|
||||
benchmarks.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from openjarvis.core.types import Trace, TraceStep
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
from openjarvis.skills.overlay import SkillOverlay, write_overlay
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SkillOptimizationResult:
|
||||
"""Result of optimizing a single skill."""
|
||||
|
||||
skill_name: str
|
||||
status: str # "optimized" | "skipped" | "error"
|
||||
trace_count: int = 0
|
||||
overlay_path: Optional[Path] = None
|
||||
error: str = ""
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _OptimizerOutput:
|
||||
"""Internal: extracted output from the underlying DSPy/GEPA optimizer."""
|
||||
|
||||
description: str = ""
|
||||
few_shot: List[Dict[str, str]] = field(default_factory=list)
|
||||
|
||||
|
||||
class SkillOptimizer:
|
||||
"""Per-skill optimization wrapper around DSPyAgentOptimizer / GEPA.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
min_traces_per_skill:
|
||||
Minimum trace count for a skill to be eligible for optimization.
|
||||
optimizer:
|
||||
``"dspy"`` or ``"gepa"``. Determines which underlying optimizer is
|
||||
called inside ``_run_dspy`` / ``_run_gepa``.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
min_traces_per_skill: int = 20,
|
||||
optimizer: str = "dspy",
|
||||
) -> None:
|
||||
self._min_traces = min_traces_per_skill
|
||||
self._optimizer = optimizer
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Public API
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def optimize(
|
||||
self,
|
||||
trace_store: Any,
|
||||
skill_manager: SkillManager,
|
||||
*,
|
||||
overlay_dir: Optional[Path] = None,
|
||||
) -> Dict[str, SkillOptimizationResult]:
|
||||
"""Run the per-skill optimization loop.
|
||||
|
||||
Returns a dict mapping skill name to result. Writes overlay TOML
|
||||
files for each skill that produced output.
|
||||
"""
|
||||
if overlay_dir is None:
|
||||
# Try config first; fall back to the default tree.
|
||||
try:
|
||||
from openjarvis.core.config import load_config
|
||||
|
||||
cfg = load_config()
|
||||
cfg_dir = getattr(
|
||||
getattr(cfg.learning, "skills", None),
|
||||
"overlay_dir",
|
||||
None,
|
||||
)
|
||||
if cfg_dir:
|
||||
overlay_dir = Path(cfg_dir).expanduser()
|
||||
except Exception:
|
||||
pass
|
||||
if overlay_dir is None:
|
||||
overlay_dir = Path(
|
||||
"~/.openjarvis/learning/skills/"
|
||||
).expanduser()
|
||||
overlay_dir = Path(overlay_dir).expanduser()
|
||||
|
||||
traces = trace_store.list_traces(limit=10000)
|
||||
buckets = self._bucket_traces_by_skill(traces)
|
||||
|
||||
results: Dict[str, SkillOptimizationResult] = {}
|
||||
for skill_name, skill_traces in buckets.items():
|
||||
if len(skill_traces) < self._min_traces:
|
||||
results[skill_name] = SkillOptimizationResult(
|
||||
skill_name=skill_name,
|
||||
status="skipped",
|
||||
trace_count=len(skill_traces),
|
||||
)
|
||||
continue
|
||||
|
||||
try:
|
||||
if self._optimizer == "gepa":
|
||||
output = self._run_gepa(skill_name, skill_traces)
|
||||
else:
|
||||
output = self._run_dspy(skill_name, skill_traces)
|
||||
except Exception as exc:
|
||||
LOGGER.warning("Skill optimizer failed for '%s': %s", skill_name, exc)
|
||||
results[skill_name] = SkillOptimizationResult(
|
||||
skill_name=skill_name,
|
||||
status="error",
|
||||
trace_count=len(skill_traces),
|
||||
error=str(exc),
|
||||
)
|
||||
continue
|
||||
|
||||
overlay = SkillOverlay(
|
||||
skill_name=skill_name,
|
||||
optimizer=self._optimizer,
|
||||
optimized_at=datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
trace_count=len(skill_traces),
|
||||
description=output.description,
|
||||
few_shot=list(output.few_shot),
|
||||
)
|
||||
path = write_overlay(overlay, overlay_dir)
|
||||
results[skill_name] = SkillOptimizationResult(
|
||||
skill_name=skill_name,
|
||||
status="optimized",
|
||||
trace_count=len(skill_traces),
|
||||
overlay_path=path,
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Bucketing
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _bucket_traces_by_skill(self, traces: List[Trace]) -> Dict[str, List[Trace]]:
|
||||
"""Group traces by the skill names they invoked.
|
||||
|
||||
A trace is added to a skill's bucket once for each unique skill it
|
||||
invoked. Traces with no skill-tagged tool calls are dropped entirely.
|
||||
"""
|
||||
buckets: Dict[str, List[Trace]] = {}
|
||||
for trace in traces:
|
||||
skill_names = self._extract_skill_names(trace)
|
||||
for name in skill_names:
|
||||
buckets.setdefault(name, []).append(trace)
|
||||
return buckets
|
||||
|
||||
@staticmethod
|
||||
def _extract_skill_names(trace: Trace) -> List[str]:
|
||||
"""Return the unique skill names invoked in *trace*'s tool steps."""
|
||||
seen: List[str] = []
|
||||
steps = getattr(trace, "steps", []) or []
|
||||
for step in steps:
|
||||
metadata: Dict[str, Any] = {}
|
||||
if isinstance(step, dict):
|
||||
metadata = step.get("metadata", {}) or {}
|
||||
elif isinstance(step, TraceStep):
|
||||
metadata = step.metadata or {}
|
||||
else:
|
||||
metadata = getattr(step, "metadata", {}) or {}
|
||||
skill_name = metadata.get("skill")
|
||||
if skill_name and skill_name not in seen:
|
||||
seen.append(skill_name)
|
||||
return seen
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Underlying optimizers (mockable in tests)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _run_dspy(
|
||||
self,
|
||||
skill_name: str,
|
||||
skill_traces: List[Trace],
|
||||
) -> _OptimizerOutput:
|
||||
"""Run DSPyAgentOptimizer on the bucket and extract description + few-shot.
|
||||
|
||||
In Plan 2A this is intentionally simple — it pulls the few highest-
|
||||
feedback traces as candidate few-shot examples and uses the agent
|
||||
optimizer's output `system_prompt` as the new skill description if
|
||||
non-empty. Plan 2B will measure and refine.
|
||||
"""
|
||||
from openjarvis.core.config import DSPyOptimizerConfig
|
||||
from openjarvis.learning.agents.dspy_optimizer import DSPyAgentOptimizer
|
||||
|
||||
cfg = DSPyOptimizerConfig(
|
||||
min_traces=max(1, self._min_traces),
|
||||
max_bootstrapped_demos=4,
|
||||
max_labeled_demos=4,
|
||||
)
|
||||
optimizer = DSPyAgentOptimizer(cfg)
|
||||
|
||||
# Build a tiny in-memory trace store shim for the underlying optimizer
|
||||
class _BucketStore:
|
||||
def __init__(self, _traces):
|
||||
self._traces = _traces
|
||||
|
||||
def list_traces(self, *, limit=100, **_kwargs):
|
||||
return list(self._traces[:limit])
|
||||
|
||||
try:
|
||||
updates = optimizer.optimize(_BucketStore(skill_traces)) or {}
|
||||
except Exception as exc:
|
||||
LOGGER.warning(
|
||||
"DSPyAgentOptimizer raised for '%s' (using empty output): %s",
|
||||
skill_name,
|
||||
exc,
|
||||
)
|
||||
updates = {}
|
||||
|
||||
description = str(updates.get("system_prompt", "")) if updates else ""
|
||||
few_shot_raw = updates.get("few_shot_examples", []) if updates else []
|
||||
few_shot: List[Dict[str, str]] = []
|
||||
for item in few_shot_raw or []:
|
||||
if isinstance(item, dict):
|
||||
few_shot.append(
|
||||
{
|
||||
"input": str(item.get("input", "")),
|
||||
"output": str(item.get("output", "")),
|
||||
}
|
||||
)
|
||||
|
||||
# Fallback: if DSPy produced nothing, derive few-shot from top traces
|
||||
if not few_shot:
|
||||
top = sorted(
|
||||
skill_traces,
|
||||
key=lambda t: t.feedback if t.feedback is not None else 0.0,
|
||||
reverse=True,
|
||||
)[:4]
|
||||
for tr in top:
|
||||
if tr.query and tr.result:
|
||||
few_shot.append({"input": tr.query, "output": tr.result})
|
||||
|
||||
return _OptimizerOutput(description=description, few_shot=few_shot)
|
||||
|
||||
def _run_gepa(
|
||||
self,
|
||||
skill_name: str,
|
||||
skill_traces: List[Trace],
|
||||
) -> _OptimizerOutput:
|
||||
"""Run GEPAAgentOptimizer on the bucket. Same shape as _run_dspy."""
|
||||
from openjarvis.core.config import GEPAOptimizerConfig
|
||||
from openjarvis.learning.agents.gepa_optimizer import GEPAAgentOptimizer
|
||||
|
||||
cfg = GEPAOptimizerConfig(min_traces=max(1, self._min_traces))
|
||||
optimizer = GEPAAgentOptimizer(cfg)
|
||||
|
||||
class _BucketStore:
|
||||
def __init__(self, _traces):
|
||||
self._traces = _traces
|
||||
|
||||
def list_traces(self, *, limit=100, **_kwargs):
|
||||
return list(self._traces[:limit])
|
||||
|
||||
try:
|
||||
updates = optimizer.optimize(_BucketStore(skill_traces)) or {}
|
||||
except Exception as exc:
|
||||
LOGGER.warning(
|
||||
"GEPAAgentOptimizer raised for '%s' (using empty output): %s",
|
||||
skill_name,
|
||||
exc,
|
||||
)
|
||||
updates = {}
|
||||
|
||||
description = str(updates.get("system_prompt", "")) if updates else ""
|
||||
|
||||
# Parse GEPA's few_shot_examples output (matches DSPy path semantics)
|
||||
few_shot_raw = updates.get("few_shot_examples", []) if updates else []
|
||||
few_shot: List[Dict[str, str]] = []
|
||||
for item in few_shot_raw or []:
|
||||
if isinstance(item, dict):
|
||||
few_shot.append(
|
||||
{
|
||||
"input": str(item.get("input", "")),
|
||||
"output": str(item.get("output", "")),
|
||||
}
|
||||
)
|
||||
|
||||
# Fallback: if GEPA produced nothing, derive few-shot from top traces
|
||||
if not few_shot:
|
||||
top = sorted(
|
||||
skill_traces,
|
||||
key=lambda t: t.feedback if t.feedback is not None else 0.0,
|
||||
reverse=True,
|
||||
)[:4]
|
||||
for tr in top:
|
||||
if tr.query and tr.result:
|
||||
few_shot.append({"input": tr.query, "output": tr.result})
|
||||
return _OptimizerOutput(description=description, few_shot=few_shot)
|
||||
|
||||
|
||||
__all__ = ["SkillOptimizer", "SkillOptimizationResult"]
|
||||
@@ -98,6 +98,32 @@ class LearningOrchestrator:
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
|
||||
# 0. Skill optimization (Plan 2A C2) — runs INDEPENDENTLY of the
|
||||
# routing/agent SFT pipeline. Skills are tagged via trace metadata
|
||||
# rather than mined as SFT pairs, so they can be optimized even when
|
||||
# there's no other training data available.
|
||||
try:
|
||||
from openjarvis.core.config import load_config
|
||||
|
||||
cfg = load_config()
|
||||
skills_cfg = getattr(cfg.learning, "skills", None)
|
||||
if skills_cfg is not None and skills_cfg.auto_optimize:
|
||||
skill_results = self._maybe_optimize_skills(
|
||||
auto_optimize=True,
|
||||
optimizer=skills_cfg.optimizer,
|
||||
min_traces_per_skill=skills_cfg.min_traces_per_skill,
|
||||
)
|
||||
if skill_results is not None:
|
||||
result["skill_optimization"] = {
|
||||
name: {
|
||||
"status": r.status,
|
||||
"trace_count": r.trace_count,
|
||||
}
|
||||
for name, r in skill_results.items()
|
||||
}
|
||||
except Exception as exc:
|
||||
logger.warning("Skill auto-optimize probe failed: %s", exc)
|
||||
|
||||
# 1. Mine training data from traces
|
||||
sft_pairs = self._miner.extract_sft_pairs(agent=agent_id)
|
||||
routing_pairs = self._miner.extract_routing_pairs(agent=agent_id)
|
||||
@@ -194,5 +220,37 @@ class LearningOrchestrator:
|
||||
logger.warning("LoRA training failed: %s", exc)
|
||||
return {"status": "error", "reason": str(exc)}
|
||||
|
||||
def _maybe_optimize_skills(
|
||||
self,
|
||||
*,
|
||||
auto_optimize: bool = False,
|
||||
optimizer: str = "dspy",
|
||||
min_traces_per_skill: int = 20,
|
||||
) -> Optional[dict]:
|
||||
"""Optionally run the skill optimizer.
|
||||
|
||||
Called from :meth:`run` when ``learning.skills.auto_optimize`` is
|
||||
true. Returns the per-skill result dict or ``None`` if disabled.
|
||||
"""
|
||||
if not auto_optimize:
|
||||
return None
|
||||
try:
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.learning.agents.skill_optimizer import SkillOptimizer
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover()
|
||||
opt = SkillOptimizer(
|
||||
min_traces_per_skill=min_traces_per_skill,
|
||||
optimizer=optimizer,
|
||||
)
|
||||
return opt.optimize(self._trace_store, mgr)
|
||||
except Exception as exc:
|
||||
import logging
|
||||
|
||||
logging.getLogger(__name__).warning("Skill auto-optimize failed: %s", exc)
|
||||
return None
|
||||
|
||||
|
||||
__all__ = ["LearningOrchestrator"]
|
||||
|
||||
@@ -17,6 +17,9 @@ class SystemPromptBuilder:
|
||||
skill_index: Optional[List[Tuple[str, str]]] = None,
|
||||
session_context: Optional[str] = None,
|
||||
previous_state: Optional[str] = None,
|
||||
skill_catalog_xml: Optional[str] = None,
|
||||
skill_few_shot: Optional[List[str]] = None,
|
||||
skill_few_shot_examples: Optional[List[str]] = None,
|
||||
) -> None:
|
||||
self._agent_template = agent_template
|
||||
self._mf_config = memory_files_config or MemoryFilesConfig()
|
||||
@@ -24,6 +27,12 @@ class SystemPromptBuilder:
|
||||
self._skill_index = skill_index or []
|
||||
self._session_context = session_context
|
||||
self._previous_state = previous_state
|
||||
self._skill_catalog_xml = skill_catalog_xml
|
||||
# Allow either name; skill_few_shot_examples is the Plan 2A canonical name.
|
||||
if skill_few_shot_examples is not None:
|
||||
self._skill_few_shot = list(skill_few_shot_examples)
|
||||
else:
|
||||
self._skill_few_shot = list(skill_few_shot or [])
|
||||
self._frozen_prefix: Optional[str] = None
|
||||
|
||||
def build(self) -> str:
|
||||
@@ -57,7 +66,10 @@ class SystemPromptBuilder:
|
||||
)
|
||||
if user:
|
||||
sections.append(f"## User Profile\n\n{user}")
|
||||
if self._skill_index:
|
||||
# XML skill catalog (preferred over legacy markdown list)
|
||||
if self._skill_catalog_xml:
|
||||
sections.append("## Available Skills\n\n" + self._skill_catalog_xml)
|
||||
elif self._skill_index:
|
||||
skill_lines = []
|
||||
for name, desc in self._skill_index:
|
||||
truncated = desc[: self._sp_config.skill_desc_max_chars]
|
||||
@@ -65,6 +77,9 @@ class SystemPromptBuilder:
|
||||
truncated = truncated[:-3] + "..."
|
||||
skill_lines.append(f"- **{name}**: {truncated}")
|
||||
sections.append("## Available Skills\n\n" + "\n".join(skill_lines))
|
||||
if self._skill_few_shot:
|
||||
examples = "\n\n".join(self._skill_few_shot)
|
||||
sections.append("## Skill Examples\n\n" + examples)
|
||||
return "\n\n".join(sections)
|
||||
|
||||
def _load_file(self, path_str: str, max_chars: int) -> str:
|
||||
|
||||
@@ -1,8 +1,46 @@
|
||||
"""Skill system — reusable multi-tool compositions."""
|
||||
|
||||
from openjarvis.skills.executor import SkillExecutor
|
||||
from openjarvis.skills.loader import load_skill
|
||||
from openjarvis.skills.dependency import (
|
||||
DependencyCycleError,
|
||||
DepthExceededError,
|
||||
build_dependency_graph,
|
||||
compute_capability_union,
|
||||
validate_dependencies,
|
||||
)
|
||||
from openjarvis.skills.executor import SkillExecutor, SkillResult
|
||||
from openjarvis.skills.importer import ImportResult, SkillImporter
|
||||
from openjarvis.skills.loader import (
|
||||
discover_skills,
|
||||
load_skill,
|
||||
load_skill_directory,
|
||||
load_skill_markdown,
|
||||
)
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
from openjarvis.skills.parser import SkillParseError, SkillParser
|
||||
from openjarvis.skills.tool_adapter import SkillTool
|
||||
from openjarvis.skills.tool_translator import TOOL_TRANSLATION, ToolTranslator
|
||||
from openjarvis.skills.types import SkillManifest, SkillStep
|
||||
|
||||
__all__ = ["SkillExecutor", "SkillManifest", "SkillStep", "SkillTool", "load_skill"]
|
||||
__all__ = [
|
||||
"DependencyCycleError",
|
||||
"DepthExceededError",
|
||||
"ImportResult",
|
||||
"SkillExecutor",
|
||||
"SkillImporter",
|
||||
"SkillManager",
|
||||
"SkillManifest",
|
||||
"SkillParseError",
|
||||
"SkillParser",
|
||||
"SkillResult",
|
||||
"SkillStep",
|
||||
"SkillTool",
|
||||
"TOOL_TRANSLATION",
|
||||
"ToolTranslator",
|
||||
"build_dependency_graph",
|
||||
"compute_capability_union",
|
||||
"discover_skills",
|
||||
"load_skill",
|
||||
"load_skill_directory",
|
||||
"load_skill_markdown",
|
||||
"validate_dependencies",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Dependency graph: cycle detection, topological sort, capability union."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import deque
|
||||
from typing import TYPE_CHECKING, Dict, List, Set
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from openjarvis.skills.types import SkillManifest
|
||||
|
||||
|
||||
class DependencyCycleError(Exception):
|
||||
"""Raised when a cycle is detected in the skill dependency graph."""
|
||||
|
||||
|
||||
class DepthExceededError(Exception):
|
||||
"""Raised when the dependency depth exceeds the configured maximum."""
|
||||
|
||||
|
||||
def build_dependency_graph(skills: Dict[str, "SkillManifest"]) -> Dict[str, Set[str]]:
|
||||
"""Build a directed graph of skill dependencies.
|
||||
|
||||
Each key maps to the set of skill names it directly depends on, drawn from
|
||||
both the ``depends`` field and any ``skill_name`` references in steps.
|
||||
Only edges pointing to skills that are present in the *skills* dict are
|
||||
included (unknown refs are silently dropped so that the graph stays clean
|
||||
for topological analysis).
|
||||
|
||||
Args:
|
||||
skills: Mapping of skill name → SkillManifest.
|
||||
|
||||
Returns:
|
||||
Dict mapping each skill name to the set of its direct dependencies.
|
||||
"""
|
||||
graph: Dict[str, Set[str]] = {name: set() for name in skills}
|
||||
|
||||
for name, manifest in skills.items():
|
||||
# Explicit depends list
|
||||
for dep in manifest.depends:
|
||||
if dep in skills:
|
||||
graph[name].add(dep)
|
||||
|
||||
# Implicit deps via step skill_name references
|
||||
for step in manifest.steps:
|
||||
if step.skill_name and step.skill_name in skills:
|
||||
graph[name].add(step.skill_name)
|
||||
|
||||
return graph
|
||||
|
||||
|
||||
def validate_dependencies(
|
||||
skills: Dict[str, "SkillManifest"],
|
||||
*,
|
||||
max_depth: int = 5,
|
||||
) -> List[str]:
|
||||
"""Validate the skill dependency graph and return a topological ordering.
|
||||
|
||||
Uses Kahn's algorithm for topological sort and cycle detection. After a
|
||||
valid ordering is found, a DFS checks that no skill's transitive dependency
|
||||
chain exceeds *max_depth*. Dependencies that reference unknown skills are
|
||||
silently skipped.
|
||||
|
||||
Args:
|
||||
skills: Mapping of skill name → SkillManifest.
|
||||
max_depth: Maximum allowed dependency chain depth (default 5).
|
||||
|
||||
Returns:
|
||||
List of skill names in valid topological order (dependencies first).
|
||||
|
||||
Raises:
|
||||
DependencyCycleError: If the graph contains a cycle.
|
||||
DepthExceededError: If any skill's dependency chain depth exceeds max_depth.
|
||||
"""
|
||||
graph = build_dependency_graph(skills)
|
||||
|
||||
# --- Kahn's algorithm ---
|
||||
in_degree: Dict[str, int] = {name: 0 for name in graph}
|
||||
for name in graph:
|
||||
for dep in graph[name]:
|
||||
in_degree[dep] # ensure present (already is, but be explicit)
|
||||
|
||||
# Count how many skills depend on each node
|
||||
reverse: Dict[str, Set[str]] = {name: set() for name in graph}
|
||||
for name, deps in graph.items():
|
||||
for dep in deps:
|
||||
reverse[dep].add(name)
|
||||
# re-compute in_degree from scratch below
|
||||
|
||||
# Compute in-degree: number of dependencies each skill has (within the graph)
|
||||
in_degree = {name: len(deps) for name, deps in graph.items()}
|
||||
|
||||
queue: deque[str] = deque(name for name, deg in in_degree.items() if deg == 0)
|
||||
order: List[str] = []
|
||||
|
||||
while queue:
|
||||
node = queue.popleft()
|
||||
order.append(node)
|
||||
for dependent in reverse[node]:
|
||||
in_degree[dependent] -= 1
|
||||
if in_degree[dependent] == 0:
|
||||
queue.append(dependent)
|
||||
|
||||
if len(order) != len(graph):
|
||||
raise DependencyCycleError(
|
||||
"Cycle detected in skill dependency graph. "
|
||||
f"Skills involved: {set(graph) - set(order)}"
|
||||
)
|
||||
|
||||
# --- Depth enforcement via DFS ---
|
||||
depth_cache: Dict[str, int] = {}
|
||||
|
||||
def _depth(name: str, visiting: Set[str]) -> int:
|
||||
if name in depth_cache:
|
||||
return depth_cache[name]
|
||||
deps = graph.get(name, set())
|
||||
if not deps:
|
||||
depth_cache[name] = 1
|
||||
return 1
|
||||
visiting.add(name)
|
||||
max_child = max(_depth(d, visiting) for d in deps)
|
||||
visiting.discard(name)
|
||||
result = max_child + 1
|
||||
depth_cache[name] = result
|
||||
return result
|
||||
|
||||
for name in graph:
|
||||
d = _depth(name, set())
|
||||
if d > max_depth:
|
||||
raise DepthExceededError(
|
||||
f"Skill '{name}' depth {d} exceeds max_depth={max_depth}"
|
||||
)
|
||||
|
||||
return order
|
||||
|
||||
|
||||
def compute_capability_union(
|
||||
skill_name: str,
|
||||
skills: Dict[str, "SkillManifest"],
|
||||
) -> List[str]:
|
||||
"""Compute the union of all required_capabilities transitively needed by a skill.
|
||||
|
||||
Performs a DFS over the dependency graph, collecting ``required_capabilities``
|
||||
from the skill itself and all of its transitive dependencies. Duplicates are
|
||||
removed while preserving a deterministic order (first-seen wins).
|
||||
|
||||
Args:
|
||||
skill_name: Name of the root skill to start from.
|
||||
skills: Mapping of skill name → SkillManifest.
|
||||
|
||||
Returns:
|
||||
Deduplicated list of capability strings. Returns an empty list if the
|
||||
skill is not found.
|
||||
"""
|
||||
if skill_name not in skills:
|
||||
return []
|
||||
|
||||
graph = build_dependency_graph(skills)
|
||||
seen_skills: Set[str] = set()
|
||||
seen_caps: Set[str] = set()
|
||||
caps_ordered: List[str] = []
|
||||
|
||||
def _dfs(name: str) -> None:
|
||||
if name in seen_skills:
|
||||
return
|
||||
seen_skills.add(name)
|
||||
manifest = skills.get(name)
|
||||
if manifest is None:
|
||||
return
|
||||
for cap in manifest.required_capabilities:
|
||||
if cap not in seen_caps:
|
||||
seen_caps.add(cap)
|
||||
caps_ordered.append(cap)
|
||||
for dep in graph.get(name, set()):
|
||||
_dfs(dep)
|
||||
|
||||
_dfs(skill_name)
|
||||
return caps_ordered
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DependencyCycleError",
|
||||
"DepthExceededError",
|
||||
"build_dependency_graph",
|
||||
"validate_dependencies",
|
||||
"compute_capability_union",
|
||||
]
|
||||
@@ -5,7 +5,7 @@ from __future__ import annotations
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
from openjarvis.core.events import EventBus, EventType
|
||||
from openjarvis.core.types import ToolCall, ToolResult
|
||||
@@ -21,6 +21,10 @@ class SkillResult:
|
||||
context: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
# Resolver callback: given a skill name and the current context, returns a SkillResult.
|
||||
SkillResolver = Callable[[str, Dict[str, Any]], SkillResult]
|
||||
|
||||
|
||||
class SkillExecutor:
|
||||
"""Execute a skill manifest step-by-step.
|
||||
|
||||
@@ -36,6 +40,11 @@ class SkillExecutor:
|
||||
) -> None:
|
||||
self._tool_executor = tool_executor
|
||||
self._bus = bus
|
||||
self._skill_resolver: Optional[SkillResolver] = None
|
||||
|
||||
def set_skill_resolver(self, resolver: SkillResolver) -> None:
|
||||
"""Register a callback used to delegate ``skill_name`` steps."""
|
||||
self._skill_resolver = resolver
|
||||
|
||||
def run(
|
||||
self,
|
||||
@@ -54,25 +63,34 @@ class SkillExecutor:
|
||||
)
|
||||
|
||||
for i, step in enumerate(manifest.steps):
|
||||
step_id = step.tool_name or step.skill_name
|
||||
|
||||
# Render template
|
||||
try:
|
||||
rendered = self._render_template(step.arguments_template, ctx)
|
||||
except Exception as exc:
|
||||
result = ToolResult(
|
||||
tool_name=step.tool_name,
|
||||
tool_name=step_id,
|
||||
content=f"Template rendering error: {exc}",
|
||||
success=False,
|
||||
)
|
||||
all_results.append(result)
|
||||
break
|
||||
|
||||
# Execute
|
||||
tool_call = ToolCall(
|
||||
id=f"skill_{manifest.name}_{i}",
|
||||
name=step.tool_name,
|
||||
arguments=rendered,
|
||||
)
|
||||
result = self._tool_executor.execute(tool_call)
|
||||
if step.skill_name:
|
||||
# Delegate to sub-skill resolver
|
||||
result = self._run_sub_skill(
|
||||
step.skill_name, rendered, ctx, manifest.name, i
|
||||
)
|
||||
else:
|
||||
# Execute via tool executor
|
||||
tool_call = ToolCall(
|
||||
id=f"skill_{manifest.name}_{i}",
|
||||
name=step.tool_name,
|
||||
arguments=rendered,
|
||||
)
|
||||
result = self._tool_executor.execute(tool_call)
|
||||
|
||||
all_results.append(result)
|
||||
|
||||
if not result.success:
|
||||
@@ -97,6 +115,54 @@ class SkillExecutor:
|
||||
context=ctx,
|
||||
)
|
||||
|
||||
def _run_sub_skill(
|
||||
self,
|
||||
skill_name: str,
|
||||
rendered_args: str,
|
||||
parent_ctx: Dict[str, Any],
|
||||
parent_skill: str,
|
||||
step_index: int,
|
||||
) -> ToolResult:
|
||||
"""Invoke the skill resolver and convert its result to a ToolResult."""
|
||||
if self._skill_resolver is None:
|
||||
return ToolResult(
|
||||
tool_name=skill_name,
|
||||
content=f"No skill resolver registered for sub-skill '{skill_name}'",
|
||||
success=False,
|
||||
)
|
||||
|
||||
# Parse rendered args and merge into a copy of the parent context
|
||||
try:
|
||||
args: Dict[str, Any] = json.loads(rendered_args)
|
||||
except json.JSONDecodeError:
|
||||
args = {}
|
||||
|
||||
child_ctx = {**parent_ctx, **args}
|
||||
|
||||
sub_result: SkillResult = self._skill_resolver(skill_name, child_ctx)
|
||||
|
||||
# Expose the final context value under the first output_key, or the
|
||||
# last step's content, as the synthetic "content" of this ToolResult.
|
||||
content: Any = ""
|
||||
if sub_result.context:
|
||||
# Return the last stored value from the child's context that is
|
||||
# not already in the parent context (i.e. the output of the sub-skill).
|
||||
new_keys = [k for k in sub_result.context if k not in parent_ctx]
|
||||
if new_keys:
|
||||
content = sub_result.context[new_keys[-1]]
|
||||
else:
|
||||
content = (
|
||||
list(sub_result.context.values())[-1] if sub_result.context else ""
|
||||
)
|
||||
elif sub_result.step_results:
|
||||
content = sub_result.step_results[-1].content
|
||||
|
||||
return ToolResult(
|
||||
tool_name=skill_name,
|
||||
content=content,
|
||||
success=sub_result.success,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _render_template(template: str, ctx: Dict[str, Any]) -> str:
|
||||
"""Simple {key} placeholder rendering."""
|
||||
@@ -111,4 +177,4 @@ class SkillExecutor:
|
||||
return re.sub(r"\{(\w+)\}", _replace, template)
|
||||
|
||||
|
||||
__all__ = ["SkillExecutor", "SkillResult"]
|
||||
__all__ = ["SkillExecutor", "SkillResolver", "SkillResult"]
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
"""SkillImporter — install ResolvedSkill instances into ~/.openjarvis/skills/.
|
||||
|
||||
Steps performed by ``import_skill``:
|
||||
|
||||
1. Parse the source SKILL.md through SkillParser (strict + tolerant).
|
||||
2. Translate tool references in the markdown body via ToolTranslator.
|
||||
3. Decide on scripts (default-skip; opt-in via with_scripts=True).
|
||||
4. Write to disk at <target_root>/<source>/<name>/:
|
||||
- translated SKILL.md
|
||||
- references/, assets/, templates/ (always copied)
|
||||
- scripts/ (only if approved)
|
||||
- .source metadata file
|
||||
5. Return ImportResult with status, warnings, translated/missing tools.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
import yaml
|
||||
|
||||
from openjarvis.skills.parser import SkillParser
|
||||
from openjarvis.skills.sources.base import ResolvedSkill
|
||||
from openjarvis.skills.tool_translator import ToolTranslator
|
||||
|
||||
# Subdirectories of a skill that are always copied (never gated by --with-scripts)
|
||||
COPIED_SUBDIRS = ("references", "assets", "templates")
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ImportResult:
|
||||
"""Result of importing a single skill."""
|
||||
|
||||
success: bool = True
|
||||
skipped: bool = False
|
||||
target_path: Path | None = None
|
||||
translated_tools: List[str] = field(default_factory=list)
|
||||
untranslated_tools: List[str] = field(default_factory=list)
|
||||
scripts_imported: bool = False
|
||||
warnings: List[str] = field(default_factory=list)
|
||||
|
||||
|
||||
class SkillImporter:
|
||||
"""Install resolved skills into the user skills directory."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
parser: SkillParser,
|
||||
tool_translator: ToolTranslator,
|
||||
target_root: Path | None = None,
|
||||
) -> None:
|
||||
self._parser = parser
|
||||
self._translator = tool_translator
|
||||
if target_root is None:
|
||||
target_root = Path("~/.openjarvis/skills/").expanduser()
|
||||
self._target_root = Path(target_root)
|
||||
|
||||
def import_skill(
|
||||
self,
|
||||
resolved: ResolvedSkill,
|
||||
*,
|
||||
with_scripts: bool = False,
|
||||
force: bool = False,
|
||||
) -> ImportResult:
|
||||
"""Install *resolved* into ``<target_root>/<source>/<name>/``.
|
||||
|
||||
Returns an :class:`ImportResult` with status, paths, translated
|
||||
tools, untranslated tools, and warnings.
|
||||
"""
|
||||
result = ImportResult()
|
||||
target_dir = self._target_root / resolved.source / resolved.name
|
||||
result.target_path = target_dir
|
||||
|
||||
# Skip if already installed and force is False
|
||||
if target_dir.exists() and not force:
|
||||
result.skipped = True
|
||||
result.warnings.append(
|
||||
f"Skill already installed at {target_dir} (use force=True to overwrite)"
|
||||
)
|
||||
return result
|
||||
|
||||
# 1. Parse source SKILL.md
|
||||
source_md = resolved.path / "SKILL.md"
|
||||
if not source_md.exists():
|
||||
source_md = resolved.path / "skill.md"
|
||||
if not source_md.exists():
|
||||
result.success = False
|
||||
result.warnings.append(f"No SKILL.md found in {resolved.path}")
|
||||
return result
|
||||
|
||||
try:
|
||||
frontmatter, body = self._read_skill_md(source_md)
|
||||
self._parser.parse_frontmatter(frontmatter, markdown_content=body)
|
||||
except Exception as exc:
|
||||
result.success = False
|
||||
result.warnings.append(f"Parse error: {exc}")
|
||||
return result
|
||||
|
||||
# 2. Translate tool references
|
||||
translated_body, untranslated = self._translator.translate_markdown(body)
|
||||
result.untranslated_tools = untranslated
|
||||
# Compute the list of translations actually applied
|
||||
applied: List[str] = []
|
||||
for ext, internal in self._translator._table.items():
|
||||
if ext in body and ext not in translated_body:
|
||||
applied.append(f"{ext}->{internal}")
|
||||
result.translated_tools = applied
|
||||
|
||||
# 3. Write the target directory
|
||||
if target_dir.exists():
|
||||
shutil.rmtree(target_dir)
|
||||
target_dir.mkdir(parents=True)
|
||||
|
||||
# 3a. Translated SKILL.md
|
||||
new_md = self._render_skill_md(frontmatter, translated_body)
|
||||
(target_dir / "SKILL.md").write_text(new_md, encoding="utf-8")
|
||||
|
||||
# 3b. Always-copied subdirs
|
||||
for subdir in COPIED_SUBDIRS:
|
||||
src_sub = resolved.path / subdir
|
||||
if src_sub.exists():
|
||||
shutil.copytree(src_sub, target_dir / subdir)
|
||||
|
||||
# 3c. Scripts (gated by with_scripts)
|
||||
scripts_src = resolved.path / "scripts"
|
||||
if scripts_src.exists() and with_scripts:
|
||||
shutil.copytree(scripts_src, target_dir / "scripts")
|
||||
result.scripts_imported = True
|
||||
elif scripts_src.exists():
|
||||
result.warnings.append(
|
||||
"Skipped scripts/ directory (use with_scripts=True to import)"
|
||||
)
|
||||
|
||||
# 4. Write .source metadata
|
||||
self._write_source_metadata(target_dir, resolved, result)
|
||||
|
||||
return result
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _read_skill_md(self, path: Path) -> tuple[dict, str]:
|
||||
"""Parse a SKILL.md file into (frontmatter dict, markdown body)."""
|
||||
raw = path.read_text(encoding="utf-8")
|
||||
if not raw.startswith("---"):
|
||||
return {}, raw
|
||||
rest = raw[3:].lstrip("\n")
|
||||
end = rest.find("\n---")
|
||||
if end == -1:
|
||||
return {}, raw
|
||||
fm_text = rest[:end]
|
||||
body = rest[end + 4 :].lstrip("\n")
|
||||
try:
|
||||
fm = yaml.safe_load(fm_text)
|
||||
if not isinstance(fm, dict):
|
||||
fm = {}
|
||||
except yaml.YAMLError:
|
||||
fm = {}
|
||||
return fm, body
|
||||
|
||||
def _render_skill_md(self, frontmatter: dict, body: str) -> str:
|
||||
"""Re-serialize a SKILL.md file from frontmatter dict + body."""
|
||||
fm_text = yaml.safe_dump(frontmatter, sort_keys=False, default_flow_style=False)
|
||||
return f"---\n{fm_text}---\n\n{body}"
|
||||
|
||||
def _write_source_metadata(
|
||||
self,
|
||||
target_dir: Path,
|
||||
resolved: ResolvedSkill,
|
||||
result: ImportResult,
|
||||
) -> None:
|
||||
"""Write the .source TOML provenance file."""
|
||||
installed_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
translated_str = ", ".join(f'"{t}"' for t in result.translated_tools)
|
||||
missing_str = ", ".join(f'"{t}"' for t in result.untranslated_tools)
|
||||
scripts_lower = "true" if result.scripts_imported else "false"
|
||||
|
||||
content = (
|
||||
f'source = "{resolved.source}:{resolved.name}"\n'
|
||||
f'commit = "{resolved.commit}"\n'
|
||||
f'category = "{resolved.category}"\n'
|
||||
f'installed_at = "{installed_at}"\n'
|
||||
f"translated_tools = [{translated_str}]\n"
|
||||
f"missing_tools = [{missing_str}]\n"
|
||||
f"scripts_imported = {scripts_lower}\n"
|
||||
)
|
||||
(target_dir / ".source").write_text(content, encoding="utf-8")
|
||||
|
||||
|
||||
__all__ = ["ImportResult", "SkillImporter"]
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Skill index — load and search a remote/local skill catalog."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
try:
|
||||
import tomllib
|
||||
except ModuleNotFoundError:
|
||||
import tomli as tomllib # type: ignore[no-redef]
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SkillIndexEntry:
|
||||
"""A single entry in the skill index catalog."""
|
||||
|
||||
name: str
|
||||
version: str
|
||||
description: str
|
||||
author: str
|
||||
source: str
|
||||
sha256: str
|
||||
tags: list[str] = field(default_factory=list)
|
||||
required_capabilities: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
class SkillIndex:
|
||||
"""Catalog of available skills loaded from an ``index.toml`` file.
|
||||
|
||||
``index.toml`` format::
|
||||
|
||||
[[skills]]
|
||||
name = "research"
|
||||
version = "0.1.0"
|
||||
description = "Research a topic"
|
||||
author = "openjarvis"
|
||||
source = "github.com/openjarvis/skills/research"
|
||||
sha256 = "abc123"
|
||||
tags = ["research"]
|
||||
required_capabilities = ["network:fetch"]
|
||||
"""
|
||||
|
||||
def __init__(self, index_dir: str | Path) -> None:
|
||||
self._index_dir = Path(index_dir)
|
||||
self._entries: dict[str, SkillIndexEntry] = {}
|
||||
self._load()
|
||||
|
||||
def _load(self) -> None:
|
||||
"""Parse ``index.toml`` from the index directory if it exists."""
|
||||
index_file = self._index_dir / "index.toml"
|
||||
if not index_file.exists():
|
||||
return
|
||||
|
||||
with open(index_file, "rb") as fh:
|
||||
data = tomllib.load(fh)
|
||||
|
||||
for skill_data in data.get("skills", []):
|
||||
entry = SkillIndexEntry(
|
||||
name=skill_data.get("name", ""),
|
||||
version=skill_data.get("version", "0.1.0"),
|
||||
description=skill_data.get("description", ""),
|
||||
author=skill_data.get("author", ""),
|
||||
source=skill_data.get("source", ""),
|
||||
sha256=skill_data.get("sha256", ""),
|
||||
tags=skill_data.get("tags", []),
|
||||
required_capabilities=skill_data.get("required_capabilities", []),
|
||||
)
|
||||
self._entries[entry.name] = entry
|
||||
|
||||
@property
|
||||
def entries(self) -> dict[str, SkillIndexEntry]:
|
||||
"""Return all entries keyed by skill name."""
|
||||
return self._entries
|
||||
|
||||
def search(self, query: str) -> list[SkillIndexEntry]:
|
||||
"""Return entries whose name, description, or tags match *query*.
|
||||
|
||||
Matching is case-insensitive substring search across name, description,
|
||||
and each tag value.
|
||||
"""
|
||||
q = query.lower()
|
||||
results: list[SkillIndexEntry] = []
|
||||
for entry in self._entries.values():
|
||||
if (
|
||||
q in entry.name.lower()
|
||||
or q in entry.description.lower()
|
||||
or any(q in tag.lower() for tag in entry.tags)
|
||||
):
|
||||
results.append(entry)
|
||||
return results
|
||||
|
||||
def get(self, name: str) -> Optional[SkillIndexEntry]:
|
||||
"""Look up a skill by exact name, returning ``None`` if not found."""
|
||||
return self._entries.get(name)
|
||||
|
||||
|
||||
__all__ = ["SkillIndexEntry", "SkillIndex"]
|
||||
@@ -5,6 +5,8 @@ from __future__ import annotations
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
import yaml
|
||||
|
||||
from openjarvis.skills.types import SkillManifest, SkillStep
|
||||
|
||||
try:
|
||||
@@ -12,6 +14,34 @@ try:
|
||||
except ModuleNotFoundError:
|
||||
import tomli as tomllib # type: ignore[no-redef]
|
||||
|
||||
import logging
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _read_source_metadata(path: Path) -> dict:
|
||||
"""Read .source TOML file if present, returning the parsed dict.
|
||||
|
||||
Returns an empty dict if the file is missing or malformed. Never
|
||||
raises — bad sidecar files should never break skill loading. Logs
|
||||
a warning when a malformed file is encountered so users can debug
|
||||
why their imported source provenance is missing.
|
||||
"""
|
||||
source_path = path / ".source"
|
||||
if not source_path.exists():
|
||||
return {}
|
||||
try:
|
||||
with open(source_path, "rb") as fh:
|
||||
return tomllib.load(fh)
|
||||
except Exception as exc:
|
||||
LOGGER.warning(
|
||||
"Malformed .source sidecar at %s (skill source provenance "
|
||||
"will be missing): %s",
|
||||
source_path,
|
||||
exc,
|
||||
)
|
||||
return {}
|
||||
|
||||
|
||||
def load_skill(
|
||||
path: str | Path,
|
||||
@@ -53,7 +83,8 @@ def load_skill(
|
||||
for step_data in skill_data.get("steps", []):
|
||||
steps.append(
|
||||
SkillStep(
|
||||
tool_name=step_data["tool_name"],
|
||||
tool_name=step_data.get("tool_name", ""),
|
||||
skill_name=step_data.get("skill_name", ""),
|
||||
arguments_template=step_data.get("arguments_template", "{}"),
|
||||
output_key=step_data.get("output_key", ""),
|
||||
)
|
||||
@@ -68,6 +99,10 @@ def load_skill(
|
||||
required_capabilities=skill_data.get("required_capabilities", []),
|
||||
signature=skill_data.get("signature", ""),
|
||||
metadata=skill_data.get("metadata", {}),
|
||||
tags=skill_data.get("tags", []),
|
||||
depends=skill_data.get("depends", []),
|
||||
user_invocable=skill_data.get("user_invocable", True),
|
||||
disable_model_invocation=skill_data.get("disable_model_invocation", False),
|
||||
)
|
||||
|
||||
# Verify signature if requested
|
||||
@@ -108,18 +143,177 @@ def load_skill(
|
||||
return manifest
|
||||
|
||||
|
||||
def load_skill_markdown(path: str | Path) -> SkillManifest:
|
||||
"""Load a skill manifest from a SKILL.md file via SkillParser.
|
||||
|
||||
Parses YAML frontmatter (between ``---`` delimiters) and the markdown
|
||||
body, then runs them through :class:`SkillParser` for strict validation
|
||||
and tolerant field mapping.
|
||||
"""
|
||||
from openjarvis.skills.parser import SkillParseError, SkillParser
|
||||
|
||||
path = Path(path)
|
||||
raw = path.read_text(encoding="utf-8")
|
||||
|
||||
frontmatter: dict = {}
|
||||
markdown_content = raw
|
||||
|
||||
if raw.startswith("---"):
|
||||
rest = raw[3:]
|
||||
if rest.startswith("\n"):
|
||||
rest = rest[1:]
|
||||
end_idx = rest.find("\n---")
|
||||
if end_idx != -1:
|
||||
yaml_block = rest[:end_idx]
|
||||
try:
|
||||
frontmatter = yaml.safe_load(yaml_block) or {}
|
||||
except yaml.YAMLError:
|
||||
frontmatter = {}
|
||||
after = rest[end_idx + 4 :]
|
||||
if after.startswith("\n"):
|
||||
after = after[1:]
|
||||
markdown_content = after
|
||||
|
||||
# Default name to file stem if missing (preserves legacy behavior)
|
||||
if "name" not in frontmatter:
|
||||
frontmatter["name"] = path.stem
|
||||
if "description" not in frontmatter:
|
||||
frontmatter["description"] = frontmatter.get("name", path.stem)
|
||||
|
||||
parser = SkillParser()
|
||||
try:
|
||||
return parser.parse_frontmatter(frontmatter, markdown_content=markdown_content)
|
||||
except SkillParseError:
|
||||
# Legacy fallback — build manifest directly without strict validation
|
||||
return SkillManifest(
|
||||
name=str(frontmatter.get("name", path.stem)),
|
||||
description=str(frontmatter.get("description", "")),
|
||||
version=str(frontmatter.get("version", "0.1.0")),
|
||||
author=str(frontmatter.get("author", "")),
|
||||
tags=list(frontmatter.get("tags", []) or []),
|
||||
required_capabilities=list(
|
||||
frontmatter.get("required_capabilities", []) or []
|
||||
),
|
||||
user_invocable=bool(frontmatter.get("user_invocable", True)),
|
||||
disable_model_invocation=bool(
|
||||
frontmatter.get("disable_model_invocation", False)
|
||||
),
|
||||
markdown_content=markdown_content,
|
||||
)
|
||||
|
||||
|
||||
def load_skill_directory(path: str | Path) -> SkillManifest:
|
||||
"""Load a skill from a directory containing ``skill.toml`` and/or ``SKILL.md``.
|
||||
|
||||
- If only ``skill.toml`` is present the manifest is loaded from TOML.
|
||||
- If only ``SKILL.md`` is present the manifest is loaded from markdown.
|
||||
- If both are present the TOML manifest takes precedence for structured
|
||||
fields and ``markdown_content`` is merged in from the markdown file.
|
||||
- If neither is present a ``FileNotFoundError`` is raised.
|
||||
"""
|
||||
path = Path(path)
|
||||
toml_path = path / "skill.toml"
|
||||
md_path = path / "SKILL.md"
|
||||
|
||||
has_toml = toml_path.exists()
|
||||
has_md = md_path.exists()
|
||||
|
||||
if not has_toml and not has_md:
|
||||
raise FileNotFoundError(
|
||||
f"No skill.toml or SKILL.md found in directory '{path}'"
|
||||
)
|
||||
|
||||
if has_toml:
|
||||
manifest = load_skill(toml_path)
|
||||
if has_md:
|
||||
md_manifest = load_skill_markdown(md_path)
|
||||
# Merge markdown content into the TOML-sourced manifest
|
||||
manifest = SkillManifest(
|
||||
name=manifest.name,
|
||||
version=manifest.version,
|
||||
description=manifest.description,
|
||||
author=manifest.author,
|
||||
steps=manifest.steps,
|
||||
required_capabilities=manifest.required_capabilities,
|
||||
signature=manifest.signature,
|
||||
metadata=manifest.metadata,
|
||||
tags=manifest.tags,
|
||||
depends=manifest.depends,
|
||||
user_invocable=manifest.user_invocable,
|
||||
disable_model_invocation=manifest.disable_model_invocation,
|
||||
markdown_content=md_manifest.markdown_content,
|
||||
)
|
||||
else:
|
||||
# Only markdown present
|
||||
manifest = load_skill_markdown(md_path)
|
||||
|
||||
# Promote .source file's source field into manifest.metadata.openjarvis.source
|
||||
source_data = _read_source_metadata(path)
|
||||
if source_data:
|
||||
# Extract just the source name (e.g. "hermes" from "hermes:apple-notes")
|
||||
source_str = source_data.get("source", "")
|
||||
source_name = source_str.partition(":")[0] if source_str else ""
|
||||
if source_name:
|
||||
new_metadata = dict(manifest.metadata) if manifest.metadata else {}
|
||||
oj = dict(new_metadata.get("openjarvis", {}) or {})
|
||||
oj["source"] = source_name
|
||||
new_metadata["openjarvis"] = oj
|
||||
manifest.metadata = new_metadata
|
||||
|
||||
return manifest
|
||||
|
||||
|
||||
def discover_skills(directory: str | Path) -> list[SkillManifest]:
|
||||
"""Scan directory for TOML skill files and load them."""
|
||||
"""Scan a directory for skill definitions and load them.
|
||||
|
||||
Handles three layouts:
|
||||
- Flat ``*.toml`` files directly inside *directory*.
|
||||
- Skill directories: ``<directory>/<name>/{skill.toml,SKILL.md}``.
|
||||
- Sourced layout: ``<directory>/<source>/<name>/{skill.toml,SKILL.md}``.
|
||||
"""
|
||||
directory = Path(directory).expanduser()
|
||||
if not directory.exists():
|
||||
return []
|
||||
manifests = []
|
||||
|
||||
manifests: list[SkillManifest] = []
|
||||
|
||||
# Flat *.toml files at the top level
|
||||
for toml_file in sorted(directory.glob("*.toml")):
|
||||
try:
|
||||
manifests.append(load_skill(toml_file))
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
# Walk one or two levels deep looking for skill packages
|
||||
for child in sorted(directory.iterdir()):
|
||||
if not child.is_dir():
|
||||
continue
|
||||
# Direct skill package: <child>/{skill.toml,SKILL.md}
|
||||
if (child / "skill.toml").exists() or (child / "SKILL.md").exists():
|
||||
try:
|
||||
manifests.append(load_skill_directory(child))
|
||||
except Exception:
|
||||
continue
|
||||
continue
|
||||
|
||||
# Sourced layout: <child>/<grandchild>/{skill.toml,SKILL.md}
|
||||
for grandchild in sorted(child.iterdir()):
|
||||
if not grandchild.is_dir():
|
||||
continue
|
||||
if (grandchild / "skill.toml").exists() or (
|
||||
grandchild / "SKILL.md"
|
||||
).exists():
|
||||
try:
|
||||
manifests.append(load_skill_directory(grandchild))
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return manifests
|
||||
|
||||
|
||||
__all__ = ["load_skill", "discover_skills"]
|
||||
__all__ = [
|
||||
"load_skill",
|
||||
"load_skill_markdown",
|
||||
"load_skill_directory",
|
||||
"discover_skills",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,396 @@
|
||||
"""SkillManager — coordinates skill discovery, catalog, tool wrapping, and execution."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.dependency import validate_dependencies
|
||||
from openjarvis.skills.executor import SkillExecutor, SkillResult
|
||||
from openjarvis.skills.loader import discover_skills
|
||||
from openjarvis.skills.tool_adapter import SkillTool
|
||||
from openjarvis.skills.types import SkillManifest
|
||||
from openjarvis.tools._stubs import BaseTool, ToolExecutor
|
||||
|
||||
|
||||
class SkillManager:
|
||||
"""Coordinate skill discovery, resolution, catalog generation, and execution.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
bus:
|
||||
Event bus for publishing lifecycle events.
|
||||
capability_policy:
|
||||
Optional capability policy passed through to tool executors.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
bus: EventBus,
|
||||
*,
|
||||
capability_policy: Optional[Any] = None,
|
||||
overlay_dir: Optional[Path] = None,
|
||||
) -> None:
|
||||
self._bus = bus
|
||||
self._capability_policy = capability_policy
|
||||
self._skills: Dict[str, SkillManifest] = {}
|
||||
self._tool_executor: Optional[ToolExecutor] = None
|
||||
if overlay_dir is None:
|
||||
# Try to read from config first; fall back to the default
|
||||
# ~/.openjarvis/learning/skills/ if config can't be loaded.
|
||||
try:
|
||||
from openjarvis.core.config import load_config
|
||||
|
||||
cfg = load_config()
|
||||
cfg_dir = getattr(
|
||||
getattr(cfg.learning, "skills", None),
|
||||
"overlay_dir",
|
||||
None,
|
||||
)
|
||||
if cfg_dir:
|
||||
overlay_dir = Path(cfg_dir).expanduser()
|
||||
except Exception:
|
||||
pass
|
||||
if overlay_dir is None:
|
||||
overlay_dir = Path(
|
||||
"~/.openjarvis/learning/skills/"
|
||||
).expanduser()
|
||||
self._overlay_dir = Path(overlay_dir).expanduser()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Discovery
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def discover(self, paths: Optional[List[Path]] = None) -> None:
|
||||
"""Scan directories in order and register skills.
|
||||
|
||||
First-seen name wins (workspace path listed first = highest precedence).
|
||||
After loading, the full dependency graph is validated and any
|
||||
sidecar overlays in ``overlay_dir`` are applied to discovered skills.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
paths:
|
||||
Directories to scan. If *None* or empty, no skills are loaded
|
||||
from disk — but ``_load_overlays()`` still runs (in case the
|
||||
caller had previously seeded ``self._skills`` directly).
|
||||
"""
|
||||
if paths:
|
||||
for directory in paths:
|
||||
manifests = discover_skills(directory)
|
||||
for manifest in manifests:
|
||||
# First-seen wins: do not overwrite an already-registered skill
|
||||
if manifest.name not in self._skills:
|
||||
self._skills[manifest.name] = manifest
|
||||
|
||||
# Validate the dependency graph after loading skills
|
||||
if self._skills:
|
||||
validate_dependencies(self._skills)
|
||||
|
||||
# Load optimization overlays (Plan 2A) — always runs, even when no
|
||||
# paths are provided, so callers can apply overlays to skills loaded
|
||||
# via other means.
|
||||
self._load_overlays()
|
||||
|
||||
def _load_overlays(self) -> None:
|
||||
"""Apply optimization overlays to discovered skills.
|
||||
|
||||
For each skill, look for ``<overlay_dir>/<skill-name>/optimized.toml``.
|
||||
If present, override the manifest description and stash few-shot
|
||||
examples under ``manifest.metadata.openjarvis.few_shot``.
|
||||
|
||||
Bad overlays are silently ignored — they should not break discovery.
|
||||
"""
|
||||
from openjarvis.skills.overlay import SkillOverlayLoader
|
||||
|
||||
loader = SkillOverlayLoader(self._overlay_dir)
|
||||
for name, manifest in self._skills.items():
|
||||
overlay = loader.load(name)
|
||||
if overlay is None:
|
||||
continue
|
||||
if overlay.description:
|
||||
manifest.description = overlay.description
|
||||
if overlay.few_shot:
|
||||
new_metadata = dict(manifest.metadata) if manifest.metadata else {}
|
||||
oj = dict(new_metadata.get("openjarvis", {}) or {})
|
||||
oj["few_shot"] = list(overlay.few_shot)
|
||||
new_metadata["openjarvis"] = oj
|
||||
manifest.metadata = new_metadata
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Resolve / introspect
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def resolve(self, name: str) -> SkillManifest:
|
||||
"""Return the manifest for a skill by name.
|
||||
|
||||
Raises
|
||||
------
|
||||
KeyError
|
||||
If *name* is not registered.
|
||||
"""
|
||||
if name not in self._skills:
|
||||
raise KeyError(f"Skill '{name}' not found")
|
||||
return self._skills[name]
|
||||
|
||||
def skill_names(self) -> List[str]:
|
||||
"""Return the names of all registered skills."""
|
||||
return list(self._skills.keys())
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Tool wrapping
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_skill_tools(
|
||||
self, *, tool_executor: Optional[ToolExecutor] = None
|
||||
) -> List[BaseTool]:
|
||||
"""Wrap each registered skill as a :class:`SkillTool` (a :class:`BaseTool`).
|
||||
|
||||
Parameters
|
||||
----------
|
||||
tool_executor:
|
||||
Tool executor to use when running skill pipelines. Falls back to
|
||||
the one set via :meth:`set_tool_executor` if not provided here.
|
||||
|
||||
Returns
|
||||
-------
|
||||
list[BaseTool]
|
||||
One :class:`SkillTool` per registered skill.
|
||||
"""
|
||||
executor = tool_executor or self._tool_executor
|
||||
tools: List[BaseTool] = []
|
||||
|
||||
for manifest in self._skills.values():
|
||||
real_executor = executor or _NullToolExecutor()
|
||||
skill_exec = SkillExecutor(real_executor, bus=self._bus)
|
||||
|
||||
# Wire sub-skill resolver so nested skill_name steps can delegate back
|
||||
skill_exec.set_skill_resolver(self._make_resolver())
|
||||
|
||||
skill_tool = SkillTool(manifest, skill_exec, skill_manager=self)
|
||||
tools.append(skill_tool)
|
||||
|
||||
return tools
|
||||
|
||||
def _make_resolver(self):
|
||||
"""Return a resolver callback that delegates sub-skill execution."""
|
||||
|
||||
def _resolver(name: str, context: Dict[str, Any]) -> SkillResult:
|
||||
manifest = self.resolve(name)
|
||||
skill_exec = SkillExecutor(
|
||||
self._tool_executor or _NullToolExecutor(),
|
||||
bus=self._bus,
|
||||
)
|
||||
skill_exec.set_skill_resolver(_resolver)
|
||||
return skill_exec.run(manifest, initial_context=context)
|
||||
|
||||
return _resolver
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Catalog
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_catalog_xml(self) -> str:
|
||||
"""Generate an ``<available_skills>`` XML catalog.
|
||||
|
||||
Skills with ``disable_model_invocation=True`` are excluded so that
|
||||
internal or automation-only skills are not surfaced to the model.
|
||||
"""
|
||||
lines: List[str] = ["<available_skills>"]
|
||||
|
||||
for manifest in self._skills.values():
|
||||
if manifest.disable_model_invocation:
|
||||
continue
|
||||
escaped_name = html.escape(manifest.name)
|
||||
escaped_desc = html.escape(manifest.description or manifest.name)
|
||||
lines.append(
|
||||
f" <skill name={escaped_name!r} description={escaped_desc!r} />"
|
||||
)
|
||||
|
||||
lines.append("</available_skills>")
|
||||
return "\n".join(lines)
|
||||
|
||||
def get_few_shot_examples(self) -> List[str]:
|
||||
"""Return formatted few-shot example strings ready for system prompt.
|
||||
|
||||
Pulls from ``manifest.metadata.openjarvis.few_shot`` for every
|
||||
registered skill. Returns one formatted string per example.
|
||||
"""
|
||||
examples: List[str] = []
|
||||
for name, manifest in self._skills.items():
|
||||
oj = manifest.metadata.get("openjarvis", {}) if manifest.metadata else {}
|
||||
few_shot = oj.get("few_shot", []) or []
|
||||
for ex in few_shot:
|
||||
if not isinstance(ex, dict):
|
||||
continue
|
||||
inp = str(ex.get("input", ""))
|
||||
out = str(ex.get("output", ""))
|
||||
if inp or out:
|
||||
examples.append(f"### {name}\nInput: {inp}\nOutput: {out}")
|
||||
return examples
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Trace-driven skill discovery (Plan 2A)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def discover_from_traces(
|
||||
self,
|
||||
trace_store: Any,
|
||||
*,
|
||||
min_frequency: int = 3,
|
||||
min_outcome: float = 0.5,
|
||||
output_dir: Optional[Path] = None,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Mine the trace store for recurring tool sequences.
|
||||
|
||||
For each recurring sequence found by :class:`SkillDiscovery`, write
|
||||
a TOML skill manifest into *output_dir* (default
|
||||
``~/.openjarvis/skills/discovered/``). Returns a list of dicts with
|
||||
``name`` and ``path`` for each manifest written.
|
||||
|
||||
Names are normalized to spec-compliant kebab-case (lowercase with
|
||||
hyphens, no underscores) so the resulting manifests load cleanly
|
||||
through the discovery walker.
|
||||
"""
|
||||
from openjarvis.learning.agents.skill_discovery import SkillDiscovery
|
||||
|
||||
traces = trace_store.list_traces(limit=10000)
|
||||
discovery = SkillDiscovery(
|
||||
min_frequency=min_frequency,
|
||||
min_outcome=min_outcome,
|
||||
)
|
||||
discovered = discovery.analyze_traces(traces)
|
||||
|
||||
if output_dir is None:
|
||||
output_dir = Path("~/.openjarvis/skills/discovered/").expanduser()
|
||||
output_dir = Path(output_dir).expanduser()
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
written: List[Dict[str, Any]] = []
|
||||
for skill in discovered:
|
||||
name = self._normalize_skill_name(skill.name)
|
||||
skill_subdir = output_dir / name
|
||||
skill_subdir.mkdir(parents=True, exist_ok=True)
|
||||
toml_path = skill_subdir / "skill.toml"
|
||||
toml_path.write_text(
|
||||
self._serialize_discovered_skill(name, skill),
|
||||
encoding="utf-8",
|
||||
)
|
||||
written.append({"name": name, "path": str(toml_path)})
|
||||
|
||||
return written
|
||||
|
||||
@staticmethod
|
||||
def _normalize_skill_name(raw_name: str) -> str:
|
||||
"""Convert an arbitrary discovered name to a spec-compliant kebab name.
|
||||
|
||||
- Lowercase
|
||||
- Replace underscores and whitespace with hyphens
|
||||
- Collapse runs of hyphens
|
||||
- Strip leading/trailing hyphens
|
||||
"""
|
||||
import re
|
||||
|
||||
normalized = raw_name.lower()
|
||||
normalized = re.sub(r"[_\s]+", "-", normalized)
|
||||
normalized = re.sub(r"-+", "-", normalized)
|
||||
normalized = normalized.strip("-")
|
||||
if not normalized:
|
||||
normalized = "discovered-skill"
|
||||
return normalized
|
||||
|
||||
@staticmethod
|
||||
def _serialize_discovered_skill(name: str, skill: Any) -> str:
|
||||
"""Serialize a DiscoveredSkill into a spec-compliant skill.toml."""
|
||||
lines: List[str] = ["[skill]"]
|
||||
lines.append(f'name = "{name}"')
|
||||
lines.append('version = "0.1.0"')
|
||||
# Truncate description to spec max 1024 chars
|
||||
description = (skill.description or f"Discovered skill: {name}")[:1024]
|
||||
# Escape backslashes and double quotes for basic TOML strings
|
||||
description = description.replace("\\", "\\\\").replace('"', '\\"')
|
||||
lines.append(f'description = "{description}"')
|
||||
lines.append('author = "openjarvis (auto-discovered)"')
|
||||
lines.append('tags = ["auto-discovered"]')
|
||||
lines.append("")
|
||||
|
||||
for tool_name in skill.tool_sequence:
|
||||
lines.append("[[skill.steps]]")
|
||||
lines.append(f'tool_name = "{tool_name}"')
|
||||
lines.append('arguments_template = "{}"')
|
||||
lines.append('output_key = ""')
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Execution
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def execute(
|
||||
self,
|
||||
name: str,
|
||||
context: Optional[Dict[str, Any]] = None,
|
||||
) -> SkillResult:
|
||||
"""Resolve and execute a skill by name.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
name:
|
||||
Skill name to execute.
|
||||
context:
|
||||
Initial context dict passed to the executor.
|
||||
|
||||
Returns
|
||||
-------
|
||||
SkillResult
|
||||
"""
|
||||
manifest = self.resolve(name)
|
||||
executor = SkillExecutor(
|
||||
self._tool_executor or _NullToolExecutor(),
|
||||
bus=self._bus,
|
||||
)
|
||||
executor.set_skill_resolver(self._make_resolver())
|
||||
return executor.run(manifest, initial_context=context)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Configuration
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def set_tool_executor(self, tool_executor: ToolExecutor) -> None:
|
||||
"""Attach a :class:`ToolExecutor` for running tool steps in skill pipelines."""
|
||||
self._tool_executor = tool_executor
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Lifecycle stubs (not yet implemented)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def install(self, source: Any, *, verify: bool = True) -> None:
|
||||
"""Install a skill from a remote source. Not yet implemented."""
|
||||
raise NotImplementedError("Skill installation not yet implemented")
|
||||
|
||||
def remove(self, name: Any) -> None:
|
||||
"""Remove a skill by name. Not yet implemented."""
|
||||
raise NotImplementedError("Skill removal not yet implemented")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Internal helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _NullToolExecutor(ToolExecutor):
|
||||
"""A no-op ToolExecutor used when no real executor is available.
|
||||
|
||||
Allows SkillTool/SkillExecutor construction to succeed even before a
|
||||
real tool executor is wired in; any actual tool call will produce an
|
||||
error ToolResult rather than raising.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__(tools=[], bus=None)
|
||||
|
||||
|
||||
__all__ = ["SkillManager"]
|
||||
@@ -0,0 +1,151 @@
|
||||
"""SkillOverlay — sidecar storage for optimization output (Plan 2A).
|
||||
|
||||
Optimization results from DSPy/GEPA are written to
|
||||
``~/.openjarvis/learning/skills/<skill-name>/optimized.toml``. This module
|
||||
provides a small loader and writer for that overlay format.
|
||||
|
||||
The overlay is a strict TOML file with a single ``[optimized]`` section
|
||||
followed by zero or more ``[[optimized.few_shot]]`` array tables.
|
||||
|
||||
Example:
|
||||
|
||||
[optimized]
|
||||
skill_name = "research-and-summarize"
|
||||
optimizer = "dspy"
|
||||
optimized_at = "2026-04-08T14:30:00Z"
|
||||
trace_count = 47
|
||||
description = "Search the web for a topic and produce a structured summary"
|
||||
|
||||
[[optimized.few_shot]]
|
||||
input = "transformer attention mechanisms"
|
||||
output = "## Recent Advances ..."
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
try:
|
||||
import tomllib
|
||||
except ModuleNotFoundError:
|
||||
import tomli as tomllib # type: ignore[no-redef]
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SkillOverlay:
|
||||
"""A single optimization overlay for one skill."""
|
||||
|
||||
skill_name: str
|
||||
optimizer: str # "dspy" or "gepa"
|
||||
optimized_at: str # ISO 8601 UTC timestamp
|
||||
trace_count: int
|
||||
description: str
|
||||
few_shot: List[Dict[str, str]] = field(default_factory=list)
|
||||
|
||||
|
||||
class SkillOverlayLoader:
|
||||
"""Read overlay files from a sidecar directory tree.
|
||||
|
||||
Layout: ``<root>/<skill-name>/optimized.toml``
|
||||
"""
|
||||
|
||||
def __init__(self, root: Path) -> None:
|
||||
self._root = Path(root).expanduser()
|
||||
|
||||
def load(self, skill_name: str) -> Optional[SkillOverlay]:
|
||||
"""Load the overlay for *skill_name*.
|
||||
|
||||
Returns ``None`` if the overlay file is missing or malformed.
|
||||
Never raises — bad overlay files should not break skill loading.
|
||||
"""
|
||||
path = self._root / skill_name / "optimized.toml"
|
||||
if not path.exists():
|
||||
return None
|
||||
try:
|
||||
with open(path, "rb") as fh:
|
||||
data = tomllib.load(fh)
|
||||
except Exception as exc:
|
||||
LOGGER.warning(
|
||||
"Failed to load overlay for skill '%s' at %s: %s",
|
||||
skill_name,
|
||||
path,
|
||||
exc,
|
||||
)
|
||||
return None
|
||||
|
||||
opt = data.get("optimized", {})
|
||||
if not isinstance(opt, dict) or not opt:
|
||||
LOGGER.warning(
|
||||
"Overlay file %s missing [optimized] section",
|
||||
path,
|
||||
)
|
||||
return None
|
||||
|
||||
few_shot_raw = opt.get("few_shot", []) or []
|
||||
few_shot: List[Dict[str, str]] = []
|
||||
if isinstance(few_shot_raw, list):
|
||||
for item in few_shot_raw:
|
||||
if isinstance(item, dict):
|
||||
few_shot.append(
|
||||
{
|
||||
"input": str(item.get("input", "")),
|
||||
"output": str(item.get("output", "")),
|
||||
}
|
||||
)
|
||||
|
||||
return SkillOverlay(
|
||||
skill_name=str(opt.get("skill_name", skill_name)),
|
||||
optimizer=str(opt.get("optimizer", "")),
|
||||
optimized_at=str(opt.get("optimized_at", "")),
|
||||
trace_count=int(opt.get("trace_count", 0)),
|
||||
description=str(opt.get("description", "")),
|
||||
few_shot=few_shot,
|
||||
)
|
||||
|
||||
|
||||
def write_overlay(overlay: SkillOverlay, root: Path) -> Path:
|
||||
"""Write *overlay* to ``<root>/<skill-name>/optimized.toml``.
|
||||
|
||||
Creates the directory structure if needed. Returns the path written.
|
||||
"""
|
||||
root = Path(root).expanduser()
|
||||
skill_dir = root / overlay.skill_name
|
||||
skill_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = skill_dir / "optimized.toml"
|
||||
|
||||
lines: List[str] = ["[optimized]"]
|
||||
lines.append(f'skill_name = "{_escape(overlay.skill_name)}"')
|
||||
lines.append(f'optimizer = "{_escape(overlay.optimizer)}"')
|
||||
lines.append(f'optimized_at = "{_escape(overlay.optimized_at)}"')
|
||||
lines.append(f"trace_count = {int(overlay.trace_count)}")
|
||||
lines.append(f'description = "{_escape(overlay.description)}"')
|
||||
lines.append("")
|
||||
|
||||
for example in overlay.few_shot:
|
||||
lines.append("[[optimized.few_shot]]")
|
||||
lines.append(f'input = "{_escape(example.get("input", ""))}"')
|
||||
lines.append(f'output = "{_escape(example.get("output", ""))}"')
|
||||
lines.append("")
|
||||
|
||||
path.write_text("\n".join(lines), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _escape(text: str) -> str:
|
||||
"""Minimal TOML string escaping for the basic-string format."""
|
||||
return (
|
||||
str(text)
|
||||
.replace("\\", "\\\\")
|
||||
.replace('"', '\\"')
|
||||
.replace("\n", "\\n")
|
||||
.replace("\r", "\\r")
|
||||
.replace("\t", "\\t")
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["SkillOverlay", "SkillOverlayLoader", "write_overlay"]
|
||||
@@ -0,0 +1,231 @@
|
||||
"""SkillParser — strict spec validation + tolerant field mapping.
|
||||
|
||||
The parser is the single chokepoint for converting raw frontmatter dicts
|
||||
into normalized SkillManifest instances. It runs two passes:
|
||||
|
||||
1. Strict pass — validates required fields, length limits, naming rules.
|
||||
2. Tolerant pass — maps non-spec top-level fields to their canonical
|
||||
locations under metadata.openjarvis.* via FIELD_MAPPING.
|
||||
|
||||
The mapping table is data, not code paths. Adding support for a new
|
||||
vendor's fields means adding entries to FIELD_MAPPING — no logic changes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Any, Dict
|
||||
|
||||
from openjarvis.skills.types import SkillManifest
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Spec constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MAX_NAME_LENGTH = 64
|
||||
MAX_DESCRIPTION_LENGTH = 1024
|
||||
MAX_COMPATIBILITY_LENGTH = 500
|
||||
|
||||
# Spec-allowed top-level frontmatter fields
|
||||
SPEC_FIELDS = frozenset(
|
||||
{"name", "description", "license", "compatibility", "metadata", "allowed-tools"}
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Field mapping table — non-spec top-level fields → canonical locations
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Each entry maps a non-spec top-level field name to (target_kind, attr).
|
||||
# When target_kind is "field" the value is set directly on the SkillManifest
|
||||
# dataclass attribute. When target_kind is "openjarvis_meta" the value is
|
||||
# stored under manifest.metadata["openjarvis"][attr].
|
||||
FIELD_MAPPING: Dict[str, tuple[str, str]] = {
|
||||
"version": ("field", "version"),
|
||||
"author": ("field", "author"),
|
||||
"tags": ("field", "tags"),
|
||||
"depends": ("field", "depends"),
|
||||
"required_capabilities": ("field", "required_capabilities"),
|
||||
"user_invocable": ("field", "user_invocable"),
|
||||
"disable_model_invocation": ("field", "disable_model_invocation"),
|
||||
"platforms": ("openjarvis_meta", "platforms"),
|
||||
"prerequisites": ("openjarvis_meta", "prerequisites"),
|
||||
}
|
||||
|
||||
# Naming pattern: lowercase alnum + hyphens, no leading/trailing/consecutive hyphens
|
||||
_NAME_PATTERN = re.compile(r"^[a-z0-9](?:[a-z0-9]|-(?!-))*[a-z0-9]$|^[a-z0-9]$")
|
||||
|
||||
|
||||
class SkillParseError(ValueError):
|
||||
"""Raised when a skill frontmatter cannot be parsed."""
|
||||
|
||||
|
||||
class SkillParser:
|
||||
"""Parse SKILL.md frontmatter into a SkillManifest.
|
||||
|
||||
Two-pass design:
|
||||
|
||||
- :meth:`parse_frontmatter` runs the strict pass + tolerant pass and
|
||||
returns a normalized :class:`SkillManifest`.
|
||||
"""
|
||||
|
||||
def parse_frontmatter(
|
||||
self,
|
||||
frontmatter: Dict[str, Any],
|
||||
*,
|
||||
markdown_content: str = "",
|
||||
) -> SkillManifest:
|
||||
"""Parse a frontmatter dict, returning a SkillManifest.
|
||||
|
||||
Runs strict validation first, then applies tolerant field mapping.
|
||||
"""
|
||||
self._validate_strict(frontmatter)
|
||||
return self._build_manifest(frontmatter, markdown_content)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Strict pass
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _validate_strict(self, frontmatter: Dict[str, Any]) -> None:
|
||||
"""Validate required fields, length limits, and naming rules."""
|
||||
# Required fields
|
||||
if "name" not in frontmatter:
|
||||
raise SkillParseError("Missing required field in frontmatter: name")
|
||||
if "description" not in frontmatter:
|
||||
raise SkillParseError("Missing required field in frontmatter: description")
|
||||
|
||||
name = frontmatter["name"]
|
||||
description = frontmatter["description"]
|
||||
|
||||
# Type check
|
||||
if not isinstance(name, str):
|
||||
raise SkillParseError(f"Field 'name' must be a string, got {type(name)}")
|
||||
if not isinstance(description, str):
|
||||
raise SkillParseError(
|
||||
f"Field 'description' must be a string, got {type(description)}"
|
||||
)
|
||||
|
||||
# Length limits
|
||||
if len(name) == 0 or len(name) > MAX_NAME_LENGTH:
|
||||
raise SkillParseError(
|
||||
f"Field 'name' must be 1-{MAX_NAME_LENGTH} chars, got {len(name)}"
|
||||
)
|
||||
if len(description) == 0 or len(description) > MAX_DESCRIPTION_LENGTH:
|
||||
raise SkillParseError(
|
||||
f"Field 'description' must be 1-{MAX_DESCRIPTION_LENGTH} chars, "
|
||||
f"got {len(description)}"
|
||||
)
|
||||
|
||||
# Naming rules
|
||||
self._validate_name(name)
|
||||
|
||||
# compatibility length (optional field)
|
||||
compat = frontmatter.get("compatibility")
|
||||
if compat is not None:
|
||||
if not isinstance(compat, str):
|
||||
raise SkillParseError("Field 'compatibility' must be a string")
|
||||
if len(compat) > MAX_COMPATIBILITY_LENGTH:
|
||||
raise SkillParseError(
|
||||
f"Field 'compatibility' exceeds {MAX_COMPATIBILITY_LENGTH} chars"
|
||||
)
|
||||
|
||||
def _validate_name(self, name: str) -> None:
|
||||
"""Validate the spec naming rules for a skill name."""
|
||||
if name != name.lower():
|
||||
raise SkillParseError(f"Skill name '{name}' must be lowercase")
|
||||
if name.startswith("-") or name.endswith("-"):
|
||||
raise SkillParseError(
|
||||
f"Skill name '{name}' must not start or end with a hyphen"
|
||||
)
|
||||
if "--" in name:
|
||||
raise SkillParseError(
|
||||
f"Skill name '{name}' must not contain consecutive hyphens"
|
||||
)
|
||||
for ch in name:
|
||||
if not (ch.isalnum() or ch == "-"):
|
||||
raise SkillParseError(
|
||||
f"Skill name '{name}' contains invalid character '{ch}'; "
|
||||
f"only lowercase alphanumeric and hyphens are allowed"
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Build manifest (placeholder — Task 2 adds tolerant pass)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _build_manifest(
|
||||
self,
|
||||
frontmatter: Dict[str, Any],
|
||||
markdown_content: str,
|
||||
) -> SkillManifest:
|
||||
"""Construct a SkillManifest from validated frontmatter.
|
||||
|
||||
Runs the tolerant pass: applies FIELD_MAPPING for non-spec fields,
|
||||
captures unknown fields under metadata.openjarvis.original_frontmatter,
|
||||
and merges metadata.openjarvis.* into canonical fields.
|
||||
"""
|
||||
# Start with required + optional spec fields
|
||||
manifest = SkillManifest(
|
||||
name=frontmatter["name"],
|
||||
description=frontmatter["description"],
|
||||
markdown_content=markdown_content,
|
||||
)
|
||||
|
||||
# Pre-existing metadata block (if any)
|
||||
raw_metadata = frontmatter.get("metadata") or {}
|
||||
if not isinstance(raw_metadata, dict):
|
||||
raw_metadata = {}
|
||||
# Initialize openjarvis namespace
|
||||
oj_meta = dict(raw_metadata.get("openjarvis") or {})
|
||||
|
||||
# Apply FIELD_MAPPING for non-spec top-level fields
|
||||
unmapped: Dict[str, Any] = {}
|
||||
for key, value in frontmatter.items():
|
||||
if key in SPEC_FIELDS:
|
||||
continue
|
||||
if key in FIELD_MAPPING:
|
||||
target, attr = FIELD_MAPPING[key]
|
||||
if target == "field":
|
||||
setattr(manifest, attr, value)
|
||||
else: # "openjarvis_meta"
|
||||
oj_meta[attr] = value
|
||||
else:
|
||||
unmapped[key] = value
|
||||
LOGGER.warning(
|
||||
"Unmapped frontmatter field '%s' in skill '%s' "
|
||||
"(value preserved in metadata.openjarvis.original_frontmatter)",
|
||||
key,
|
||||
manifest.name,
|
||||
)
|
||||
|
||||
# Merge metadata.openjarvis.* into canonical fields (these override
|
||||
# top-level mappings since they are explicit OpenJarvis-namespaced).
|
||||
for key in (
|
||||
"version",
|
||||
"author",
|
||||
"tags",
|
||||
"depends",
|
||||
"required_capabilities",
|
||||
"user_invocable",
|
||||
"disable_model_invocation",
|
||||
):
|
||||
if key in oj_meta:
|
||||
setattr(manifest, key, oj_meta[key])
|
||||
|
||||
# Preserve unmapped fields under metadata.openjarvis.original_frontmatter
|
||||
if unmapped:
|
||||
oj_meta["original_frontmatter"] = unmapped
|
||||
|
||||
# Stash the openjarvis metadata block back
|
||||
if oj_meta:
|
||||
new_metadata = dict(raw_metadata)
|
||||
new_metadata["openjarvis"] = oj_meta
|
||||
manifest.metadata = new_metadata
|
||||
else:
|
||||
manifest.metadata = raw_metadata
|
||||
|
||||
return manifest
|
||||
|
||||
|
||||
__all__ = ["SkillParseError", "SkillParser", "SPEC_FIELDS", "FIELD_MAPPING"]
|
||||
@@ -0,0 +1,65 @@
|
||||
"""Skill security — capability validation and trust tiers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import Enum
|
||||
from typing import List, Set
|
||||
|
||||
from openjarvis.skills.types import SkillManifest
|
||||
|
||||
DANGEROUS_CAPABILITIES: frozenset[str] = frozenset(
|
||||
{"shell:execute", "network:listen", "filesystem:write"}
|
||||
)
|
||||
|
||||
|
||||
class TrustTier(str, Enum):
|
||||
"""Trust tier for a skill, ordered from most to least trusted."""
|
||||
|
||||
BUNDLED = "bundled"
|
||||
INDEXED = "indexed"
|
||||
WORKSPACE = "workspace"
|
||||
UNREVIEWED = "unreviewed"
|
||||
|
||||
|
||||
def classify_trust_tier(
|
||||
*,
|
||||
is_bundled: bool = False,
|
||||
is_workspace: bool = False,
|
||||
has_signature: bool = False,
|
||||
in_index: bool = False,
|
||||
) -> TrustTier:
|
||||
"""Return the trust tier for a skill based on its provenance.
|
||||
|
||||
Priority (highest to lowest): bundled > workspace > indexed > unreviewed.
|
||||
"""
|
||||
if is_bundled:
|
||||
return TrustTier.BUNDLED
|
||||
if is_workspace:
|
||||
return TrustTier.WORKSPACE
|
||||
if has_signature and in_index:
|
||||
return TrustTier.INDEXED
|
||||
return TrustTier.UNREVIEWED
|
||||
|
||||
|
||||
def validate_capabilities(manifest: SkillManifest, allowed: Set[str]) -> List[str]:
|
||||
"""Return a list of capabilities required by *manifest* that are not in *allowed*.
|
||||
|
||||
An empty list means the manifest is fully authorized.
|
||||
"""
|
||||
return [cap for cap in manifest.required_capabilities if cap not in allowed]
|
||||
|
||||
|
||||
def has_dangerous_capabilities(manifest: SkillManifest) -> List[str]:
|
||||
"""Return the subset of *manifest*'s required capabilities that are dangerous."""
|
||||
return [
|
||||
cap for cap in manifest.required_capabilities if cap in DANGEROUS_CAPABILITIES
|
||||
]
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DANGEROUS_CAPABILITIES",
|
||||
"TrustTier",
|
||||
"classify_trust_tier",
|
||||
"validate_capabilities",
|
||||
"has_dangerous_capabilities",
|
||||
]
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Skill source resolvers — Hermes, OpenClaw, generic GitHub."""
|
||||
|
||||
from openjarvis.skills.sources.base import ResolvedSkill, SourceResolver
|
||||
from openjarvis.skills.sources.github import GitHubResolver
|
||||
from openjarvis.skills.sources.hermes import HERMES_REPO_URL, HermesResolver
|
||||
from openjarvis.skills.sources.openclaw import OPENCLAW_REPO_URL, OpenClawResolver
|
||||
|
||||
__all__ = [
|
||||
"GitHubResolver",
|
||||
"HERMES_REPO_URL",
|
||||
"HermesResolver",
|
||||
"OPENCLAW_REPO_URL",
|
||||
"OpenClawResolver",
|
||||
"ResolvedSkill",
|
||||
"SourceResolver",
|
||||
]
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Source resolver ABC + ResolvedSkill dataclass."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ResolvedSkill:
|
||||
"""A skill found in an upstream source, ready to import.
|
||||
|
||||
Lightweight — does not include the full SKILL.md body. The importer
|
||||
reads the file from *path* when actually installing.
|
||||
"""
|
||||
|
||||
name: str
|
||||
source: str
|
||||
path: Path
|
||||
category: str
|
||||
description: str
|
||||
commit: str
|
||||
sidecar_data: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class SourceResolver(ABC):
|
||||
"""Abstract base for a skill source resolver.
|
||||
|
||||
Implementations clone or pull an upstream repo into a cache directory,
|
||||
walk the cache to find SKILL.md files, and return ResolvedSkill objects
|
||||
that the importer can install.
|
||||
"""
|
||||
|
||||
name: str = ""
|
||||
|
||||
@abstractmethod
|
||||
def cache_dir(self) -> Path:
|
||||
"""Where this source clones its repo."""
|
||||
|
||||
@abstractmethod
|
||||
def sync(self) -> None:
|
||||
"""Clone or pull the upstream repo into the cache."""
|
||||
|
||||
@abstractmethod
|
||||
def list_skills(self) -> List[ResolvedSkill]:
|
||||
"""Walk the cache directory and return all discoverable skills."""
|
||||
|
||||
def resolve(self, query: str) -> List[ResolvedSkill]:
|
||||
"""Filter list_skills() by name (substring match).
|
||||
|
||||
Empty *query* returns all skills.
|
||||
"""
|
||||
all_skills = self.list_skills()
|
||||
if not query:
|
||||
return all_skills
|
||||
q = query.lower()
|
||||
return [s for s in all_skills if q in s.name.lower()]
|
||||
|
||||
def filter_by_category(self, category: str) -> List[ResolvedSkill]:
|
||||
"""Return skills whose category exactly matches *category*."""
|
||||
return [s for s in self.list_skills() if s.category == category]
|
||||
|
||||
|
||||
__all__ = ["ResolvedSkill", "SourceResolver"]
|
||||
@@ -0,0 +1,121 @@
|
||||
"""GitHubResolver — generic resolver for any GitHub repo containing skills.
|
||||
|
||||
Performs a recursive walk for SKILL.md (or skill.md) files anywhere
|
||||
under the cache directory.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
import yaml
|
||||
|
||||
from openjarvis.skills.sources.base import ResolvedSkill, SourceResolver
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GitHubResolver(SourceResolver):
|
||||
"""Generic resolver for any GitHub repo containing SKILL.md files."""
|
||||
|
||||
name = "github"
|
||||
|
||||
def __init__(self, cache_root: Path, repo_url: str) -> None:
|
||||
self._cache_root = Path(cache_root)
|
||||
self._repo_url = repo_url
|
||||
|
||||
def cache_dir(self) -> Path:
|
||||
return self._cache_root
|
||||
|
||||
def sync(self) -> None:
|
||||
if self._cache_root.exists() and (self._cache_root / ".git").exists():
|
||||
subprocess.run(
|
||||
["git", "-C", str(self._cache_root), "pull", "--ff-only"],
|
||||
check=True,
|
||||
)
|
||||
else:
|
||||
self._cache_root.parent.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
["git", "clone", self._repo_url, str(self._cache_root)],
|
||||
check=True,
|
||||
)
|
||||
|
||||
def list_skills(self) -> List[ResolvedSkill]:
|
||||
if not self._cache_root.exists():
|
||||
return []
|
||||
|
||||
results: List[ResolvedSkill] = []
|
||||
commit = self._read_commit()
|
||||
seen_dirs: set[Path] = set()
|
||||
|
||||
# Recursive walk for SKILL.md or skill.md
|
||||
for pattern in ("SKILL.md", "skill.md"):
|
||||
for skill_md in sorted(self._cache_root.rglob(pattern)):
|
||||
# Skip files inside .git
|
||||
if ".git" in skill_md.parts:
|
||||
continue
|
||||
skill_dir = skill_md.parent
|
||||
if skill_dir in seen_dirs:
|
||||
continue
|
||||
seen_dirs.add(skill_dir)
|
||||
|
||||
name, description = self._read_preview(
|
||||
skill_md, default_name=skill_dir.name
|
||||
)
|
||||
# Use the immediate parent directory of the skill dir as category
|
||||
try:
|
||||
category = skill_dir.parent.relative_to(self._cache_root).as_posix()
|
||||
except ValueError:
|
||||
category = ""
|
||||
|
||||
results.append(
|
||||
ResolvedSkill(
|
||||
name=name,
|
||||
source=self.name,
|
||||
path=skill_dir,
|
||||
category=category,
|
||||
description=description,
|
||||
commit=commit,
|
||||
)
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
def _read_preview(self, skill_md: Path, default_name: str) -> tuple[str, str]:
|
||||
try:
|
||||
raw = skill_md.read_text(encoding="utf-8")
|
||||
except Exception:
|
||||
return default_name, ""
|
||||
if not raw.startswith("---"):
|
||||
return default_name, ""
|
||||
rest = raw[3:].lstrip("\n")
|
||||
end = rest.find("\n---")
|
||||
if end == -1:
|
||||
return default_name, ""
|
||||
try:
|
||||
fm = yaml.safe_load(rest[:end])
|
||||
except yaml.YAMLError:
|
||||
return default_name, ""
|
||||
if not isinstance(fm, dict):
|
||||
return default_name, ""
|
||||
return str(fm.get("name", default_name)), str(fm.get("description", ""))
|
||||
|
||||
def _read_commit(self) -> str:
|
||||
if not (self._cache_root / ".git").exists():
|
||||
return ""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "-C", str(self._cache_root), "rev-parse", "HEAD"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return result.stdout.strip()
|
||||
except subprocess.CalledProcessError:
|
||||
return ""
|
||||
|
||||
|
||||
__all__ = ["GitHubResolver"]
|
||||
@@ -0,0 +1,130 @@
|
||||
"""HermesResolver — resolves skills from NousResearch/hermes-agent.
|
||||
|
||||
Layout:
|
||||
skills/<category>/<skill-name>/SKILL.md
|
||||
skills/<category>/DESCRIPTION.md (skipped)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
import yaml
|
||||
|
||||
from openjarvis.skills.sources.base import ResolvedSkill, SourceResolver
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
HERMES_REPO_URL = "https://github.com/NousResearch/hermes-agent.git"
|
||||
|
||||
|
||||
class HermesResolver(SourceResolver):
|
||||
"""Resolves skills from the Hermes Agent repository."""
|
||||
|
||||
name = "hermes"
|
||||
|
||||
def __init__(self, cache_root: Path | None = None) -> None:
|
||||
if cache_root is None:
|
||||
cache_root = Path("~/.openjarvis/skill-cache/hermes/").expanduser()
|
||||
self._cache_root = Path(cache_root)
|
||||
|
||||
def cache_dir(self) -> Path:
|
||||
return self._cache_root
|
||||
|
||||
def sync(self) -> None:
|
||||
"""Clone or pull the Hermes repo into the cache directory."""
|
||||
if self._cache_root.exists() and (self._cache_root / ".git").exists():
|
||||
subprocess.run(
|
||||
["git", "-C", str(self._cache_root), "pull", "--ff-only"],
|
||||
check=True,
|
||||
)
|
||||
else:
|
||||
self._cache_root.parent.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
["git", "clone", HERMES_REPO_URL, str(self._cache_root)],
|
||||
check=True,
|
||||
)
|
||||
|
||||
def list_skills(self) -> List[ResolvedSkill]:
|
||||
"""Walk skills/<category>/<skill>/SKILL.md."""
|
||||
skills_root = self._cache_root / "skills"
|
||||
if not skills_root.exists():
|
||||
return []
|
||||
|
||||
results: List[ResolvedSkill] = []
|
||||
commit = self._read_commit()
|
||||
|
||||
for category_dir in sorted(skills_root.iterdir()):
|
||||
if not category_dir.is_dir():
|
||||
continue
|
||||
category = category_dir.name
|
||||
for skill_dir in sorted(category_dir.iterdir()):
|
||||
if not skill_dir.is_dir():
|
||||
continue # skip DESCRIPTION.md and other files
|
||||
skill_md = skill_dir / "SKILL.md"
|
||||
if not skill_md.exists():
|
||||
continue
|
||||
|
||||
# Read minimal frontmatter for the preview
|
||||
name, description = self._read_preview(
|
||||
skill_md, default_name=skill_dir.name
|
||||
)
|
||||
results.append(
|
||||
ResolvedSkill(
|
||||
name=name,
|
||||
source=self.name,
|
||||
path=skill_dir,
|
||||
category=category,
|
||||
description=description,
|
||||
commit=commit,
|
||||
)
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
def _read_preview(self, skill_md: Path, default_name: str) -> tuple[str, str]:
|
||||
"""Read just enough frontmatter to populate ResolvedSkill preview fields."""
|
||||
try:
|
||||
raw = skill_md.read_text(encoding="utf-8")
|
||||
except Exception:
|
||||
return default_name, ""
|
||||
|
||||
if not raw.startswith("---"):
|
||||
return default_name, ""
|
||||
|
||||
rest = raw[3:].lstrip("\n")
|
||||
end = rest.find("\n---")
|
||||
if end == -1:
|
||||
return default_name, ""
|
||||
try:
|
||||
fm = yaml.safe_load(rest[:end])
|
||||
except yaml.YAMLError:
|
||||
return default_name, ""
|
||||
if not isinstance(fm, dict):
|
||||
return default_name, ""
|
||||
|
||||
return (
|
||||
str(fm.get("name", default_name)),
|
||||
str(fm.get("description", "")),
|
||||
)
|
||||
|
||||
def _read_commit(self) -> str:
|
||||
"""Return the current HEAD SHA of the cached repo, or empty string."""
|
||||
if not (self._cache_root / ".git").exists():
|
||||
return ""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "-C", str(self._cache_root), "rev-parse", "HEAD"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return result.stdout.strip()
|
||||
except subprocess.CalledProcessError:
|
||||
return ""
|
||||
|
||||
|
||||
__all__ = ["HermesResolver", "HERMES_REPO_URL"]
|
||||
@@ -0,0 +1,129 @@
|
||||
"""OpenClawResolver — resolves skills from the OpenClaw skill index.
|
||||
|
||||
Layout:
|
||||
skills/<owner>/<skill-name>/SKILL.md
|
||||
skills/<owner>/<skill-name>/_meta.json (optional sidecar registry data)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
import yaml
|
||||
|
||||
from openjarvis.skills.sources.base import ResolvedSkill, SourceResolver
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
OPENCLAW_REPO_URL = "https://github.com/openclaw/skills.git"
|
||||
|
||||
|
||||
class OpenClawResolver(SourceResolver):
|
||||
"""Resolves skills from the OpenClaw skill index."""
|
||||
|
||||
name = "openclaw"
|
||||
|
||||
def __init__(self, cache_root: Path | None = None) -> None:
|
||||
if cache_root is None:
|
||||
cache_root = Path("~/.openjarvis/skill-cache/openclaw/").expanduser()
|
||||
self._cache_root = Path(cache_root)
|
||||
|
||||
def cache_dir(self) -> Path:
|
||||
return self._cache_root
|
||||
|
||||
def sync(self) -> None:
|
||||
if self._cache_root.exists() and (self._cache_root / ".git").exists():
|
||||
subprocess.run(
|
||||
["git", "-C", str(self._cache_root), "pull", "--ff-only"],
|
||||
check=True,
|
||||
)
|
||||
else:
|
||||
self._cache_root.parent.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
["git", "clone", OPENCLAW_REPO_URL, str(self._cache_root)],
|
||||
check=True,
|
||||
)
|
||||
|
||||
def list_skills(self) -> List[ResolvedSkill]:
|
||||
skills_root = self._cache_root / "skills"
|
||||
if not skills_root.exists():
|
||||
return []
|
||||
|
||||
results: List[ResolvedSkill] = []
|
||||
commit = self._read_commit()
|
||||
|
||||
for owner_dir in sorted(skills_root.iterdir()):
|
||||
if not owner_dir.is_dir():
|
||||
continue
|
||||
for skill_dir in sorted(owner_dir.iterdir()):
|
||||
if not skill_dir.is_dir():
|
||||
continue
|
||||
skill_md = skill_dir / "SKILL.md"
|
||||
if not skill_md.exists():
|
||||
continue
|
||||
|
||||
name, description = self._read_preview(
|
||||
skill_md, default_name=skill_dir.name
|
||||
)
|
||||
sidecar = self._read_sidecar(skill_dir / "_meta.json")
|
||||
results.append(
|
||||
ResolvedSkill(
|
||||
name=name,
|
||||
source=self.name,
|
||||
path=skill_dir,
|
||||
category=owner_dir.name,
|
||||
description=description,
|
||||
commit=commit,
|
||||
sidecar_data=sidecar,
|
||||
)
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
def _read_preview(self, skill_md: Path, default_name: str) -> tuple[str, str]:
|
||||
try:
|
||||
raw = skill_md.read_text(encoding="utf-8")
|
||||
except Exception:
|
||||
return default_name, ""
|
||||
if not raw.startswith("---"):
|
||||
return default_name, ""
|
||||
rest = raw[3:].lstrip("\n")
|
||||
end = rest.find("\n---")
|
||||
if end == -1:
|
||||
return default_name, ""
|
||||
try:
|
||||
fm = yaml.safe_load(rest[:end])
|
||||
except yaml.YAMLError:
|
||||
return default_name, ""
|
||||
if not isinstance(fm, dict):
|
||||
return default_name, ""
|
||||
return str(fm.get("name", default_name)), str(fm.get("description", ""))
|
||||
|
||||
def _read_sidecar(self, sidecar_path: Path) -> dict:
|
||||
if not sidecar_path.exists():
|
||||
return {}
|
||||
try:
|
||||
return json.loads(sidecar_path.read_text(encoding="utf-8"))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return {}
|
||||
|
||||
def _read_commit(self) -> str:
|
||||
if not (self._cache_root / ".git").exists():
|
||||
return ""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "-C", str(self._cache_root), "rev-parse", "HEAD"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
return result.stdout.strip()
|
||||
except subprocess.CalledProcessError:
|
||||
return ""
|
||||
|
||||
|
||||
__all__ = ["OpenClawResolver", "OPENCLAW_REPO_URL"]
|
||||
@@ -2,18 +2,30 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Dict
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional, Set
|
||||
|
||||
from openjarvis.core.types import ToolResult
|
||||
from openjarvis.skills.executor import SkillExecutor
|
||||
from openjarvis.skills.types import SkillManifest
|
||||
from openjarvis.tools._stubs import BaseTool, ToolSpec
|
||||
|
||||
_PLACEHOLDER_RE = re.compile(r"\{(\w+)\}")
|
||||
|
||||
|
||||
class SkillTool(BaseTool):
|
||||
"""Wraps a SkillManifest as a BaseTool that agents can invoke.
|
||||
|
||||
Follows the same adapter pattern as MCPToolAdapter.
|
||||
|
||||
Parameters
|
||||
----------
|
||||
manifest:
|
||||
The skill manifest to wrap.
|
||||
executor:
|
||||
A :class:`SkillExecutor` used to run the skill pipeline.
|
||||
skill_manager:
|
||||
Optional skill manager (reserved for sub-skill delegation).
|
||||
"""
|
||||
|
||||
tool_id: str
|
||||
@@ -22,50 +34,162 @@ class SkillTool(BaseTool):
|
||||
self,
|
||||
manifest: SkillManifest,
|
||||
executor: SkillExecutor,
|
||||
*,
|
||||
skill_manager: Optional[Any] = None,
|
||||
) -> None:
|
||||
self._manifest = manifest
|
||||
self._executor = executor
|
||||
self._skill_manager = skill_manager
|
||||
self.tool_id = f"skill_{manifest.name}"
|
||||
self._parameters = self._build_parameters()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Parameter extraction
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _build_parameters(self) -> Dict[str, Any]:
|
||||
"""Auto-extract input parameters from the manifest.
|
||||
|
||||
For pipeline skills, scan each step's ``arguments_template`` for
|
||||
``{placeholder}`` patterns. Subtract the ``output_key`` values
|
||||
produced by *prior* steps so only externally-supplied parameters
|
||||
are exposed.
|
||||
|
||||
For instruction-only skills (no steps), expose a single optional
|
||||
``task`` parameter.
|
||||
"""
|
||||
if not self._manifest.steps:
|
||||
# Instruction-only / markdown-only skill
|
||||
return {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"task": {
|
||||
"type": "string",
|
||||
"description": "Optional task description or context.",
|
||||
}
|
||||
},
|
||||
}
|
||||
|
||||
produced: Set[str] = set()
|
||||
input_params: List[str] = []
|
||||
seen: Set[str] = set()
|
||||
|
||||
for step in self._manifest.steps:
|
||||
# Find all {placeholder} tokens in this step's template
|
||||
placeholders = _PLACEHOLDER_RE.findall(step.arguments_template)
|
||||
for ph in placeholders:
|
||||
if ph not in produced and ph not in seen:
|
||||
input_params.append(ph)
|
||||
seen.add(ph)
|
||||
|
||||
# After processing this step, its output_key becomes available
|
||||
# for subsequent steps and should NOT be surfaced as an input
|
||||
if step.output_key:
|
||||
produced.add(step.output_key)
|
||||
|
||||
properties: Dict[str, Any] = {}
|
||||
for param in input_params:
|
||||
properties[param] = {
|
||||
"type": "string",
|
||||
"description": f"Input value for '{param}'.",
|
||||
}
|
||||
|
||||
return {
|
||||
"type": "object",
|
||||
"properties": properties,
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Result metadata
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _build_result_metadata(self, *, steps_run: int) -> Dict[str, Any]:
|
||||
"""Build the metadata dict attached to ToolResult.
|
||||
|
||||
Includes the skill name, source provenance, and kind so downstream
|
||||
consumers (TraceCollector, SkillOptimizer) can bucket invocations.
|
||||
"""
|
||||
oj_meta = self._manifest.metadata.get("openjarvis", {}) or {}
|
||||
source = oj_meta.get("source", "user")
|
||||
kind = "executable" if self._manifest.steps else "instructional"
|
||||
return {
|
||||
"skill": self._manifest.name,
|
||||
"skill_source": source,
|
||||
"skill_kind": kind,
|
||||
"steps": steps_run,
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# BaseTool interface
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
@property
|
||||
def spec(self) -> ToolSpec:
|
||||
return ToolSpec(
|
||||
name=f"skill_{self._manifest.name}",
|
||||
description=self._manifest.description or f"Skill: {self._manifest.name}",
|
||||
parameters={
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"input": {
|
||||
"type": "string",
|
||||
"description": "Input text for the skill pipeline.",
|
||||
},
|
||||
"context": {
|
||||
"type": "object",
|
||||
"description": "Additional context key-value pairs.",
|
||||
},
|
||||
},
|
||||
},
|
||||
parameters=self._parameters,
|
||||
category="skill",
|
||||
required_capabilities=self._manifest.required_capabilities,
|
||||
)
|
||||
|
||||
def execute(self, **params: Any) -> ToolResult:
|
||||
initial_ctx: Dict[str, Any] = params.get("context", {})
|
||||
if "input" in params:
|
||||
initial_ctx["input"] = params["input"]
|
||||
"""Execute the skill.
|
||||
|
||||
result = self._executor.run(self._manifest, initial_context=initial_ctx)
|
||||
If the manifest has pipeline steps, run them via the executor and
|
||||
collect the last step's output. If ``markdown_content`` is present,
|
||||
append it to the content. Returns a combined :class:`ToolResult`.
|
||||
"""
|
||||
tool_name = self.spec.name
|
||||
content_parts: List[str] = []
|
||||
|
||||
if self._manifest.steps:
|
||||
# Build initial context from all supplied params
|
||||
initial_ctx: Dict[str, Any] = {k: v for k, v in params.items()}
|
||||
|
||||
result = self._executor.run(self._manifest, initial_context=initial_ctx)
|
||||
|
||||
if not result.success:
|
||||
# Propagate failure immediately
|
||||
error_content = (
|
||||
result.step_results[-1].content
|
||||
if result.step_results
|
||||
else "Pipeline failed with no step results."
|
||||
)
|
||||
return ToolResult(
|
||||
tool_name=tool_name,
|
||||
content=error_content,
|
||||
success=False,
|
||||
metadata=self._build_result_metadata(
|
||||
steps_run=len(result.step_results)
|
||||
),
|
||||
)
|
||||
|
||||
# Use the last step's output as the primary content
|
||||
if result.step_results:
|
||||
last_step = result.step_results[-1]
|
||||
last_output = (
|
||||
result.context.get(
|
||||
# Prefer the keyed output if available
|
||||
self._manifest.steps[-1].output_key,
|
||||
last_step.content,
|
||||
)
|
||||
if self._manifest.steps[-1].output_key
|
||||
else last_step.content
|
||||
)
|
||||
content_parts.append(str(last_output))
|
||||
|
||||
# Append markdown content if present (hybrid or instruction-only)
|
||||
if self._manifest.markdown_content:
|
||||
content_parts.append(self._manifest.markdown_content)
|
||||
|
||||
combined = "\n\n".join(filter(None, content_parts))
|
||||
|
||||
return ToolResult(
|
||||
tool_name=self.spec.name,
|
||||
content=result.context.get(
|
||||
result.step_results[-1].tool_name if result.step_results else "",
|
||||
result.step_results[-1].content if result.step_results else "",
|
||||
)
|
||||
if result.step_results
|
||||
else "",
|
||||
success=result.success,
|
||||
metadata={"skill": self._manifest.name, "steps": len(result.step_results)},
|
||||
tool_name=tool_name,
|
||||
content=combined,
|
||||
success=True,
|
||||
metadata=self._build_result_metadata(steps_run=len(self._manifest.steps)),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
"""ToolTranslator — translate external tool names to OpenJarvis equivalents.
|
||||
|
||||
External skill libraries (Hermes Agent, OpenClaw) reference tools by Claude
|
||||
Code's standard tool names (Bash, Read, Write, etc.). OpenJarvis uses
|
||||
different names (shell_exec, file_read, file_write). This module translates
|
||||
those references in skill markdown bodies and ``allowed-tools`` fields.
|
||||
|
||||
The translation table is small (~10 entries) covering Claude Code's standard
|
||||
tools and grows as we encounter new vendor tools.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Dict, List, Tuple
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Translation table — external name → OpenJarvis name
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
TOOL_TRANSLATION: Dict[str, str] = {
|
||||
"Bash": "shell_exec",
|
||||
"Read": "file_read",
|
||||
"Write": "file_write",
|
||||
"Edit": "file_edit",
|
||||
"Glob": "file_glob",
|
||||
"Grep": "file_grep",
|
||||
"WebFetch": "web_search",
|
||||
"WebSearch": "web_search",
|
||||
"Task": "delegate_agent",
|
||||
"NotebookEdit": "notebook_edit",
|
||||
}
|
||||
|
||||
|
||||
class ToolTranslator:
|
||||
"""Rewrite tool references in markdown bodies and allowed-tools fields.
|
||||
|
||||
Uses a word-boundary regex so partial matches like 'Reader' or
|
||||
'Reading' are not rewritten.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
translation_table: Dict[str, str] | None = None,
|
||||
) -> None:
|
||||
self._table = dict(translation_table or TOOL_TRANSLATION)
|
||||
# Word-boundary pattern matching any external name
|
||||
names = sorted(self._table.keys(), key=len, reverse=True)
|
||||
if names:
|
||||
self._pattern = re.compile(
|
||||
r"\b(" + "|".join(re.escape(n) for n in names) + r")\b"
|
||||
)
|
||||
else:
|
||||
self._pattern = None
|
||||
|
||||
def translate_markdown(self, body: str) -> Tuple[str, List[str]]:
|
||||
"""Translate tool references in a markdown body.
|
||||
|
||||
Returns
|
||||
-------
|
||||
tuple[str, list[str]]
|
||||
(rewritten body, list of untranslated tool names found)
|
||||
"""
|
||||
if not body or self._pattern is None:
|
||||
return body, []
|
||||
|
||||
def _sub(match: re.Match) -> str:
|
||||
return self._table.get(match.group(1), match.group(1))
|
||||
|
||||
new_body = self._pattern.sub(_sub, body)
|
||||
|
||||
# Find untranslated tool-like references (CamelCase words that look
|
||||
# like tool names but aren't in the table). Conservative heuristic:
|
||||
# words with internal uppercase (true CamelCase) followed by ' tool'
|
||||
# or a word boundary, length 3-30, not in the translation table.
|
||||
untranslated: List[str] = []
|
||||
# Require at least one uppercase letter after position 0 (true CamelCase)
|
||||
candidate_pattern = re.compile(r"\b([A-Z][a-z]+[A-Z][a-zA-Z]*)(?:\s+tool|\b)")
|
||||
for cand in candidate_pattern.findall(body):
|
||||
if cand not in self._table and cand not in untranslated:
|
||||
if 3 <= len(cand) <= 30:
|
||||
untranslated.append(cand)
|
||||
return new_body, untranslated
|
||||
|
||||
def translate_allowed_tools(self, allowed: str) -> Tuple[str, List[str]]:
|
||||
"""Translate the space-delimited allowed-tools field.
|
||||
|
||||
Tokens may have parenthesized arguments like ``Bash(git:*)``. Only
|
||||
the prefix before the first ``(`` is translated.
|
||||
"""
|
||||
if not allowed:
|
||||
return allowed, []
|
||||
|
||||
out_tokens: List[str] = []
|
||||
untranslated: List[str] = []
|
||||
for token in allowed.split():
|
||||
# Split off any (args) suffix
|
||||
if "(" in token:
|
||||
head, _, tail = token.partition("(")
|
||||
tail = "(" + tail
|
||||
else:
|
||||
head, tail = token, ""
|
||||
|
||||
if head in self._table:
|
||||
out_tokens.append(self._table[head] + tail)
|
||||
else:
|
||||
out_tokens.append(token)
|
||||
if head not in untranslated:
|
||||
untranslated.append(head)
|
||||
|
||||
return " ".join(out_tokens), untranslated
|
||||
|
||||
|
||||
__all__ = ["TOOL_TRANSLATION", "ToolTranslator"]
|
||||
@@ -10,7 +10,8 @@ from typing import Any, Dict, List
|
||||
class SkillStep:
|
||||
"""A single step in a skill pipeline."""
|
||||
|
||||
tool_name: str
|
||||
tool_name: str = ""
|
||||
skill_name: str = "" # invoke another skill instead of a tool
|
||||
arguments_template: str = "{}" # Jinja2-style template
|
||||
output_key: str = "" # Key to store result in context
|
||||
|
||||
@@ -27,6 +28,11 @@ class SkillManifest:
|
||||
required_capabilities: List[str] = field(default_factory=list)
|
||||
signature: str = "" # Base64-encoded Ed25519 signature
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
tags: List[str] = field(default_factory=list)
|
||||
depends: List[str] = field(default_factory=list)
|
||||
user_invocable: bool = True
|
||||
disable_model_invocation: bool = False
|
||||
markdown_content: str = "" # loaded from SKILL.md
|
||||
|
||||
def manifest_bytes(self) -> bytes:
|
||||
"""Serialize the manifest (excluding signature) for signing/verification."""
|
||||
@@ -40,12 +46,15 @@ class SkillManifest:
|
||||
"steps": [
|
||||
{
|
||||
"tool_name": s.tool_name,
|
||||
"skill_name": s.skill_name,
|
||||
"arguments_template": s.arguments_template,
|
||||
"output_key": s.output_key,
|
||||
}
|
||||
for s in self.steps
|
||||
],
|
||||
"required_capabilities": self.required_capabilities,
|
||||
"tags": self.tags,
|
||||
"depends": self.depends,
|
||||
}
|
||||
return json.dumps(data, sort_keys=True).encode()
|
||||
|
||||
|
||||
@@ -49,6 +49,7 @@ class JarvisSystem:
|
||||
agent_scheduler: Optional[Any] = None # AgentScheduler
|
||||
agent_executor: Optional[Any] = None # AgentExecutor
|
||||
speech_backend: Optional[Any] = None # SpeechBackend
|
||||
skill_manager: Optional[Any] = None # SkillManager
|
||||
_learning_orchestrator: Optional[Any] = None # LearningOrchestrator
|
||||
_mcp_clients: List = field(default_factory=list)
|
||||
|
||||
@@ -202,6 +203,12 @@ class JarvisSystem:
|
||||
if getattr(agent_cls, "accepts_tools", False):
|
||||
agent_kwargs["tools"] = agent_tools
|
||||
agent_kwargs["max_turns"] = self.config.agent.max_turns
|
||||
# Plan 2B I3: forward optimized few-shot examples to agents
|
||||
# that accept them. Older agents that don't accept the kwarg
|
||||
# are handled by the existing TypeError fallback below.
|
||||
examples = getattr(self, "_skill_few_shot_examples", None)
|
||||
if examples:
|
||||
agent_kwargs["skill_few_shot_examples"] = examples
|
||||
if system_prompt is not None:
|
||||
agent_kwargs["system_prompt"] = system_prompt
|
||||
if self.capability_policy is not None:
|
||||
@@ -662,6 +669,37 @@ class SystemBuilder:
|
||||
# Build tool executor
|
||||
tool_executor = ToolExecutor(tool_list, bus) if tool_list else None
|
||||
|
||||
# Resolve skills
|
||||
skill_manager = None
|
||||
skill_few_shot_examples: List[str] = []
|
||||
if config.skills.enabled:
|
||||
try:
|
||||
from pathlib import Path
|
||||
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
skill_manager = SkillManager(
|
||||
bus, capability_policy=sec.capability_policy
|
||||
)
|
||||
skill_paths = [Path(config.skills.skills_dir).expanduser()]
|
||||
workspace_skills = Path("./skills")
|
||||
if workspace_skills.exists():
|
||||
skill_paths.insert(0, workspace_skills)
|
||||
skill_manager.discover(paths=skill_paths)
|
||||
if tool_executor:
|
||||
skill_manager.set_tool_executor(tool_executor)
|
||||
skill_tools = skill_manager.get_skill_tools(
|
||||
tool_executor=tool_executor,
|
||||
)
|
||||
tool_list.extend(skill_tools)
|
||||
if tool_list:
|
||||
tool_executor = ToolExecutor(tool_list, bus)
|
||||
# Plan 2B I3: capture optimized few-shot examples so
|
||||
# _run_agent can forward them to tool-using agents.
|
||||
skill_few_shot_examples = skill_manager.get_few_shot_examples()
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to initialize skills: %s", exc)
|
||||
|
||||
# Resolve agent name
|
||||
agent_name = self._agent_name or config.agent.default_agent
|
||||
|
||||
@@ -777,8 +815,12 @@ class SystemBuilder:
|
||||
agent_scheduler=agent_scheduler,
|
||||
agent_executor=agent_executor,
|
||||
speech_backend=speech_backend,
|
||||
skill_manager=skill_manager,
|
||||
)
|
||||
system._learning_orchestrator = learning_orchestrator
|
||||
# Plan 2B I3: stash few-shot examples on the system so _run_agent
|
||||
# can include them in agent_kwargs for tool-using agents.
|
||||
system._skill_few_shot_examples = skill_few_shot_examples
|
||||
# Transfer MCP clients so JarvisSystem.close() can shut them down
|
||||
system._mcp_clients = list(getattr(self, "_mcp_clients", []))
|
||||
# Wire system reference — must happen before scheduler.start()
|
||||
|
||||
@@ -269,6 +269,13 @@ class ToolExecutor:
|
||||
# Emit end event
|
||||
if self._bus:
|
||||
result_text = str(result.content)[:10240] if result.content else ""
|
||||
# Pass through ToolResult.metadata so downstream consumers
|
||||
# (TraceCollector → TraceStep.metadata → SkillOptimizer) can
|
||||
# see skill-tagged invocations. Filter to JSON-serializable
|
||||
# values only — internal objects like TaintSet (added by the
|
||||
# taint auto-detect above) must not leak to event subscribers
|
||||
# since the trace store will JSON-serialize them later.
|
||||
event_metadata = self._json_safe_metadata(result.metadata)
|
||||
self._bus.publish(
|
||||
EventType.TOOL_CALL_END,
|
||||
{
|
||||
@@ -276,11 +283,41 @@ class ToolExecutor:
|
||||
"success": result.success,
|
||||
"latency": latency,
|
||||
"result": result_text,
|
||||
"metadata": event_metadata,
|
||||
},
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
@staticmethod
|
||||
def _json_safe_metadata(metadata: Optional[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
"""Return a copy of *metadata* containing only JSON-serializable values.
|
||||
|
||||
``ToolExecutor`` annotates ``ToolResult.metadata`` with internal
|
||||
objects (currently ``_taint: TaintSet``). Those are useful for
|
||||
in-process security checks but cannot be serialized when the
|
||||
``TraceCollector`` writes ``TraceStep.metadata`` to JSON in the
|
||||
SQLite trace store. This helper drops any keys whose value is
|
||||
not JSON-safe — silently, since the missing data is not
|
||||
load-bearing for downstream consumers.
|
||||
"""
|
||||
if not metadata:
|
||||
return {}
|
||||
|
||||
import json
|
||||
|
||||
safe: Dict[str, Any] = {}
|
||||
for key, value in metadata.items():
|
||||
if not isinstance(key, str):
|
||||
continue
|
||||
try:
|
||||
json.dumps(value)
|
||||
except (TypeError, ValueError):
|
||||
# Skip non-serializable values (e.g. TaintSet)
|
||||
continue
|
||||
safe[key] = value
|
||||
return safe
|
||||
|
||||
def available_tools(self) -> List[ToolSpec]:
|
||||
"""Return specs for all available tools."""
|
||||
return [t.spec for t in self._tools.values()]
|
||||
|
||||
@@ -182,6 +182,10 @@ class TraceCollector:
|
||||
def _on_tool_end(self, event: Any) -> None:
|
||||
start = getattr(self, "_tool_start_time", event.timestamp)
|
||||
start_data = getattr(self, "_tool_start_data", {})
|
||||
# Pull through any metadata the tool attached to its ToolResult
|
||||
# (e.g. SkillTool's skill/skill_source/skill_kind tags) so the
|
||||
# SkillOptimizer can bucket traces by skill name.
|
||||
result_metadata = event.data.get("metadata") or {}
|
||||
self._current_steps.append(
|
||||
TraceStep(
|
||||
step_type=StepType.TOOL_CALL,
|
||||
@@ -197,6 +201,7 @@ class TraceCollector:
|
||||
"success": event.data.get("success", False),
|
||||
"result": event.data.get("result", ""),
|
||||
},
|
||||
metadata=dict(result_metadata),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Tests for jarvis bench skills CLI command (Plan 2B)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from openjarvis.cli import cli
|
||||
|
||||
|
||||
class TestBenchSkillsCommand:
|
||||
def test_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["bench", "skills", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "condition" in result.output.lower()
|
||||
|
||||
def test_runs_all_conditions_with_mocked_runner(self, tmp_path: Path) -> None:
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
ConditionComparison,
|
||||
ConditionResult,
|
||||
SkillBenchmarkConfig,
|
||||
)
|
||||
|
||||
# Build a fake comparison the mocked runner will return
|
||||
fake_results = {
|
||||
cond: ConditionResult(
|
||||
condition=cond,
|
||||
seeds=[42],
|
||||
per_seed_pass_rate={42: 0.5},
|
||||
mean_pass_rate=0.5,
|
||||
stddev_pass_rate=0.0,
|
||||
per_task_results={},
|
||||
skill_invocation_counts={},
|
||||
total_tokens=10,
|
||||
total_runtime_seconds=1.0,
|
||||
)
|
||||
for cond in (
|
||||
"no_skills",
|
||||
"skills_on",
|
||||
"skills_optimized_dspy",
|
||||
"skills_optimized_gepa",
|
||||
)
|
||||
}
|
||||
fake_cmp = ConditionComparison(
|
||||
config=SkillBenchmarkConfig(output_dir=tmp_path),
|
||||
started_at="2026-04-08T00:00:00Z",
|
||||
ended_at="2026-04-08T00:01:00Z",
|
||||
results=fake_results,
|
||||
deltas={"skills_on - no_skills": 0.0},
|
||||
)
|
||||
|
||||
with patch(
|
||||
"openjarvis.evals.skill_benchmark.SkillBenchmarkRunner.run_all_conditions",
|
||||
return_value=fake_cmp,
|
||||
):
|
||||
with patch(
|
||||
"openjarvis.evals.skill_benchmark.SkillBenchmarkRunner.write_report",
|
||||
return_value=tmp_path / "fake-report.md",
|
||||
):
|
||||
result = CliRunner().invoke(
|
||||
cli,
|
||||
[
|
||||
"bench",
|
||||
"skills",
|
||||
"--max-samples",
|
||||
"1",
|
||||
"--seeds",
|
||||
"42",
|
||||
"--output-dir",
|
||||
str(tmp_path),
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "no_skills" in result.output
|
||||
assert "skills_optimized_dspy" in result.output
|
||||
|
||||
def test_runs_single_condition(self, tmp_path: Path) -> None:
|
||||
from openjarvis.evals.skill_benchmark import ConditionResult
|
||||
|
||||
fake_result = ConditionResult(
|
||||
condition="skills_optimized_dspy",
|
||||
seeds=[42],
|
||||
per_seed_pass_rate={42: 0.7},
|
||||
mean_pass_rate=0.7,
|
||||
stddev_pass_rate=0.0,
|
||||
per_task_results={},
|
||||
skill_invocation_counts={},
|
||||
total_tokens=20,
|
||||
total_runtime_seconds=2.0,
|
||||
)
|
||||
|
||||
with patch(
|
||||
"openjarvis.evals.skill_benchmark.SkillBenchmarkRunner.run_condition",
|
||||
return_value=fake_result,
|
||||
):
|
||||
result = CliRunner().invoke(
|
||||
cli,
|
||||
[
|
||||
"bench",
|
||||
"skills",
|
||||
"--condition",
|
||||
"skills_optimized_dspy",
|
||||
"--seeds",
|
||||
"42",
|
||||
"--max-samples",
|
||||
"1",
|
||||
"--output-dir",
|
||||
str(tmp_path),
|
||||
],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "skills_optimized_dspy" in result.output
|
||||
assert "0.700" in result.output or "0.7" in result.output
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Tests for jarvis optimize skills CLI command (Plan 2A)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from openjarvis.cli import cli
|
||||
|
||||
|
||||
class TestOptimizeSkillsCommand:
|
||||
def test_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["optimize", "skills", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_dry_run_no_traces(self, tmp_path: Path) -> None:
|
||||
class _EmptyStore:
|
||||
def list_traces(self, *, limit: int = 100, **kwargs):
|
||||
return []
|
||||
|
||||
with patch(
|
||||
"openjarvis.cli.optimize_cmd._get_trace_store",
|
||||
return_value=_EmptyStore(),
|
||||
):
|
||||
result = CliRunner().invoke(cli, ["optimize", "skills", "--dry-run"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_optimize_runs_with_mocked_optimizer(self, tmp_path: Path) -> None:
|
||||
from openjarvis.core.types import StepType, Trace, TraceStep
|
||||
from openjarvis.learning.agents.skill_optimizer import (
|
||||
SkillOptimizationResult,
|
||||
)
|
||||
|
||||
def _trace(skill_name="research-skill"):
|
||||
return Trace(
|
||||
query=f"q for {skill_name}",
|
||||
steps=[
|
||||
TraceStep(
|
||||
step_type=StepType.TOOL_CALL,
|
||||
timestamp=0.0,
|
||||
input={"tool": f"skill_{skill_name}", "arguments": {}},
|
||||
output={"success": True, "result": "ok"},
|
||||
metadata={
|
||||
"skill": skill_name,
|
||||
"skill_kind": "instructional",
|
||||
},
|
||||
),
|
||||
],
|
||||
outcome="success",
|
||||
feedback=1.0,
|
||||
result="ok",
|
||||
)
|
||||
|
||||
class _Store:
|
||||
def list_traces(self, *, limit: int = 100, **kwargs):
|
||||
return [_trace() for _ in range(25)]
|
||||
|
||||
fake_results = {
|
||||
"research-skill": SkillOptimizationResult(
|
||||
skill_name="research-skill",
|
||||
status="optimized",
|
||||
trace_count=25,
|
||||
overlay_path=tmp_path / "research-skill" / "optimized.toml",
|
||||
),
|
||||
}
|
||||
|
||||
with patch(
|
||||
"openjarvis.cli.optimize_cmd._get_trace_store",
|
||||
return_value=_Store(),
|
||||
):
|
||||
with patch(
|
||||
"openjarvis.learning.agents.skill_optimizer.SkillOptimizer.optimize",
|
||||
return_value=fake_results,
|
||||
):
|
||||
result = CliRunner().invoke(
|
||||
cli, ["optimize", "skills", "--policy", "dspy"]
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "research-skill" in result.output
|
||||
assert "optimized" in result.output
|
||||
@@ -2,6 +2,10 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from openjarvis.cli import cli
|
||||
@@ -27,3 +31,225 @@ class TestSkillCmd:
|
||||
assert "install" in result.output
|
||||
assert "remove" in result.output
|
||||
assert "search" in result.output
|
||||
|
||||
def test_skill_run_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["skill", "run", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_skill_info_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["skill", "info", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_skill_update_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["skill", "update", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_skill_list_shows_discovered_skills(self, tmp_path: Path) -> None:
|
||||
skill_dir = tmp_path / "my_skill"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "skill.toml").write_text(
|
||||
textwrap.dedent("""\
|
||||
[skill]
|
||||
name = "my_skill"
|
||||
description = "A test skill"
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "echo"
|
||||
output_key = "x"
|
||||
""")
|
||||
)
|
||||
with patch(
|
||||
"openjarvis.cli.skill_cmd._get_skill_paths",
|
||||
return_value=[tmp_path],
|
||||
):
|
||||
result = CliRunner().invoke(cli, ["skill", "list"])
|
||||
assert result.exit_code == 0
|
||||
assert "my_skill" in result.output
|
||||
|
||||
def test_skill_info_shows_details(self, tmp_path: Path) -> None:
|
||||
skill_dir = tmp_path / "info_skill"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "skill.toml").write_text(
|
||||
textwrap.dedent("""\
|
||||
[skill]
|
||||
name = "info_skill"
|
||||
description = "Detailed skill"
|
||||
author = "test_author"
|
||||
tags = ["research", "test"]
|
||||
required_capabilities = ["network:fetch"]
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "echo"
|
||||
output_key = "x"
|
||||
""")
|
||||
)
|
||||
with patch(
|
||||
"openjarvis.cli.skill_cmd._get_skill_paths",
|
||||
return_value=[tmp_path],
|
||||
):
|
||||
result = CliRunner().invoke(cli, ["skill", "info", "info_skill"])
|
||||
assert result.exit_code == 0
|
||||
assert "info_skill" in result.output
|
||||
assert "test_author" in result.output
|
||||
|
||||
|
||||
class TestSkillInstallCommand:
|
||||
def test_install_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["skill", "install", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "source" in result.output.lower()
|
||||
|
||||
def test_install_invalid_source_format(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["skill", "install", "invalid"])
|
||||
assert result.exit_code != 0
|
||||
assert "format" in result.output.lower() or "source" in result.output.lower()
|
||||
|
||||
|
||||
class TestSkillSyncCommand:
|
||||
def test_sync_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["skill", "sync", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
|
||||
class TestSkillSourcesCommand:
|
||||
def test_sources_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["skill", "sources", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_sources_lists_configured_sources(self, tmp_path: Path) -> None:
|
||||
from unittest.mock import patch
|
||||
|
||||
from openjarvis.core.config import (
|
||||
JarvisConfig,
|
||||
SkillsConfig,
|
||||
SkillSourceConfig,
|
||||
)
|
||||
|
||||
cfg = JarvisConfig()
|
||||
cfg.skills = SkillsConfig(
|
||||
sources=[
|
||||
SkillSourceConfig(source="hermes"),
|
||||
SkillSourceConfig(source="openclaw"),
|
||||
]
|
||||
)
|
||||
with patch("openjarvis.cli.skill_cmd.load_config", return_value=cfg):
|
||||
result = CliRunner().invoke(cli, ["skill", "sources"])
|
||||
assert result.exit_code == 0
|
||||
assert "hermes" in result.output
|
||||
assert "openclaw" in result.output
|
||||
|
||||
|
||||
class TestSkillDiscoverCommand:
|
||||
def test_discover_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["skill", "discover", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_discover_dry_run_no_traces(self, tmp_path: Path) -> None:
|
||||
from unittest.mock import patch
|
||||
|
||||
# Patch the trace store to return nothing
|
||||
class _EmptyStore:
|
||||
def list_traces(self, *, limit: int = 100, **kwargs):
|
||||
return []
|
||||
|
||||
with patch(
|
||||
"openjarvis.cli.skill_cmd._get_trace_store",
|
||||
return_value=_EmptyStore(),
|
||||
):
|
||||
result = CliRunner().invoke(cli, ["skill", "discover", "--dry-run"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_discover_writes_when_not_dry_run(self, tmp_path: Path) -> None:
|
||||
from unittest.mock import patch
|
||||
|
||||
from openjarvis.core.types import StepType, Trace, TraceStep
|
||||
|
||||
def _trace():
|
||||
return Trace(
|
||||
query="q",
|
||||
steps=[
|
||||
TraceStep(
|
||||
step_type=StepType.TOOL_CALL,
|
||||
timestamp=0.0,
|
||||
input={"tool": "web_search", "arguments": {}},
|
||||
output={"success": True, "result": "ok"},
|
||||
metadata={},
|
||||
),
|
||||
TraceStep(
|
||||
step_type=StepType.TOOL_CALL,
|
||||
timestamp=0.0,
|
||||
input={"tool": "calculator", "arguments": {}},
|
||||
output={"success": True, "result": "ok"},
|
||||
metadata={},
|
||||
),
|
||||
],
|
||||
outcome="success",
|
||||
feedback=1.0,
|
||||
)
|
||||
|
||||
class _Store:
|
||||
def list_traces(self, *, limit: int = 100, **kwargs):
|
||||
return [_trace() for _ in range(5)]
|
||||
|
||||
output_dir = tmp_path / "discovered"
|
||||
with patch(
|
||||
"openjarvis.cli.skill_cmd._get_trace_store",
|
||||
return_value=_Store(),
|
||||
):
|
||||
with patch(
|
||||
"openjarvis.cli.skill_cmd._get_discovered_dir",
|
||||
return_value=output_dir,
|
||||
):
|
||||
result = CliRunner().invoke(
|
||||
cli, ["skill", "discover", "--min-frequency", "3"]
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert output_dir.exists()
|
||||
# At least one manifest should have been written
|
||||
assert any(output_dir.rglob("skill.toml"))
|
||||
|
||||
|
||||
class TestSkillShowOverlayCommand:
|
||||
def test_show_overlay_help(self) -> None:
|
||||
result = CliRunner().invoke(cli, ["skill", "show-overlay", "--help"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
def test_show_overlay_missing(self, tmp_path: Path) -> None:
|
||||
from unittest.mock import patch
|
||||
|
||||
with patch(
|
||||
"openjarvis.cli.skill_cmd._get_overlay_dir",
|
||||
return_value=tmp_path / "no-such",
|
||||
):
|
||||
result = CliRunner().invoke(cli, ["skill", "show-overlay", "nonexistent"])
|
||||
assert result.exit_code != 0
|
||||
|
||||
def test_show_overlay_displays_optimized(self, tmp_path: Path) -> None:
|
||||
from unittest.mock import patch
|
||||
|
||||
from openjarvis.skills.overlay import SkillOverlay, write_overlay
|
||||
|
||||
write_overlay(
|
||||
SkillOverlay(
|
||||
skill_name="research-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T14:30:00Z",
|
||||
trace_count=42,
|
||||
description="Better description",
|
||||
few_shot=[{"input": "q", "output": "a"}],
|
||||
),
|
||||
tmp_path,
|
||||
)
|
||||
|
||||
with patch(
|
||||
"openjarvis.cli.skill_cmd._get_overlay_dir",
|
||||
return_value=tmp_path,
|
||||
):
|
||||
result = CliRunner().invoke(
|
||||
cli, ["skill", "show-overlay", "research-skill"]
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "research-skill" in result.output
|
||||
assert "Better description" in result.output
|
||||
assert "42" in result.output
|
||||
assert "dspy" in result.output
|
||||
|
||||
@@ -19,6 +19,7 @@ from openjarvis.core.registry import (
|
||||
MemoryRegistry,
|
||||
ModelRegistry,
|
||||
RouterPolicyRegistry,
|
||||
SkillRegistry,
|
||||
SpeechRegistry,
|
||||
ToolRegistry,
|
||||
TTSRegistry,
|
||||
@@ -40,6 +41,7 @@ def _clean_registries() -> None:
|
||||
CompressionRegistry.clear()
|
||||
ConnectorRegistry.clear()
|
||||
TTSRegistry.clear()
|
||||
SkillRegistry.clear()
|
||||
reset_event_bus()
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Tests for skills.sources config section."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from openjarvis.core.config import SkillsConfig, SkillSourceConfig
|
||||
|
||||
|
||||
class TestSkillSourceConfig:
|
||||
def test_default_filter_empty(self):
|
||||
cfg = SkillSourceConfig(source="hermes")
|
||||
assert cfg.source == "hermes"
|
||||
assert cfg.filter == {}
|
||||
assert cfg.auto_update is False
|
||||
assert cfg.url == ""
|
||||
|
||||
def test_with_filter(self):
|
||||
cfg = SkillSourceConfig(
|
||||
source="hermes",
|
||||
filter={"category": ["research", "coding"]},
|
||||
)
|
||||
assert cfg.filter["category"] == ["research", "coding"]
|
||||
|
||||
|
||||
class TestSkillsConfigWithSources:
|
||||
def test_default_no_sources(self):
|
||||
cfg = SkillsConfig()
|
||||
assert cfg.enabled is True
|
||||
assert cfg.auto_sync is False
|
||||
assert cfg.sources == []
|
||||
|
||||
def test_auto_sync_can_be_enabled(self):
|
||||
cfg = SkillsConfig(auto_sync=True)
|
||||
assert cfg.auto_sync is True
|
||||
|
||||
def test_can_add_sources(self):
|
||||
cfg = SkillsConfig(
|
||||
sources=[
|
||||
SkillSourceConfig(source="hermes"),
|
||||
SkillSourceConfig(source="openclaw"),
|
||||
]
|
||||
)
|
||||
assert len(cfg.sources) == 2
|
||||
assert cfg.sources[0].source == "hermes"
|
||||
@@ -0,0 +1,40 @@
|
||||
"""Tests for SkillsLearningConfig and its wiring into LearningConfig."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from openjarvis.core.config import LearningConfig, SkillsLearningConfig
|
||||
|
||||
|
||||
class TestSkillsLearningConfig:
|
||||
def test_defaults(self):
|
||||
cfg = SkillsLearningConfig()
|
||||
assert cfg.auto_optimize is False
|
||||
assert cfg.optimizer == "dspy"
|
||||
assert cfg.min_traces_per_skill == 20
|
||||
assert cfg.optimization_interval_seconds == 86400
|
||||
assert cfg.overlay_dir == "~/.openjarvis/learning/skills/"
|
||||
|
||||
def test_can_be_constructed_with_all_fields(self):
|
||||
cfg = SkillsLearningConfig(
|
||||
auto_optimize=True,
|
||||
optimizer="gepa",
|
||||
min_traces_per_skill=10,
|
||||
optimization_interval_seconds=3600,
|
||||
overlay_dir="/tmp/overlays/",
|
||||
)
|
||||
assert cfg.auto_optimize is True
|
||||
assert cfg.optimizer == "gepa"
|
||||
assert cfg.min_traces_per_skill == 10
|
||||
assert cfg.optimization_interval_seconds == 3600
|
||||
assert cfg.overlay_dir == "/tmp/overlays/"
|
||||
|
||||
|
||||
class TestLearningConfigSkillsField:
|
||||
def test_skills_field_present(self):
|
||||
cfg = LearningConfig()
|
||||
assert hasattr(cfg, "skills")
|
||||
assert isinstance(cfg.skills, SkillsLearningConfig)
|
||||
|
||||
def test_skills_field_default_disabled(self):
|
||||
cfg = LearningConfig()
|
||||
assert cfg.skills.auto_optimize is False
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Tests for JarvisAgentBackend skills_enabled / overlay_dir kwargs (Plan 2B)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class TestJarvisAgentBackendSkillsKwargs:
|
||||
def test_default_skills_enabled_true(self):
|
||||
from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
|
||||
|
||||
# Construction should not raise even if no model/engine is reachable
|
||||
# because we don't call generate(). The kwargs themselves should be
|
||||
# accepted and applied to the SystemBuilder config.
|
||||
try:
|
||||
backend = JarvisAgentBackend(
|
||||
engine_key="ollama",
|
||||
agent_name="native_react",
|
||||
tools=[],
|
||||
skills_enabled=True,
|
||||
)
|
||||
except RuntimeError:
|
||||
# If no engine is available we still consider the test passing
|
||||
# for this kwarg-acceptance check; the kwarg was applied before
|
||||
# the engine resolution failed.
|
||||
return
|
||||
assert backend._system.config.skills.enabled is True
|
||||
|
||||
def test_skills_enabled_false_disables_skills(self):
|
||||
from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
|
||||
|
||||
try:
|
||||
backend = JarvisAgentBackend(
|
||||
engine_key="ollama",
|
||||
agent_name="native_react",
|
||||
tools=[],
|
||||
skills_enabled=False,
|
||||
)
|
||||
except RuntimeError:
|
||||
return
|
||||
assert backend._system.config.skills.enabled is False
|
||||
assert backend._system.skill_manager is None
|
||||
|
||||
def test_overlay_dir_kwarg_applied_to_config(self, tmp_path: Path):
|
||||
from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
|
||||
|
||||
custom = tmp_path / "custom-overlays"
|
||||
try:
|
||||
backend = JarvisAgentBackend(
|
||||
engine_key="ollama",
|
||||
agent_name="native_react",
|
||||
tools=[],
|
||||
skills_enabled=True,
|
||||
overlay_dir=custom,
|
||||
)
|
||||
except RuntimeError:
|
||||
return
|
||||
# The config should reflect our overlay_dir override
|
||||
assert backend._system.config.learning.skills.overlay_dir == str(custom)
|
||||
|
||||
def test_init_signature_accepts_kwargs_without_engine(self):
|
||||
"""Even without a real engine, the __init__ signature must accept
|
||||
the new kwargs without raising TypeError."""
|
||||
import inspect
|
||||
|
||||
from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
|
||||
|
||||
sig = inspect.signature(JarvisAgentBackend.__init__)
|
||||
assert "skills_enabled" in sig.parameters
|
||||
assert "overlay_dir" in sig.parameters
|
||||
@@ -0,0 +1,382 @@
|
||||
"""Tests for SkillBenchmarkRunner (Plan 2B)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
class TestSkillBenchmarkConfigDefaults:
|
||||
def test_defaults(self):
|
||||
from openjarvis.evals.skill_benchmark import SkillBenchmarkConfig
|
||||
|
||||
cfg = SkillBenchmarkConfig()
|
||||
assert cfg.benchmark == "pinchbench"
|
||||
assert cfg.model == "qwen3.5:9b"
|
||||
assert cfg.engine == "ollama"
|
||||
assert cfg.agent == "native_react"
|
||||
assert cfg.seeds == [42, 43, 44]
|
||||
assert cfg.max_samples is None
|
||||
assert "shell_exec" in cfg.tools
|
||||
assert "web_search" in cfg.tools
|
||||
|
||||
def test_construct_with_overrides(self):
|
||||
from openjarvis.evals.skill_benchmark import SkillBenchmarkConfig
|
||||
|
||||
cfg = SkillBenchmarkConfig(
|
||||
benchmark="pinchbench",
|
||||
model="other-model",
|
||||
seeds=[1, 2],
|
||||
max_samples=5,
|
||||
)
|
||||
assert cfg.model == "other-model"
|
||||
assert cfg.seeds == [1, 2]
|
||||
assert cfg.max_samples == 5
|
||||
|
||||
|
||||
class TestConditionResult:
|
||||
def test_create(self):
|
||||
from openjarvis.evals.skill_benchmark import ConditionResult
|
||||
|
||||
r = ConditionResult(
|
||||
condition="no_skills",
|
||||
seeds=[42, 43, 44],
|
||||
per_seed_pass_rate={42: 0.30, 43: 0.32, 44: 0.28},
|
||||
mean_pass_rate=0.30,
|
||||
stddev_pass_rate=0.02,
|
||||
per_task_results={"task_001": [True, False, True]},
|
||||
skill_invocation_counts={},
|
||||
total_tokens=1000,
|
||||
total_runtime_seconds=120.0,
|
||||
)
|
||||
assert r.condition == "no_skills"
|
||||
assert r.mean_pass_rate == 0.30
|
||||
assert r.skill_invocation_counts == {}
|
||||
|
||||
|
||||
class TestConditionComparison:
|
||||
def test_create(self):
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
ConditionComparison,
|
||||
ConditionResult,
|
||||
SkillBenchmarkConfig,
|
||||
)
|
||||
|
||||
cfg = SkillBenchmarkConfig()
|
||||
cmp = ConditionComparison(
|
||||
config=cfg,
|
||||
started_at="2026-04-08T00:00:00Z",
|
||||
ended_at="2026-04-08T01:00:00Z",
|
||||
results={
|
||||
"no_skills": ConditionResult(
|
||||
condition="no_skills",
|
||||
seeds=[42],
|
||||
per_seed_pass_rate={42: 0.30},
|
||||
mean_pass_rate=0.30,
|
||||
stddev_pass_rate=0.0,
|
||||
per_task_results={},
|
||||
skill_invocation_counts={},
|
||||
total_tokens=0,
|
||||
total_runtime_seconds=0.0,
|
||||
),
|
||||
},
|
||||
deltas={},
|
||||
)
|
||||
assert "no_skills" in cmp.results
|
||||
assert cmp.config.benchmark == "pinchbench"
|
||||
|
||||
|
||||
class TestBuildBackendForCondition:
|
||||
def _make_runner(self, tmp_path: Path):
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
SkillBenchmarkConfig,
|
||||
SkillBenchmarkRunner,
|
||||
)
|
||||
|
||||
cfg = SkillBenchmarkConfig(
|
||||
engine="ollama",
|
||||
model="qwen3.5:9b",
|
||||
tools=[],
|
||||
seeds=[42],
|
||||
output_dir=tmp_path,
|
||||
overlay_dir_dspy=tmp_path / "dspy",
|
||||
overlay_dir_gepa=tmp_path / "gepa",
|
||||
)
|
||||
return SkillBenchmarkRunner(cfg)
|
||||
|
||||
def test_unknown_condition_raises(self, tmp_path: Path):
|
||||
runner = self._make_runner(tmp_path)
|
||||
with pytest.raises(ValueError, match="condition"):
|
||||
runner._backend_kwargs_for_condition("not_a_condition")
|
||||
|
||||
def test_no_skills_kwargs(self, tmp_path: Path):
|
||||
runner = self._make_runner(tmp_path)
|
||||
kw = runner._backend_kwargs_for_condition("no_skills")
|
||||
assert kw["skills_enabled"] is False
|
||||
assert kw["overlay_dir"] is None
|
||||
|
||||
def test_skills_on_kwargs(self, tmp_path: Path):
|
||||
runner = self._make_runner(tmp_path)
|
||||
kw = runner._backend_kwargs_for_condition("skills_on")
|
||||
assert kw["skills_enabled"] is True
|
||||
# skills_on uses an empty/missing overlay dir so no overlays load
|
||||
assert kw["overlay_dir"] is not None
|
||||
assert "skills_on_empty_overlays" in str(kw["overlay_dir"])
|
||||
|
||||
def test_skills_optimized_dspy_kwargs(self, tmp_path: Path):
|
||||
runner = self._make_runner(tmp_path)
|
||||
kw = runner._backend_kwargs_for_condition("skills_optimized_dspy")
|
||||
assert kw["skills_enabled"] is True
|
||||
assert kw["overlay_dir"] == tmp_path / "dspy"
|
||||
|
||||
def test_skills_optimized_gepa_kwargs(self, tmp_path: Path):
|
||||
runner = self._make_runner(tmp_path)
|
||||
kw = runner._backend_kwargs_for_condition("skills_optimized_gepa")
|
||||
assert kw["skills_enabled"] is True
|
||||
assert kw["overlay_dir"] == tmp_path / "gepa"
|
||||
|
||||
|
||||
class TestRunCondition:
|
||||
def _make_runner(self, tmp_path: Path):
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
SkillBenchmarkConfig,
|
||||
SkillBenchmarkRunner,
|
||||
)
|
||||
|
||||
cfg = SkillBenchmarkConfig(
|
||||
seeds=[42, 43],
|
||||
max_samples=2,
|
||||
output_dir=tmp_path,
|
||||
)
|
||||
return SkillBenchmarkRunner(cfg)
|
||||
|
||||
def test_run_condition_aggregates_per_seed(
|
||||
self, tmp_path: Path, monkeypatch: Any
|
||||
) -> None:
|
||||
"""run_condition runs the eval once per seed and aggregates the
|
||||
results into a ConditionResult."""
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
SkillBenchmarkRunner,
|
||||
)
|
||||
|
||||
runner = self._make_runner(tmp_path)
|
||||
|
||||
# Stub _run_single_seed to return synthetic per-seed data
|
||||
seed_results = {
|
||||
42: {
|
||||
"pass_rate": 0.5,
|
||||
"per_task": {"task_001": True, "task_002": False},
|
||||
"skill_invocations": {"research-skill": 1},
|
||||
"total_tokens": 100,
|
||||
"total_runtime_seconds": 10.0,
|
||||
},
|
||||
43: {
|
||||
"pass_rate": 1.0,
|
||||
"per_task": {"task_001": True, "task_002": True},
|
||||
"skill_invocations": {"research-skill": 2},
|
||||
"total_tokens": 200,
|
||||
"total_runtime_seconds": 20.0,
|
||||
},
|
||||
}
|
||||
|
||||
def fake_run_single_seed(self_unused, condition, seed):
|
||||
return seed_results[seed]
|
||||
|
||||
monkeypatch.setattr(
|
||||
SkillBenchmarkRunner,
|
||||
"_run_single_seed",
|
||||
fake_run_single_seed,
|
||||
)
|
||||
|
||||
result = runner.run_condition("no_skills")
|
||||
|
||||
assert result.condition == "no_skills"
|
||||
assert result.seeds == [42, 43]
|
||||
assert result.per_seed_pass_rate == {42: 0.5, 43: 1.0}
|
||||
assert result.mean_pass_rate == 0.75
|
||||
assert result.stddev_pass_rate > 0.0
|
||||
# Per-task aggregation: each task gets a list of [seed1_pass, seed2_pass]
|
||||
assert result.per_task_results["task_001"] == [True, True]
|
||||
assert result.per_task_results["task_002"] == [False, True]
|
||||
# Skill invocation counts summed across seeds
|
||||
assert result.skill_invocation_counts["research-skill"] == 3
|
||||
assert result.total_tokens == 300
|
||||
assert result.total_runtime_seconds == 30.0
|
||||
|
||||
def test_run_condition_single_seed_zero_stddev(
|
||||
self, tmp_path: Path, monkeypatch: Any
|
||||
) -> None:
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
SkillBenchmarkConfig,
|
||||
SkillBenchmarkRunner,
|
||||
)
|
||||
|
||||
cfg = SkillBenchmarkConfig(
|
||||
seeds=[42],
|
||||
max_samples=1,
|
||||
output_dir=tmp_path,
|
||||
)
|
||||
runner = SkillBenchmarkRunner(cfg)
|
||||
|
||||
def fake_run_single_seed(self_unused, condition, seed):
|
||||
return {
|
||||
"pass_rate": 0.42,
|
||||
"per_task": {"task_001": False},
|
||||
"skill_invocations": {},
|
||||
"total_tokens": 50,
|
||||
"total_runtime_seconds": 5.0,
|
||||
}
|
||||
|
||||
monkeypatch.setattr(
|
||||
SkillBenchmarkRunner,
|
||||
"_run_single_seed",
|
||||
fake_run_single_seed,
|
||||
)
|
||||
|
||||
result = runner.run_condition("no_skills")
|
||||
assert result.mean_pass_rate == 0.42
|
||||
# Single seed → stddev is 0
|
||||
assert result.stddev_pass_rate == 0.0
|
||||
|
||||
|
||||
class TestRunAllConditions:
|
||||
def test_run_all_conditions_invokes_all_four(
|
||||
self, tmp_path: Path, monkeypatch: Any
|
||||
) -> None:
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
ConditionResult,
|
||||
SkillBenchmarkConfig,
|
||||
SkillBenchmarkRunner,
|
||||
)
|
||||
|
||||
cfg = SkillBenchmarkConfig(
|
||||
seeds=[42],
|
||||
max_samples=1,
|
||||
output_dir=tmp_path,
|
||||
)
|
||||
runner = SkillBenchmarkRunner(cfg)
|
||||
|
||||
invoked: list = []
|
||||
|
||||
def fake_run_condition(self_unused, condition):
|
||||
invoked.append(condition)
|
||||
return ConditionResult(
|
||||
condition=condition,
|
||||
seeds=[42],
|
||||
per_seed_pass_rate={42: 0.5},
|
||||
mean_pass_rate=0.5,
|
||||
stddev_pass_rate=0.0,
|
||||
per_task_results={},
|
||||
skill_invocation_counts={},
|
||||
total_tokens=10,
|
||||
total_runtime_seconds=1.0,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
SkillBenchmarkRunner,
|
||||
"run_condition",
|
||||
fake_run_condition,
|
||||
)
|
||||
|
||||
comparison = runner.run_all_conditions()
|
||||
assert set(invoked) == {
|
||||
"no_skills",
|
||||
"skills_on",
|
||||
"skills_optimized_dspy",
|
||||
"skills_optimized_gepa",
|
||||
}
|
||||
assert len(comparison.results) == 4
|
||||
|
||||
def test_run_all_conditions_computes_deltas(
|
||||
self, tmp_path: Path, monkeypatch: Any
|
||||
) -> None:
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
ConditionResult,
|
||||
SkillBenchmarkConfig,
|
||||
SkillBenchmarkRunner,
|
||||
)
|
||||
|
||||
cfg = SkillBenchmarkConfig(
|
||||
seeds=[42],
|
||||
max_samples=1,
|
||||
output_dir=tmp_path,
|
||||
)
|
||||
runner = SkillBenchmarkRunner(cfg)
|
||||
|
||||
rates = {
|
||||
"no_skills": 0.30,
|
||||
"skills_on": 0.35,
|
||||
"skills_optimized_dspy": 0.40,
|
||||
"skills_optimized_gepa": 0.38,
|
||||
}
|
||||
|
||||
def fake_run_condition(self_unused, condition):
|
||||
return ConditionResult(
|
||||
condition=condition,
|
||||
seeds=[42],
|
||||
per_seed_pass_rate={42: rates[condition]},
|
||||
mean_pass_rate=rates[condition],
|
||||
stddev_pass_rate=0.0,
|
||||
per_task_results={},
|
||||
skill_invocation_counts={},
|
||||
total_tokens=0,
|
||||
total_runtime_seconds=0.0,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
SkillBenchmarkRunner,
|
||||
"run_condition",
|
||||
fake_run_condition,
|
||||
)
|
||||
|
||||
comparison = runner.run_all_conditions()
|
||||
# Deltas use the same names as the conditions
|
||||
assert comparison.deltas["skills_on - no_skills"] == pytest.approx(0.05)
|
||||
assert comparison.deltas["skills_optimized_dspy - skills_on"] == pytest.approx(
|
||||
0.05
|
||||
)
|
||||
assert comparison.deltas["skills_optimized_gepa - skills_on"] == pytest.approx(
|
||||
0.03
|
||||
)
|
||||
|
||||
|
||||
class TestWriteReport:
|
||||
def test_write_report_creates_dated_markdown(self, tmp_path: Path) -> None:
|
||||
from openjarvis.evals.skill_benchmark import (
|
||||
ConditionComparison,
|
||||
ConditionResult,
|
||||
SkillBenchmarkConfig,
|
||||
SkillBenchmarkRunner,
|
||||
)
|
||||
|
||||
cfg = SkillBenchmarkConfig(output_dir=tmp_path)
|
||||
runner = SkillBenchmarkRunner(cfg)
|
||||
|
||||
result = ConditionResult(
|
||||
condition="no_skills",
|
||||
seeds=[42, 43],
|
||||
per_seed_pass_rate={42: 0.30, 43: 0.32},
|
||||
mean_pass_rate=0.31,
|
||||
stddev_pass_rate=0.014,
|
||||
per_task_results={"task_001": [True, False]},
|
||||
skill_invocation_counts={},
|
||||
total_tokens=100,
|
||||
total_runtime_seconds=12.5,
|
||||
)
|
||||
cmp = ConditionComparison(
|
||||
config=cfg,
|
||||
started_at="2026-04-08T00:00:00Z",
|
||||
ended_at="2026-04-08T00:01:00Z",
|
||||
results={"no_skills": result},
|
||||
deltas={},
|
||||
)
|
||||
|
||||
path = runner.write_report(cmp)
|
||||
assert path.exists()
|
||||
assert path.suffix == ".md"
|
||||
content = path.read_text()
|
||||
assert "no_skills" in content
|
||||
assert "0.31" in content or "31.0" in content
|
||||
assert "task_001" in content
|
||||
@@ -0,0 +1,146 @@
|
||||
"""Tests for LearningOrchestrator opt-in skill optimization (Plan 2A)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class TestOrchestratorSkillAutoOptimize:
|
||||
def test_auto_optimize_disabled_by_default_does_not_call_skill_optimizer(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
from openjarvis.learning.learning_orchestrator import (
|
||||
LearningOrchestrator,
|
||||
)
|
||||
|
||||
store = MagicMock()
|
||||
store.list_traces.return_value = []
|
||||
|
||||
orchestrator = LearningOrchestrator(
|
||||
trace_store=store,
|
||||
config_dir=tmp_path,
|
||||
)
|
||||
|
||||
with patch(
|
||||
"openjarvis.learning.agents.skill_optimizer.SkillOptimizer.optimize"
|
||||
) as mock_optimize:
|
||||
orchestrator._maybe_optimize_skills(auto_optimize=False)
|
||||
mock_optimize.assert_not_called()
|
||||
|
||||
def test_auto_optimize_enabled_calls_skill_optimizer(self, tmp_path: Path) -> None:
|
||||
from openjarvis.learning.learning_orchestrator import (
|
||||
LearningOrchestrator,
|
||||
)
|
||||
|
||||
store = MagicMock()
|
||||
store.list_traces.return_value = []
|
||||
|
||||
orchestrator = LearningOrchestrator(
|
||||
trace_store=store,
|
||||
config_dir=tmp_path,
|
||||
)
|
||||
|
||||
with patch(
|
||||
"openjarvis.learning.agents.skill_optimizer.SkillOptimizer.optimize",
|
||||
return_value={},
|
||||
) as mock_optimize:
|
||||
orchestrator._maybe_optimize_skills(
|
||||
auto_optimize=True,
|
||||
optimizer="dspy",
|
||||
min_traces_per_skill=5,
|
||||
)
|
||||
mock_optimize.assert_called_once()
|
||||
|
||||
|
||||
class TestOrchestratorRunSkillTrigger:
|
||||
"""End-to-end: LearningOrchestrator.run() invokes _maybe_optimize_skills
|
||||
when learning.skills.auto_optimize is true (Plan 2A C2 fix)."""
|
||||
|
||||
def _make_store(self) -> MagicMock:
|
||||
store = MagicMock()
|
||||
# Just enough surface area for orchestrator.run() to short-circuit
|
||||
store.list_traces.return_value = []
|
||||
return store
|
||||
|
||||
def _make_config(self, *, auto_optimize: bool):
|
||||
from openjarvis.core.config import (
|
||||
JarvisConfig,
|
||||
LearningConfig,
|
||||
SkillsLearningConfig,
|
||||
)
|
||||
|
||||
cfg = JarvisConfig()
|
||||
cfg.learning = LearningConfig()
|
||||
cfg.learning.skills = SkillsLearningConfig(
|
||||
auto_optimize=auto_optimize,
|
||||
optimizer="dspy",
|
||||
min_traces_per_skill=5,
|
||||
)
|
||||
return cfg
|
||||
|
||||
def test_run_does_not_call_skill_optimizer_when_disabled(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
from openjarvis.learning.learning_orchestrator import (
|
||||
LearningOrchestrator,
|
||||
)
|
||||
|
||||
store = self._make_store()
|
||||
orchestrator = LearningOrchestrator(
|
||||
trace_store=store,
|
||||
config_dir=tmp_path,
|
||||
)
|
||||
|
||||
with patch(
|
||||
"openjarvis.core.config.load_config",
|
||||
return_value=self._make_config(auto_optimize=False),
|
||||
):
|
||||
with patch(
|
||||
"openjarvis.learning.agents.skill_optimizer.SkillOptimizer.optimize"
|
||||
) as mock_optimize:
|
||||
orchestrator.run()
|
||||
mock_optimize.assert_not_called()
|
||||
|
||||
def test_run_calls_skill_optimizer_when_enabled(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
from openjarvis.learning.agents.skill_optimizer import (
|
||||
SkillOptimizationResult,
|
||||
)
|
||||
from openjarvis.learning.learning_orchestrator import (
|
||||
LearningOrchestrator,
|
||||
)
|
||||
|
||||
store = self._make_store()
|
||||
orchestrator = LearningOrchestrator(
|
||||
trace_store=store,
|
||||
config_dir=tmp_path,
|
||||
)
|
||||
|
||||
fake_results = {
|
||||
"research-skill": SkillOptimizationResult(
|
||||
skill_name="research-skill",
|
||||
status="optimized",
|
||||
trace_count=10,
|
||||
),
|
||||
}
|
||||
|
||||
with patch(
|
||||
"openjarvis.core.config.load_config",
|
||||
return_value=self._make_config(auto_optimize=True),
|
||||
):
|
||||
with patch(
|
||||
"openjarvis.learning.agents.skill_optimizer.SkillOptimizer.optimize",
|
||||
return_value=fake_results,
|
||||
) as mock_optimize:
|
||||
result = orchestrator.run()
|
||||
mock_optimize.assert_called_once()
|
||||
# The orchestrator should record the skill optimization
|
||||
# results in the returned dict
|
||||
assert "skill_optimization" in result
|
||||
assert "research-skill" in result["skill_optimization"]
|
||||
assert (
|
||||
result["skill_optimization"]["research-skill"]["status"]
|
||||
== "optimized"
|
||||
)
|
||||
@@ -0,0 +1,141 @@
|
||||
"""Tests for SkillOptimizer (Plan 2A) — buckets traces by skill, runs DSPy/GEPA."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, List
|
||||
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.core.types import StepType, Trace, TraceStep
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
|
||||
class _FakeTraceStore:
|
||||
def __init__(self, traces: List[Trace]) -> None:
|
||||
self._traces = traces
|
||||
|
||||
def list_traces(self, *, limit: int = 100, **_kwargs: Any) -> List[Trace]:
|
||||
return list(self._traces[:limit])
|
||||
|
||||
|
||||
def _make_skill_trace(skill_name: str, feedback: float = 1.0) -> Trace:
|
||||
return Trace(
|
||||
query=f"query for {skill_name}",
|
||||
agent="native_react",
|
||||
model="qwen3.5:9b",
|
||||
engine="ollama",
|
||||
steps=[
|
||||
TraceStep(
|
||||
step_type=StepType.TOOL_CALL,
|
||||
timestamp=0.0,
|
||||
input={"tool": f"skill_{skill_name}", "arguments": {}},
|
||||
output={"success": True, "result": "ok"},
|
||||
metadata={"skill": skill_name, "skill_kind": "instructional"},
|
||||
),
|
||||
],
|
||||
outcome="success",
|
||||
feedback=feedback,
|
||||
result="result text",
|
||||
)
|
||||
|
||||
|
||||
def _make_manager_with_skill(name: str, tmp_path: Path) -> SkillManager:
|
||||
skill_dir = tmp_path / "skills" / name
|
||||
skill_dir.mkdir(parents=True)
|
||||
(skill_dir / "SKILL.md").write_text(
|
||||
f"---\nname: {name}\ndescription: Original description\n---\nBody"
|
||||
)
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path / "skills"])
|
||||
return mgr
|
||||
|
||||
|
||||
class TestSkillOptimizerBucketing:
|
||||
def test_buckets_traces_by_skill_name(self, tmp_path: Path):
|
||||
from openjarvis.learning.agents.skill_optimizer import SkillOptimizer
|
||||
|
||||
traces = [
|
||||
_make_skill_trace("research-skill"),
|
||||
_make_skill_trace("research-skill"),
|
||||
_make_skill_trace("code-skill"),
|
||||
]
|
||||
optimizer = SkillOptimizer(min_traces_per_skill=1)
|
||||
buckets = optimizer._bucket_traces_by_skill(traces)
|
||||
|
||||
assert "research-skill" in buckets
|
||||
assert "code-skill" in buckets
|
||||
assert len(buckets["research-skill"]) == 2
|
||||
assert len(buckets["code-skill"]) == 1
|
||||
|
||||
def test_skips_traces_without_skill_metadata(self, tmp_path: Path):
|
||||
from openjarvis.learning.agents.skill_optimizer import SkillOptimizer
|
||||
|
||||
# A trace with no skill metadata in the tool call
|
||||
plain_trace = Trace(
|
||||
query="plain",
|
||||
steps=[
|
||||
TraceStep(
|
||||
step_type=StepType.TOOL_CALL,
|
||||
timestamp=0.0,
|
||||
input={"tool": "calculator", "arguments": {}},
|
||||
output={"success": True, "result": "42"},
|
||||
metadata={},
|
||||
),
|
||||
],
|
||||
)
|
||||
optimizer = SkillOptimizer(min_traces_per_skill=1)
|
||||
buckets = optimizer._bucket_traces_by_skill([plain_trace])
|
||||
assert buckets == {}
|
||||
|
||||
|
||||
class TestSkillOptimizerOptimize:
|
||||
def test_skips_skills_below_min_traces(self, tmp_path: Path):
|
||||
from openjarvis.learning.agents.skill_optimizer import SkillOptimizer
|
||||
|
||||
traces = [_make_skill_trace("research-skill") for _ in range(3)]
|
||||
store = _FakeTraceStore(traces)
|
||||
mgr = _make_manager_with_skill("research-skill", tmp_path)
|
||||
|
||||
optimizer = SkillOptimizer(min_traces_per_skill=20)
|
||||
results = optimizer.optimize(store, mgr, overlay_dir=tmp_path / "overlays")
|
||||
|
||||
assert "research-skill" in results
|
||||
assert results["research-skill"].status == "skipped"
|
||||
assert results["research-skill"].trace_count == 3
|
||||
|
||||
def test_optimizes_skill_with_enough_traces(
|
||||
self, tmp_path: Path, monkeypatch: Any
|
||||
) -> None:
|
||||
from openjarvis.learning.agents.skill_optimizer import (
|
||||
SkillOptimizer,
|
||||
_OptimizerOutput,
|
||||
)
|
||||
|
||||
traces = [_make_skill_trace("research-skill") for _ in range(25)]
|
||||
store = _FakeTraceStore(traces)
|
||||
mgr = _make_manager_with_skill("research-skill", tmp_path)
|
||||
|
||||
# Mock the underlying DSPy call
|
||||
def fake_run(self_unused, skill_name, skill_traces):
|
||||
return _OptimizerOutput(
|
||||
description="An optimized description",
|
||||
few_shot=[
|
||||
{"input": "hello", "output": "world"},
|
||||
],
|
||||
)
|
||||
|
||||
monkeypatch.setattr(SkillOptimizer, "_run_dspy", fake_run)
|
||||
|
||||
optimizer = SkillOptimizer(min_traces_per_skill=10)
|
||||
results = optimizer.optimize(store, mgr, overlay_dir=tmp_path / "overlays")
|
||||
|
||||
assert results["research-skill"].status == "optimized"
|
||||
assert results["research-skill"].trace_count == 25
|
||||
# Overlay file should exist
|
||||
overlay_path = tmp_path / "overlays" / "research-skill" / "optimized.toml"
|
||||
assert overlay_path.exists()
|
||||
# And contain the optimized description
|
||||
content = overlay_path.read_text()
|
||||
assert "An optimized description" in content
|
||||
assert "hello" in content
|
||||
assert "world" in content
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Tests for SystemPromptBuilder skill few-shot injection (Plan 2A)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from openjarvis.prompt.builder import SystemPromptBuilder
|
||||
|
||||
|
||||
class TestSystemPromptBuilderFewShot:
|
||||
def test_no_few_shot_no_section(self):
|
||||
builder = SystemPromptBuilder(
|
||||
agent_template="You are an agent.",
|
||||
)
|
||||
prompt = builder.build()
|
||||
assert "## Skill Examples" not in prompt
|
||||
|
||||
def test_few_shot_section_appears_in_prompt(self):
|
||||
builder = SystemPromptBuilder(
|
||||
agent_template="You are an agent.",
|
||||
skill_few_shot_examples=[
|
||||
"### research-skill\nInput: q1\nOutput: a1",
|
||||
"### code-skill\nInput: q2\nOutput: a2",
|
||||
],
|
||||
)
|
||||
prompt = builder.build()
|
||||
assert "## Skill Examples" in prompt
|
||||
assert "research-skill" in prompt
|
||||
assert "code-skill" in prompt
|
||||
assert "Input: q1" in prompt
|
||||
assert "Output: a2" in prompt
|
||||
|
||||
def test_empty_few_shot_list_no_section(self):
|
||||
builder = SystemPromptBuilder(
|
||||
agent_template="You are an agent.",
|
||||
skill_few_shot_examples=[],
|
||||
)
|
||||
prompt = builder.build()
|
||||
assert "## Skill Examples" not in prompt
|
||||
@@ -0,0 +1,83 @@
|
||||
"""End-to-end CLI tests for skill install + sync against fake sources."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
from openjarvis.cli import cli
|
||||
|
||||
|
||||
def _build_fake_hermes_cache(cache_root: Path) -> None:
|
||||
"""Build a minimal fake Hermes cache for testing."""
|
||||
skills_root = cache_root / "skills"
|
||||
(skills_root / "research" / "research-skill").mkdir(parents=True)
|
||||
(skills_root / "research" / "research-skill" / "SKILL.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: research-skill
|
||||
description: A research skill
|
||||
---
|
||||
|
||||
Use the Bash tool to fetch data, then Read the file.
|
||||
""")
|
||||
)
|
||||
# Fake .git so HermesResolver doesn't try to clone
|
||||
(cache_root / ".git").mkdir()
|
||||
|
||||
|
||||
class TestCliInstallE2E:
|
||||
def test_install_hermes_skill_e2e(self, tmp_path: Path) -> None:
|
||||
"""jarvis skill install hermes:research-skill installs to target dir."""
|
||||
from openjarvis.skills.sources.hermes import HermesResolver
|
||||
|
||||
cache = tmp_path / "hermes-cache"
|
||||
target = tmp_path / "target"
|
||||
_build_fake_hermes_cache(cache)
|
||||
|
||||
# Create a HermesResolver pointed at the fake cache
|
||||
def _make_resolver(*_args, **_kwargs):
|
||||
r = HermesResolver.__new__(HermesResolver)
|
||||
r._cache_root = cache
|
||||
return r
|
||||
|
||||
# Patch the helper that builds resolvers in the CLI
|
||||
with patch(
|
||||
"openjarvis.cli.skill_cmd._get_resolver",
|
||||
lambda src, url="": _make_resolver(),
|
||||
):
|
||||
# Patch HermesResolver.sync to no-op (cache is already built)
|
||||
with patch.object(HermesResolver, "sync", lambda self: None):
|
||||
# Patch the SkillImporter constructor used inside the install
|
||||
# command so it writes to our test target instead of
|
||||
# ~/.openjarvis/skills/.
|
||||
from openjarvis.skills.importer import SkillImporter as _SI
|
||||
|
||||
original_init = _SI.__init__
|
||||
|
||||
def patched_init(self, parser, tool_translator, target_root=None):
|
||||
original_init(
|
||||
self,
|
||||
parser=parser,
|
||||
tool_translator=tool_translator,
|
||||
target_root=target,
|
||||
)
|
||||
|
||||
with patch.object(_SI, "__init__", patched_init):
|
||||
result = CliRunner().invoke(
|
||||
cli, ["skill", "install", "hermes:research-skill"]
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "Installed" in result.output
|
||||
|
||||
installed = target / "hermes" / "research-skill" / "SKILL.md"
|
||||
assert installed.exists(), (
|
||||
f"Expected {installed} to exist; output:\n{result.output}"
|
||||
)
|
||||
body = installed.read_text()
|
||||
assert "shell_exec" in body
|
||||
assert "file_read" in body
|
||||
@@ -0,0 +1,251 @@
|
||||
"""Tests for dependency graph and cycle detection (Task 3)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from openjarvis.skills.types import SkillManifest, SkillStep
|
||||
|
||||
|
||||
def _manifest(name, depends=None, capabilities=None, steps=None):
|
||||
return SkillManifest(
|
||||
name=name,
|
||||
depends=depends or [],
|
||||
required_capabilities=capabilities or [],
|
||||
steps=steps or [],
|
||||
)
|
||||
|
||||
|
||||
class TestBuildDependencyGraph:
|
||||
def test_no_dependencies(self):
|
||||
from openjarvis.skills.dependency import build_dependency_graph
|
||||
|
||||
skills = {"a": _manifest("a")}
|
||||
graph = build_dependency_graph(skills)
|
||||
assert graph == {"a": set()}
|
||||
|
||||
def test_simple_dependency(self):
|
||||
from openjarvis.skills.dependency import build_dependency_graph
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a"),
|
||||
"b": _manifest("b", depends=["a"]),
|
||||
}
|
||||
graph = build_dependency_graph(skills)
|
||||
assert "a" in graph["b"]
|
||||
assert graph["a"] == set()
|
||||
|
||||
def test_step_skill_name_edges(self):
|
||||
from openjarvis.skills.dependency import build_dependency_graph
|
||||
|
||||
steps = [SkillStep(skill_name="base_skill", output_key="r")]
|
||||
skills = {
|
||||
"base_skill": _manifest("base_skill"),
|
||||
"composite": _manifest("composite", steps=steps),
|
||||
}
|
||||
graph = build_dependency_graph(skills)
|
||||
assert "base_skill" in graph["composite"]
|
||||
|
||||
def test_combined_depends_and_steps(self):
|
||||
from openjarvis.skills.dependency import build_dependency_graph
|
||||
|
||||
steps = [SkillStep(skill_name="step_dep", output_key="r")]
|
||||
skills = {
|
||||
"explicit_dep": _manifest("explicit_dep"),
|
||||
"step_dep": _manifest("step_dep"),
|
||||
"main": _manifest("main", depends=["explicit_dep"], steps=steps),
|
||||
}
|
||||
graph = build_dependency_graph(skills)
|
||||
assert "explicit_dep" in graph["main"]
|
||||
assert "step_dep" in graph["main"]
|
||||
|
||||
def test_multiple_skills_no_overlap(self):
|
||||
from openjarvis.skills.dependency import build_dependency_graph
|
||||
|
||||
skills = {
|
||||
"x": _manifest("x"),
|
||||
"y": _manifest("y"),
|
||||
}
|
||||
graph = build_dependency_graph(skills)
|
||||
assert graph["x"] == set()
|
||||
assert graph["y"] == set()
|
||||
|
||||
|
||||
class TestValidateDependencies:
|
||||
def test_valid_topological_sort_simple(self):
|
||||
from openjarvis.skills.dependency import validate_dependencies
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a"),
|
||||
"b": _manifest("b", depends=["a"]),
|
||||
}
|
||||
order = validate_dependencies(skills)
|
||||
assert order.index("a") < order.index("b")
|
||||
|
||||
def test_valid_topological_sort_chain(self):
|
||||
from openjarvis.skills.dependency import validate_dependencies
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a"),
|
||||
"b": _manifest("b", depends=["a"]),
|
||||
"c": _manifest("c", depends=["b"]),
|
||||
}
|
||||
order = validate_dependencies(skills)
|
||||
assert order.index("a") < order.index("b")
|
||||
assert order.index("b") < order.index("c")
|
||||
|
||||
def test_no_dependencies_returns_all_skills(self):
|
||||
from openjarvis.skills.dependency import validate_dependencies
|
||||
|
||||
skills = {
|
||||
"x": _manifest("x"),
|
||||
"y": _manifest("y"),
|
||||
}
|
||||
order = validate_dependencies(skills)
|
||||
assert set(order) == {"x", "y"}
|
||||
|
||||
def test_cycle_detection_raises(self):
|
||||
from openjarvis.skills.dependency import (
|
||||
DependencyCycleError,
|
||||
validate_dependencies,
|
||||
)
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a", depends=["b"]),
|
||||
"b": _manifest("b", depends=["a"]),
|
||||
}
|
||||
with pytest.raises(DependencyCycleError):
|
||||
validate_dependencies(skills)
|
||||
|
||||
def test_cycle_detection_three_nodes(self):
|
||||
from openjarvis.skills.dependency import (
|
||||
DependencyCycleError,
|
||||
validate_dependencies,
|
||||
)
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a", depends=["c"]),
|
||||
"b": _manifest("b", depends=["a"]),
|
||||
"c": _manifest("c", depends=["b"]),
|
||||
}
|
||||
with pytest.raises(DependencyCycleError):
|
||||
validate_dependencies(skills)
|
||||
|
||||
def test_depth_exceeded_raises(self):
|
||||
from openjarvis.skills.dependency import (
|
||||
DepthExceededError,
|
||||
validate_dependencies,
|
||||
)
|
||||
|
||||
# Chain of depth 6 exceeds default max_depth=5
|
||||
skills = {
|
||||
"a": _manifest("a"),
|
||||
"b": _manifest("b", depends=["a"]),
|
||||
"c": _manifest("c", depends=["b"]),
|
||||
"d": _manifest("d", depends=["c"]),
|
||||
"e": _manifest("e", depends=["d"]),
|
||||
"f": _manifest("f", depends=["e"]),
|
||||
}
|
||||
with pytest.raises(DepthExceededError):
|
||||
validate_dependencies(skills, max_depth=5)
|
||||
|
||||
def test_depth_within_limit_passes(self):
|
||||
from openjarvis.skills.dependency import validate_dependencies
|
||||
|
||||
# Chain of depth 3 is within max_depth=5
|
||||
skills = {
|
||||
"a": _manifest("a"),
|
||||
"b": _manifest("b", depends=["a"]),
|
||||
"c": _manifest("c", depends=["b"]),
|
||||
}
|
||||
order = validate_dependencies(skills, max_depth=5)
|
||||
assert len(order) == 3
|
||||
|
||||
def test_missing_dependency_silently_skipped(self):
|
||||
from openjarvis.skills.dependency import validate_dependencies
|
||||
|
||||
# "missing" is not in skills — should not raise
|
||||
skills = {
|
||||
"a": _manifest("a", depends=["missing"]),
|
||||
}
|
||||
order = validate_dependencies(skills)
|
||||
assert "a" in order
|
||||
|
||||
def test_custom_max_depth(self):
|
||||
from openjarvis.skills.dependency import (
|
||||
DepthExceededError,
|
||||
validate_dependencies,
|
||||
)
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a"),
|
||||
"b": _manifest("b", depends=["a"]),
|
||||
"c": _manifest("c", depends=["b"]),
|
||||
}
|
||||
# Chain depth 2 exceeds max_depth=1
|
||||
with pytest.raises(DepthExceededError):
|
||||
validate_dependencies(skills, max_depth=1)
|
||||
|
||||
|
||||
class TestComputeCapabilityUnion:
|
||||
def test_single_skill_no_deps(self):
|
||||
from openjarvis.skills.dependency import compute_capability_union
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a", capabilities=["read_files"]),
|
||||
}
|
||||
caps = compute_capability_union("a", skills)
|
||||
assert "read_files" in caps
|
||||
|
||||
def test_transitive_capabilities(self):
|
||||
from openjarvis.skills.dependency import compute_capability_union
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a", capabilities=["network"]),
|
||||
"b": _manifest("b", depends=["a"], capabilities=["disk_write"]),
|
||||
}
|
||||
caps = compute_capability_union("b", skills)
|
||||
assert "network" in caps
|
||||
assert "disk_write" in caps
|
||||
|
||||
def test_deep_transitive_capabilities(self):
|
||||
from openjarvis.skills.dependency import compute_capability_union
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a", capabilities=["cap_a"]),
|
||||
"b": _manifest("b", depends=["a"], capabilities=["cap_b"]),
|
||||
"c": _manifest("c", depends=["b"], capabilities=["cap_c"]),
|
||||
}
|
||||
caps = compute_capability_union("c", skills)
|
||||
assert "cap_a" in caps
|
||||
assert "cap_b" in caps
|
||||
assert "cap_c" in caps
|
||||
|
||||
def test_no_capabilities(self):
|
||||
from openjarvis.skills.dependency import compute_capability_union
|
||||
|
||||
skills = {
|
||||
"a": _manifest("a"),
|
||||
"b": _manifest("b", depends=["a"]),
|
||||
}
|
||||
caps = compute_capability_union("b", skills)
|
||||
assert caps == []
|
||||
|
||||
def test_dedup_capabilities(self):
|
||||
from openjarvis.skills.dependency import compute_capability_union
|
||||
|
||||
# Both a and b claim "network"
|
||||
skills = {
|
||||
"a": _manifest("a", capabilities=["network"]),
|
||||
"b": _manifest("b", depends=["a"], capabilities=["network", "disk"]),
|
||||
}
|
||||
caps = compute_capability_union("b", skills)
|
||||
assert caps.count("network") == 1
|
||||
assert "disk" in caps
|
||||
|
||||
def test_missing_skill_returns_empty(self):
|
||||
from openjarvis.skills.dependency import compute_capability_union
|
||||
|
||||
skills = {}
|
||||
caps = compute_capability_union("nonexistent", skills)
|
||||
assert caps == []
|
||||
@@ -0,0 +1,204 @@
|
||||
"""Tests for SkillImporter — installs ResolvedSkill instances on disk."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from openjarvis.skills.importer import SkillImporter
|
||||
from openjarvis.skills.parser import SkillParser
|
||||
from openjarvis.skills.sources.base import ResolvedSkill
|
||||
from openjarvis.skills.tool_translator import ToolTranslator
|
||||
|
||||
|
||||
def _make_resolved(tmp_path: Path, body: str = "Body") -> ResolvedSkill:
|
||||
"""Create a fake source skill directory and return a ResolvedSkill."""
|
||||
src_dir = tmp_path / "source" / "my-skill"
|
||||
src_dir.mkdir(parents=True)
|
||||
(src_dir / "SKILL.md").write_text(
|
||||
f"---\nname: my-skill\ndescription: A test skill\n---\n{body}"
|
||||
)
|
||||
return ResolvedSkill(
|
||||
name="my-skill",
|
||||
source="hermes",
|
||||
path=src_dir,
|
||||
category="testing",
|
||||
description="A test skill",
|
||||
commit="abc123",
|
||||
)
|
||||
|
||||
|
||||
class TestImportSkill:
|
||||
def test_imports_to_sourced_subdir(self, tmp_path: Path):
|
||||
target_root = tmp_path / "skills"
|
||||
importer = SkillImporter(
|
||||
parser=SkillParser(),
|
||||
tool_translator=ToolTranslator(),
|
||||
target_root=target_root,
|
||||
)
|
||||
resolved = _make_resolved(tmp_path)
|
||||
result = importer.import_skill(resolved)
|
||||
|
||||
assert result.success
|
||||
target = target_root / "hermes" / "my-skill"
|
||||
assert target.exists()
|
||||
assert (target / "SKILL.md").exists()
|
||||
|
||||
def test_writes_source_metadata_file(self, tmp_path: Path):
|
||||
target_root = tmp_path / "skills"
|
||||
importer = SkillImporter(
|
||||
parser=SkillParser(),
|
||||
tool_translator=ToolTranslator(),
|
||||
target_root=target_root,
|
||||
)
|
||||
resolved = _make_resolved(tmp_path)
|
||||
importer.import_skill(resolved)
|
||||
|
||||
source_file = target_root / "hermes" / "my-skill" / ".source"
|
||||
assert source_file.exists()
|
||||
content = source_file.read_text()
|
||||
assert "source = " in content
|
||||
assert "abc123" in content
|
||||
assert "scripts_imported = false" in content
|
||||
|
||||
def test_translates_tool_references_in_body(self, tmp_path: Path):
|
||||
target_root = tmp_path / "skills"
|
||||
importer = SkillImporter(
|
||||
parser=SkillParser(),
|
||||
tool_translator=ToolTranslator(),
|
||||
target_root=target_root,
|
||||
)
|
||||
resolved = _make_resolved(
|
||||
tmp_path, body="First use the Bash tool, then Read the file."
|
||||
)
|
||||
result = importer.import_skill(resolved)
|
||||
|
||||
installed = target_root / "hermes" / "my-skill" / "SKILL.md"
|
||||
body = installed.read_text()
|
||||
assert "shell_exec" in body
|
||||
assert "file_read" in body
|
||||
assert "Bash" not in body
|
||||
assert "Bash->shell_exec" in str(result.translated_tools)
|
||||
|
||||
def test_scripts_skipped_by_default(self, tmp_path: Path):
|
||||
target_root = tmp_path / "skills"
|
||||
# Source has a scripts/ directory
|
||||
src_dir = tmp_path / "source" / "my-skill"
|
||||
src_dir.mkdir(parents=True)
|
||||
(src_dir / "SKILL.md").write_text("---\nname: my-skill\ndescription: x\n---\n")
|
||||
scripts_dir = src_dir / "scripts"
|
||||
scripts_dir.mkdir()
|
||||
(scripts_dir / "helper.py").write_text("print('hi')")
|
||||
|
||||
resolved = ResolvedSkill(
|
||||
name="my-skill",
|
||||
source="hermes",
|
||||
path=src_dir,
|
||||
category="x",
|
||||
description="x",
|
||||
commit="a",
|
||||
)
|
||||
importer = SkillImporter(
|
||||
parser=SkillParser(),
|
||||
tool_translator=ToolTranslator(),
|
||||
target_root=target_root,
|
||||
)
|
||||
result = importer.import_skill(resolved)
|
||||
|
||||
target = target_root / "hermes" / "my-skill"
|
||||
assert not (target / "scripts").exists()
|
||||
assert result.scripts_imported is False
|
||||
|
||||
def test_scripts_imported_with_flag(self, tmp_path: Path):
|
||||
target_root = tmp_path / "skills"
|
||||
src_dir = tmp_path / "source" / "my-skill"
|
||||
src_dir.mkdir(parents=True)
|
||||
(src_dir / "SKILL.md").write_text("---\nname: my-skill\ndescription: x\n---\n")
|
||||
scripts_dir = src_dir / "scripts"
|
||||
scripts_dir.mkdir()
|
||||
(scripts_dir / "helper.py").write_text("print('hi')")
|
||||
|
||||
resolved = ResolvedSkill(
|
||||
name="my-skill",
|
||||
source="hermes",
|
||||
path=src_dir,
|
||||
category="x",
|
||||
description="x",
|
||||
commit="a",
|
||||
)
|
||||
importer = SkillImporter(
|
||||
parser=SkillParser(),
|
||||
tool_translator=ToolTranslator(),
|
||||
target_root=target_root,
|
||||
)
|
||||
result = importer.import_skill(resolved, with_scripts=True)
|
||||
|
||||
target = target_root / "hermes" / "my-skill"
|
||||
assert (target / "scripts" / "helper.py").exists()
|
||||
assert result.scripts_imported is True
|
||||
|
||||
def test_references_assets_always_copied(self, tmp_path: Path):
|
||||
target_root = tmp_path / "skills"
|
||||
src_dir = tmp_path / "source" / "my-skill"
|
||||
src_dir.mkdir(parents=True)
|
||||
(src_dir / "SKILL.md").write_text("---\nname: my-skill\ndescription: x\n---\n")
|
||||
(src_dir / "references").mkdir()
|
||||
(src_dir / "references" / "REFERENCE.md").write_text("# Reference")
|
||||
(src_dir / "assets").mkdir()
|
||||
(src_dir / "assets" / "template.txt").write_text("template")
|
||||
|
||||
resolved = ResolvedSkill(
|
||||
name="my-skill",
|
||||
source="hermes",
|
||||
path=src_dir,
|
||||
category="x",
|
||||
description="x",
|
||||
commit="a",
|
||||
)
|
||||
importer = SkillImporter(
|
||||
parser=SkillParser(),
|
||||
tool_translator=ToolTranslator(),
|
||||
target_root=target_root,
|
||||
)
|
||||
importer.import_skill(resolved)
|
||||
|
||||
target = target_root / "hermes" / "my-skill"
|
||||
assert (target / "references" / "REFERENCE.md").exists()
|
||||
assert (target / "assets" / "template.txt").exists()
|
||||
|
||||
def test_force_overwrites_existing_install(self, tmp_path: Path):
|
||||
target_root = tmp_path / "skills"
|
||||
importer = SkillImporter(
|
||||
parser=SkillParser(),
|
||||
tool_translator=ToolTranslator(),
|
||||
target_root=target_root,
|
||||
)
|
||||
resolved = _make_resolved(tmp_path, body="Original body")
|
||||
importer.import_skill(resolved)
|
||||
|
||||
# Modify source and re-import with force
|
||||
(resolved.path / "SKILL.md").write_text(
|
||||
"---\nname: my-skill\ndescription: A test skill\n---\nUpdated body"
|
||||
)
|
||||
importer.import_skill(resolved, force=True)
|
||||
|
||||
installed = target_root / "hermes" / "my-skill" / "SKILL.md"
|
||||
assert "Updated body" in installed.read_text()
|
||||
|
||||
def test_install_without_force_skips_existing(self, tmp_path: Path):
|
||||
target_root = tmp_path / "skills"
|
||||
importer = SkillImporter(
|
||||
parser=SkillParser(),
|
||||
tool_translator=ToolTranslator(),
|
||||
target_root=target_root,
|
||||
)
|
||||
resolved = _make_resolved(tmp_path, body="Original")
|
||||
importer.import_skill(resolved)
|
||||
|
||||
(resolved.path / "SKILL.md").write_text(
|
||||
"---\nname: my-skill\ndescription: A test skill\n---\nNew body"
|
||||
)
|
||||
result = importer.import_skill(resolved, force=False)
|
||||
assert result.skipped
|
||||
|
||||
installed = target_root / "hermes" / "my-skill" / "SKILL.md"
|
||||
assert "Original" in installed.read_text()
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Tests for skill index — sync and search."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
|
||||
from openjarvis.skills.index import SkillIndex
|
||||
|
||||
|
||||
class TestSkillIndex:
|
||||
def test_load_index(self, tmp_path: Path):
|
||||
index_file = tmp_path / "index.toml"
|
||||
index_file.write_text(
|
||||
textwrap.dedent("""\
|
||||
[[skills]]
|
||||
name = "research"
|
||||
version = "0.1.0"
|
||||
description = "Research a topic"
|
||||
author = "openjarvis"
|
||||
source = "github.com/openjarvis/skills/research"
|
||||
sha256 = "abc123"
|
||||
tags = ["research"]
|
||||
required_capabilities = ["network:fetch"]
|
||||
|
||||
[[skills]]
|
||||
name = "summarize"
|
||||
version = "0.2.0"
|
||||
description = "Summarize text"
|
||||
author = "community"
|
||||
source = "github.com/user/summarize"
|
||||
sha256 = "def456"
|
||||
tags = ["nlp"]
|
||||
required_capabilities = []
|
||||
""")
|
||||
)
|
||||
index = SkillIndex(tmp_path)
|
||||
assert len(index.entries) == 2
|
||||
assert index.entries["research"].version == "0.1.0"
|
||||
|
||||
def test_search_by_name(self, tmp_path: Path):
|
||||
index_file = tmp_path / "index.toml"
|
||||
index_file.write_text(
|
||||
textwrap.dedent("""\
|
||||
[[skills]]
|
||||
name = "web_research"
|
||||
version = "0.1.0"
|
||||
description = "Search the web"
|
||||
author = "openjarvis"
|
||||
source = "github.com/openjarvis/skills/web_research"
|
||||
sha256 = "abc"
|
||||
tags = ["research"]
|
||||
required_capabilities = []
|
||||
|
||||
[[skills]]
|
||||
name = "code_review"
|
||||
version = "0.1.0"
|
||||
description = "Review code"
|
||||
author = "openjarvis"
|
||||
source = "github.com/openjarvis/skills/code_review"
|
||||
sha256 = "def"
|
||||
tags = ["coding"]
|
||||
required_capabilities = []
|
||||
""")
|
||||
)
|
||||
index = SkillIndex(tmp_path)
|
||||
results = index.search("research")
|
||||
assert len(results) == 1
|
||||
assert results[0].name == "web_research"
|
||||
|
||||
def test_search_by_tag(self, tmp_path: Path):
|
||||
index_file = tmp_path / "index.toml"
|
||||
index_file.write_text(
|
||||
textwrap.dedent("""\
|
||||
[[skills]]
|
||||
name = "skill_a"
|
||||
version = "0.1.0"
|
||||
description = "First"
|
||||
author = "x"
|
||||
source = "github.com/x/a"
|
||||
sha256 = "a"
|
||||
tags = ["nlp", "research"]
|
||||
required_capabilities = []
|
||||
""")
|
||||
)
|
||||
index = SkillIndex(tmp_path)
|
||||
results = index.search("nlp")
|
||||
assert len(results) == 1
|
||||
|
||||
def test_search_no_results(self, tmp_path: Path):
|
||||
index_file = tmp_path / "index.toml"
|
||||
index_file.write_text(
|
||||
"[[skills]]\n"
|
||||
'name = "a"\nversion = "0.1.0"\n'
|
||||
'description = ""\nauthor = ""\n'
|
||||
'source = ""\nsha256 = ""\n'
|
||||
"tags = []\nrequired_capabilities = []\n"
|
||||
)
|
||||
index = SkillIndex(tmp_path)
|
||||
results = index.search("zzzzz")
|
||||
assert results == []
|
||||
|
||||
def test_missing_index_file(self, tmp_path: Path):
|
||||
index = SkillIndex(tmp_path)
|
||||
assert index.entries == {}
|
||||
@@ -0,0 +1,175 @@
|
||||
"""Live integration tests for the skills system.
|
||||
|
||||
These tests require a running Ollama instance with qwen3.5:4b.
|
||||
Mark: live
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from openjarvis.core.events import EventBus, EventType
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
from openjarvis.skills.tool_adapter import SkillTool
|
||||
from openjarvis.system import SystemBuilder
|
||||
|
||||
|
||||
@pytest.mark.live
|
||||
class TestSkillSystemIntegration:
|
||||
"""Integration tests verifying skills flow end-to-end with a real engine."""
|
||||
|
||||
def test_system_builder_discovers_skills(self):
|
||||
"""SystemBuilder.build() discovers installed skills and adds them to tools."""
|
||||
system = SystemBuilder().engine("ollama").model("qwen3.5:4b").build()
|
||||
try:
|
||||
assert system.skill_manager is not None, "SkillManager should be created"
|
||||
skill_names = system.skill_manager.skill_names()
|
||||
assert len(skill_names) > 0, f"Should discover skills, got: {skill_names}"
|
||||
print(f" Discovered skills: {skill_names}")
|
||||
|
||||
# Verify skill tools are in the system tools list
|
||||
skill_tools = [t for t in system.tools if isinstance(t, SkillTool)]
|
||||
assert len(skill_tools) > 0, "Skill tools should be in system tools"
|
||||
print(f" Skill tools in system: {[t.spec.name for t in skill_tools]}")
|
||||
finally:
|
||||
if hasattr(system, "close"):
|
||||
try:
|
||||
system.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def test_skill_catalog_in_system_prompt(self):
|
||||
"""The skill catalog XML should be generated correctly."""
|
||||
bus = EventBus()
|
||||
mgr = SkillManager(bus=bus)
|
||||
from pathlib import Path
|
||||
|
||||
mgr.discover(paths=[Path("~/.openjarvis/skills/").expanduser()])
|
||||
|
||||
catalog = mgr.get_catalog_xml()
|
||||
assert "<available_skills>" in catalog
|
||||
assert "research-and-summarize" in catalog
|
||||
assert "code-explainer" in catalog
|
||||
assert "math-solver" in catalog
|
||||
print(f" Catalog XML:\n{catalog}")
|
||||
|
||||
def test_skill_tool_invocation_returns_content(self):
|
||||
"""Invoking a skill tool returns meaningful content."""
|
||||
bus = EventBus(record_history=True)
|
||||
mgr = SkillManager(bus=bus)
|
||||
from pathlib import Path
|
||||
|
||||
mgr.discover(paths=[Path("~/.openjarvis/skills/").expanduser()])
|
||||
|
||||
# Test instruction-only skill
|
||||
tools = mgr.get_skill_tools()
|
||||
code_explainer = next(
|
||||
(t for t in tools if "code-explainer" in t.spec.name), None
|
||||
)
|
||||
assert code_explainer is not None, "code-explainer skill should exist"
|
||||
|
||||
result = code_explainer.execute(task="explain a for loop")
|
||||
assert result.success
|
||||
assert (
|
||||
"programming language" in result.content.lower()
|
||||
or "plain language" in result.content.lower()
|
||||
), f"Should return instructions, got: {result.content[:200]}"
|
||||
print(f" code-explainer returned: {result.content[:200]}...")
|
||||
|
||||
# Verify the bus is reachable (instruction-only skills don't run
|
||||
# a pipeline, so no SKILL_EXECUTE_* events here — tool-mode skills
|
||||
# emit them, covered in TestSkillEventsAndTracing below)
|
||||
assert bus.history is not None
|
||||
|
||||
def test_agent_ask_with_skills_available(self):
|
||||
"""Agent can answer a query with skills available in the tool list."""
|
||||
system = (
|
||||
SystemBuilder()
|
||||
.engine("ollama")
|
||||
.model("qwen3.5:4b")
|
||||
.tools(
|
||||
[
|
||||
"think",
|
||||
"calculator",
|
||||
]
|
||||
)
|
||||
.build()
|
||||
)
|
||||
try:
|
||||
# Skills should be auto-discovered and added
|
||||
assert system.skill_manager is not None
|
||||
skill_count = len(system.skill_manager.skill_names())
|
||||
total_tools = len(system.tools)
|
||||
print(f" Skills: {skill_count}, Total tools: {total_tools}")
|
||||
|
||||
# The tools list should include both regular tools and skill tools
|
||||
tool_names = [t.spec.name for t in system.tools]
|
||||
print(f" Tool names: {tool_names}")
|
||||
|
||||
# Ask a simple question — agent should work normally
|
||||
result = system.ask("What is 2 + 2?")
|
||||
assert "content" in result
|
||||
content = result["content"]
|
||||
assert "4" in content, f"Expected '4' in response, got: {content[:200]}"
|
||||
print(f" Agent response: {content[:200]}")
|
||||
finally:
|
||||
if hasattr(system, "close"):
|
||||
try:
|
||||
system.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
@pytest.mark.live
|
||||
class TestSkillEventsAndTracing:
|
||||
"""Verify skills emit proper events for the Learning pipeline."""
|
||||
|
||||
def test_skill_execution_emits_events(self):
|
||||
"""Running a structured skill emits SKILL_EXECUTE_START/END events."""
|
||||
from openjarvis.core.types import ToolResult
|
||||
from openjarvis.skills.executor import SkillExecutor
|
||||
from openjarvis.skills.types import SkillManifest, SkillStep
|
||||
from openjarvis.tools._stubs import BaseTool, ToolExecutor, ToolSpec
|
||||
|
||||
class EchoTool(BaseTool):
|
||||
tool_id = "echo"
|
||||
|
||||
@property
|
||||
def spec(self):
|
||||
return ToolSpec(name="echo", description="Echo")
|
||||
|
||||
def execute(self, **params):
|
||||
return ToolResult(
|
||||
tool_name="echo",
|
||||
content=params.get("text", "echoed"),
|
||||
success=True,
|
||||
)
|
||||
|
||||
bus = EventBus(record_history=True)
|
||||
te = ToolExecutor([EchoTool()])
|
||||
executor = SkillExecutor(te, bus=bus)
|
||||
|
||||
manifest = SkillManifest(
|
||||
name="test_traced",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "hello"}',
|
||||
output_key="result",
|
||||
),
|
||||
],
|
||||
)
|
||||
result = executor.run(manifest)
|
||||
assert result.success
|
||||
|
||||
events = [(e.event_type, e.data) for e in bus.history]
|
||||
event_types = [e[0] for e in events]
|
||||
assert EventType.SKILL_EXECUTE_START in event_types
|
||||
assert EventType.SKILL_EXECUTE_END in event_types
|
||||
|
||||
# Verify event data
|
||||
start_event = next(e for e in events if e[0] == EventType.SKILL_EXECUTE_START)
|
||||
assert start_event[1]["skill"] == "test_traced"
|
||||
end_event = next(e for e in events if e[0] == EventType.SKILL_EXECUTE_END)
|
||||
assert end_event[1]["success"] is True
|
||||
print(" Events correctly emitted with skill metadata")
|
||||
@@ -0,0 +1,215 @@
|
||||
"""Tests for SKILL.md and directory-based skill loading."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
|
||||
from openjarvis.skills.loader import load_skill_directory, load_skill_markdown
|
||||
|
||||
|
||||
class TestLoadSkillMarkdown:
|
||||
def test_load_markdown_with_frontmatter(self, tmp_path: Path):
|
||||
md = tmp_path / "SKILL.md"
|
||||
md.write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: research
|
||||
required_capabilities: [network:fetch]
|
||||
---
|
||||
|
||||
When asked to research a topic:
|
||||
1. Break into sub-questions
|
||||
2. Search each one
|
||||
""")
|
||||
)
|
||||
manifest = load_skill_markdown(md)
|
||||
assert manifest.name == "research"
|
||||
assert manifest.required_capabilities == ["network:fetch"]
|
||||
assert "Break into sub-questions" in manifest.markdown_content
|
||||
|
||||
def test_load_markdown_no_frontmatter(self, tmp_path: Path):
|
||||
md = tmp_path / "SKILL.md"
|
||||
md.write_text("Just some instructions.\n")
|
||||
manifest = load_skill_markdown(md)
|
||||
assert manifest.name == "SKILL"
|
||||
assert manifest.markdown_content == "Just some instructions.\n"
|
||||
|
||||
|
||||
class TestLoadSkillDirectory:
|
||||
def test_directory_with_toml_only(self, tmp_path: Path):
|
||||
skill_dir = tmp_path / "my_skill"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "skill.toml").write_text(
|
||||
textwrap.dedent("""\
|
||||
[skill]
|
||||
name = "my_skill"
|
||||
description = "A test skill"
|
||||
tags = ["test"]
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "echo"
|
||||
output_key = "result"
|
||||
""")
|
||||
)
|
||||
manifest = load_skill_directory(skill_dir)
|
||||
assert manifest.name == "my_skill"
|
||||
assert manifest.tags == ["test"]
|
||||
assert len(manifest.steps) == 1
|
||||
assert manifest.markdown_content == ""
|
||||
|
||||
def test_directory_with_markdown_only(self, tmp_path: Path):
|
||||
skill_dir = tmp_path / "guide_skill"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "SKILL.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: guide_skill
|
||||
required_capabilities: [filesystem:read]
|
||||
---
|
||||
|
||||
Follow these steps when reviewing code.
|
||||
""")
|
||||
)
|
||||
manifest = load_skill_directory(skill_dir)
|
||||
assert manifest.name == "guide_skill"
|
||||
assert manifest.required_capabilities == ["filesystem:read"]
|
||||
assert "reviewing code" in manifest.markdown_content
|
||||
|
||||
def test_directory_with_both(self, tmp_path: Path):
|
||||
skill_dir = tmp_path / "hybrid"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "skill.toml").write_text(
|
||||
textwrap.dedent("""\
|
||||
[skill]
|
||||
name = "hybrid"
|
||||
description = "Hybrid skill"
|
||||
tags = ["hybrid"]
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "web_search"
|
||||
arguments_template = '{"query": "{query}"}'
|
||||
output_key = "results"
|
||||
""")
|
||||
)
|
||||
(skill_dir / "SKILL.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: hybrid
|
||||
---
|
||||
|
||||
Present results with citations.
|
||||
""")
|
||||
)
|
||||
manifest = load_skill_directory(skill_dir)
|
||||
assert manifest.name == "hybrid"
|
||||
assert len(manifest.steps) == 1
|
||||
assert "citations" in manifest.markdown_content
|
||||
|
||||
def test_directory_with_depends_and_skill_steps(self, tmp_path: Path):
|
||||
skill_dir = tmp_path / "composed"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "skill.toml").write_text(
|
||||
textwrap.dedent("""\
|
||||
[skill]
|
||||
name = "composed"
|
||||
depends = ["summarize"]
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "web_search"
|
||||
arguments_template = '{"query": "{query}"}'
|
||||
output_key = "raw"
|
||||
|
||||
[[skill.steps]]
|
||||
skill_name = "summarize"
|
||||
arguments_template = '{"text": "{raw}"}'
|
||||
output_key = "summary"
|
||||
""")
|
||||
)
|
||||
manifest = load_skill_directory(skill_dir)
|
||||
assert manifest.depends == ["summarize"]
|
||||
assert manifest.steps[1].skill_name == "summarize"
|
||||
assert manifest.steps[1].tool_name == ""
|
||||
|
||||
|
||||
class TestParserDelegation:
|
||||
def test_load_skill_markdown_uses_parser_for_validation(self, tmp_path: Path):
|
||||
"""SKILL.md loading goes through SkillParser, gaining strict validation."""
|
||||
from openjarvis.skills.loader import load_skill_markdown
|
||||
|
||||
md = tmp_path / "SKILL.md"
|
||||
md.write_text("---\nname: my-skill\ndescription: x\n---\nBody")
|
||||
manifest = load_skill_markdown(md)
|
||||
assert manifest.name == "my-skill"
|
||||
assert manifest.description == "x"
|
||||
assert "Body" in manifest.markdown_content
|
||||
|
||||
def test_load_skill_markdown_handles_legacy_top_level_fields(self, tmp_path: Path):
|
||||
"""Legacy SKILL.md with top-level tags/version/etc. still works."""
|
||||
from openjarvis.skills.loader import load_skill_markdown
|
||||
|
||||
md = tmp_path / "SKILL.md"
|
||||
md.write_text(
|
||||
"---\n"
|
||||
"name: legacy-skill\n"
|
||||
"description: x\n"
|
||||
"version: 2.0.0\n"
|
||||
"tags: [a, b]\n"
|
||||
"required_capabilities: [network:fetch]\n"
|
||||
"---\n"
|
||||
"Body"
|
||||
)
|
||||
manifest = load_skill_markdown(md)
|
||||
assert manifest.version == "2.0.0"
|
||||
assert manifest.tags == ["a", "b"]
|
||||
assert manifest.required_capabilities == ["network:fetch"]
|
||||
|
||||
|
||||
class TestLoadSkillDirectorySourcePromotion:
|
||||
def test_dot_source_file_promoted_to_metadata(self, tmp_path: Path):
|
||||
"""A .source file in the skill directory promotes its source field
|
||||
into manifest.metadata.openjarvis.source."""
|
||||
from openjarvis.skills.loader import load_skill_directory
|
||||
|
||||
skill_dir = tmp_path / "imported-skill"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "SKILL.md").write_text(
|
||||
"---\nname: imported-skill\ndescription: x\n---\nBody"
|
||||
)
|
||||
(skill_dir / ".source").write_text(
|
||||
'source = "hermes:apple-notes"\n'
|
||||
'commit = "abc123"\n'
|
||||
'category = "apple"\n'
|
||||
'installed_at = "2026-04-04T22:30:00Z"\n'
|
||||
"translated_tools = []\n"
|
||||
"missing_tools = []\n"
|
||||
"scripts_imported = false\n"
|
||||
)
|
||||
manifest = load_skill_directory(skill_dir)
|
||||
oj = manifest.metadata.get("openjarvis", {})
|
||||
assert oj.get("source") == "hermes"
|
||||
|
||||
def test_no_dot_source_file_no_metadata(self, tmp_path: Path):
|
||||
"""When no .source file is present, metadata.openjarvis.source is unset."""
|
||||
from openjarvis.skills.loader import load_skill_directory
|
||||
|
||||
skill_dir = tmp_path / "user-skill"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "SKILL.md").write_text(
|
||||
"---\nname: user-skill\ndescription: x\n---\nBody"
|
||||
)
|
||||
manifest = load_skill_directory(skill_dir)
|
||||
oj = manifest.metadata.get("openjarvis", {})
|
||||
assert "source" not in oj
|
||||
|
||||
def test_malformed_dot_source_does_not_crash(self, tmp_path: Path):
|
||||
"""A malformed .source file is ignored, not crashed on."""
|
||||
from openjarvis.skills.loader import load_skill_directory
|
||||
|
||||
skill_dir = tmp_path / "bad"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "SKILL.md").write_text("---\nname: bad\ndescription: x\n---\nBody")
|
||||
(skill_dir / ".source").write_text("this is not valid toml = [[[")
|
||||
# Should load the manifest without raising
|
||||
manifest = load_skill_directory(skill_dir)
|
||||
assert manifest.name == "bad"
|
||||
@@ -0,0 +1,520 @@
|
||||
"""Tests for SkillManager — discovery, catalog, tools, and resolve."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.core.types import ToolResult
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
from openjarvis.tools._stubs import BaseTool, ToolExecutor, ToolSpec # noqa: F401
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _write_toml_skill(directory: Path, name: str, description: str = "") -> None:
|
||||
skill_dir = directory / name
|
||||
skill_dir.mkdir(parents=True, exist_ok=True)
|
||||
(skill_dir / "skill.toml").write_text(
|
||||
textwrap.dedent(f"""\
|
||||
[skill]
|
||||
name = "{name}"
|
||||
description = "{description or name}"
|
||||
tags = ["test"]
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "echo"
|
||||
arguments_template = '{{"text": "{{input}}"}}'
|
||||
output_key = "result"
|
||||
""")
|
||||
)
|
||||
|
||||
|
||||
class EchoTool(BaseTool):
|
||||
tool_id = "echo"
|
||||
|
||||
@property
|
||||
def spec(self) -> ToolSpec:
|
||||
return ToolSpec(name="echo", description="Echo input")
|
||||
|
||||
def execute(self, **params) -> ToolResult:
|
||||
return ToolResult(
|
||||
tool_name="echo", content=params.get("text", ""), success=True
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestSkillManagerDiscovery
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSkillManagerDiscovery:
|
||||
def test_discover_single_dir(self, tmp_path: Path) -> None:
|
||||
"""discover() loads skills from a single directory."""
|
||||
_write_toml_skill(tmp_path, "alpha")
|
||||
_write_toml_skill(tmp_path, "beta")
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
|
||||
names = mgr.skill_names()
|
||||
assert "alpha" in names
|
||||
assert "beta" in names
|
||||
|
||||
def test_discover_precedence_workspace_over_user(self, tmp_path: Path) -> None:
|
||||
"""First path (workspace) wins when the same name appears in multiple dirs."""
|
||||
workspace = tmp_path / "workspace"
|
||||
user = tmp_path / "user"
|
||||
workspace.mkdir()
|
||||
user.mkdir()
|
||||
|
||||
# Both dirs have a skill named "shared"
|
||||
_write_toml_skill(workspace, "shared", description="workspace version")
|
||||
_write_toml_skill(user, "shared", description="user version")
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
# workspace is listed first → highest precedence
|
||||
mgr.discover(paths=[workspace, user])
|
||||
|
||||
manifest = mgr.resolve("shared")
|
||||
assert manifest.description == "workspace version"
|
||||
|
||||
def test_discover_empty_dir(self, tmp_path: Path) -> None:
|
||||
"""discover() does not raise on an empty directory."""
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
assert mgr.skill_names() == []
|
||||
|
||||
def test_discover_nonexistent_dir(self, tmp_path: Path) -> None:
|
||||
"""discover() silently skips paths that do not exist."""
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path / "does_not_exist"])
|
||||
assert mgr.skill_names() == []
|
||||
|
||||
def test_discover_multiple_dirs_accumulates(self, tmp_path: Path) -> None:
|
||||
"""Skills from all directories are accumulated (modulo precedence)."""
|
||||
dir_a = tmp_path / "a"
|
||||
dir_b = tmp_path / "b"
|
||||
dir_a.mkdir()
|
||||
dir_b.mkdir()
|
||||
|
||||
_write_toml_skill(dir_a, "skill_a")
|
||||
_write_toml_skill(dir_b, "skill_b")
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[dir_a, dir_b])
|
||||
|
||||
names = mgr.skill_names()
|
||||
assert "skill_a" in names
|
||||
assert "skill_b" in names
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestSkillManagerCatalog
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSkillManagerCatalog:
|
||||
def test_catalog_xml_contains_skill_names(self, tmp_path: Path) -> None:
|
||||
"""get_catalog_xml() includes discovered skill names."""
|
||||
_write_toml_skill(tmp_path, "greet", description="Greet the user")
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
|
||||
xml = mgr.get_catalog_xml()
|
||||
assert "greet" in xml
|
||||
assert "<available_skills>" in xml
|
||||
|
||||
def test_catalog_xml_empty_when_no_skills(self) -> None:
|
||||
"""get_catalog_xml() returns a well-formed empty block when no skills loaded."""
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
xml = mgr.get_catalog_xml()
|
||||
assert "<available_skills>" in xml
|
||||
assert "</available_skills>" in xml
|
||||
|
||||
def test_catalog_excludes_model_disabled_skills(self, tmp_path: Path) -> None:
|
||||
"""Skills with disable_model_invocation=true are excluded from the catalog."""
|
||||
skill_dir = tmp_path / "hidden"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "skill.toml").write_text(
|
||||
textwrap.dedent("""\
|
||||
[skill]
|
||||
name = "hidden"
|
||||
description = "Hidden skill"
|
||||
disable_model_invocation = true
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "echo"
|
||||
output_key = "x"
|
||||
""")
|
||||
)
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
xml = mgr.get_catalog_xml()
|
||||
assert "hidden" not in xml
|
||||
|
||||
def test_catalog_escapes_xml_special_chars(self, tmp_path: Path) -> None:
|
||||
"""Descriptions with XML-special chars are escaped in the catalog."""
|
||||
skill_dir = tmp_path / "special"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "skill.toml").write_text(
|
||||
textwrap.dedent("""\
|
||||
[skill]
|
||||
name = "special"
|
||||
description = "A skill with <tags> & 'quotes'"
|
||||
tags = ["test"]
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "echo"
|
||||
output_key = "r"
|
||||
""")
|
||||
)
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
xml = mgr.get_catalog_xml()
|
||||
# Raw < must not appear unescaped inside the XML body
|
||||
assert "<tags>" not in xml
|
||||
assert "<" in xml or "special" in xml # description is escaped or safe
|
||||
|
||||
def test_catalog_includes_visible_but_not_hidden(self, tmp_path: Path) -> None:
|
||||
"""Catalog includes user_invocable skills and excludes model-disabled ones."""
|
||||
_write_toml_skill(tmp_path, "visible")
|
||||
skill_dir = tmp_path / "invisible"
|
||||
skill_dir.mkdir()
|
||||
(skill_dir / "skill.toml").write_text(
|
||||
textwrap.dedent("""\
|
||||
[skill]
|
||||
name = "invisible"
|
||||
description = "Not for models"
|
||||
disable_model_invocation = true
|
||||
|
||||
[[skill.steps]]
|
||||
tool_name = "echo"
|
||||
output_key = "r"
|
||||
""")
|
||||
)
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
xml = mgr.get_catalog_xml()
|
||||
assert "visible" in xml
|
||||
assert "invisible" not in xml
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestSkillManagerTools
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSkillManagerTools:
|
||||
def test_get_skill_tools_returns_base_tool_instances(self, tmp_path: Path) -> None:
|
||||
"""get_skill_tools() returns a list of BaseTool instances."""
|
||||
_write_toml_skill(tmp_path, "tool_skill")
|
||||
|
||||
tool_executor = ToolExecutor([EchoTool()])
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
mgr.set_tool_executor(tool_executor)
|
||||
|
||||
tools = mgr.get_skill_tools()
|
||||
assert len(tools) == 1
|
||||
assert isinstance(tools[0], BaseTool)
|
||||
|
||||
def test_get_skill_tools_accepts_tool_executor_kwarg(self, tmp_path: Path) -> None:
|
||||
"""get_skill_tools() accepts an optional tool_executor keyword argument."""
|
||||
_write_toml_skill(tmp_path, "kwarg_skill")
|
||||
|
||||
tool_executor = ToolExecutor([EchoTool()])
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
|
||||
tools = mgr.get_skill_tools(tool_executor=tool_executor)
|
||||
assert len(tools) == 1
|
||||
assert isinstance(tools[0], BaseTool)
|
||||
|
||||
def test_get_skill_tools_empty_when_no_skills(self) -> None:
|
||||
"""get_skill_tools() returns an empty list when no skills are loaded."""
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
tools = mgr.get_skill_tools()
|
||||
assert tools == []
|
||||
|
||||
def test_get_skill_tools_names_prefixed(self, tmp_path: Path) -> None:
|
||||
"""Each SkillTool has a name prefixed with 'skill_'."""
|
||||
_write_toml_skill(tmp_path, "my_skill")
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
|
||||
tools = mgr.get_skill_tools()
|
||||
assert any(t.spec.name == "skill_my_skill" for t in tools)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# TestSkillManagerResolve
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSkillManagerResolve:
|
||||
def test_resolve_existing_skill(self, tmp_path: Path) -> None:
|
||||
"""resolve() returns the SkillManifest for a known skill name."""
|
||||
_write_toml_skill(tmp_path, "resolve_me")
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
|
||||
manifest = mgr.resolve("resolve_me")
|
||||
assert manifest.name == "resolve_me"
|
||||
|
||||
def test_resolve_missing_raises_key_error(self) -> None:
|
||||
"""resolve() raises KeyError for an unknown skill name."""
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
with pytest.raises(KeyError):
|
||||
mgr.resolve("nonexistent_skill")
|
||||
|
||||
def test_resolve_after_multiple_discovers(self, tmp_path: Path) -> None:
|
||||
"""resolve() works correctly after calling discover() multiple times."""
|
||||
dir_a = tmp_path / "a"
|
||||
dir_b = tmp_path / "b"
|
||||
dir_a.mkdir()
|
||||
dir_b.mkdir()
|
||||
|
||||
_write_toml_skill(dir_a, "skill_one")
|
||||
_write_toml_skill(dir_b, "skill_two")
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[dir_a])
|
||||
mgr.discover(paths=[dir_b])
|
||||
|
||||
assert mgr.resolve("skill_one").name == "skill_one"
|
||||
assert mgr.resolve("skill_two").name == "skill_two"
|
||||
|
||||
|
||||
class TestSkillManagerSourcedLayout:
|
||||
def test_discovers_skills_under_source_subdirs(self, tmp_path: Path):
|
||||
"""SkillManager.discover() finds skills in <source>/<name>/ layout."""
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
# Build hermes/<name>/ and openclaw/<name>/ subdirs
|
||||
for source, name in [("hermes", "apple-notes"), ("openclaw", "etherscan")]:
|
||||
d = tmp_path / source / name
|
||||
d.mkdir(parents=True)
|
||||
(d / "SKILL.md").write_text(
|
||||
f"---\nname: {name}\ndescription: from {source}\n---\nBody"
|
||||
)
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
names = mgr.skill_names()
|
||||
assert "apple-notes" in names
|
||||
assert "etherscan" in names
|
||||
|
||||
def test_flat_and_sourced_layout_coexist(self, tmp_path: Path):
|
||||
"""Both flat ./<name>/ and ./<source>/<name>/ are discovered."""
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
# Flat layout
|
||||
flat = tmp_path / "my-flat-skill"
|
||||
flat.mkdir()
|
||||
(flat / "SKILL.md").write_text(
|
||||
"---\nname: my-flat-skill\ndescription: flat\n---\n"
|
||||
)
|
||||
|
||||
# Sourced layout
|
||||
sourced = tmp_path / "hermes" / "my-sourced-skill"
|
||||
sourced.mkdir(parents=True)
|
||||
(sourced / "SKILL.md").write_text(
|
||||
"---\nname: my-sourced-skill\ndescription: sourced\n---\n"
|
||||
)
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path])
|
||||
names = mgr.skill_names()
|
||||
assert "my-flat-skill" in names
|
||||
assert "my-sourced-skill" in names
|
||||
|
||||
|
||||
class TestSkillManagerOverlayLoading:
|
||||
def test_overlay_description_overrides_manifest(self, tmp_path: Path):
|
||||
"""When an overlay exists, the optimized description replaces
|
||||
the manifest's description after discover()."""
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
from openjarvis.skills.overlay import SkillOverlay, write_overlay
|
||||
|
||||
skill_dir = tmp_path / "skills" / "research-skill"
|
||||
skill_dir.mkdir(parents=True)
|
||||
(skill_dir / "SKILL.md").write_text(
|
||||
"---\nname: research-skill\ndescription: Original description\n---\nBody"
|
||||
)
|
||||
|
||||
overlay_dir = tmp_path / "overlays"
|
||||
write_overlay(
|
||||
SkillOverlay(
|
||||
skill_name="research-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T14:30:00Z",
|
||||
trace_count=25,
|
||||
description="A much better optimized description",
|
||||
),
|
||||
overlay_dir,
|
||||
)
|
||||
|
||||
mgr = SkillManager(bus=EventBus(), overlay_dir=overlay_dir)
|
||||
mgr.discover(paths=[tmp_path / "skills"])
|
||||
|
||||
manifest = mgr.resolve("research-skill")
|
||||
assert manifest.description == "A much better optimized description"
|
||||
|
||||
def test_overlay_few_shot_stored_in_metadata(self, tmp_path: Path):
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
from openjarvis.skills.overlay import SkillOverlay, write_overlay
|
||||
|
||||
skill_dir = tmp_path / "skills" / "test-skill"
|
||||
skill_dir.mkdir(parents=True)
|
||||
(skill_dir / "SKILL.md").write_text(
|
||||
"---\nname: test-skill\ndescription: x\n---\nBody"
|
||||
)
|
||||
|
||||
overlay_dir = tmp_path / "overlays"
|
||||
write_overlay(
|
||||
SkillOverlay(
|
||||
skill_name="test-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T14:30:00Z",
|
||||
trace_count=20,
|
||||
description="x",
|
||||
few_shot=[
|
||||
{"input": "q1", "output": "a1"},
|
||||
{"input": "q2", "output": "a2"},
|
||||
],
|
||||
),
|
||||
overlay_dir,
|
||||
)
|
||||
|
||||
mgr = SkillManager(bus=EventBus(), overlay_dir=overlay_dir)
|
||||
mgr.discover(paths=[tmp_path / "skills"])
|
||||
|
||||
manifest = mgr.resolve("test-skill")
|
||||
oj = manifest.metadata.get("openjarvis", {})
|
||||
few_shot = oj.get("few_shot", [])
|
||||
assert len(few_shot) == 2
|
||||
assert few_shot[0]["input"] == "q1"
|
||||
|
||||
def test_no_overlay_dir_does_not_crash(self, tmp_path: Path):
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
skill_dir = tmp_path / "skills" / "test-skill"
|
||||
skill_dir.mkdir(parents=True)
|
||||
(skill_dir / "SKILL.md").write_text(
|
||||
"---\nname: test-skill\ndescription: original\n---\nBody"
|
||||
)
|
||||
|
||||
# No overlay_dir argument — should still work
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
mgr.discover(paths=[tmp_path / "skills"])
|
||||
manifest = mgr.resolve("test-skill")
|
||||
assert manifest.description == "original"
|
||||
|
||||
def test_get_few_shot_examples_returns_formatted_strings(self, tmp_path: Path):
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
from openjarvis.skills.overlay import SkillOverlay, write_overlay
|
||||
|
||||
skill_dir = tmp_path / "skills" / "fs-skill"
|
||||
skill_dir.mkdir(parents=True)
|
||||
(skill_dir / "SKILL.md").write_text(
|
||||
"---\nname: fs-skill\ndescription: x\n---\nBody"
|
||||
)
|
||||
|
||||
overlay_dir = tmp_path / "overlays"
|
||||
write_overlay(
|
||||
SkillOverlay(
|
||||
skill_name="fs-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T14:30:00Z",
|
||||
trace_count=20,
|
||||
description="x",
|
||||
few_shot=[
|
||||
{"input": "what is X?", "output": "X is Y"},
|
||||
],
|
||||
),
|
||||
overlay_dir,
|
||||
)
|
||||
|
||||
mgr = SkillManager(bus=EventBus(), overlay_dir=overlay_dir)
|
||||
mgr.discover(paths=[tmp_path / "skills"])
|
||||
|
||||
examples = mgr.get_few_shot_examples()
|
||||
assert len(examples) >= 1
|
||||
assert any("what is X?" in s and "X is Y" in s for s in examples)
|
||||
|
||||
def test_overlay_dir_read_from_config_when_not_explicit(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
"""Plan 2A I1 fix: SkillManager picks up overlay_dir from
|
||||
cfg.learning.skills.overlay_dir when no explicit value is passed."""
|
||||
from unittest.mock import patch
|
||||
|
||||
from openjarvis.core.config import (
|
||||
JarvisConfig,
|
||||
LearningConfig,
|
||||
SkillsLearningConfig,
|
||||
)
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
cfg = JarvisConfig()
|
||||
cfg.learning = LearningConfig()
|
||||
cfg.learning.skills = SkillsLearningConfig(
|
||||
overlay_dir=str(tmp_path / "configured-overlays")
|
||||
)
|
||||
|
||||
with patch("openjarvis.core.config.load_config", return_value=cfg):
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
assert (
|
||||
mgr._overlay_dir == (tmp_path / "configured-overlays").expanduser()
|
||||
)
|
||||
|
||||
def test_discover_with_empty_paths_still_loads_overlays(
|
||||
self, tmp_path: Path
|
||||
) -> None:
|
||||
"""Plan 2A I2 fix: discover() with no paths still applies overlays
|
||||
to skills that were seeded by other means."""
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
from openjarvis.skills.overlay import SkillOverlay, write_overlay
|
||||
from openjarvis.skills.types import SkillManifest
|
||||
|
||||
overlay_dir = tmp_path / "overlays"
|
||||
write_overlay(
|
||||
SkillOverlay(
|
||||
skill_name="seeded-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T14:30:00Z",
|
||||
trace_count=15,
|
||||
description="Optimized seeded description",
|
||||
),
|
||||
overlay_dir,
|
||||
)
|
||||
|
||||
mgr = SkillManager(bus=EventBus(), overlay_dir=overlay_dir)
|
||||
# Seed a skill directly (simulating a non-disk source)
|
||||
mgr._skills["seeded-skill"] = SkillManifest(
|
||||
name="seeded-skill",
|
||||
description="Original description",
|
||||
markdown_content="Body",
|
||||
)
|
||||
# Call discover() with no paths — should still load overlays
|
||||
mgr.discover()
|
||||
|
||||
manifest = mgr.resolve("seeded-skill")
|
||||
assert manifest.description == "Optimized seeded description"
|
||||
@@ -0,0 +1,111 @@
|
||||
"""Tests for SkillManager.discover_from_traces (Plan 2A)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, List
|
||||
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.core.types import StepType, Trace, TraceStep
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
|
||||
|
||||
class _FakeTraceStore:
|
||||
"""In-memory trace store stub for tests."""
|
||||
|
||||
def __init__(self, traces: List[Trace]) -> None:
|
||||
self._traces = traces
|
||||
|
||||
def list_traces(self, *, limit: int = 100, **_kwargs: Any) -> List[Trace]:
|
||||
return list(self._traces[:limit])
|
||||
|
||||
|
||||
def _make_trace(tools: List[str], outcome: str = "success") -> Trace:
|
||||
steps = [
|
||||
TraceStep(
|
||||
step_type=StepType.TOOL_CALL,
|
||||
timestamp=0.0,
|
||||
input={"tool": tool, "arguments": {}},
|
||||
output={"success": True, "result": ""},
|
||||
metadata={},
|
||||
)
|
||||
for tool in tools
|
||||
]
|
||||
return Trace(
|
||||
query="test query",
|
||||
agent="native_react",
|
||||
model="qwen3.5:9b",
|
||||
engine="ollama",
|
||||
steps=steps,
|
||||
outcome=outcome,
|
||||
feedback=1.0 if outcome == "success" else 0.0,
|
||||
)
|
||||
|
||||
|
||||
class TestDiscoverFromTraces:
|
||||
def test_writes_manifest_for_recurring_sequence(self, tmp_path: Path):
|
||||
# Five identical successful traces of (web_search, calculator)
|
||||
traces = [_make_trace(["web_search", "calculator"]) for _ in range(5)]
|
||||
store = _FakeTraceStore(traces)
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
output_dir = tmp_path / "discovered"
|
||||
written = mgr.discover_from_traces(
|
||||
store, min_frequency=3, output_dir=output_dir
|
||||
)
|
||||
|
||||
assert len(written) >= 1
|
||||
# Discovery should produce at least one manifest file
|
||||
assert any(Path(item["path"]).exists() for item in written)
|
||||
|
||||
def test_respects_min_frequency_filter(self, tmp_path: Path):
|
||||
# Only 2 occurrences — below default min_frequency of 3
|
||||
traces = [_make_trace(["web_search", "calculator"]) for _ in range(2)]
|
||||
store = _FakeTraceStore(traces)
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
output_dir = tmp_path / "discovered"
|
||||
written = mgr.discover_from_traces(
|
||||
store, min_frequency=3, output_dir=output_dir
|
||||
)
|
||||
|
||||
assert written == []
|
||||
|
||||
def test_empty_trace_store_returns_empty_list(self, tmp_path: Path):
|
||||
store = _FakeTraceStore([])
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
output_dir = tmp_path / "discovered"
|
||||
written = mgr.discover_from_traces(store, output_dir=output_dir)
|
||||
assert written == []
|
||||
|
||||
def test_discovered_skills_use_hyphen_naming(self, tmp_path: Path):
|
||||
traces = [_make_trace(["web_search", "calculator"]) for _ in range(5)]
|
||||
store = _FakeTraceStore(traces)
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
output_dir = tmp_path / "discovered"
|
||||
written = mgr.discover_from_traces(
|
||||
store, min_frequency=3, output_dir=output_dir
|
||||
)
|
||||
|
||||
for item in written:
|
||||
name = item["name"]
|
||||
# Plan 1 spec: lowercase, only hyphens (no underscores)
|
||||
assert name == name.lower()
|
||||
assert "_" not in name
|
||||
|
||||
def test_round_trip_discover_then_load(self, tmp_path: Path):
|
||||
"""Discovered skills are subsequently loadable via discover()."""
|
||||
traces = [_make_trace(["web_search", "calculator"]) for _ in range(5)]
|
||||
store = _FakeTraceStore(traces)
|
||||
|
||||
mgr = SkillManager(bus=EventBus())
|
||||
output_dir = tmp_path / "discovered"
|
||||
mgr.discover_from_traces(store, min_frequency=3, output_dir=output_dir)
|
||||
|
||||
# Now load them
|
||||
mgr2 = SkillManager(bus=EventBus())
|
||||
mgr2.discover(paths=[output_dir])
|
||||
# At least one of the discovered skills should be loadable
|
||||
names = mgr2.skill_names()
|
||||
assert len(names) >= 1
|
||||
@@ -0,0 +1,167 @@
|
||||
"""Tests for native_react skill_few_shot_examples wiring (Plan 2B I3 fix)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from openjarvis.agents.native_react import REACT_SYSTEM_PROMPT, NativeReActAgent
|
||||
|
||||
|
||||
class _StubEngine:
|
||||
def generate(self, *args, **kwargs):
|
||||
return {"content": "Final Answer: stub", "usage": {}}
|
||||
|
||||
|
||||
class TestToolUsingAgentAcceptsKwarg:
|
||||
def test_default_empty_list(self):
|
||||
agent = NativeReActAgent(
|
||||
engine=_StubEngine(),
|
||||
model="stub",
|
||||
tools=[],
|
||||
)
|
||||
assert agent._skill_few_shot_examples == []
|
||||
|
||||
def test_explicit_examples_stored(self):
|
||||
examples = ["### research-skill\nInput: q\nOutput: a"]
|
||||
agent = NativeReActAgent(
|
||||
engine=_StubEngine(),
|
||||
model="stub",
|
||||
tools=[],
|
||||
skill_few_shot_examples=examples,
|
||||
)
|
||||
assert agent._skill_few_shot_examples == examples
|
||||
|
||||
def test_none_resolves_to_empty_list(self):
|
||||
agent = NativeReActAgent(
|
||||
engine=_StubEngine(),
|
||||
model="stub",
|
||||
tools=[],
|
||||
skill_few_shot_examples=None,
|
||||
)
|
||||
assert agent._skill_few_shot_examples == []
|
||||
|
||||
|
||||
class TestReactSystemPromptPlaceholder:
|
||||
def test_format_with_empty_examples(self):
|
||||
rendered = REACT_SYSTEM_PROMPT.format(
|
||||
tool_descriptions="No tools available.",
|
||||
skill_examples="",
|
||||
)
|
||||
assert "## Skill Examples" not in rendered
|
||||
assert "No tools available." in rendered
|
||||
# ReAct format spec must still be present
|
||||
assert "Thought:" in rendered
|
||||
assert "Action:" in rendered
|
||||
assert "Final Answer:" in rendered
|
||||
|
||||
def test_format_with_non_empty_examples(self):
|
||||
examples_block = (
|
||||
"## Skill Examples\n\n### research-skill\nInput: q\nOutput: a\n\n"
|
||||
)
|
||||
rendered = REACT_SYSTEM_PROMPT.format(
|
||||
tool_descriptions="No tools available.",
|
||||
skill_examples=examples_block,
|
||||
)
|
||||
assert "## Skill Examples" in rendered
|
||||
assert "research-skill" in rendered
|
||||
assert "Input: q" in rendered
|
||||
assert "Output: a" in rendered
|
||||
|
||||
|
||||
class TestSystemBuilderCapturesFewShot:
|
||||
def test_skill_manager_examples_stored_on_system(self, tmp_path):
|
||||
"""SystemBuilder.build() pulls examples from SkillManager and
|
||||
stashes them on the JarvisSystem instance for _run_agent to
|
||||
forward to tool-using agents."""
|
||||
|
||||
from openjarvis.skills.manager import SkillManager
|
||||
from openjarvis.skills.overlay import SkillOverlay, write_overlay
|
||||
from openjarvis.skills.types import SkillManifest
|
||||
|
||||
# Build an overlay so the manager picks up real few-shot examples
|
||||
overlay_dir = tmp_path / "overlays"
|
||||
write_overlay(
|
||||
SkillOverlay(
|
||||
skill_name="seeded-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T00:00:00Z",
|
||||
trace_count=10,
|
||||
description="Optimized",
|
||||
few_shot=[{"input": "ping", "output": "pong"}],
|
||||
),
|
||||
overlay_dir,
|
||||
)
|
||||
|
||||
from openjarvis.core.events import EventBus
|
||||
|
||||
mgr = SkillManager(bus=EventBus(), overlay_dir=overlay_dir)
|
||||
mgr._skills["seeded-skill"] = SkillManifest(
|
||||
name="seeded-skill",
|
||||
description="Original",
|
||||
markdown_content="Body",
|
||||
)
|
||||
mgr.discover() # applies overlay
|
||||
|
||||
# The captured examples should now contain our seeded one
|
||||
examples = mgr.get_few_shot_examples()
|
||||
assert len(examples) >= 1
|
||||
assert any("ping" in s and "pong" in s for s in examples)
|
||||
|
||||
|
||||
class TestRunAgentForwardsExamples:
|
||||
def test_run_agent_passes_examples_to_tool_using_agent(self):
|
||||
"""_run_agent injects system._skill_few_shot_examples into
|
||||
agent_kwargs when the agent class has accepts_tools=True."""
|
||||
from unittest.mock import patch
|
||||
|
||||
from openjarvis.agents._stubs import AgentResult
|
||||
from openjarvis.system import JarvisSystem
|
||||
|
||||
captured_kwargs: dict = {}
|
||||
|
||||
class _CapturingAgent:
|
||||
accepts_tools = True
|
||||
|
||||
def __init__(self, engine, model, **kwargs):
|
||||
captured_kwargs.update(kwargs)
|
||||
|
||||
def run(self, query, context=None, **kw):
|
||||
return AgentResult(content="ok", turns=1)
|
||||
|
||||
# Build a minimal JarvisSystem with the captured examples
|
||||
system = JarvisSystem.__new__(JarvisSystem)
|
||||
system.config = MagicMock()
|
||||
system.config.intelligence.temperature = 0.0
|
||||
system.config.intelligence.max_tokens = 100
|
||||
system.config.agent.max_turns = 5
|
||||
system.bus = MagicMock()
|
||||
system.engine = MagicMock()
|
||||
system.engine_key = "stub"
|
||||
system.model = "stub"
|
||||
system.tools = []
|
||||
system.tool_executor = None
|
||||
system.memory_backend = None
|
||||
system.channel_backend = None
|
||||
system.trace_store = None
|
||||
system.capability_policy = None
|
||||
system.session_store = None
|
||||
system._skill_few_shot_examples = ["### research-skill\nInput: q\nOutput: a"]
|
||||
system._mcp_clients = []
|
||||
|
||||
with patch(
|
||||
"openjarvis.core.registry.AgentRegistry.get",
|
||||
return_value=_CapturingAgent,
|
||||
):
|
||||
system._run_agent(
|
||||
query="test",
|
||||
messages=[],
|
||||
agent_name="capturing",
|
||||
tool_names=None,
|
||||
temperature=0.0,
|
||||
max_tokens=100,
|
||||
)
|
||||
|
||||
assert "skill_few_shot_examples" in captured_kwargs
|
||||
assert captured_kwargs["skill_few_shot_examples"] == [
|
||||
"### research-skill\nInput: q\nOutput: a"
|
||||
]
|
||||
@@ -0,0 +1,99 @@
|
||||
"""Tests for the skills overlay loader/writer (Plan 2A)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from openjarvis.skills.overlay import (
|
||||
SkillOverlay,
|
||||
SkillOverlayLoader,
|
||||
write_overlay,
|
||||
)
|
||||
|
||||
|
||||
class TestSkillOverlayDataclass:
|
||||
def test_create_minimal(self):
|
||||
ov = SkillOverlay(
|
||||
skill_name="my-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T14:30:00Z",
|
||||
trace_count=47,
|
||||
description="Optimized description",
|
||||
)
|
||||
assert ov.skill_name == "my-skill"
|
||||
assert ov.optimizer == "dspy"
|
||||
assert ov.few_shot == []
|
||||
|
||||
def test_create_with_few_shot(self):
|
||||
ov = SkillOverlay(
|
||||
skill_name="my-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T14:30:00Z",
|
||||
trace_count=47,
|
||||
description="Optimized description",
|
||||
few_shot=[
|
||||
{"input": "transformer attention", "output": "## Recent Advances..."},
|
||||
],
|
||||
)
|
||||
assert len(ov.few_shot) == 1
|
||||
assert ov.few_shot[0]["input"] == "transformer attention"
|
||||
|
||||
|
||||
class TestWriteOverlay:
|
||||
def test_write_creates_file(self, tmp_path: Path):
|
||||
overlay_dir = tmp_path / "learning" / "skills"
|
||||
ov = SkillOverlay(
|
||||
skill_name="test-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T14:30:00Z",
|
||||
trace_count=20,
|
||||
description="Optimized",
|
||||
)
|
||||
path = write_overlay(ov, overlay_dir)
|
||||
assert path.exists()
|
||||
assert path.name == "optimized.toml"
|
||||
assert path.parent.name == "test-skill"
|
||||
|
||||
def test_write_then_read_roundtrip(self, tmp_path: Path):
|
||||
overlay_dir = tmp_path / "learning" / "skills"
|
||||
ov = SkillOverlay(
|
||||
skill_name="test-skill",
|
||||
optimizer="dspy",
|
||||
optimized_at="2026-04-08T14:30:00Z",
|
||||
trace_count=42,
|
||||
description="Better description",
|
||||
few_shot=[
|
||||
{"input": "q1", "output": "a1"},
|
||||
{"input": "q2", "output": "a2"},
|
||||
],
|
||||
)
|
||||
write_overlay(ov, overlay_dir)
|
||||
|
||||
loader = SkillOverlayLoader(overlay_dir)
|
||||
loaded = loader.load("test-skill")
|
||||
assert loaded is not None
|
||||
assert loaded.skill_name == "test-skill"
|
||||
assert loaded.optimizer == "dspy"
|
||||
assert loaded.trace_count == 42
|
||||
assert loaded.description == "Better description"
|
||||
assert len(loaded.few_shot) == 2
|
||||
assert loaded.few_shot[0]["input"] == "q1"
|
||||
|
||||
|
||||
class TestSkillOverlayLoader:
|
||||
def test_load_missing_returns_none(self, tmp_path: Path):
|
||||
overlay_dir = tmp_path / "learning" / "skills"
|
||||
loader = SkillOverlayLoader(overlay_dir)
|
||||
assert loader.load("nonexistent") is None
|
||||
|
||||
def test_load_malformed_returns_none(self, tmp_path: Path):
|
||||
overlay_dir = tmp_path / "learning" / "skills" / "broken"
|
||||
overlay_dir.mkdir(parents=True)
|
||||
(overlay_dir / "optimized.toml").write_text("not valid toml = [[[")
|
||||
|
||||
loader = SkillOverlayLoader(tmp_path / "learning" / "skills")
|
||||
assert loader.load("broken") is None
|
||||
|
||||
def test_load_directory_does_not_exist(self, tmp_path: Path):
|
||||
loader = SkillOverlayLoader(tmp_path / "does-not-exist")
|
||||
assert loader.load("anything") is None
|
||||
@@ -0,0 +1,213 @@
|
||||
"""Tests for the SkillParser strict + tolerant passes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from openjarvis.skills.parser import SkillParseError, SkillParser
|
||||
|
||||
|
||||
class TestStrictRequiredFields:
|
||||
def test_missing_name_raises(self):
|
||||
parser = SkillParser()
|
||||
frontmatter = {"description": "x"}
|
||||
with pytest.raises(SkillParseError, match="name"):
|
||||
parser.parse_frontmatter(frontmatter)
|
||||
|
||||
def test_missing_description_raises(self):
|
||||
parser = SkillParser()
|
||||
frontmatter = {"name": "test-skill"}
|
||||
with pytest.raises(SkillParseError, match="description"):
|
||||
parser.parse_frontmatter(frontmatter)
|
||||
|
||||
def test_minimal_valid_frontmatter(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{"name": "minimal", "description": "Does something useful"}
|
||||
)
|
||||
assert manifest.name == "minimal"
|
||||
assert manifest.description == "Does something useful"
|
||||
|
||||
|
||||
class TestStrictNamingRules:
|
||||
def test_uppercase_rejected(self):
|
||||
parser = SkillParser()
|
||||
with pytest.raises(SkillParseError, match="lowercase"):
|
||||
parser.parse_frontmatter({"name": "MySkill", "description": "x"})
|
||||
|
||||
def test_underscore_rejected(self):
|
||||
parser = SkillParser()
|
||||
with pytest.raises(SkillParseError, match="hyphen"):
|
||||
parser.parse_frontmatter({"name": "my_skill", "description": "x"})
|
||||
|
||||
def test_leading_hyphen_rejected(self):
|
||||
parser = SkillParser()
|
||||
with pytest.raises(SkillParseError, match="hyphen"):
|
||||
parser.parse_frontmatter({"name": "-skill", "description": "x"})
|
||||
|
||||
def test_trailing_hyphen_rejected(self):
|
||||
parser = SkillParser()
|
||||
with pytest.raises(SkillParseError, match="hyphen"):
|
||||
parser.parse_frontmatter({"name": "skill-", "description": "x"})
|
||||
|
||||
def test_consecutive_hyphens_rejected(self):
|
||||
parser = SkillParser()
|
||||
with pytest.raises(SkillParseError, match="consecutive"):
|
||||
parser.parse_frontmatter({"name": "my--skill", "description": "x"})
|
||||
|
||||
def test_valid_kebab_name_accepted(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{"name": "my-skill-123", "description": "x"}
|
||||
)
|
||||
assert manifest.name == "my-skill-123"
|
||||
|
||||
|
||||
class TestStrictLengthLimits:
|
||||
def test_name_over_64_chars_rejected(self):
|
||||
parser = SkillParser()
|
||||
too_long = "a" * 65
|
||||
with pytest.raises(SkillParseError, match="64"):
|
||||
parser.parse_frontmatter({"name": too_long, "description": "x"})
|
||||
|
||||
def test_description_over_1024_chars_rejected(self):
|
||||
parser = SkillParser()
|
||||
too_long = "a" * 1025
|
||||
with pytest.raises(SkillParseError, match="1024"):
|
||||
parser.parse_frontmatter({"name": "test", "description": too_long})
|
||||
|
||||
|
||||
class TestTolerantFieldMapping:
|
||||
def test_top_level_version_mapped_to_metadata(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"version": "1.2.3",
|
||||
}
|
||||
)
|
||||
assert manifest.version == "1.2.3"
|
||||
|
||||
def test_top_level_author_mapped_to_metadata(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"author": "alice",
|
||||
}
|
||||
)
|
||||
assert manifest.author == "alice"
|
||||
|
||||
def test_top_level_tags_mapped_to_metadata(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"tags": ["research", "nlp"],
|
||||
}
|
||||
)
|
||||
assert manifest.tags == ["research", "nlp"]
|
||||
|
||||
def test_top_level_required_capabilities_mapped(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"required_capabilities": ["network:fetch"],
|
||||
}
|
||||
)
|
||||
assert manifest.required_capabilities == ["network:fetch"]
|
||||
|
||||
def test_top_level_depends_mapped(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"depends": ["summarize"],
|
||||
}
|
||||
)
|
||||
assert manifest.depends == ["summarize"]
|
||||
|
||||
def test_top_level_user_invocable_mapped(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"user_invocable": False,
|
||||
}
|
||||
)
|
||||
assert manifest.user_invocable is False
|
||||
|
||||
def test_top_level_disable_model_invocation_mapped(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"disable_model_invocation": True,
|
||||
}
|
||||
)
|
||||
assert manifest.disable_model_invocation is True
|
||||
|
||||
def test_metadata_openjarvis_namespace_used(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"metadata": {
|
||||
"openjarvis": {
|
||||
"version": "9.9.9",
|
||||
"tags": ["already", "mapped"],
|
||||
},
|
||||
},
|
||||
}
|
||||
)
|
||||
assert manifest.version == "9.9.9"
|
||||
assert manifest.tags == ["already", "mapped"]
|
||||
|
||||
def test_unmapped_field_logs_warning(self, caplog):
|
||||
import logging
|
||||
|
||||
parser = SkillParser()
|
||||
with caplog.at_level(logging.WARNING):
|
||||
parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"weird_vendor_field": "something",
|
||||
}
|
||||
)
|
||||
assert any("weird_vendor_field" in record.message for record in caplog.records)
|
||||
|
||||
def test_original_frontmatter_preserved(self):
|
||||
parser = SkillParser()
|
||||
original = {
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"weird_field": "preserved",
|
||||
}
|
||||
manifest = parser.parse_frontmatter(original)
|
||||
# Stored under metadata.openjarvis.original_frontmatter
|
||||
oj = manifest.metadata.get("openjarvis", {})
|
||||
assert "original_frontmatter" in oj
|
||||
assert oj["original_frontmatter"]["weird_field"] == "preserved"
|
||||
|
||||
def test_platforms_list_mapped_to_compatibility_string(self):
|
||||
parser = SkillParser()
|
||||
manifest = parser.parse_frontmatter(
|
||||
{
|
||||
"name": "test",
|
||||
"description": "x",
|
||||
"platforms": ["macos", "linux"],
|
||||
}
|
||||
)
|
||||
# Platforms get rendered into a compatibility string under metadata
|
||||
oj = manifest.metadata.get("openjarvis", {})
|
||||
assert "platforms" in oj or "compatibility" in oj
|
||||
@@ -0,0 +1,59 @@
|
||||
"""Tests for skill security — capability validation and trust tiers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from openjarvis.skills.security import (
|
||||
TrustTier,
|
||||
classify_trust_tier,
|
||||
has_dangerous_capabilities,
|
||||
validate_capabilities,
|
||||
)
|
||||
from openjarvis.skills.types import SkillManifest
|
||||
|
||||
|
||||
class TestTrustTiers:
|
||||
def test_bundled_tier(self):
|
||||
assert classify_trust_tier(is_bundled=True) == TrustTier.BUNDLED
|
||||
|
||||
def test_indexed_tier(self):
|
||||
assert (
|
||||
classify_trust_tier(has_signature=True, in_index=True) == TrustTier.INDEXED
|
||||
)
|
||||
|
||||
def test_unreviewed_tier(self):
|
||||
assert (
|
||||
classify_trust_tier(has_signature=False, in_index=False)
|
||||
== TrustTier.UNREVIEWED
|
||||
)
|
||||
|
||||
def test_workspace_tier(self):
|
||||
assert classify_trust_tier(is_workspace=True) == TrustTier.WORKSPACE
|
||||
|
||||
|
||||
class TestCapabilityValidation:
|
||||
def test_valid_capabilities(self):
|
||||
manifest = SkillManifest(name="test", required_capabilities=["network:fetch"])
|
||||
allowed = {"network:fetch", "filesystem:read"}
|
||||
assert validate_capabilities(manifest, allowed) == []
|
||||
|
||||
def test_missing_capability(self):
|
||||
manifest = SkillManifest(name="test", required_capabilities=["shell:execute"])
|
||||
allowed = {"network:fetch"}
|
||||
violations = validate_capabilities(manifest, allowed)
|
||||
assert "shell:execute" in violations
|
||||
|
||||
|
||||
class TestDangerousCapabilities:
|
||||
def test_detects_dangerous(self):
|
||||
manifest = SkillManifest(
|
||||
name="test", required_capabilities=["shell:execute", "network:fetch"]
|
||||
)
|
||||
dangerous = has_dangerous_capabilities(manifest)
|
||||
assert "shell:execute" in dangerous
|
||||
assert "network:fetch" not in dangerous
|
||||
|
||||
def test_no_dangerous(self):
|
||||
manifest = SkillManifest(
|
||||
name="test", required_capabilities=["network:fetch", "filesystem:read"]
|
||||
)
|
||||
assert has_dangerous_capabilities(manifest) == []
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Tests for SkillTool metadata tagging (Plan 2A trace tagging)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from openjarvis.core.types import ToolResult
|
||||
from openjarvis.skills.executor import SkillExecutor
|
||||
from openjarvis.skills.tool_adapter import SkillTool
|
||||
from openjarvis.skills.types import SkillManifest, SkillStep
|
||||
from openjarvis.tools._stubs import BaseTool, ToolExecutor, ToolSpec
|
||||
|
||||
|
||||
class _EchoTool(BaseTool):
|
||||
tool_id = "echo"
|
||||
|
||||
@property
|
||||
def spec(self):
|
||||
return ToolSpec(name="echo", description="echo")
|
||||
|
||||
def execute(self, **params):
|
||||
return ToolResult(
|
||||
tool_name="echo",
|
||||
content=params.get("text", ""),
|
||||
success=True,
|
||||
)
|
||||
|
||||
|
||||
class TestSkillToolMetadataTagging:
|
||||
def _make_tool(self, manifest: SkillManifest) -> SkillTool:
|
||||
executor = SkillExecutor(ToolExecutor([_EchoTool()]))
|
||||
return SkillTool(manifest, executor)
|
||||
|
||||
def test_metadata_includes_skill_name(self):
|
||||
manifest = SkillManifest(
|
||||
name="my-skill",
|
||||
description="A test skill",
|
||||
markdown_content="Just instructions",
|
||||
)
|
||||
tool = self._make_tool(manifest)
|
||||
result = tool.execute(task="hello")
|
||||
assert result.metadata["skill"] == "my-skill"
|
||||
|
||||
def test_skill_source_defaults_to_user(self):
|
||||
manifest = SkillManifest(
|
||||
name="my-skill",
|
||||
description="A test skill",
|
||||
markdown_content="Just instructions",
|
||||
)
|
||||
tool = self._make_tool(manifest)
|
||||
result = tool.execute(task="hello")
|
||||
assert result.metadata["skill_source"] == "user"
|
||||
|
||||
def test_skill_source_propagates_from_manifest(self):
|
||||
manifest = SkillManifest(
|
||||
name="apple-notes",
|
||||
description="Apple Notes",
|
||||
markdown_content="Use memo",
|
||||
metadata={"openjarvis": {"source": "hermes"}},
|
||||
)
|
||||
tool = self._make_tool(manifest)
|
||||
result = tool.execute(task="create a note")
|
||||
assert result.metadata["skill_source"] == "hermes"
|
||||
|
||||
def test_skill_kind_instructional_when_no_steps(self):
|
||||
manifest = SkillManifest(
|
||||
name="explainer",
|
||||
description="Explains things",
|
||||
markdown_content="Explain step by step",
|
||||
)
|
||||
tool = self._make_tool(manifest)
|
||||
result = tool.execute(task="explain a loop")
|
||||
assert result.metadata["skill_kind"] == "instructional"
|
||||
|
||||
def test_skill_kind_executable_when_steps_present(self):
|
||||
manifest = SkillManifest(
|
||||
name="echo-skill",
|
||||
description="Echoes input",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "{input}"}',
|
||||
output_key="result",
|
||||
),
|
||||
],
|
||||
)
|
||||
tool = self._make_tool(manifest)
|
||||
result = tool.execute(input="hello")
|
||||
assert result.metadata["skill_kind"] == "executable"
|
||||
|
||||
def test_failed_pipeline_still_tags_metadata(self):
|
||||
manifest = SkillManifest(
|
||||
name="broken",
|
||||
description="Calls nonexistent tool",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="nonexistent_tool",
|
||||
arguments_template="{}",
|
||||
output_key="x",
|
||||
),
|
||||
],
|
||||
)
|
||||
tool = self._make_tool(manifest)
|
||||
result = tool.execute()
|
||||
assert result.success is False
|
||||
assert result.metadata["skill"] == "broken"
|
||||
assert result.metadata["skill_kind"] == "executable"
|
||||
@@ -0,0 +1,212 @@
|
||||
"""End-to-end integration test for skill trace tagging (Plan 2A C1 fix).
|
||||
|
||||
Verifies the full flow:
|
||||
SkillTool.execute()
|
||||
→ ToolExecutor publishes TOOL_CALL_END with metadata
|
||||
→ TraceCollector copies metadata into TraceStep.metadata
|
||||
→ SkillOptimizer can bucket traces by skill name
|
||||
|
||||
If this test passes, the trace tagging path is wired end-to-end.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from openjarvis.core.events import EventBus
|
||||
from openjarvis.core.types import StepType, ToolCall, ToolResult
|
||||
from openjarvis.skills.executor import SkillExecutor
|
||||
from openjarvis.skills.tool_adapter import SkillTool
|
||||
from openjarvis.skills.types import SkillManifest
|
||||
from openjarvis.tools._stubs import BaseTool, ToolExecutor, ToolSpec
|
||||
|
||||
|
||||
class TestSkillTraceTaggingEndToEnd:
|
||||
def test_skill_metadata_flows_to_trace_step(self) -> None:
|
||||
"""SkillTool metadata reaches TraceCollector via the event bus."""
|
||||
bus = EventBus(record_history=True)
|
||||
|
||||
# Subscribe a small collector that mimics TraceCollector._on_tool_end
|
||||
captured_metadata: dict = {}
|
||||
|
||||
def _on_tool_end(event):
|
||||
nonlocal captured_metadata
|
||||
captured_metadata = event.data.get("metadata", {})
|
||||
|
||||
from openjarvis.core.events import EventType
|
||||
|
||||
bus.subscribe(EventType.TOOL_CALL_END, _on_tool_end)
|
||||
|
||||
# Build a SkillTool wrapping an instructional manifest
|
||||
manifest = SkillManifest(
|
||||
name="research-skill",
|
||||
description="x",
|
||||
markdown_content="Just instructions.",
|
||||
metadata={"openjarvis": {"source": "hermes"}},
|
||||
)
|
||||
skill_executor = SkillExecutor(ToolExecutor([], bus=bus))
|
||||
skill_tool = SkillTool(manifest, skill_executor)
|
||||
|
||||
# Build a ToolExecutor that knows about the SkillTool, with the bus
|
||||
tool_executor = ToolExecutor([skill_tool], bus=bus)
|
||||
# Invoke through the executor (this is what the agent would do)
|
||||
tool_executor.execute(
|
||||
ToolCall(id="t1", name="skill_research-skill", arguments="{}")
|
||||
)
|
||||
|
||||
# Now the published TOOL_CALL_END event should carry the metadata
|
||||
assert captured_metadata.get("skill") == "research-skill"
|
||||
assert captured_metadata.get("skill_source") == "hermes"
|
||||
assert captured_metadata.get("skill_kind") == "instructional"
|
||||
|
||||
def test_trace_collector_writes_metadata_to_step(self) -> None:
|
||||
"""A real TraceCollector populates TraceStep.metadata from the event."""
|
||||
from openjarvis.traces.collector import TraceCollector
|
||||
|
||||
bus = EventBus(record_history=True)
|
||||
|
||||
# Set up the SkillTool
|
||||
manifest = SkillManifest(
|
||||
name="my-skill",
|
||||
description="x",
|
||||
markdown_content="Body",
|
||||
metadata={"openjarvis": {"source": "openclaw"}},
|
||||
)
|
||||
skill_executor = SkillExecutor(ToolExecutor([], bus=bus))
|
||||
skill_tool = SkillTool(manifest, skill_executor)
|
||||
tool_executor = ToolExecutor([skill_tool], bus=bus)
|
||||
|
||||
# Stub agent that just calls the skill once
|
||||
class _StubAgent:
|
||||
agent_id = "stub"
|
||||
|
||||
def run(self, query, context=None, **kwargs):
|
||||
from openjarvis.agents._stubs import AgentResult
|
||||
|
||||
tool_executor.execute(
|
||||
ToolCall(
|
||||
id="t1",
|
||||
name="skill_my-skill",
|
||||
arguments="{}",
|
||||
)
|
||||
)
|
||||
return AgentResult(content="done", turns=1)
|
||||
|
||||
collector = TraceCollector(
|
||||
agent=_StubAgent(),
|
||||
store=None, # in-memory only
|
||||
bus=bus,
|
||||
)
|
||||
collector.run("test")
|
||||
|
||||
# Inspect the captured trace
|
||||
trace = collector.last_trace
|
||||
assert trace is not None, "Collector should have captured a trace"
|
||||
|
||||
tool_steps = [s for s in trace.steps if s.step_type == StepType.TOOL_CALL]
|
||||
assert len(tool_steps) >= 1, "Should have captured at least one tool step"
|
||||
|
||||
first = tool_steps[0]
|
||||
assert first.metadata.get("skill") == "my-skill", (
|
||||
f"Expected metadata.skill='my-skill', got {first.metadata!r}"
|
||||
)
|
||||
assert first.metadata.get("skill_source") == "openclaw"
|
||||
assert first.metadata.get("skill_kind") == "instructional"
|
||||
|
||||
|
||||
class _TaintingTool(BaseTool):
|
||||
"""Test tool whose result triggers the auto_detect_taint codepath.
|
||||
|
||||
Returns content containing strings the security taint scanner picks
|
||||
up as user-input or external — those add a `_taint: TaintSet` to
|
||||
the result metadata before TOOL_CALL_END is published.
|
||||
"""
|
||||
|
||||
tool_id = "tainting"
|
||||
|
||||
@property
|
||||
def spec(self):
|
||||
return ToolSpec(name="tainting", description="emit tainted output")
|
||||
|
||||
def execute(self, **params):
|
||||
return ToolResult(
|
||||
tool_name="tainting",
|
||||
content="user said: hello world",
|
||||
success=True,
|
||||
)
|
||||
|
||||
|
||||
class TestEventMetadataIsJsonSafe:
|
||||
"""Plan 2B regression: TOOL_CALL_END payload metadata must be JSON
|
||||
serializable so the trace store can persist it without crashing.
|
||||
|
||||
Bug surfaced by Plan 2B Task 11 smoke test, where every PinchBench
|
||||
task failed with `Object of type TaintSet is not JSON serializable`
|
||||
because ToolExecutor was passing through the internal `_taint` key
|
||||
that the security auto-detect adds.
|
||||
"""
|
||||
|
||||
def test_event_metadata_excludes_non_json_objects(self):
|
||||
from openjarvis.core.events import EventType
|
||||
|
||||
bus = EventBus(record_history=True)
|
||||
captured: dict = {}
|
||||
|
||||
def _on_tool_end(event):
|
||||
nonlocal captured
|
||||
captured = dict(event.data.get("metadata") or {})
|
||||
|
||||
bus.subscribe(EventType.TOOL_CALL_END, _on_tool_end)
|
||||
|
||||
tool_executor = ToolExecutor([_TaintingTool()], bus=bus)
|
||||
tool_executor.execute(ToolCall(id="t1", name="tainting", arguments="{}"))
|
||||
|
||||
# The published metadata must be JSON serializable end-to-end
|
||||
# — TraceCollector will eventually feed this to TraceStore.save()
|
||||
# which calls json.dumps() on TraceStep.metadata.
|
||||
try:
|
||||
json.dumps(captured)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise AssertionError(
|
||||
f"Published TOOL_CALL_END metadata is not JSON serializable: "
|
||||
f"{exc}. Keys: {list(captured.keys())}"
|
||||
)
|
||||
|
||||
# And the internal _taint key must NOT be present (it would have
|
||||
# been the offender if json.dumps had failed)
|
||||
assert "_taint" not in captured, (
|
||||
f"_taint key leaked into event metadata: {captured!r}"
|
||||
)
|
||||
|
||||
def test_skill_metadata_still_present_after_filtering(self):
|
||||
"""The JSON-safe filter must NOT drop legitimate skill metadata."""
|
||||
from openjarvis.core.events import EventType
|
||||
|
||||
bus = EventBus(record_history=True)
|
||||
captured: dict = {}
|
||||
|
||||
def _on_tool_end(event):
|
||||
nonlocal captured
|
||||
captured = dict(event.data.get("metadata") or {})
|
||||
|
||||
bus.subscribe(EventType.TOOL_CALL_END, _on_tool_end)
|
||||
|
||||
manifest = SkillManifest(
|
||||
name="my-skill",
|
||||
description="x",
|
||||
markdown_content="Body",
|
||||
metadata={"openjarvis": {"source": "hermes"}},
|
||||
)
|
||||
skill_executor = SkillExecutor(ToolExecutor([], bus=bus))
|
||||
skill_tool = SkillTool(manifest, skill_executor)
|
||||
|
||||
tool_executor = ToolExecutor([skill_tool], bus=bus)
|
||||
tool_executor.execute(ToolCall(id="t1", name="skill_my-skill", arguments="{}"))
|
||||
|
||||
# Skill keys must survive the filter
|
||||
assert captured.get("skill") == "my-skill"
|
||||
assert captured.get("skill_source") == "hermes"
|
||||
assert captured.get("skill_kind") == "instructional"
|
||||
|
||||
# And it must still be JSON-serializable
|
||||
json.dumps(captured)
|
||||
@@ -148,6 +148,129 @@ class TestSkillExecutor:
|
||||
assert EventType.SKILL_EXECUTE_END in event_types
|
||||
|
||||
|
||||
class TestSkillStepExtended:
|
||||
def test_step_with_skill_name(self):
|
||||
step = SkillStep(skill_name="summarize", output_key="result")
|
||||
assert step.skill_name == "summarize"
|
||||
assert step.tool_name == ""
|
||||
|
||||
def test_step_either_tool_or_skill(self):
|
||||
step_tool = SkillStep(tool_name="web_search")
|
||||
step_skill = SkillStep(skill_name="summarize")
|
||||
assert step_tool.tool_name == "web_search"
|
||||
assert step_tool.skill_name == ""
|
||||
assert step_skill.skill_name == "summarize"
|
||||
assert step_skill.tool_name == ""
|
||||
|
||||
|
||||
class TestSkillManifestExtended:
|
||||
def test_manifest_with_tags_and_depends(self):
|
||||
manifest = SkillManifest(
|
||||
name="test",
|
||||
tags=["research"],
|
||||
depends=["summarize"],
|
||||
)
|
||||
assert manifest.tags == ["research"]
|
||||
assert manifest.depends == ["summarize"]
|
||||
|
||||
def test_manifest_with_invocation_flags(self):
|
||||
manifest = SkillManifest(
|
||||
name="test",
|
||||
user_invocable=False,
|
||||
disable_model_invocation=True,
|
||||
)
|
||||
assert not manifest.user_invocable
|
||||
assert manifest.disable_model_invocation
|
||||
|
||||
def test_manifest_with_markdown_content(self):
|
||||
manifest = SkillManifest(
|
||||
name="test",
|
||||
markdown_content="When asked to research...",
|
||||
)
|
||||
assert manifest.markdown_content == "When asked to research..."
|
||||
|
||||
def test_manifest_bytes_includes_new_fields(self):
|
||||
manifest = SkillManifest(
|
||||
name="test",
|
||||
tags=["a"],
|
||||
depends=["b"],
|
||||
steps=[],
|
||||
)
|
||||
data = manifest.manifest_bytes()
|
||||
assert b"tags" in data
|
||||
assert b"depends" in data
|
||||
|
||||
|
||||
class TestSkillExecutorSubSkills:
|
||||
def test_sub_skill_delegation(self):
|
||||
"""Executor delegates skill_name steps to a skill resolver."""
|
||||
tools = [EchoTool(), UpperTool()]
|
||||
tool_executor = ToolExecutor(tools)
|
||||
executor = SkillExecutor(tool_executor)
|
||||
|
||||
child_manifest = SkillManifest(
|
||||
name="upper_skill",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="upper",
|
||||
arguments_template='{"text": "{text}"}',
|
||||
output_key="uppered",
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
def resolve_skill(name, context):
|
||||
from openjarvis.skills.executor import SkillResult
|
||||
|
||||
if name == "upper_skill":
|
||||
return executor.run(child_manifest, initial_context=context)
|
||||
return SkillResult(skill_name=name, success=False)
|
||||
|
||||
executor.set_skill_resolver(resolve_skill)
|
||||
|
||||
parent_manifest = SkillManifest(
|
||||
name="parent",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "hello"}',
|
||||
output_key="echoed",
|
||||
),
|
||||
SkillStep(
|
||||
skill_name="upper_skill",
|
||||
arguments_template='{"text": "{echoed}"}',
|
||||
output_key="result",
|
||||
),
|
||||
],
|
||||
)
|
||||
result = executor.run(parent_manifest)
|
||||
assert result.success
|
||||
assert result.context.get("result") == "HELLO"
|
||||
|
||||
def test_sub_skill_failure_stops_pipeline(self):
|
||||
tools = [EchoTool()]
|
||||
tool_executor = ToolExecutor(tools)
|
||||
executor = SkillExecutor(tool_executor)
|
||||
|
||||
def resolve_skill(name, context):
|
||||
from openjarvis.skills.executor import SkillResult
|
||||
|
||||
return SkillResult(skill_name=name, success=False)
|
||||
|
||||
executor.set_skill_resolver(resolve_skill)
|
||||
|
||||
manifest = SkillManifest(
|
||||
name="parent",
|
||||
steps=[
|
||||
SkillStep(skill_name="nonexistent", output_key="x"),
|
||||
SkillStep(tool_name="echo", output_key="y"),
|
||||
],
|
||||
)
|
||||
result = executor.run(manifest)
|
||||
assert not result.success
|
||||
assert len(result.step_results) == 1
|
||||
|
||||
|
||||
class TestSkillTool:
|
||||
def test_skill_as_tool(self):
|
||||
from openjarvis.skills.tool_adapter import SkillTool
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
"""Tests for SourceResolver ABC and resolver implementations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
from openjarvis.skills.sources.base import ResolvedSkill, SourceResolver
|
||||
|
||||
|
||||
class _FakeResolver(SourceResolver):
|
||||
"""In-memory resolver used for ABC tests."""
|
||||
|
||||
name = "fake"
|
||||
|
||||
def __init__(self, skills: List[ResolvedSkill]) -> None:
|
||||
self._skills = skills
|
||||
|
||||
def cache_dir(self) -> Path:
|
||||
return Path("/tmp/fake-cache")
|
||||
|
||||
def sync(self) -> None:
|
||||
pass
|
||||
|
||||
def list_skills(self) -> List[ResolvedSkill]:
|
||||
return list(self._skills)
|
||||
|
||||
|
||||
class TestResolvedSkill:
|
||||
def test_create(self):
|
||||
skill = ResolvedSkill(
|
||||
name="my-skill",
|
||||
source="hermes",
|
||||
path=Path("/tmp/x"),
|
||||
category="research",
|
||||
description="Does research",
|
||||
commit="abc123",
|
||||
)
|
||||
assert skill.name == "my-skill"
|
||||
assert skill.source == "hermes"
|
||||
|
||||
def test_default_sidecar_data(self):
|
||||
skill = ResolvedSkill(
|
||||
name="x",
|
||||
source="x",
|
||||
path=Path("/tmp"),
|
||||
category="",
|
||||
description="",
|
||||
commit="",
|
||||
)
|
||||
assert skill.sidecar_data == {}
|
||||
|
||||
|
||||
class TestSourceResolverABC:
|
||||
def test_resolve_filters_by_name(self):
|
||||
skills = [
|
||||
ResolvedSkill(
|
||||
name="research-it",
|
||||
source="fake",
|
||||
path=Path("/tmp/r"),
|
||||
category="research",
|
||||
description="x",
|
||||
commit="a",
|
||||
),
|
||||
ResolvedSkill(
|
||||
name="code-it",
|
||||
source="fake",
|
||||
path=Path("/tmp/c"),
|
||||
category="coding",
|
||||
description="x",
|
||||
commit="a",
|
||||
),
|
||||
]
|
||||
resolver = _FakeResolver(skills)
|
||||
results = resolver.resolve("research-it")
|
||||
assert len(results) == 1
|
||||
assert results[0].name == "research-it"
|
||||
|
||||
def test_resolve_filters_by_partial_name(self):
|
||||
skills = [
|
||||
ResolvedSkill(
|
||||
name="research-it",
|
||||
source="fake",
|
||||
path=Path("/tmp/r"),
|
||||
category="research",
|
||||
description="x",
|
||||
commit="a",
|
||||
),
|
||||
ResolvedSkill(
|
||||
name="code-it",
|
||||
source="fake",
|
||||
path=Path("/tmp/c"),
|
||||
category="coding",
|
||||
description="x",
|
||||
commit="a",
|
||||
),
|
||||
]
|
||||
resolver = _FakeResolver(skills)
|
||||
# 'res' should match 'research-it'
|
||||
results = resolver.resolve("res")
|
||||
assert len(results) == 1
|
||||
|
||||
def test_resolve_empty_query_returns_all(self):
|
||||
skills = [
|
||||
ResolvedSkill(
|
||||
name="a",
|
||||
source="fake",
|
||||
path=Path("/tmp/a"),
|
||||
category="x",
|
||||
description="x",
|
||||
commit="a",
|
||||
),
|
||||
ResolvedSkill(
|
||||
name="b",
|
||||
source="fake",
|
||||
path=Path("/tmp/b"),
|
||||
category="x",
|
||||
description="x",
|
||||
commit="a",
|
||||
),
|
||||
]
|
||||
resolver = _FakeResolver(skills)
|
||||
assert len(resolver.resolve("")) == 2
|
||||
|
||||
def test_filter_by_category(self):
|
||||
skills = [
|
||||
ResolvedSkill(
|
||||
name="a",
|
||||
source="fake",
|
||||
path=Path("/tmp/a"),
|
||||
category="research",
|
||||
description="x",
|
||||
commit="a",
|
||||
),
|
||||
ResolvedSkill(
|
||||
name="b",
|
||||
source="fake",
|
||||
path=Path("/tmp/b"),
|
||||
category="coding",
|
||||
description="x",
|
||||
commit="a",
|
||||
),
|
||||
]
|
||||
resolver = _FakeResolver(skills)
|
||||
results = resolver.filter_by_category("research")
|
||||
assert len(results) == 1
|
||||
assert results[0].category == "research"
|
||||
|
||||
|
||||
class TestHermesResolver:
|
||||
def test_lists_skills_in_two_level_layout(self, tmp_path: Path):
|
||||
from openjarvis.skills.sources.hermes import HermesResolver
|
||||
|
||||
# Build a fake Hermes layout: skills/<category>/<skill>/SKILL.md
|
||||
skills_root = tmp_path / "skills"
|
||||
(skills_root / "apple" / "apple-notes").mkdir(parents=True)
|
||||
(skills_root / "apple" / "apple-notes" / "SKILL.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: apple-notes
|
||||
description: Manage Apple Notes
|
||||
---
|
||||
Body
|
||||
""")
|
||||
)
|
||||
(skills_root / "github" / "github-pr").mkdir(parents=True)
|
||||
(skills_root / "github" / "github-pr" / "SKILL.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: github-pr
|
||||
description: Manage GitHub PRs
|
||||
---
|
||||
Body
|
||||
""")
|
||||
)
|
||||
|
||||
resolver = HermesResolver(cache_root=tmp_path)
|
||||
skills = resolver.list_skills()
|
||||
names = sorted(s.name for s in skills)
|
||||
assert names == ["apple-notes", "github-pr"]
|
||||
# Categories come from the parent directory
|
||||
for s in skills:
|
||||
if s.name == "apple-notes":
|
||||
assert s.category == "apple"
|
||||
elif s.name == "github-pr":
|
||||
assert s.category == "github"
|
||||
|
||||
def test_skips_description_md(self, tmp_path: Path):
|
||||
from openjarvis.skills.sources.hermes import HermesResolver
|
||||
|
||||
skills_root = tmp_path / "skills"
|
||||
category_dir = skills_root / "apple"
|
||||
category_dir.mkdir(parents=True)
|
||||
# DESCRIPTION.md at category level — should be skipped
|
||||
(category_dir / "DESCRIPTION.md").write_text("# Apple skills")
|
||||
(category_dir / "apple-notes").mkdir()
|
||||
(category_dir / "apple-notes" / "SKILL.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: apple-notes
|
||||
description: x
|
||||
---
|
||||
""")
|
||||
)
|
||||
|
||||
resolver = HermesResolver(cache_root=tmp_path)
|
||||
skills = resolver.list_skills()
|
||||
assert len(skills) == 1
|
||||
assert skills[0].name == "apple-notes"
|
||||
|
||||
def test_filter_by_category(self, tmp_path: Path):
|
||||
from openjarvis.skills.sources.hermes import HermesResolver
|
||||
|
||||
skills_root = tmp_path / "skills"
|
||||
for cat in ("apple", "github"):
|
||||
d = skills_root / cat / f"{cat}-skill"
|
||||
d.mkdir(parents=True)
|
||||
(d / "SKILL.md").write_text(
|
||||
textwrap.dedent(f"""\
|
||||
---
|
||||
name: {cat}-skill
|
||||
description: x
|
||||
---
|
||||
""")
|
||||
)
|
||||
|
||||
resolver = HermesResolver(cache_root=tmp_path)
|
||||
results = resolver.filter_by_category("apple")
|
||||
assert len(results) == 1
|
||||
assert results[0].category == "apple"
|
||||
|
||||
|
||||
class TestOpenClawResolver:
|
||||
def test_lists_skills_in_owner_layout(self, tmp_path: Path):
|
||||
from openjarvis.skills.sources.openclaw import OpenClawResolver
|
||||
|
||||
skills_root = tmp_path / "skills"
|
||||
# OpenClaw layout: skills/<owner>/<skill>/SKILL.md
|
||||
d = skills_root / "alice" / "etherscan"
|
||||
d.mkdir(parents=True)
|
||||
(d / "SKILL.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: etherscan
|
||||
description: Query EVM chain data
|
||||
---
|
||||
Body
|
||||
""")
|
||||
)
|
||||
|
||||
resolver = OpenClawResolver(cache_root=tmp_path)
|
||||
skills = resolver.list_skills()
|
||||
assert len(skills) == 1
|
||||
assert skills[0].name == "etherscan"
|
||||
assert skills[0].category == "alice"
|
||||
|
||||
def test_reads_meta_json_sidecar(self, tmp_path: Path):
|
||||
import json
|
||||
|
||||
from openjarvis.skills.sources.openclaw import OpenClawResolver
|
||||
|
||||
skills_root = tmp_path / "skills"
|
||||
d = skills_root / "alice" / "etherscan"
|
||||
d.mkdir(parents=True)
|
||||
(d / "SKILL.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: etherscan
|
||||
description: x
|
||||
---
|
||||
""")
|
||||
)
|
||||
(d / "_meta.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"owner": "alice",
|
||||
"slug": "etherscan",
|
||||
"latest": {"version": "1.0.0", "commit": "abc123"},
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
resolver = OpenClawResolver(cache_root=tmp_path)
|
||||
skills = resolver.list_skills()
|
||||
assert len(skills) == 1
|
||||
assert skills[0].sidecar_data["owner"] == "alice"
|
||||
assert skills[0].sidecar_data["latest"]["version"] == "1.0.0"
|
||||
|
||||
|
||||
class TestGitHubResolver:
|
||||
def test_recursive_walk_finds_skill_md(self, tmp_path: Path):
|
||||
from openjarvis.skills.sources.github import GitHubResolver
|
||||
|
||||
# Arbitrary nested layout
|
||||
d = tmp_path / "any" / "depth" / "my-skill"
|
||||
d.mkdir(parents=True)
|
||||
(d / "SKILL.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: my-skill
|
||||
description: x
|
||||
---
|
||||
Body
|
||||
""")
|
||||
)
|
||||
|
||||
resolver = GitHubResolver(cache_root=tmp_path, repo_url="https://example.com/x")
|
||||
skills = resolver.list_skills()
|
||||
assert len(skills) == 1
|
||||
assert skills[0].name == "my-skill"
|
||||
|
||||
def test_finds_lowercase_skill_md(self, tmp_path: Path):
|
||||
from openjarvis.skills.sources.github import GitHubResolver
|
||||
|
||||
d = tmp_path / "my-skill"
|
||||
d.mkdir()
|
||||
(d / "skill.md").write_text(
|
||||
textwrap.dedent("""\
|
||||
---
|
||||
name: my-skill
|
||||
description: x
|
||||
---
|
||||
""")
|
||||
)
|
||||
|
||||
resolver = GitHubResolver(cache_root=tmp_path, repo_url="https://example.com/x")
|
||||
skills = resolver.list_skills()
|
||||
assert len(skills) == 1
|
||||
@@ -0,0 +1,202 @@
|
||||
"""Tests for SkillTool v2 — parameter extraction and markdown support."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from openjarvis.core.types import ToolResult
|
||||
from openjarvis.skills.executor import SkillExecutor
|
||||
from openjarvis.skills.tool_adapter import SkillTool
|
||||
from openjarvis.skills.types import SkillManifest, SkillStep
|
||||
from openjarvis.tools._stubs import BaseTool, ToolExecutor, ToolSpec
|
||||
|
||||
|
||||
class EchoTool(BaseTool):
|
||||
tool_id = "echo"
|
||||
|
||||
@property
|
||||
def spec(self):
|
||||
return ToolSpec(name="echo", description="Echo input")
|
||||
|
||||
def execute(self, **params):
|
||||
return ToolResult(
|
||||
tool_name="echo", content=params.get("text", ""), success=True
|
||||
)
|
||||
|
||||
|
||||
def _make_executor(*extra_tools):
|
||||
tools = [EchoTool(), *extra_tools]
|
||||
tool_executor = ToolExecutor(tools)
|
||||
return SkillExecutor(tool_executor)
|
||||
|
||||
|
||||
class TestParameterExtraction:
|
||||
def test_pipeline_params_extracted(self):
|
||||
"""Placeholders in arguments_template become input params."""
|
||||
manifest = SkillManifest(
|
||||
name="pipe_skill",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "{query}"}',
|
||||
output_key="echoed",
|
||||
)
|
||||
],
|
||||
)
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
props = skill_tool.spec.parameters.get("properties", {})
|
||||
assert "query" in props
|
||||
|
||||
def test_output_keys_not_exposed_as_params(self):
|
||||
"""output_key values from prior steps are not exposed as input params."""
|
||||
manifest = SkillManifest(
|
||||
name="chain_skill",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "hello"}',
|
||||
output_key="echoed",
|
||||
),
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "{echoed}"}',
|
||||
output_key="result",
|
||||
),
|
||||
],
|
||||
)
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
props = skill_tool.spec.parameters.get("properties", {})
|
||||
# "echoed" is produced by step 1, so it should NOT be an input param
|
||||
assert "echoed" not in props
|
||||
|
||||
def test_instruction_only_skill_gets_task_param(self):
|
||||
"""Skills with no steps (markdown-only) expose an optional 'task' param."""
|
||||
manifest = SkillManifest(
|
||||
name="md_only",
|
||||
markdown_content="When asked to research, follow these steps...",
|
||||
steps=[],
|
||||
)
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
props = skill_tool.spec.parameters.get("properties", {})
|
||||
assert "task" in props
|
||||
|
||||
def test_no_extra_params_for_pipeline_skill(self):
|
||||
"""Pipeline skills don't get a spurious 'task' param."""
|
||||
manifest = SkillManifest(
|
||||
name="echo_skill",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "{msg}"}',
|
||||
output_key="out",
|
||||
)
|
||||
],
|
||||
)
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
props = skill_tool.spec.parameters.get("properties", {})
|
||||
assert "msg" in props
|
||||
assert "task" not in props
|
||||
|
||||
def test_multiple_placeholders_across_steps(self):
|
||||
"""Multiple distinct placeholders across steps are all exposed."""
|
||||
manifest = SkillManifest(
|
||||
name="multi_param",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "{first}"}',
|
||||
output_key="out1",
|
||||
),
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "{second}"}',
|
||||
output_key="out2",
|
||||
),
|
||||
],
|
||||
)
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
props = skill_tool.spec.parameters.get("properties", {})
|
||||
assert "first" in props
|
||||
assert "second" in props
|
||||
|
||||
|
||||
class TestMarkdownReturn:
|
||||
def test_instruction_only_returns_markdown(self):
|
||||
"""Instruction-only skill (no steps) returns markdown_content as result."""
|
||||
manifest = SkillManifest(
|
||||
name="md_skill",
|
||||
markdown_content="## Instructions\nDo the thing.",
|
||||
steps=[],
|
||||
)
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
result = skill_tool.execute(task="some task")
|
||||
assert result.success
|
||||
assert "## Instructions" in result.content
|
||||
|
||||
def test_instruction_only_no_task_param_still_works(self):
|
||||
"""Instruction-only skill works even without task param."""
|
||||
manifest = SkillManifest(
|
||||
name="md_skill2",
|
||||
markdown_content="Follow these instructions.",
|
||||
steps=[],
|
||||
)
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
result = skill_tool.execute()
|
||||
assert result.success
|
||||
assert "Follow these instructions." in result.content
|
||||
|
||||
def test_hybrid_returns_pipeline_result_and_markdown(self):
|
||||
"""Hybrid skill (steps + markdown) returns both pipeline output and markdown."""
|
||||
manifest = SkillManifest(
|
||||
name="hybrid_skill",
|
||||
markdown_content="## Guidance\nExtra context.",
|
||||
steps=[
|
||||
SkillStep(
|
||||
tool_name="echo",
|
||||
arguments_template='{"text": "{input}"}',
|
||||
output_key="result",
|
||||
)
|
||||
],
|
||||
)
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
result = skill_tool.execute(input="hello world")
|
||||
assert result.success
|
||||
# Pipeline output is present
|
||||
assert "hello world" in result.content
|
||||
# Markdown is appended
|
||||
assert "## Guidance" in result.content
|
||||
|
||||
def test_pipeline_failure_is_propagated(self):
|
||||
"""If the pipeline fails, the ToolResult is not successful."""
|
||||
manifest = SkillManifest(
|
||||
name="broken_skill",
|
||||
steps=[
|
||||
SkillStep(tool_name="nonexistent_tool", output_key="x"),
|
||||
],
|
||||
)
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
result = skill_tool.execute()
|
||||
assert not result.success
|
||||
|
||||
def test_tool_id_uses_skill_prefix(self):
|
||||
"""tool_id is prefixed with 'skill_'."""
|
||||
manifest = SkillManifest(name="my_skill", steps=[])
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
assert skill_tool.tool_id == "skill_my_skill"
|
||||
|
||||
def test_spec_name_uses_skill_prefix(self):
|
||||
"""spec.name is prefixed with 'skill_'."""
|
||||
manifest = SkillManifest(name="another_skill", steps=[])
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
assert skill_tool.spec.name == "skill_another_skill"
|
||||
|
||||
def test_spec_category_is_skill(self):
|
||||
"""spec.category is always 'skill'."""
|
||||
manifest = SkillManifest(name="cat_skill", steps=[])
|
||||
skill_tool = SkillTool(manifest, _make_executor())
|
||||
assert skill_tool.spec.category == "skill"
|
||||
|
||||
def test_skill_manager_kwarg_accepted(self):
|
||||
"""Constructor accepts optional skill_manager keyword argument."""
|
||||
manifest = SkillManifest(name="mgr_skill", steps=[])
|
||||
# Should not raise
|
||||
skill_tool = SkillTool(manifest, _make_executor(), skill_manager=None)
|
||||
assert skill_tool.tool_id == "skill_mgr_skill"
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Tests for ToolTranslator — translate external tool names to OpenJarvis."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from openjarvis.skills.tool_translator import TOOL_TRANSLATION, ToolTranslator
|
||||
|
||||
|
||||
class TestTranslationTable:
|
||||
def test_bash_translates_to_shell_exec(self):
|
||||
assert TOOL_TRANSLATION["Bash"] == "shell_exec"
|
||||
|
||||
def test_read_translates_to_file_read(self):
|
||||
assert TOOL_TRANSLATION["Read"] == "file_read"
|
||||
|
||||
def test_write_translates_to_file_write(self):
|
||||
assert TOOL_TRANSLATION["Write"] == "file_write"
|
||||
|
||||
def test_websearch_translates(self):
|
||||
assert TOOL_TRANSLATION["WebSearch"] == "web_search"
|
||||
|
||||
|
||||
class TestMarkdownTranslation:
|
||||
def test_translate_inline_tool_reference(self):
|
||||
translator = ToolTranslator()
|
||||
body = "Use the Bash tool to run commands."
|
||||
new_body, untranslated = translator.translate_markdown(body)
|
||||
assert "shell_exec" in new_body
|
||||
assert "Bash" not in new_body
|
||||
assert untranslated == []
|
||||
|
||||
def test_translate_multiple_tools(self):
|
||||
translator = ToolTranslator()
|
||||
body = "First use Read to load the file, then Write to save it."
|
||||
new_body, _ = translator.translate_markdown(body)
|
||||
assert "file_read" in new_body
|
||||
assert "file_write" in new_body
|
||||
|
||||
def test_unknown_tool_collected(self):
|
||||
translator = ToolTranslator()
|
||||
body = "Use the QuantumComputer tool."
|
||||
new_body, untranslated = translator.translate_markdown(body)
|
||||
assert "QuantumComputer" in untranslated
|
||||
assert "QuantumComputer" in new_body # left in place
|
||||
|
||||
def test_word_boundary_respected(self):
|
||||
"""'Read' should not match 'Reader' or 'Reading'."""
|
||||
translator = ToolTranslator()
|
||||
body = "The Reader processes Reading material."
|
||||
new_body, _ = translator.translate_markdown(body)
|
||||
assert "Reader" in new_body
|
||||
assert "Reading" in new_body
|
||||
assert "file_read" not in new_body
|
||||
|
||||
def test_empty_body(self):
|
||||
translator = ToolTranslator()
|
||||
new_body, untranslated = translator.translate_markdown("")
|
||||
assert new_body == ""
|
||||
assert untranslated == []
|
||||
|
||||
|
||||
class TestAllowedToolsTranslation:
|
||||
def test_translate_space_delimited(self):
|
||||
translator = ToolTranslator()
|
||||
new_field, untranslated = translator.translate_allowed_tools("Bash Read Write")
|
||||
assert "shell_exec" in new_field
|
||||
assert "file_read" in new_field
|
||||
assert "file_write" in new_field
|
||||
assert untranslated == []
|
||||
|
||||
def test_translate_with_arguments(self):
|
||||
translator = ToolTranslator()
|
||||
# Spec example: Bash(git:*) Bash(jq:*) Read
|
||||
new_field, _ = translator.translate_allowed_tools("Bash(git:*) Bash(jq:*) Read")
|
||||
assert "shell_exec(git:*)" in new_field
|
||||
assert "shell_exec(jq:*)" in new_field
|
||||
assert "file_read" in new_field
|
||||
|
||||
def test_unknown_in_allowed_tools_collected(self):
|
||||
translator = ToolTranslator()
|
||||
new_field, untranslated = translator.translate_allowed_tools("Bash UnknownTool")
|
||||
assert "UnknownTool" in untranslated
|
||||
Reference in New Issue
Block a user