OpenJarvis/CHANGELOG.md

Changelog

All notable changes to OpenJarvis are documented in this file.

The format is based on Keep a Changelog.

---

[Unreleased]

Added

• AI stack support for evaluating other agentic frameworks via subprocess. New evals/backends/external/ subpackage wraps Hermes Agent and OpenClaw as one-shot subprocess backends behind the existing InferenceBackend ABC; new evals/comparison/ toolkit provides path + commit-pin enforcement (third_party.py), config templating (make_configs.py), and LaTeX table generation (table_gen.py).
• New optional extra framework-comparison (depends on polars).
• New pytest marker live_external for integration tests requiring real foreign-framework installations.

Changed

• JarvisAgentBackend.generate_full and JarvisDirectBackend.generate_full now return the spec §6.2 extended fields (energy_joules, peak_power_w, tool_calls, turn_count, framework, framework_commit, error) for cross-framework comparison parity. Existing callers that didn't read these fields are unaffected.
• _third_party.toml no longer ships user-specific default paths. Set HERMES_AGENT_PATH and OPENCLAW_PATH env vars to point at your local checkouts before running the framework-comparison harness; missing or empty paths now raise ThirdPartyNotFoundError with an actionable hint.

Skills System (Plans 1, 2A, 2B)

• Skills core — every skill is a tool. Skills appear in a system prompt catalog, agents invoke them on demand, content (pipeline results, markdown instructions, or both) gets injected into context.

  • SkillManifest + SkillStep types with tags, depends, invocation flags, markdown content
  • SkillManager — discovery, precedence resolution, catalog XML generation, tool wrapping
  • SkillTool(BaseTool) — auto-extracts parameters from step argument templates
  • SkillExecutor — sequential pipeline execution with sub-skill delegation
  • Dependency graph with cycle detection, max depth enforcement, capability unions
  • Security: four trust tiers (bundled/indexed/unreviewed/workspace), capability-gated enforcement
  • Skill index module for git-backed registry search

• agentskills.io spec adoption — canonical SKILL.md format with YAML frontmatter following the agentskills.io open standard.

  • SkillParser with strict spec validation + tolerant field mapping via FIELD_MAPPING table
  • ToolTranslator for external tool name translation (Bash -> shell_exec, Read -> file_read, etc.)
  • Source resolvers: HermesResolver, OpenClawResolver, GitHubResolver
  • SkillImporter with provenance tracking (.source metadata files), optional script import
  • Sourced subdirectory layout (~/.openjarvis/skills/<source>/<name>/)

• Skills learning loop — trace tagging, pattern discovery, DSPy/GEPA optimization.

  • Trace metadata tagging: skill, skill_source, skill_kind flow through ToolExecutor -> TraceCollector -> TraceStep
  • SkillDiscovery wired into SkillManager.discover_from_traces() with kebab name normalization
  • SkillOptimizer — per-skill DSPy/GEPA wrapper that buckets traces and writes sidecar overlays
  • SkillOverlay — sidecar storage at ~/.openjarvis/learning/skills/<name>/optimized.toml
  • SkillManager._load_overlays() applies optimized descriptions + few-shot examples at discovery time
  • LearningOrchestrator._maybe_optimize_skills() — opt-in auto-trigger

• Skills benchmark harness — 4-condition PinchBench evaluation.

  • I3 fix: skill_few_shot_examples wired through SystemBuilder -> _run_agent -> ToolUsingAgent -> native_react.REACT_SYSTEM_PROMPT
  • SkillBenchmarkRunner — 4-condition x N-seed x M-task sweep with markdown report
  • JarvisAgentBackend accepts skills_enabled and overlay_dir kwargs
  • Conditions: no_skills, skills_on, skills_optimized_dspy, skills_optimized_gepa

• CLI commands:

  • jarvis skill list / info / run / install / sync / sources / update / remove / search
  • jarvis skill discover — mine traces for recurring tool patterns
  • jarvis skill show-overlay — inspect optimization output
  • jarvis optimize skills — run DSPy/GEPA per-skill optimization
  • jarvis bench skills — run the PinchBench skills benchmark

• Agent prompt improvement:

  • native_react.REACT_SYSTEM_PROMPT now includes "Using Skills" guidance that teaches agents to distinguish executable vs. instructional skill responses
  • {skill_examples} placeholder for optimized few-shot example injection

• Configuration:

  • [skills] section: enabled, skills_dir, active, auto_discover, auto_sync, max_depth, sandbox_dangerous
  • [[skills.sources]] section: source, url, filter, auto_update
  • [learning.skills] section: auto_optimize, optimizer, min_traces_per_skill, optimization_interval_seconds, overlay_dir
  • SkillSourceConfig and SkillsLearningConfig dataclasses

• Documentation:

  • docs/user-guide/skills.md — comprehensive user guide
  • docs/architecture/skills.md — technical deep-dive
  • docs/tutorials/skills-workflow.md — end-to-end tutorial
  • docs/getting-started/configuration.md — expanded with skills config sections
  • CLAUDE.md — updated architecture section

Fixed

• Trace metadata flow — ToolResult.metadata now propagates through TOOL_CALL_END event to TraceStep.metadata (was silently dropped at the event-bus boundary)
• TaintSet JSON serialization — ToolExecutor._json_safe_metadata() filters non-JSON-serializable values (like TaintSet) from event payloads before they reach TraceStore
• Non-dict YAML frontmatter — source resolvers handle yaml.safe_load() returning a string instead of a dict (discovered on real OpenClaw imports)
• OpenClaw category/name queries — jarvis skill install openclaw:owner/slug now correctly splits into category + name match
• SkillDiscovery trace compatibility — _extract_tool_sequence reads from step.input["tool"] (the actual TraceStep format), not the nonexistent step.tool_name attribute
• LearningOrchestrator skill trigger — _maybe_optimize_skills runs BEFORE the SFT-data short-circuit (skills are tagged via trace metadata, not mined as SFT pairs)
• PinchBenchScorer constructor — SkillBenchmarkRunner constructs PinchBenchScorer(judge_backend, model) instead of no-args
• EvalRunner results access — reads per-task data from eval_runner.results property, not nonexistent summary.results