Files
OpenJarvis/tests/evals
Jon Saad-FalconandClaude Opus 4.6 2aebcd7d77 feat: Phase 23 — Differentiated functionalities
Trace-driven learning pipeline:
- TrainingDataMiner: extract SFT/routing/agent pairs from traces
- LoRATrainer: fine-tune local models from trace-derived data
- AgentConfigEvolver: rewrite agent configs from trace analysis
- LearningOrchestrator: coordinate mine→train→evolve cycle, wired into SystemBuilder

Eval framework (15 real IPW benchmarks):
- Datasets: SuperGPQA, GPQA, MMLU-Pro, MATH-500, Natural Reasoning, HLE,
  SimpleQA, WildChat, IPW, GAIA, FRAMES, SWE-bench, SWEfficiency,
  TerminalBench, TerminalBench Native
- Scorers: MCQ extraction, LLM-judge, exact match, structural validation
- CLI: jarvis eval list|run|compare|report

Composable abstractions:
- Recipe system: TOML composition of all 5 pillars (3 built-in recipes)
- Agent templates: 15 pre-configured TOML manifests with system prompts
- Bundled skills: 20 ready-to-use TOML skill manifests
- Operator recipes: researcher (4h), correspondent (5min), sentinel (2h)

102 files changed, ~11,500 lines added. 3241 tests pass (44 skipped).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 05:34:46 +00:00
..