Tarun SureshandClaude Opus 4.6
bf24ffc527
feat: joint multi-benchmark optimization with LLM-guided search
Add Claude Opus-guided optimization loop that jointly optimizes agent
configs across multiple benchmarks (TerminalBench-native, GAIA, HLE)
with weighted accuracy aggregation and Pareto frontier computation.
Key additions:
- MultiBenchTrialRunner with native terminal-bench v2 Docker execution
- LLM optimizer: fixed params injection, structured trial feedback
- OptimizationStore (SQLite) with per-benchmark score persistence
- System prompt passthrough from optimizer → eval → agent
- Custom terminal-bench agent (OpenJarvisTerminalBenchAgent) avoiding
LiteLLM serialization issues with agent_import_path
- TOML configs for Qwen3-235B joint agentic optimization
- CLI: `jarvis optimize` command with dry-run support
Early results on Qwen3-235B-A22B-Instruct-2507-FP8 (4xA100):
Best config: native_openhands, temp=0.0, 20 turns → 5.6% weighted acc
(TB2=5%, GAIA=6%, HLE=6%)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 18:44:24 +00:00
..
2026-02-27 19:14:05 +00:00
2026-03-05 05:38:27 +00:00
2026-02-28 17:37:02 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-02-26 20:09:07 +00:00
2026-02-24 03:34:05 +00:00
2026-03-05 05:47:29 +00:00
2026-02-22 05:57:13 +00:00
2026-03-02 22:33:57 +00:00
2026-03-05 05:29:42 +00:00
2026-03-04 19:35:33 -08:00
2026-02-27 19:14:05 +00:00
2026-03-05 05:29:42 +00:00
2026-02-17 00:52:48 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-02-27 19:14:05 +00:00
2026-03-05 05:29:42 +00:00
2026-03-03 19:08:15 +00:00
2026-03-05 05:59:32 +00:00
2026-03-05 05:29:42 +00:00
2026-02-23 18:32:32 +00:00
2026-03-02 22:33:57 +00:00
2026-03-05 05:29:42 +00:00
2026-02-27 19:14:05 +00:00
2026-02-17 00:52:48 +00:00
2026-03-03 04:33:15 +00:00
2026-03-02 23:10:19 +00:00
2026-03-04 23:49:23 +00:00
2026-03-02 22:33:57 +00:00
2026-02-24 03:34:05 +00:00
2026-03-05 05:29:42 +00:00
2026-03-06 18:44:24 +00:00
2026-03-05 00:01:52 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:47:29 +00:00
2026-03-04 23:43:54 +00:00
2026-03-02 05:34:46 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 05:29:42 +00:00
2026-03-05 01:04:30 +00:00