mirror of
https://github.com/open-jarvis/OpenJarvis.git
synced 2026-07-28 14:07:55 +00:00
450397a2ad
* feat(evals): add TauBench V2 native integration + GAIA eval configs Integrate TauBench V2 (τ²-bench) multi-turn customer service benchmark natively through OpenJarvis's inference engine. The agent's LLM calls go through OpenJarvis while tau2-bench handles the orchestration, user simulation, domain tools, database, and evaluation. ## TauBench Integration Architecture: JarvisHalfDuplexAgent bridges OpenJarvis's engine into tau2's Orchestrator as a drop-in agent replacement. This enables testing how well OpenJarvis's Intelligence + Engine handles multi-turn customer service tasks with real tool calling and database mutations. Key features: - Native OpenJarvis engine for agent LLM calls - tau2's UserSimulator for realistic customer interactions - Domain tools (airline, retail, telecom) with mock databases - Full evaluation: DB state checks, action matching, NL assertions - Test-split filtering for leaderboard-comparable results - Pass^k multi-trial support (default 3 trials per task) - Qwen thinking-mode disabled for clean tool call parsing - Gemini thought_signature handling for multi-turn conversations Files: - datasets/taubench.py: Dataset provider with test-split filtering - execution/taubench_env.py: JarvisHalfDuplexAgent + simulation runner - scorers/taubench.py: Scorer reading tau2 evaluation rewards - CLI registration and KNOWN_BENCHMARKS update ## Results (test split, pass^3, 60 tasks) | Model | TauBench | Leaderboard | |--------------------|----------|-------------| | Claude Opus 4.6 | 86.67% | 84.8% | | Nemotron-3-Super | 86.67% | — | | Qwen3.5-397B | 81.67% | 95.6% | | GPT-5.4 | 81.67% | 91.5% | | Qwen3.5-122B | 80.00% | 93.6% | | Qwen3.5-35B | 77.27% | 89.2% | | Gemini 3.1 Pro | 58.33% | ~87% | ## GAIA Eval Configs Added configs for GPT-5.4, Gemini 3.1 Pro, Nemotron, Qwen 122B, Qwen 35B, and existing GAIA rerun configs for multiple models. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: lint errors in taubench integration Remove unused imports and sort import blocks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: E501 line too long in slack_connector.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove unrelated GAIA configs from PR Keep only TauBench configs that were created and tested in this PR. GAIA configs are pre-existing or belong in a separate PR. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>