mirror of
https://github.com/open-jarvis/OpenJarvis.git
synced 2026-07-28 14:07:55 +00:00
1d24c64f11
* fix(evals): PinchBench harness fixes — scores from 26% to 84% Multiple infrastructure bugs prevented PinchBench from producing accurate scores. This commit fixes the eval harness so model scores match the official leaderboard (Qwen3.5-397B: 26% → 84%, GPT-5.4: 5% → 58%). Key fixes: - EvalRunner: wrap generation in PinchBenchTaskEnv context so workspace files persist through grading (was deleting before scorer ran) - EvalRunner: detect task_env datasets and force sequential processing (CWD changes aren't thread-safe) - EvalRunner: fix episode_mode auto-detection to check for real iter_episodes() override instead of hasattr() (always True) - Scorer: use "params" field in transcripts to match PinchBench grade() functions (was "arguments") - Scorer: add None guard in _trace_to_transcript for tool_calls - Scorer: capture final assistant text response in transcript so text-only tasks (like sanity check) can be graded - Scorer: add _tool_results_to_transcript() helper for EvalRunner path - native_openhands: add native function-calling support (tools= parameter) with text-based fallback, matching monitor_operative - Tools: add Python fallbacks for file_read, file_write, shell_exec, calculator, think, http_request when openjarvis_rust unavailable - Security: add Python fallback for is_sensitive_file() - Config: add "pinchbench" to KNOWN_BENCHMARKS - Add PinchBench eval configs for Qwen3.5-397B and GPT-5.4 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(evals): fix hybrid grading crash when grading_weights is None The 4 tasks with grading_type=hybrid (task_10, task_13, task_16_market, task_22) crashed because grading_weights was explicitly None in task metadata. dict.get("key", default) returns None (not the default) when the key exists with value None. Use `or` to coalesce None to defaults. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(evals): handle MagicMock datasets in runner type checks The iter_episodes and create_task_env type identity checks fail with AttributeError when the dataset is a MagicMock (used in tracker tests). Wrap in try/except to default to False for non-DatasetProvider objects. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>