* fix(evals): PinchBench harness fixes — scores from 26% to 84%
Multiple infrastructure bugs prevented PinchBench from producing
accurate scores. This commit fixes the eval harness so model scores
match the official leaderboard (Qwen3.5-397B: 26% → 84%, GPT-5.4:
5% → 58%).
Key fixes:
- EvalRunner: wrap generation in PinchBenchTaskEnv context so workspace
files persist through grading (was deleting before scorer ran)
- EvalRunner: detect task_env datasets and force sequential processing
(CWD changes aren't thread-safe)
- EvalRunner: fix episode_mode auto-detection to check for real
iter_episodes() override instead of hasattr() (always True)
- Scorer: use "params" field in transcripts to match PinchBench grade()
functions (was "arguments")
- Scorer: add None guard in _trace_to_transcript for tool_calls
- Scorer: capture final assistant text response in transcript so
text-only tasks (like sanity check) can be graded
- Scorer: add _tool_results_to_transcript() helper for EvalRunner path
- native_openhands: add native function-calling support (tools=
parameter) with text-based fallback, matching monitor_operative
- Tools: add Python fallbacks for file_read, file_write, shell_exec,
calculator, think, http_request when openjarvis_rust unavailable
- Security: add Python fallback for is_sensitive_file()
- Config: add "pinchbench" to KNOWN_BENCHMARKS
- Add PinchBench eval configs for Qwen3.5-397B and GPT-5.4
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): fix hybrid grading crash when grading_weights is None
The 4 tasks with grading_type=hybrid (task_10, task_13, task_16_market,
task_22) crashed because grading_weights was explicitly None in task
metadata. dict.get("key", default) returns None (not the default) when
the key exists with value None. Use `or` to coalesce None to defaults.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): handle MagicMock datasets in runner type checks
The iter_episodes and create_task_env type identity checks fail with
AttributeError when the dataset is a MagicMock (used in tracker tests).
Wrap in try/except to default to False for non-DatasetProvider objects.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a `tool_calls` list field to `TurnTrace` capturing name, arguments,
and result for each tool invocation, enabling PinchBench transcript integration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The FLOPs formula changed from quadratic (P*N*(N+1)) to linear
(2*P*T_evaluated) with KV-cache awareness. Update the test
expectation from ~100x to ~10x for 10x token increase.
The energy_wh_saved and flops_saved values were orders of magnitude too
high because a scaling factor that grows linearly with N was applied to
the energy calculation, making it scale as O(N³) instead of O(N²).
Replace the buggy scale-factor approach with a direct FLOP-to-energy
conversion using each provider's per-token constants. Add regression
tests to prevent recurrence.
Closes#95
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
