* fix(evals): add tool_choice=auto + fix traces thread safety
Two fixes for eval accuracy and stability:
1. TauBench agent: add tool_choice="auto" to match tau2's native
LLMAgent behavior. Without this, Qwen and GPT-5.4 score 10-14pp
below leaderboard because the models don't receive explicit
tool-calling guidance.
2. SystemBuilder: apply self._traces flag to config.traces.enabled.
Previously builder.traces(False) was a no-op — traces stayed
enabled, creating SQLite connections in the main thread that
crashed when accessed from ThreadPoolExecutor worker threads
in GAIA evals ("SQLite objects created in a thread can only be
used in that same thread").
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enrich inference events with model response content and add Trace.messages
Add content, tool_calls, and finish_reason fields to INFERENCE_END events
published by InstrumentedEngine. For non-instrumented engines, BaseAgent._generate()
now publishes INFERENCE_START/END events with the same rich data. Add _message_to_dict
helper and messages field to Trace dataclass for full conversation capture.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: enhance TraceCollector with rich content, tool details, and messages
Capture model response content, tool_calls, and finish_reason in GENERATE
steps; store tool arguments and result text in TOOL_CALL steps; extract
conversation messages from AgentResult.metadata into Trace.messages; and
implement the last_trace property. Also adds a messages column to the
TraceStore schema so messages survive the SQLite round-trip.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: agents store conversation messages in AgentResult.metadata
Both NativeReActAgent and MonitorOperativeAgent now serialize their
internal messages list via _message_to_dict and include it in the
returned AgentResult.metadata under the "messages" key. This enables
TraceCollector to capture full conversation traces.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: wire TraceCollector into system._run_agent and JarvisAgentBackend
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: persist rich trace data in eval trace JSONL files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add TerminalBench config for Qwen 3.5-122B
* fix: add SQLite migration for traces.messages column on existing databases
* fix: check trace_store instead of shared config for trace enablement
* feat(evals): add ToolCall-15, LiveCodeBench, LiveResearchBench + telemetry
Three new benchmark integrations and full telemetry wiring for the
NeurIPS 2026 IPW/IPJ experiments.
## New Benchmarks
### ToolCall-15
15-scenario tool calling accuracy benchmark across 5 categories.
All scenarios defined inline with deterministic scoring (0/1/2 per
scenario). Fast to run (~5min/model) — ideal for optimization loops.
### LiveCodeBench
Competitive programming from LeetCode/AtCoder/CodeForces via
HuggingFace dataset. Sandboxed code execution with per-test
timeouts. Single-turn generation via jarvis-direct backend.
### LiveResearchBench
100 expert-curated deep research tasks. LLM-as-judge scoring
across 4 dimensions (comprehensiveness, insight, instruction
following, readability). Uses web_search tool for live research.
## Telemetry Wiring
- FLOPs estimation: 2 * active_params * total_tokens (MoE-aware)
- Energy/power capture flows from InstrumentedEngine through
backends to EvalResult and RunSummary
- New telemetry_summary section in output JSON with IPW/IPJ
- JarvisDirectBackend now propagates gpu_metrics flag
- TauBench forwards telemetry flags to SystemBuilder
## Experiment Plan
Added docs/experiments/neurips-2026-plan.md tracking the full
experiment matrix: 9 models x 7 benchmarks across NVIDIA, AMD,
and Apple hardware stacks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): PinchBench harness fixes — scores from 26% to 84%
Multiple infrastructure bugs prevented PinchBench from producing
accurate scores. This commit fixes the eval harness so model scores
match the official leaderboard (Qwen3.5-397B: 26% → 84%, GPT-5.4:
5% → 58%).
Key fixes:
- EvalRunner: wrap generation in PinchBenchTaskEnv context so workspace
files persist through grading (was deleting before scorer ran)
- EvalRunner: detect task_env datasets and force sequential processing
(CWD changes aren't thread-safe)
- EvalRunner: fix episode_mode auto-detection to check for real
iter_episodes() override instead of hasattr() (always True)
- Scorer: use "params" field in transcripts to match PinchBench grade()
functions (was "arguments")
- Scorer: add None guard in _trace_to_transcript for tool_calls
- Scorer: capture final assistant text response in transcript so
text-only tasks (like sanity check) can be graded
- Scorer: add _tool_results_to_transcript() helper for EvalRunner path
- native_openhands: add native function-calling support (tools=
parameter) with text-based fallback, matching monitor_operative
- Tools: add Python fallbacks for file_read, file_write, shell_exec,
calculator, think, http_request when openjarvis_rust unavailable
- Security: add Python fallback for is_sensitive_file()
- Config: add "pinchbench" to KNOWN_BENCHMARKS
- Add PinchBench eval configs for Qwen3.5-397B and GPT-5.4
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): fix hybrid grading crash when grading_weights is None
The 4 tasks with grading_type=hybrid (task_10, task_13, task_16_market,
task_22) crashed because grading_weights was explicitly None in task
metadata. dict.get("key", default) returns None (not the default) when
the key exists with value None. Use `or` to coalesce None to defaults.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): handle MagicMock datasets in runner type checks
The iter_episodes and create_task_env type identity checks fail with
AttributeError when the dataset is a MagicMock (used in tracker tests).
Wrap in try/except to default to False for non-DatasetProvider objects.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a `tool_calls` list field to `TurnTrace` capturing name, arguments,
and result for each tool invocation, enabling PinchBench transcript integration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The FLOPs formula changed from quadratic (P*N*(N+1)) to linear
(2*P*T_evaluated) with KV-cache awareness. Update the test
expectation from ~100x to ~10x for 10x token increase.
The energy_wh_saved and flops_saved values were orders of magnitude too
high because a scaling factor that grows linearly with N was applied to
the energy calculation, making it scale as O(N³) instead of O(N²).
Replace the buggy scale-factor approach with a direct FLOP-to-energy
conversion using each provider's per-token constants. Add regression
tests to prevent recurrence.
Closes#95
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
