Two failure classes hit by a downstream team:
1) tmux/task-env death (TerminalBenchTaskEnv.__enter__, terminalbench_env.py):
create_session ran inside the spin_up_terminal generator-CM with no
exception safety — a tmux failure leaked the docker compose project
(down deferred to GC, never if the env was retained) and surfaced as an
opaque mid-run death. Now: exception-safe __enter__ with an idempotent
_teardown(), a tmux/asciinema preflight in the container BEFORE the
agent loop (TaskEnvironmentError naming the task image + remedy), and
fail-that-task-cleanly semantics — the failure is recorded as a harness
error (QueryTrace.error_kind="harness_error"), the container is downed,
and the run continues.
2) OpenHands in-container SETUP hang misattributed as a model result:
harness.run() had no bound (terminal-bench runs installed-agent setup
with max_timeout_sec=inf inside the per-trial agent budget), and the
summary conversion read the nonexistent results.trial_results attr and
hardcoded errors=0, folding zero-model-request trials into resolve-rate
as model misses. Now: global_agent_timeout_sec / global_timeout_multiplier
are threaded config -> backend -> Harness kwargs (default 1800 s bound on
SETUP+RUN; configurable per [run]/[[benchmarks]] TOML), and
summarize_benchmark_results() classifies harness/infra failures out of
the accuracy denominator keyed on token usage (zero/missing tokens +
unresolved = the agent never contacted the model), NOT failure_mode —
terminal-bench 0.2.18 leaves failure_mode UNSET on success AND on
genuine misses, so a failure_mode-based check would misflag every real
model miss. Genuine misses (tokens>0, is_resolved=false) stay in the
denominator.
QueryTrace gains error/error_kind (wired through to_dict/from_dict so the
fields actually reach traces.jsonl; backward-compatible loads), the
AgenticRunner flags zero-model-contact traces and TaskEnvironmentError as
harness errors, and export/summary/console output exclude harness errors
from resolve-rate while reporting them loudly.
terminal-bench stays an undeclared dep on purpose: it requires Python
>=3.12 while this project supports >=3.10,<3.14, so an unmarked extra
would break uv lock. pyproject/uv.lock untouched.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
