Closes#219. Replace synchronous httpx calls in async SendBlue and model-management handlers with awaited httpx.AsyncClient (context-managed close); run Whisper transcription and engine.list_models via asyncio.to_thread so they don't block the event loop; and harden TelemetryStore/aggregator SQLite for concurrency (WAL, synchronous=NORMAL, busy_timeout=5000, plus a write-serializing lock on the shared connection). Adds async-usage assertions and a real 8-thread concurrent-write test. Related: #570 (async httpx, different issue #559).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#388. Parse Google Calendar all-day events from start.date instead of stamping them with the current time; treat generic next/upcoming calendar-event queries as gcalendar timeline requests that return nearest-future events first (UTC-normalized, instant-aware comparison that handles tz offsets and all-day events); and update the research planner guidance to route such queries with sources=[gcalendar] + a today-onward time_range. Real in-memory KnowledgeStore integration tests cover ordering, tz normalization, all-day inclusion, and source narrowing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes#575. Web Deep Research was hardcoded to OllamaEngine + DEFAULT_PLANNER_MODEL, ignoring the user's configured/active engine and model. Resolve the planner from [deep_research] override -> live app chat engine + selected model -> config defaults -> legacy Ollama, pass the chat picker's model from the frontend into /api/research, record the actual planner engine in telemetry, and refuse to silently fall back to a different engine (raise an actionable error instead). Adds config support and focused tests for resolution and the route. Related: #576 (duplicate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes#505. Make the desktop backend resilient to a missing/unbuilt openjarvis_rust extension: declare openjarvis-rust as a uv-managed desktop path dependency so 'uv sync --extra desktop' owns the PyO3 build (instead of pruning an undeclared package); add ~/.cargo/bin to the subprocess PATH and fail early with Rust / Windows Build Tools guidance when the toolchain is missing; verify 'import openjarvis_rust' before starting jarvis serve; and add a TCP bind preflight for port 8000 to catch non-HTTP listeners the /health probe can't classify. Includes the uv.lock entry for the new path dependency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(user-guide): document SOUL/MEMORY/USER.md persona files (#604)
The persistent-memory showcase links to the User Guide: Agents page for how
SOUL.md / MEMORY.md / USER.md are loaded at conversation start, but that page
never covered them (site search for the filenames returns nothing).
Add a "Persistent Persona" section to user-guide/agents.md: the three files and
what each holds, where they live (config dir + [memory_files]), how they load
(after the agent template, cached per conversation, per-section truncation),
named personas (--persona / personas/<name>/), and editing by hand or via the
memory_manage / user_profile_manage tools. Cross-links the distinct retrieval
memory backend to resolve the reporter's confusion.
Closes#604
* docs(user-guide): clarify memory_manage/user_profile_manage target the default MEMORY.md/USER.md
Closes#608. Make Message.content officially Optional (str | None) with a Message.text accessor that treats None as empty, and count tool-call IDs/names/arguments, tool-result IDs, and reasoning/thinking metadata in estimate_prompt_tokens — all are replayed into later prompt turns, so they belong in the estimate. Extends estimator and message-type regression tests.
Closes#607. Assistant tool-call turns can carry content=None, which crashed token estimation (len(m.content)) and think-tag stripping. Normalize with 'content or ""', route native OpenHands truncation through the shared estimate_prompt_tokens, and add tests for the estimator, the truncation helper, and an end-to-end tool-call run with None content.
Addresses #605. Prefer an already-installed Ollama model before attempting a startup download (matching the requested tag, else a preferred non-embedding installed model); fall back through installed -> FALLBACK_MODEL -> error, reusing installed models at each failure point; persist the resolved model only for first-run/default so an explicit user choice is never overwritten. Refactors the model logic into testable helpers with unit coverage.
Desktop release builds failed on all platforms with 'Found version mismatched Tauri packages' because @tauri-apps/api and @tauri-apps/cli were pinned at 2.10.1 while the tauri Rust crate resolved to 2.11.3. Bump both npm packages to the 2.11 line (api 2.11.1, cli 2.11.4) so they share the crate's major.minor. Plugins were already aligned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docs-site leaderboard (docs/javascripts/leaderboard.js) reads the public
Supabase anon key from window.OPENJARVIS_SUPABASE_ANON_KEY, but nothing set it,
so the published leaderboard always rendered "Leaderboard not configured yet".
Add a generated config file (leaderboard-config.js) loaded before
leaderboard.js that supplies the global, and inject its value at docs-build
time from the existing VITE_SUPABASE_ANON_KEY repo secret. The committed
default is empty, so local `mkdocs build` and fork PRs (no secret) degrade
gracefully. The anon key is public by design (Supabase RLS protects the data).
- docs/javascripts/leaderboard-config.js: empty-default global declaration.
- mkdocs.yml: load leaderboard-config.js before leaderboard.js.
- docs.yml: write the config from the secret (read via env, JSON-encoded into a
JS string literal to avoid injection) before `mkdocs build`.
- tests/deployment/test_docs_leaderboard.py: guard the wiring + load order.
Verified with a local `mkdocs build`: the generated config ships in site/ and
loads before leaderboard.js.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#582. Route fact-store construction through a new FactStoreRegistry (local backend registered by default); align the default facts path with get_config_dir(); wire completed chat exchanges (streamed and non-streamed) through the EventBus so the memory service captures them consistently; reload the local fact store from disk before operations so external clears don't resurrect stale facts; make the affected config/persona/memory/CLI/route tests hermetic; and refresh uv.lock with the current resolver (locks pytest-xdist + transitive deps, drops py3.14 artifacts since the project constrains Python <3.14).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build and install the mandatory openjarvis_rust wheel in the CPU, NVIDIA, ROCm, and sandbox Docker images. Rust 1.88 (matching the workspace MSRV / rust-toolchain.toml) and maturin are installed only in the builder stage, the module's import is verified during the build, and maturin is removed before the runtime artifacts are copied so build tooling never ships. The frontend leaderboard anon key is an optional empty-by-default build arg (post-#589), so default images cleanly disable the leaderboard. Adds static deployment coverage for the native build path. Closes#584.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PyPI publishing had been broken since v1.0.3.dev851: #587 made VITE_SUPABASE_ANON_KEY a hard build-time requirement, but no such secret exists, so the frontend build aborted every publish run before the PyPI upload. Decouple package buildability from the leaderboard credential: a missing anon key now disables the savings leaderboard at runtime instead of failing the build, and auto-enables when the secret is provided. Verified: npm run build with the key unset succeeds; tsc + vitest pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to #587. Pass VITE_SUPABASE_ANON_KEY into the frontend builds of both release paths: the PyPI publish workflow (wheel-bundled frontend) and the desktop tauri-action build (npm run build:tauri -> vite build). Kept strict: a missing/empty secret fails the release by design rather than shipping a placeholder key. Requires the VITE_SUPABASE_ANON_KEY repo secret to be set for releases to succeed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Route desktop cloud-key saves/status through the OS credential store (keyring with per-platform native backends: apple-native / windows-native / sync-secret-service), migrate the legacy plaintext ~/.openjarvis/cloud-keys.env into it, remove browser localStorage persistence of provider keys, and push key updates to the running server via /v1/cloud/reload (legacy env-file fallback retained). Remove hardcoded Supabase anon JWTs from frontend/docs source and make VITE_SUPABASE_ANON_KEY a required build var. Adds libdbus-1-dev to the Linux desktop build and a CI build var. Closes#220.
NOTE (post-merge follow-ups, not covered by CI): add the VITE_SUPABASE_ANON_KEY repo secret with the rotated key (release/docs builds otherwise use a placeholder), rotate the previously-committed Supabase anon key, and run a desktop save->restart->read smoke test to confirm keychain persistence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TestTraceRecording relied on the ambient ~/.openjarvis/config.toml leaving traces.enabled at its default, so it failed on any machine with traces disabled locally (passing in CI only because the runner has no config file). Pass an explicit traces-enabled config with a tmp db_path so the tests are environment-independent and parallel-safe under pytest -n auto. Relates to #582.
Pin all base images and ollama to fixed versions + @sha256 digests (no floating :latest), run Docker images as an unprivileged openjarvis user (uid 10001), replace the curl|bash NodeSource install with a digest-pinned multi-stage copy, install from the committed uv.lock via uv export --frozen --no-dev (hash-verified, --no-deps), and add systemd sandboxing (NoNewPrivileges, ProtectSystem=strict, PrivateTmp, kernel/SUID protections). Closes#228, #563, #564, #565, #566, #567.
Adds the openjarvis.memory package (LocalFactStore, FactExtractor, background MemoryService), starts/stops it in the jarvis serve and jarvis chat lifecycle, feeds completed non-streaming exchanges to it, adds [memory] config support, and adds jarvis memory list/clear CLI commands. Extraction runs on a background thread and degrades to a no-op on any failure (BrokenPipe, timeouts, unparseable output) so it can never block a reply or crash the host. Disabled by default. Closes#393, #571, #572, #573.
Run pytest with -n auto (pytest-xdist) and COVERAGE_CORE=sysmon, enable the uv cache, and switch test output to -q. Cuts the test job from ~40min to ~4min without changing what's tested or the 60% coverage gate.
Qwen3 treats /think and /no_think as soft-switch control tokens. On small
models a multi-line prompt makes the model emit one as the sole tool argument
(e.g. {"command": "/no_think"}); OpenJarvis forwards Ollama's native tool_calls
verbatim, so the operative agent executes garbage. Filter control-token-only
tool calls in both the non-streaming generate() and streaming _run_stream()
paths, keeping legitimate calls like {"command": "date"}.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously `core/config.py` defined `DEFAULT_CONFIG_DIR = Path.home() /
".openjarvis"` as a by-value module constant imported into ~45 modules, and 34
modules hardcoded `Path.home() / ".openjarvis"` directly. The installer honored
`OPENJARVIS_HOME` but the Python runtime ignored it, producing a split-brain
layout (some modules honored the override, the core config dir did not). Eval
dataset caches also scattered into `~/.cache/<benchmark>`.
This introduces a single env-aware resolver in `openjarvis/core/paths.py` and
routes every state/config/cache path through it. OpenJarvis now keeps ALL of
its state under ONE root, resolved in priority order:
1. $OPENJARVIS_HOME
2. $XDG_DATA_HOME/openjarvis (single nested dir, when XDG_DATA_HOME is set)
3. ~/.openjarvis (default — unchanged, so existing installs are
untouched and no data migration is required)
Implementation:
- New `core/paths.py`: get_config_dir / get_config_path / get_data_dir /
get_cache_dir, with a source-tree rejection guard (fails loudly per
REVIEW.md if the root resolves inside the repo).
- `core/config.py`: DEFAULT_CONFIG_DIR / DEFAULT_CONFIG_PATH are now resolved
via the env-aware resolver at import (real attributes, so existing
monkeypatch.setattr-based tests keep working). All dataclass field defaults
that pointed at ~/.openjarvis converted to default_factory so they honor the
override at instantiation.
- Routed all 34 hardcoders plus several string-literal escapees the original
audit missed: prompt_loader / description_loader (were OPENJARVIS_HOME-only,
no XDG), swebench_harness cache, tools/{memory,skill,user_profile}_manage
defaults, server trace.db fallbacks, doctor_cmd hints.
- spec_search storage/paths now delegates to the unified resolver (gains XDG);
its ConfigurationError is aliased to the core one.
- Eval dataset caches moved from ~/.cache/<name> to <root>/cache/<name>
(~/.cache/huggingface left alone — it is HF's own cache).
- Docs + installer comment + `jarvis config path` to show resolved dirs.
Read-only macOS connectors and OS service files (LaunchAgents/systemd) are
intentionally left untouched.
Fixes#462
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pasting a Google Client ID / Secret never completed OAuth: Drive (and its
Google siblings) accepted the credentials, showed no error, opened no browser,
and never appeared in Data Sources. Root cause is three coupled defects, all
reproduced at the unit level against main with a FastAPI TestClient (no Google
creds, network-free):
(A/B) POST /connect routed a `client_id:client_secret` pair into the
connector's handle_callback, which spawned a daemon thread that popped a
browser and ran its own localhost:8789 callback server. That thread fails
silently in the bundled desktop context (`except Exception: pass`), so the
connector never gained an access_token; /connect returned status "pending"
and the UI's 20x2s poll timed out with no error.
Fix: in POST /connect, an OAuth `client_id:client_secret` pair now persists
the client credentials to every Google credential file and returns an
`oauth_required` directive pointing at the in-process server flow, instead of
the silent background thread. The Google connectors' handle_callback no longer
spawns the browser thread for the pair case — it only persists the creds; the
server's /oauth/start -> /oauth/callback owns the consent round-trip.
(C) The would-be-correct server flow was itself broken: under
`from __future__ import annotations` plus a `Request` import local to the
router factory, FastAPI could not resolve the stringized `request: Request`
annotation. /oauth/start returned HTTP 422 (request mis-bound as a query
param) and /oauth/callback injected None -> AttributeError on
`request.base_url`. Fix: import `Request` at module scope and make the
callback's `request` a required injected dependency.
A malformed/blank client pair now raises HTTP 400 with the provider setup URL
instead of a perpetual silent "pending" (REVIEW.md silent-failure discipline).
Frontend: DataSourcesPage now opens the server OAuth window when /connect
returns `oauth_required`, then polls until connected; connect errors surface the
backend detail; the Drive setup steps document the "Web application" OAuth
client + server-callback redirect URI the in-process flow requires.
Tests (run on the main venv, hermetic — no ~/.openjarvis pollution):
- test_oauth_flow.py: the three handle_callback tests now assert NO browser is
opened and only client creds are persisted (was: assert background flow ran).
- test_connectors_router_oauth.py (new): reproduces + fixes all three defects via
TestClient with mocked token exchange; parametrized over gdrive/gcalendar/
gcontacts/gmail/google_tasks to prove the shared OAuth path is fixed for every
sibling and that a single consent writes the access_token to all six Google
credential files and flips is_connected() to True.
Full tests/connectors suite: 355 passed.
Relationship to PR #510: #510 rewrites all of these files (account-scoped
retrieval) but still carries all three defects. This fix is intentionally scoped
to the OAuth path and does not modify oauth.py, to minimize collision. A
maintainer can either merge this and rebase #510 on top, or port these changes
into #510. See PR body for details.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* build: adopt hatch-vcs dynamic versioning (#526)
Replace the static `version = "1.0.2"` with `dynamic = ["version"]` and
derive the version from git tags via hatch-vcs, so source and editable
checkouts report their true git describe version (e.g.
1.0.3.dev110+g<sha>) instead of a stale constant.
Config notes:
- Exclude .dev/.rc/desktop-* tags from derivation. setuptools_scm cannot
bump custom .devN tags, so the base is taken from the latest plain
release tag (vX.Y.Z) and the dev distance from commit count.
- Add fallback_version so builds without a git checkout (shallow CI
clones, Docker COPY src/, source-zip installs) resolve to a sentinel
instead of hard-failing. CI release builds inject the exact version via
SETUPTOOLS_SCM_PRETEND_VERSION.
* ci(autotag): derive dev base from the latest release tag (#526)
pyproject no longer carries a static version, so read the base from the
latest plain release tag (vX.Y.Z) reachable from HEAD instead of grepping
pyproject. .dev/.rc/desktop-* tags are excluded so they cannot be mistaken
for the release base. The computed tag (vX.Y.Z.devN) is unchanged.
* ci(pypi-publish): pin build version from tag, drop sed injection (#526)
With dynamic versioning there is no static line to sed. Pin the exact
build version from the pushed tag via SETUPTOOLS_SCM_PRETEND_VERSION so
the published version equals the tag. This is required, not cosmetic: a
naive hatch-vcs build emits 1.0.3.devN+g<sha>, and PyPI rejects local
version segments on upload.
Also add a dry_run input that targets TestPyPI instead of PyPI, for
validating the release path without a production upload.
* ci(desktop): derive dispatch-fallback version from release tag (#526)
The workflow_dispatch fallback grepped the now-removed static pyproject
version. Derive its base from the latest release tag instead (matching
autotag), and give the build-and-release checkout full history and tags
so the derivation works.
---------
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
The Claude automation workflows (claude-issues.yml, claude-review.yml)
trigger on public, attacker-controllable events (issues, issue_comment,
pull_request_review_comment) and grant the job secrets.ANTHROPIC_API_KEY
plus a write-scoped GITHUB_TOKEN with NO author-association gate.
Because issues / issue_comment / pull_request_review_comment always run in
the base-repo context with full secret access (unlike fork pull_request,
from which GitHub withholds secrets), any external GitHub user could fire
these jobs — draining the API budget and, via contents:write +
pull-requests:write, creating branches/PRs.
Fix:
- Add an author-association gate to every human-triggered, secret-bearing
if: clause, restricting to OWNER / MEMBER / COLLABORATOR. Uses the correct
event payload field per trigger: github.event.issue.author_association for
the `issues` event, github.event.comment.author_association for
issue_comment and pull_request_review_comment. workflow_dispatch stays
trusted (requires repo write to invoke).
- Drop unused id-token: write from both workflows (claude-code-action@v1 is
passed github_token directly, so OIDC is unused).
- Reduce claude-issues.yml timeout-minutes 60 -> 15.
desktop.yml and take-assign.yml are intentionally NOT touched: independently
verified as not exploitable for ANTHROPIC_API_KEY (desktop.yml's only
pull_request job uses no secrets and the trigger is plain pull_request, not
pull_request_target; take-assign.yml uses only GITHUB_TOKEN with issues:write
and no checkout/no Anthropic key). claude-review.yml's stale pull_request
auto-trigger was already removed in 3f2f46e4.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OpenAI-compatible POST /v1/chat/completions endpoint — the desktop UI's
chat backend — never injected OpenJarvis's agent.default_system_prompt when the
client omits a system message. The frontend (Chat/InputArea.tsx) posts only
user/assistant turns, so the model answered from its training identity
("I'm Claude", "I am Qwen", ...). The CLI paths ground identity via
SystemPromptBuilder / BaseAgent; the engine-direct server handlers did not.
Fix:
- Add _ensure_identity_prompt(messages, app_config) in server/routes.py: returns
messages unchanged when any has role==SYSTEM, else prepends a SYSTEM message
with the resolved identity prompt (app.state.config.agent.default_system_prompt,
else load_config()), wrapped in try/except that debug-logs on failure (no crash,
no silent swallow per REVIEW.md).
- Apply it after _to_messages() in all three engine-direct handlers:
_handle_stream, _handle_stream_tools, and _handle_direct; thread app.state.config
through. _handle_agent is left untouched (BaseAgent already injects the default).
- Harden AgentConfig.default_system_prompt so distilled models stop claiming to be
Claude/ChatGPT/Gemini and self-identify as OpenJarvis.
Tests (tests/server/test_routes.py, tests/core/test_config.py): identity prompt IS
prepended when no system message is present (stream / direct / tools paths) and is
NOT duplicated when the client supplies one; config wording anchors "OpenJarvis"
and "not Claude". Verified fail-on-unfixed against main (3 inject tests + config
wording test fail there).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(engine): add DeepSeek as a first-class cloud provider
Adds DEEPSEEK_API_KEY support to the cloud engine, wiring DeepSeek's
OpenAI-compatible API (api.deepseek.com/v1) alongside the existing
MiniMax, OpenRouter, Anthropic, and Google providers.
- Add _DEEPSEEK_MODELS list (deepseek-v4-flash, deepseek-v4-pro)
- Add _is_deepseek_model() routing predicate
- Init self._deepseek_client from DEEPSEEK_API_KEY in _init_clients()
- Add _generate_deepseek() and _stream_deepseek() methods
- Wire DeepSeek into generate(), stream(), _stream_full_openai(),
list_models(), and health()
- Add approximate pricing entries for both models
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(engine): strict cloud model routing + deepseek can_serve branch
Builds on the DeepSeek provider (PR #504) with two routing-correctness
fixes to CloudEngine._client_for_model:
1. Add the missing DeepSeek branch so can_serve('deepseek-*') agrees with
list_models()/health() when only DEEPSEEK_API_KEY is set (mirrors the
minimax branch). Without it the engine advertised deepseek models via
list_models() but refused to serve them (the #532 can_serve contract).
2. Fix#335: _client_for_model previously fell through to the OpenAI client
for ANY unrecognized model name, so an OpenAI key (even a dummy
sk-dummy... one) made can_serve('qwen3.5:0.8b') return True. With the
local engine transiently down (classic post-Windows-restart Ollama not
yet up), model-aware get_engine then mis-selected the cloud engine for a
local model and died with "OpenAI client not available". Add a positive
_is_openai_model predicate (gpt-/chatgpt-/o1/o3/o4 + _OPENAI_MODELS) and
return None for unrecognized names, so can_serve declines them. generate()
and stream() keep their OpenAI fall-through, preserving loud failure for an
explicitly-requested unknown cloud model.
Tests: DeepSeek detection/pricing/health/list_models/generate-routing/
can_serve and a #335 regression (can_serve rejects local names with an
OpenAI key; unknown model not served even with all clients set; end-to-end
get_engine does not misroute a local model with a dummy OpenAI key).
Fixes#335
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Jen Huls <me@jenhuls.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
OpenJarvis can run vision-capable local models (gemma3, qwen2.5-vl), but the
CLI had no way to send them a picture -- the Ollama engine only serialized
text. This adds end-to-end image input.
What's new
- `jarvis ask -i/--image <file>` attaches one or more images to the query.
- `jarvis ask -S/--screen` captures the primary monitor (dependency-free on
Windows via .NET; mss/Pillow fallback elsewhere).
- Vision auto-routes to direct-to-engine mode; with an explicit --agent it
warns rather than silently dropping the image.
- Privacy guard: warns before sending an image to a non-local engine,
keeping OpenJarvis local-first by default.
- Context-window default raised 8k -> 16k (JARVIS_NUM_CTX) so an image plus
a conversation fit.
Implementation
- Message.images carries base64 data; messages_to_dicts() forwards it to
Ollama's /api/chat "images" field. Text-only messages are unchanged.
- GuardrailsEngine preserves images when it rewrites a flagged message.
Tests (tests/test_vision.py, 6/6 pass, ruff-clean)
- payload forwarding, text path untouched, num_ctx override, guardrail
image preservation.
Verified on AMD RX 9070 XT (Ollama/Vulkan, 100% GPU) with gemma3:4b:
solid-color image, file image, and live screen capture all described.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Ensures that the desktop GUI correctly sends the Authorization header
when an API key is configured. This resolves the 'Failed to get response'
bug on Windows systems with enabled authentication.
Ref: #266
Co-authored-by: sanjayravit <sanjay@example.com>
Two failure classes hit by a downstream team:
1) tmux/task-env death (TerminalBenchTaskEnv.__enter__, terminalbench_env.py):
create_session ran inside the spin_up_terminal generator-CM with no
exception safety — a tmux failure leaked the docker compose project
(down deferred to GC, never if the env was retained) and surfaced as an
opaque mid-run death. Now: exception-safe __enter__ with an idempotent
_teardown(), a tmux/asciinema preflight in the container BEFORE the
agent loop (TaskEnvironmentError naming the task image + remedy), and
fail-that-task-cleanly semantics — the failure is recorded as a harness
error (QueryTrace.error_kind="harness_error"), the container is downed,
and the run continues.
2) OpenHands in-container SETUP hang misattributed as a model result:
harness.run() had no bound (terminal-bench runs installed-agent setup
with max_timeout_sec=inf inside the per-trial agent budget), and the
summary conversion read the nonexistent results.trial_results attr and
hardcoded errors=0, folding zero-model-request trials into resolve-rate
as model misses. Now: global_agent_timeout_sec / global_timeout_multiplier
are threaded config -> backend -> Harness kwargs (default 1800 s bound on
SETUP+RUN; configurable per [run]/[[benchmarks]] TOML), and
summarize_benchmark_results() classifies harness/infra failures out of
the accuracy denominator keyed on token usage (zero/missing tokens +
unresolved = the agent never contacted the model), NOT failure_mode —
terminal-bench 0.2.18 leaves failure_mode UNSET on success AND on
genuine misses, so a failure_mode-based check would misflag every real
model miss. Genuine misses (tokens>0, is_resolved=false) stay in the
denominator.
QueryTrace gains error/error_kind (wired through to_dict/from_dict so the
fields actually reach traces.jsonl; backward-compatible loads), the
AgenticRunner flags zero-model-contact traces and TaskEnvironmentError as
harness errors, and export/summary/console output exclude harness errors
from resolve-rate while reporting them loudly.
terminal-bench stays an undeclared dep on purpose: it requires Python
>=3.12 while this project supports >=3.10,<3.14, so an unmarked extra
would break uv lock. pyproject/uv.lock untouched.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
