Merge main into fix/ssrf-check, keeping both the auto-recover
logic for error-state agents and the async streaming support
for the send_message endpoint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The gemma_cpp live tests require local model weights and env vars
(GEMMA_CPP_MODEL_PATH, etc.) that are not available in CI, causing
4 failures since the gemma-cpp-engine PR was merged. Add marker
filters to the pytest invocation so live and cloud tests are skipped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): PinchBench harness fixes — scores from 26% to 84%
Multiple infrastructure bugs prevented PinchBench from producing
accurate scores. This commit fixes the eval harness so model scores
match the official leaderboard (Qwen3.5-397B: 26% → 84%, GPT-5.4:
5% → 58%).
Key fixes:
- EvalRunner: wrap generation in PinchBenchTaskEnv context so workspace
files persist through grading (was deleting before scorer ran)
- EvalRunner: detect task_env datasets and force sequential processing
(CWD changes aren't thread-safe)
- EvalRunner: fix episode_mode auto-detection to check for real
iter_episodes() override instead of hasattr() (always True)
- Scorer: use "params" field in transcripts to match PinchBench grade()
functions (was "arguments")
- Scorer: add None guard in _trace_to_transcript for tool_calls
- Scorer: capture final assistant text response in transcript so
text-only tasks (like sanity check) can be graded
- Scorer: add _tool_results_to_transcript() helper for EvalRunner path
- native_openhands: add native function-calling support (tools=
parameter) with text-based fallback, matching monitor_operative
- Tools: add Python fallbacks for file_read, file_write, shell_exec,
calculator, think, http_request when openjarvis_rust unavailable
- Security: add Python fallback for is_sensitive_file()
- Config: add "pinchbench" to KNOWN_BENCHMARKS
- Add PinchBench eval configs for Qwen3.5-397B and GPT-5.4
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): fix hybrid grading crash when grading_weights is None
The 4 tasks with grading_type=hybrid (task_10, task_13, task_16_market,
task_22) crashed because grading_weights was explicitly None in task
metadata. dict.get("key", default) returns None (not the default) when
the key exists with value None. Use `or` to coalesce None to defaults.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): handle MagicMock datasets in runner type checks
The iter_episodes and create_task_env type identity checks fail with
AttributeError when the dataset is a MagicMock (used in tracker tests).
Wrap in try/except to default to False for non-DatasetProvider objects.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Document that pygemma v0.1.3 completion() does not accept
temperature/max_tokens (params accepted for ABC compliance)
- Add model-mismatch warning to stream() for consistency with generate()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move mid-file imports to top of test file to resolve E402 violations and apply ruff formatting to both gemma_cpp.py and test_gemma_cpp.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AgenticRunner dispatches _run_body() to a ThreadPoolExecutor when a
task environment is present (for Playwright compatibility). The
TraceStore connection was created on the main thread, causing
"SQLite objects created in a thread can only be used in that same
thread" on the first agentic eval query.
Pass check_same_thread=False to sqlite3.connect(), consistent with
SchedulerStore, AgentManager, SessionStore, and TelemetryStore which
already use this flag.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Store workspace_path in record.metadata from AgenticRunner so task
envs can reuse the agent's workspace instead of creating a separate
temp dir the agent can't see
- PinchBenchTaskEnv now uses the existing workspace when available,
copies fixtures there, and sets CWD so file_read/file_write resolve
relative paths correctly
- Wrap grading in try/except so LLM judge failures don't crash the
entire run (graceful degradation to score 0.0)
- Restore CWD on exit and only clean up self-owned workspaces
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a truncated (10KB max) string representation of the tool result's
content to the TOOL_CALL_END event payload so grading functions (e.g.
PinchBench) can inspect what each tool returned.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a `tool_calls` list field to `TurnTrace` capturing name, arguments,
and result for each tool invocation, enabling PinchBench transcript integration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Merge Quick Start, Docker, and Development into single Quick Start
- Point to docs site for full documentation (Docker, cloud, dev setup)
- Add Contributing section with dev setup and roadmap pointer
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The test_init_creates_config test in test_cli.py was not updated
when the download prompt and privacy hook were added to jarvis init.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wraps long CliRunner.invoke() calls, removes unused pytest imports,
fixes import ordering, and removes unused variables in test_scan.py.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These are Claude-generated working documents that should not
be checked into the repository.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add slots=True to ScanResult dataclass
- Fix extra space in IPv6 port f-string
- Add missing tests: check_remote_access, check_icloud_sync, run_quick
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Registers the new scan command in the CLI. Adds a lightweight
privacy check at the end of jarvis init that runs disk encryption
and cloud sync checks, with pointer to jarvis scan for full audit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Prompts user to download recommended model during jarvis init.
Adds --no-download flag for CI. Shows helpful message when no
model fits available memory. Adds exo/nexa next-steps text.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The FLOPs formula changed from quadratic (P*N*(N+1)) to linear
(2*P*T_evaluated) with KV-cache awareness. Update the test
expectation from ~100x to ~10x for 10x token increase.
Refactors model pull into reusable ollama_pull() function. Adds
--engine flag to support llamacpp (GGUF) and mlx (HuggingFace)
downloads via huggingface-cli, with FileNotFoundError handling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Thread prompt_tokens_evaluated through the telemetry stack so
savings calculations use the right token count for each metric:
- Dollar cost: full prompt_tokens (what cloud providers charge)
- FLOPs/energy: prompt_tokens_evaluated (actual compute with KV
cache — subsequent turns only re-evaluate new tokens)
Ollama reports both: prompt_eval_count (cache-aware) and we estimate
full count from messages. OpenAI-compat engines report full count
only (KV caching is transparent in their API).
Changes: TelemetryRecord, store schema, aggregator, engines,
savings calculation, /v1/savings route.
New PrivacyScanner class with checks for disk encryption (FileVault/LUKS),
MDM profiles, cloud sync agents, network exposure, and screen recording.
Supports macOS and Linux with graceful skip on missing tools.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes recommend_model() returning empty string on Apple Silicon
when MLX is the recommended engine. Also adds gguf_file and
mlx_repo download metadata, and estimated_download_gb helper.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
