Comprehensive step-by-step guide covering Homebrew, uv, Rust, llama.cpp,
model download, Python 3.12 pin (PyO3 compat), and common pitfalls.
Cherry-picked from PR #131 by @gridworks — cleaned up to include only
the docs content (removed duplicate files, binary artifacts, and
unrelated lockfile changes from the original PR).
Co-Authored-By: gridworks <5502067+gridworks@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Companion to #143 which fixed the frontend to use Claude Opus 4.6 as
the sole baseline. This migration recomputes existing Supabase rows
using the exact closed-form: new = T/3.8M + 10*old/19, derived from
the original triple-provider formula.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove cent sign from cost display, use $X.XXXX format
- Compact stat cards (horizontal icon+value layout)
- Tighter config grid spacing with bolder labels
- Reduce padding and gaps throughout overview tab
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dollar savings were previously summed across all three cloud providers
(GPT-5.3 + Claude Opus 4.6 + Gemini 3.1 Pro), inflating the reported
number by ~3x. Now uses only Claude Opus 4.6 pricing as the baseline,
with an asterisk footnote on the leaderboard explaining this.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds `codex/` prefixed model support using the OpenAI Responses API —
the same protocol used by zeroclaw and other Codex-compatible tools.
Live-tested against gpt-5-mini-2025-08-07 with:
- Generate (non-streaming): confirmed working
- System prompt → instructions mapping: confirmed working
- SSE streaming: confirmed working (9 chunks)
- End-to-end via `jarvis ask`: confirmed working
Implementation:
- Default endpoint: api.openai.com/v1/responses (standard API key)
- Override via OPENAI_CODEX_BASE_URL for ChatGPT OAuth tokens
(e.g. chatgpt.com/backend-api/codex)
- Auth via OPENAI_CODEX_API_KEY env var
- Responses API format: input array, instructions field, output_text extraction
- Handles reasoning+message output blocks correctly
- SSE streaming parses response.output_text.delta events
Models: codex/gpt-4o, codex/gpt-4o-mini, codex/o3-mini,
codex/gpt-5-mini, codex/gpt-5-mini-2025-08-07
Usage:
export OPENAI_CODEX_API_KEY="your-api-key-or-oauth-token"
jarvis ask "Hello" --model codex/gpt-5-mini-2025-08-07
Closes#134
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Overview tab:
- Show Intelligence (model name) with click-to-change dropdown
- Split "Total Tokens" into "Input Tokens" and "Output Tokens"
- Model can be switched for existing agents via dropdown
Backend:
- Add input_tokens/output_tokens columns to managed_agents
- Track prompt_tokens and completion_tokens separately in executor
- Disable Ollama thinking by default (think:false) to prevent
empty responses from token exhaustion
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause of empty responses: Qwen3.5's extended thinking mode
consumes all tokens (4096) on hidden <think> tags, leaving zero
visible content. The /no_think text tag was unreliable.
Fix: pass think=false in the Ollama API payload, which properly
disables thinking at the API level. Drops token usage from ~4096
to ~5-200 per response and eliminates empty content.
Also:
- Fix token tracking to read total_tokens from metadata (was looking
for tokens_used which is never set)
- Remove the /no_think system prompt hack (superseded by API param)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Incorporates the useful new features from PR #135 (by @gridworks) on top
of the existing PrivacyScanner implementation:
- Add DNS configuration check (macOS, via scutil --dns)
- Add --json flag to `jarvis scan` for machine-readable output
- Add --no-scan flag to `jarvis init` to skip the post-init audit
- Expand remote-access process list (ngrok, tailscaled, cloudflared, ZeroTier)
- Upgrade `jarvis scan` output from plain text to Rich table
- Add GET /v1/security/scan API endpoint
- Add tests for all new features (30 tests, all passing)
Closes#133
Co-Authored-By: gridworks <5502067+gridworks@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add current_activity field to managed agents that the executor updates
at each phase of a tick (loading model, delivering messages, generating
response, retrying, finalizing). The frontend polls this every 2s and
displays the live status instead of static "Agent is thinking...".
Backend:
- Add current_activity column to managed_agents (migration)
- Add _set_activity helper to AgentExecutor
- Update activity at: start_tick, model load, message delivery,
generation, retry, finalize
- Clear activity on end_tick
Frontend:
- InteractTab polls both messages and agent status in parallel
- Shows current_activity text with pulsing indicator
- Falls back to "Agent is thinking..." if activity is empty
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fixes for the last 4 failing PinchBench tasks:
1. Tool arguments (tasks 05, 07, 13): Arguments were lost in the
pipeline — ToolExecutor stored them but system.py stripped them
from the tool_results dict, so the LLM judge saw write_file({})
instead of the actual content. Now pipe arguments through:
_stubs.py → system.py → scorer transcript.
2. Multi-session tasks (task 22): Parse `sessions` field from task
frontmatter, use first session's prompt as record.problem, and
execute remaining sessions sequentially within the workspace
context in _process_one().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
get_engine() probes all engines with health checks, which can load
different models and interfere with in-flight Ollama requests, causing
intermittent empty responses. Create a plain OllamaEngine directly
from config instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Append /no_think to system prompt to prevent Qwen3.5 from consuming
all tokens on extended thinking and producing empty visible output
- Retry once if agent returns empty content
- Add debug logging to _make_lightweight_system for engine diagnostics
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gemini 3.1+ reasoning models require a thought_signature field in
function_call parts when replaying conversation history. Without it,
the API returns 400 INVALID_ARGUMENT on every multi-turn tool call.
Changes:
- CloudEngine: capture thought_signature from Gemini responses and
store in _thought_sigs dict keyed by tool_call id
- CloudEngine: replay thought_signature when building function_call
parts for Gemini conversation history
- native_openhands: thread thought_signature through via side dict
(ToolCall uses slots, can't add dynamic attributes)
- Add PinchBench eval configs for Claude Opus 4.6, Gemini 3.1 Pro,
Nemotron-3-Super, Qwen 122B, and Qwen 35B
Impact: Gemini 3.1 Pro PinchBench score 4% → 78%
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The server's engine is wrapped in MultiEngine → InstrumentedEngine →
GuardrailsEngine. When reused from a background thread for agent ticks,
this chain returns empty content. Create a fresh OllamaEngine for each
tick instead, which reliably returns model output.
Also fixes Run Now endpoint to use the same lightweight system approach.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update STARTUP_MODEL from 2b to 4b for better quality on first launch.
Also update preferred_model() to prefer STARTUP_MODEL when it fits,
rather than always picking the third-largest model. This gives a
consistent default across machines while still falling back to
RAM-appropriate sizing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: _run_tick and _immediate_tick called SystemBuilder().build()
which picks the first model from Ollama (qwen3.5:35b) instead of the
model the server was started with (e.g. qwen3.5:9b). A 0.6B query was
running on a 35B model, causing 5+ minute stalls.
Fix: reuse the server's engine/model from app.state via a lightweight
system facade instead of rebuilding the full JarvisSystem.
Also:
- Add detailed logging to AgentExecutor (model, pending messages,
timing, content length, errors with tracebacks)
- Add logging to immediate tick lifecycle
- Remove Queue button from Interact tab (single Send button)
- Enter key now sends immediately instead of queueing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backend:
- Immediate-mode messages now trigger a background tick so the agent
actually processes and responds (previously they were just stored)
Frontend (Interact tab):
- Reverse message order so newest appear at bottom near the input box
- Filter out agent responses with empty content (blank bubbles)
- Add "Agent is thinking..." indicator with pulsing dot while processing
- Show timestamps instead of raw mode/status labels
- Poll for new messages every 3s so responses appear automatically
- Only auto-scroll to bottom on initial tab load, not on every poll
update (prevents hijacking the user's scroll position)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
_apply_toml_section only normalized TOML arrays to comma-separated strings
for real dataclass fields, but backward-compat property setters like
reward_weights also expect string input. When a user's config.toml had
an array value for a property-backed attribute, the raw list was passed
to the setter which called .split(",") on it, causing:
'list' object has no attribute 'split'
This also hardens serve.py against the same issue when reading
config.agent.tools.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merge main into fix/ssrf-check, keeping both the auto-recover
logic for error-state agents and the async streaming support
for the send_message endpoint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The model catalog listed non-existent Qwen3.5 sizes (3B, 8B, 14B) and
pointed to MLX community repos that don't exist, causing `jarvis init`
to recommend models that cannot be downloaded on Apple Silicon.
Replace with the actual Qwen3.5 model family sizes (0.8B, 2B, 9B, 27B)
and verified mlx-community repo URLs from HuggingFace.
Fixes#129
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The gemma_cpp live tests require local model weights and env vars
(GEMMA_CPP_MODEL_PATH, etc.) that are not available in CI, causing
4 failures since the gemma-cpp-engine PR was merged. Add marker
filters to the pytest invocation so live and cloud tests are skipped.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): PinchBench harness fixes — scores from 26% to 84%
Multiple infrastructure bugs prevented PinchBench from producing
accurate scores. This commit fixes the eval harness so model scores
match the official leaderboard (Qwen3.5-397B: 26% → 84%, GPT-5.4:
5% → 58%).
Key fixes:
- EvalRunner: wrap generation in PinchBenchTaskEnv context so workspace
files persist through grading (was deleting before scorer ran)
- EvalRunner: detect task_env datasets and force sequential processing
(CWD changes aren't thread-safe)
- EvalRunner: fix episode_mode auto-detection to check for real
iter_episodes() override instead of hasattr() (always True)
- Scorer: use "params" field in transcripts to match PinchBench grade()
functions (was "arguments")
- Scorer: add None guard in _trace_to_transcript for tool_calls
- Scorer: capture final assistant text response in transcript so
text-only tasks (like sanity check) can be graded
- Scorer: add _tool_results_to_transcript() helper for EvalRunner path
- native_openhands: add native function-calling support (tools=
parameter) with text-based fallback, matching monitor_operative
- Tools: add Python fallbacks for file_read, file_write, shell_exec,
calculator, think, http_request when openjarvis_rust unavailable
- Security: add Python fallback for is_sensitive_file()
- Config: add "pinchbench" to KNOWN_BENCHMARKS
- Add PinchBench eval configs for Qwen3.5-397B and GPT-5.4
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): fix hybrid grading crash when grading_weights is None
The 4 tasks with grading_type=hybrid (task_10, task_13, task_16_market,
task_22) crashed because grading_weights was explicitly None in task
metadata. dict.get("key", default) returns None (not the default) when
the key exists with value None. Use `or` to coalesce None to defaults.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(evals): handle MagicMock datasets in runner type checks
The iter_episodes and create_task_env type identity checks fail with
AttributeError when the dataset is a MagicMock (used in tracker tests).
Wrap in try/except to default to False for non-DatasetProvider objects.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Document that pygemma v0.1.3 completion() does not accept
temperature/max_tokens (params accepted for ABC compliance)
- Add model-mismatch warning to stream() for consistency with generate()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move mid-file imports to top of test file to resolve E402 violations and apply ruff formatting to both gemma_cpp.py and test_gemma_cpp.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AgenticRunner dispatches _run_body() to a ThreadPoolExecutor when a
task environment is present (for Playwright compatibility). The
TraceStore connection was created on the main thread, causing
"SQLite objects created in a thread can only be used in that same
thread" on the first agentic eval query.
Pass check_same_thread=False to sqlite3.connect(), consistent with
SchedulerStore, AgentManager, SessionStore, and TelemetryStore which
already use this flag.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
