The OpenAI-compat and Ollama engines exposed stream()/stream_full() as async def but iterated a synchronous httpx.Client.iter_lines() internally, blocking the single event loop on every inter-token read (serializing concurrent chats; one wedged upstream read froze the whole API). Convert both to a shared AsyncHTTPEngineMixin using httpx.AsyncClient + aiter_lines() with a per-event-loop pooled client and the configured timeout applied; map mid-stream transport errors (RemoteProtocolError/ReadError) to EngineConnectionError via a deliberately narrow set that keeps CancelledError/GeneratorExit propagating; handle non-2xx explicitly (incl. 3xx and a typed EngineContextLengthError for context-window overflow 400s); switch litellm streaming to acompletion; and offload the blocking non-streaming handlers and websocket generate() to asyncio.to_thread. No public API change. Strong MockTransport-based tests, including a pin that the async path never touches the sync client. Complements #618.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#608. Make Message.content officially Optional (str | None) with a Message.text accessor that treats None as empty, and count tool-call IDs/names/arguments, tool-result IDs, and reasoning/thinking metadata in estimate_prompt_tokens — all are replayed into later prompt turns, so they belong in the estimate. Extends estimator and message-type regression tests.
Closes#607. Assistant tool-call turns can carry content=None, which crashed token estimation (len(m.content)) and think-tag stripping. Normalize with 'content or ""', route native OpenHands truncation through the shared estimate_prompt_tokens, and add tests for the estimator, the truncation helper, and an end-to-end tool-call run with None content.
Qwen3 treats /think and /no_think as soft-switch control tokens. On small
models a multi-line prompt makes the model emit one as the sole tool argument
(e.g. {"command": "/no_think"}); OpenJarvis forwards Ollama's native tool_calls
verbatim, so the operative agent executes garbage. Filter control-token-only
tool calls in both the non-streaming generate() and streaming _run_stream()
paths, keeping legitimate calls like {"command": "date"}.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(engine): add DeepSeek as a first-class cloud provider
Adds DEEPSEEK_API_KEY support to the cloud engine, wiring DeepSeek's
OpenAI-compatible API (api.deepseek.com/v1) alongside the existing
MiniMax, OpenRouter, Anthropic, and Google providers.
- Add _DEEPSEEK_MODELS list (deepseek-v4-flash, deepseek-v4-pro)
- Add _is_deepseek_model() routing predicate
- Init self._deepseek_client from DEEPSEEK_API_KEY in _init_clients()
- Add _generate_deepseek() and _stream_deepseek() methods
- Wire DeepSeek into generate(), stream(), _stream_full_openai(),
list_models(), and health()
- Add approximate pricing entries for both models
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(engine): strict cloud model routing + deepseek can_serve branch
Builds on the DeepSeek provider (PR #504) with two routing-correctness
fixes to CloudEngine._client_for_model:
1. Add the missing DeepSeek branch so can_serve('deepseek-*') agrees with
list_models()/health() when only DEEPSEEK_API_KEY is set (mirrors the
minimax branch). Without it the engine advertised deepseek models via
list_models() but refused to serve them (the #532 can_serve contract).
2. Fix#335: _client_for_model previously fell through to the OpenAI client
for ANY unrecognized model name, so an OpenAI key (even a dummy
sk-dummy... one) made can_serve('qwen3.5:0.8b') return True. With the
local engine transiently down (classic post-Windows-restart Ollama not
yet up), model-aware get_engine then mis-selected the cloud engine for a
local model and died with "OpenAI client not available". Add a positive
_is_openai_model predicate (gpt-/chatgpt-/o1/o3/o4 + _OPENAI_MODELS) and
return None for unrecognized names, so can_serve declines them. generate()
and stream() keep their OpenAI fall-through, preserving loud failure for an
explicitly-requested unknown cloud model.
Tests: DeepSeek detection/pricing/health/list_models/generate-routing/
can_serve and a #335 regression (can_serve rejects local names with an
OpenAI key; unknown model not served even with all clients set; end-to-end
get_engine does not misroute a local model with a dummy OpenAI key).
Fixes#335
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Jen Huls <me@jenhuls.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
`jarvis eval run --base-url ... --api-key ...` was silently dropped for
jarvis-direct/jarvis-agent (_build_backend only forwarded the flags to
hermes/openclaw) and ignored by terminalbench-native, which hardcoded
api_base="http://localhost:8000/v1". Worse, with --base-url set the
engine-discovery fallback silently substituted ANY healthy local engine
(observed: requested vllm + healthy endpoint at --base-url, got
OllamaEngine@localhost:11434 — the requested URL was never contacted).
Changes:
- _OpenAICompatibleEngine gains an api_key param (Bearer Authorization
header on the httpx client; {ENGINE_ID}_API_KEY env fallback with
hyphen-sanitized names; no header when unset).
- New non-registered OpenAICompatEngine + normalize_openai_base_url()
(strips a single literal trailing "/v1" so request paths don't double).
- SystemBuilder.engine_instance() injects a pre-built engine; build()
health-checks it and fails loudly naming the host instead of falling
back to discovery. Discovery substitution after an explicit -e key now
logs a warning.
- JarvisDirectBackend/JarvisAgentBackend accept base_url/api_key; on
base_url they pin an OpenAICompatEngine to that endpoint with a
fail-fast pre-flight (actionable error naming the URL and probe).
- _build_backend forwards base_url/api_key to first-party backends on
the CLI path; _run_terminalbench_native receives --base-url as
api_base (single /v1 suffix) and exports OPENAI_API_KEY around the
in-process harness run (terminus-2 routes via LiteLLM).
- Suite TOML [backend.external] stays scoped to hermes/openclaw
(suite_mode=True in the suite drivers) — first-party suite semantics
are explicitly deferred. The config-host path is untouched.
- Help text updated on both CLI surfaces; KNOWN_BACKENDS now lists
hermes/openclaw/terminalbench-native.
Fixes the eval-CLI endpoint gap reported by the downstream team.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(windows): desktop backend spawn (#531) + model-aware engine selection (#532)
Two runtime bugs found during end-to-end testing on a clean Windows 11
24H2 Azure VM.
#531 - Desktop "Failed to get response": run_jarvis_command spawned the
backend with .output(), which waits for the process to exit. `jarvis
serve` never exits, so the Tauri command hung forever (the Start button
never resolved); and it ran `uv run jarvis` with no cwd, so in a packaged
install -- where the cwd isn't the checkout -- `jarvis` wasn't found and
the server never started. Now: run from find_project_root(), and for
`serve` spawn detached (.spawn()), drain stderr, and poll /health for
readiness (mirrors start_backend); short commands keep .output().
The server layer itself was verified healthy on Windows (/health and
/v1/chat/completions both 200, localhost included) -- the fault was the
Tauri spawn path.
#532 - "OpenAI client not available" after reboot: when the local engine
is down, get_engine's fallback selected CloudEngine because health() is
True if ANY provider client exists -- without checking the resolved
model's provider has a client. A user with e.g. OPENROUTER_API_KEY and a
gpt-* model then hit the OpenAI path with no client. Add
CloudEngine.can_serve(model) (checks the specific provider client via the
same routing generate()/stream() use) + a default can_serve->True on the
base engine, and make get_engine model-aware so it skips an engine that
can't serve the model -- the user falls through to the helpful "no engine
available / start ollama" message instead.
Tests: engine discovery/cloud/model-matrix + cli serve/ask suites pass
(the one ask_e2e failure is a pre-existing version-banner flake, fails
identically on main). The Tauri crate couldn't be compiled locally (no
GTK/webkit sys-libs in this env); relies on CI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(engine): cover model-aware engine selection + CloudEngine.can_serve (#532)
#533 added a `model` arg to get_engine and a can_serve() gate but shipped no
tests. Add them:
- get_engine skips a healthy engine that can't serve the requested model
(the cloud-fallback-for-unservable-model case behind #532),
- model=None preserves the legacy model-agnostic selection,
- CloudEngine.can_serve gates on the per-provider client (gpt->OpenAI,
claude->Anthropic, ...), verified empirically.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
#463 made the OpenAI-compatible engine wrap upstream HTTP errors (incl.
404) in EngineConnectionError with an actionable message, but
test_invalid_model_404 still asserted the raw httpx.HTTPStatusError, so it
broke on main once #463 landed (#463 was a stale fork PR with no CI, so it
wasn't caught pre-merge). Expect EngineConnectionError now, asserting the
httpx.HTTPStatusError is preserved as the chained cause. Whole tests/engine
suite is green again.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: forward tools through OpenRouter engine
The OpenRouter chat completion path built the request from only
`model`, `messages`, `max_tokens`, and `temperature`. `tools` and
`tool_choice` passed via `kwargs` were silently dropped, so
function definitions never reached the model. Symptom on a
managed deep_research agent: the model answered every query from
its own prior knowledge and never invoked `knowledge_search`,
`knowledge_sql`, etc.
The same path also discarded `choice.message.tool_calls` from the
response — when a model did return a tool call (verified directly
against OpenRouter with `google/gemma-4-31b-it:free` and
`nvidia/nemotron-3-ultra-550b-a55b:free`), the agent loop never
saw it.
This patch:
- forwards `tools` and `tool_choice` into the OpenAI-compatible
request in both `_generate_openrouter` (sync) and
`_stream_openrouter` (async stream),
- extracts `tool_calls` from the response in `_generate_openrouter`
in the same shape used by the OpenAI / Anthropic paths.
Verified end-to-end against a `deep_research` managed agent using
an OpenRouter preset with Gemma 4 31B + Nemotron 3 Ultra fallback:
before, the agent stated "I don't have access to your vault";
after, it calls `knowledge_search`, cites results, and produces a
structured answer.
* test(engine): regression test for OpenRouter tool forwarding
Asserts the OpenRouter path forwards tools/tool_choice to the
OpenAI-compatible API and parses tool_calls back into the result (#511).
Verified: passes on the fix, fails (KeyError 'tools') against pre-fix main.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`jarvis serve` startup was slow on two counts addressed here:
1. discover_engines() probed each registered engine's health() serially,
and every probe is a blocking network check with its own ~2s timeout —
so N dead/slow localhost ports cost N*2s. Run the probes concurrently
in a ThreadPoolExecutor; the existing healthy.sort() normalizes order,
so the result is identical to the serial version. health() impls are
read-only on per-instance HTTP clients with no shared mutable state.
2. The PyPI update check ran a blocking urlopen (up to 3s on a cache
miss) inline before dispatch, delaying every command. Move it to a
daemon thread — it's best-effort and never raises.
Together these remove ~10-30s from cold startup. Adds a regression test
asserting discovery probes overlap (concurrency), not just that output
is unchanged (covered by existing tests).
Note: the larger ~30-40s win — the duplicate SystemBuilder.build() in
serve.py — is NOT addressed here; it's not redundant (the second build
wires a JarvisSystem the inline path never constructs) and needs a
design pass. Deferred.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Some OpenAI models (e.g. gpt-5, the default for a fresh cloud install)
reject a non-default temperature with HTTP 400 "Unsupported value:
'temperature' does not support 0.7 ... Only the default (1) value is
supported." — so the user's very first prompt fails (#426).
Detect that specific 400 (param=temperature + unsupported_value/"only the
default"/"does not support") and retry the create() once without
temperature, mirroring the tools-400 retry in the Ollama and
OpenAI-compat engines. Unrelated 400s are re-raised unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apple FM's stream_response yields cumulative text snapshots, but
OpenAI-compatible clients concatenate delta.content — so streamed
responses were duplicated/stuttered (#378). Diff each snapshot against
the last and emit only the incremental suffix; fall back to the full
snapshot if the model revises earlier text, and skip empty deltas.
Rebased on #377 (apple_fm_sdk migration): uses the options= streaming
API. Adds a stubbed-SDK regression test asserting deltas are
incremental, not cumulative.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(engine): modernize apple_fm_shim for the public apple-fm-sdk
Apple released the official Foundation Models Python SDK as
`apple/python-apple-fm-sdk` (import name `apple_fm_sdk`, distribution
name `apple-fm-sdk`). The shim's `import apple_fm` predates the public
SDK and the per-call API has since moved as well; running the shim
against current `apple-fm-sdk` v0.1.1 fails on import, then on the
`/health` call shape, and again on every `respond` / `stream_response`
keyword.
This change brings the shim up to date with the real SDK without
altering its external OpenAI-compatible contract:
- Import `apple_fm_sdk` (the public name). Update the missing-package
error to point at the GitHub repo since the SDK isn't on PyPI;
installation is `uv pip install -e <clone>`.
- `SystemLanguageModel.is_available()` is an instance method and now
returns `(bool, SystemLanguageModelUnavailableReason | None)`. The
health endpoint instantiates the model and unpacks the tuple, and
surfaces the reason string when the model is unavailable.
- `LanguageModelSession.respond` and `.stream_response` no longer
accept `max_tokens` / `temperature` positionally; they take a
`GenerationOptions` instance via the `options` kwarg. The shim now
builds a `GenerationOptions(temperature=..., maximum_response_tokens=...)`
from each `ChatRequest` and threads it through both paths.
`temperature` is now actually honored (the previous code dropped it
silently).
- Add `response_model=None` to the `/v1/chat/completions` decorator so
FastAPI doesn't try to build a Pydantic field for the
`JSONResponse | StreamingResponse` union return type — that fails
with the FastAPI version pinned in the `server` extra.
- Update the module docstring to reference macOS 26 + Apple
Intelligence (the SDK's actual minimum), not macOS 15.
## How was this tested?
- `uv pip install -e ./python-apple-fm-sdk` against a local clone of
Apple's repo on macOS 26 / M5 Max with Apple Intelligence enabled.
- `uv sync --extra dev --extra server` then `uv run uvicorn
openjarvis.engine.apple_fm_shim:app --host 127.0.0.1 --port 8079`.
- `GET /health` → `{"status": "ok"}` (200).
- `GET /v1/models` → lists `apple-fm`.
- `POST /v1/chat/completions` with messages + temperature +
max_tokens → returns a real Apple Intelligence completion.
- End-to-end through OpenJarvis: add `[engine.apple_fm]
host = "http://localhost:8079"` to `~/.openjarvis/config.toml`,
then `jarvis ask --engine apple_fm --model apple-fm "..."` returns
the same Apple FM response routed through the OpenAI-compatible
engine wrapper.
- `uv run ruff check src/openjarvis/engine/apple_fm_shim.py` and
`uv run ruff format --check src/openjarvis/engine/apple_fm_shim.py`
both pass.
* test(engine): add stubbed-SDK tests for apple_fm_shim migration
Covers the apple_fm -> apple_fm_sdk migration: GenerationOptions carries
temperature + max_tokens and is passed via options= to respond() and
stream_response(), and /health unpacks the (available, reason) tuple
from SystemLanguageModel().is_available().
The real apple-fm-sdk is not installable in CI (not on PyPI; macOS 26 +
Apple Intelligence only), so the tests inject a stub SDK into
sys.modules. They verify the shim's OpenAI-compat wiring, not Apple's
real SDK behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds `codex/` prefixed model support using the OpenAI Responses API —
the same protocol used by zeroclaw and other Codex-compatible tools.
Live-tested against gpt-5-mini-2025-08-07 with:
- Generate (non-streaming): confirmed working
- System prompt → instructions mapping: confirmed working
- SSE streaming: confirmed working (9 chunks)
- End-to-end via `jarvis ask`: confirmed working
Implementation:
- Default endpoint: api.openai.com/v1/responses (standard API key)
- Override via OPENAI_CODEX_BASE_URL for ChatGPT OAuth tokens
(e.g. chatgpt.com/backend-api/codex)
- Auth via OPENAI_CODEX_API_KEY env var
- Responses API format: input array, instructions field, output_text extraction
- Handles reasoning+message output blocks correctly
- SSE streaming parses response.output_text.delta events
Models: codex/gpt-4o, codex/gpt-4o-mini, codex/o3-mini,
codex/gpt-5-mini, codex/gpt-5-mini-2025-08-07
Usage:
export OPENAI_CODEX_API_KEY="your-api-key-or-oauth-token"
jarvis ask "Hello" --model codex/gpt-5-mini-2025-08-07
Closes#134
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move mid-file imports to top of test file to resolve E402 violations and apply ruff formatting to both gemma_cpp.py and test_gemma_cpp.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add MiniMax as cloud inference provider
Add MiniMax M2.5 and M2.5-highspeed as a 5th cloud provider alongside
OpenAI, Anthropic, Google, and OpenRouter. Uses the OpenAI-compatible
API at api.minimax.io/v1 via the existing openai SDK dependency.
Changes:
- Add MiniMax client init, generate, and streaming in CloudEngine
- Add MiniMax models to model catalog with correct pricing
- Add MINIMAX_API_KEY environment variable support
- Add temperature clamping (0.01-1.0) per MiniMax API constraints
- Add 19 unit tests and 3 integration tests
- Update docs and README with MiniMax provider info
* feat: upgrade MiniMax default model to M2.7
- Add MiniMax-M2.7 and MiniMax-M2.7-highspeed to model list
- Set MiniMax-M2.7 as default model (first in list)
- Keep all previous models (M2.5, M2.5-highspeed) as alternatives
- Update pricing table with M2.7 entries
- Update model catalog with M2.7 specs
- Update docs to list all available MiniMax models
- Update unit tests (23 passing) and integration tests (3 passing)
---------
Co-authored-by: Octopus <octo-patch@users.noreply.github.com>
Add _annotate_anthropic_cache helper that annotates system messages
with cache_control for Anthropic prompt caching.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Engine discovery (#73): get_engine() now falls back to any healthy
engine when the explicitly-requested key fails, instead of returning
None. Fixes LM Studio (and other non-default engines) not being found
when deploying agents.
- FTS5 case sensitivity & scoring (#67): Add explicit unicode61 tokenizer
to the FTS5 virtual table for case-insensitive search. Replace ambiguous
`rank` column with explicit `bm25()` call with column weights (id=0,
content=1, source=0.5) to produce correct positive scores. Includes
auto-migration for existing databases.
- Interactive engine selection (#72): `jarvis init` now detects running
engines and presents an interactive picker. Also accepts `--engine`
flag to skip the prompt. Engine choice flows through to config
generation.
- Windows desktop support (#68): Add RAM detection via wmic, Windows
binary paths (LOCALAPPDATA, ProgramFiles, cargo), port cleanup via
netstat+taskkill, and USERPROFILE fallback for HOME env var.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
