OpenJarvis can run vision-capable local models (gemma3, qwen2.5-vl), but the
CLI had no way to send them a picture -- the Ollama engine only serialized
text. This adds end-to-end image input.
What's new
- `jarvis ask -i/--image <file>` attaches one or more images to the query.
- `jarvis ask -S/--screen` captures the primary monitor (dependency-free on
Windows via .NET; mss/Pillow fallback elsewhere).
- Vision auto-routes to direct-to-engine mode; with an explicit --agent it
warns rather than silently dropping the image.
- Privacy guard: warns before sending an image to a non-local engine,
keeping OpenJarvis local-first by default.
- Context-window default raised 8k -> 16k (JARVIS_NUM_CTX) so an image plus
a conversation fit.
Implementation
- Message.images carries base64 data; messages_to_dicts() forwards it to
Ollama's /api/chat "images" field. Text-only messages are unchanged.
- GuardrailsEngine preserves images when it rewrites a flagged message.
Tests (tests/test_vision.py, 6/6 pass, ruff-clean)
- payload forwarding, text path untouched, num_ctx override, guardrail
image preservation.
Verified on AMD RX 9070 XT (Ollama/Vulkan, 100% GPU) with gemma3:4b:
solid-color image, file image, and live screen capture all described.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
`jarvis serve` constructed every heavy component inline (engine discovery +
instrumentation, telemetry, memory, agent manager, per-agent tools) and then
called `SystemBuilder(config).build()` a second time inside the scheduler
block purely to feed `AgentExecutor.set_system()`. That second build
re-discovered and re-connected the engine, re-instrumented it, re-resolved
tools, re-opened the configured channel and re-created the agent manager —
~30-40s of fully redundant startup work (the headline remaining cost in #263
after engine probes were parallelised and the version check moved off the hot
path in #470).
Fix: assemble the executor's `JarvisSystem` from the components already built
inline instead of rebuilding from scratch. `AgentExecutor` only reads
`engine`, `model`, `config`, `memory_backend`, `tool_executor`,
`session_store` and `channel_backend` off the system; all are wired here. The
memory backend is now constructed just before the scheduler block (it was
built later) so the executor's system can reference it, and the primary
agent's resolved tool list is reused to build the scheduler's `ToolExecutor`
(preserving the MCP-discovered-tool pool the executor reads via
`tool_executor._tools`). `skill_manager` / the learning orchestrator are only
consumed by the orchestrator's `system.ask()` path, which the executor never
invokes, so they are intentionally omitted.
Tests: new `tests/cli/test_serve_single_build.py` patches
`SystemBuilder.build` and asserts it is never called during `jarvis serve`
startup, and that the executor still receives a system exposing
`tool_executor` / `session_store` / `memory_backend` (plus engine/model/config).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The channel `send()` contract was inconsistent across adapters. Almost
every adapter (Discord, Slack, email, WhatsApp, ...) treats the first
positional `channel` arg as the real DESTINATION id and `conversation_id`
as an optional reply/thread reference. Telegram alone treated
`conversation_id` as the destination (`chat_id = conversation_id or
channel`).
`JarvisSystem._on_channel_message` hard-coded the Telegram-shaped mapping
for ALL channels: `send(cm.channel, reply, conversation_id=cm.conversation_id)`.
Since inbound `ChannelMessage`s carry the channel TYPE label in `.channel`
("discord") and the real destination id in `.conversation_id`, this sent
the literal "discord" as the Discord channel id (HTTP 400
NUMBER_TYPE_COERCE, #515) and passed the channel id as a Discord
`message_reference` (MESSAGE_REFERENCE_UNKNOWN_MESSAGE, #516).
Fix: define and document ONE canonical contract on `BaseChannel.send` —
positional `channel` = destination id, `conversation_id` = inbound message
id used as a reply reference — and dispatch it from `_on_channel_message`
as `send(cm.conversation_id, reply, conversation_id=cm.message_id)`,
matching the already-fixed `ChannelAgent` path (#495/#459). Telegram's
`send()` is brought into line (destination = `channel`, with a
`reply_to_message_id` reply ref and a legacy `conversation_id`-only
fallback) so it keeps working unchanged.
The DiscordChannel `_gateway_loop` ChannelMessage shape locked by #495 is
untouched; `tests/agents/test_channel_agent.py` passes unchanged. The
stale `test_serve_channel_wiring.py` assertions (which encoded the old
buggy mapping from #94) are updated, and regression tests are added for
Discord (real channel id + correct message_reference), Telegram (chat id
+ reply ref + legacy fallback), and per-channel dispatch in
`_on_channel_message`.
Fixes#515Fixes#516
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`jarvis agents list` crashed with a secondary Rich MarkupError when the
underlying exception message contained markup metacharacters like
`[...]`: the error handler did `console.print(f"[red]Error: {exc}[/red]")`,
and Rich re-parsed the interpolated message as markup (#297).
Escape the dynamic message with rich.markup.escape and keep only the
static "Error:" label styled, so the original error surfaces cleanly
instead of a traceback. Adds a regression test that reproduces the
MarkupError via an exception message with brackets.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses #404 (unable to launch / fully download) and contributes to #309 (stuck on starting api server) by removing the eager numpy import paths that fail hard when a Windows host has a partially-installed or cp314-incompatible numpy.
What changed:
- `src/openjarvis/connectors/embeddings.py` / `hybrid_search.py`: numpy imports are now lazy (inside the method that needs them) with a `TYPE_CHECKING` guard for annotations. Default-argument evaluation no longer touches numpy at module import.
- `src/openjarvis/cli/__init__.py`: the `deep_research_setup` command import is now guarded behind a `try/except Exception` so an OverflowError or ImportError during its module load doesn't crash the entire CLI.
- New regression test in `tests/cli/test_cli.py`: `test_importing_cli_does_not_import_numpy` spawns a subprocess and asserts `numpy` is not in `sys.modules` after `import openjarvis.cli`. Guards against future eager-numpy regressions on Windows.
Reported by @Tentacle39 in #404 and seen alongside @xoomarx's #309. Thanks to both — the Windows-only crash signature made this hard to diagnose without your repros.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Follow-up on @TX-Huang's PR #295. The new
test_soul_md_content_reaches_engine_in_simple_agent test has one
line at 92 chars that trips ruff's E501 (88 char limit). Wrap the
ternary onto multiple lines so the file passes ruff check on CI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`SystemPromptBuilder` is fully implemented and tested in
`openjarvis.prompt.builder`, and `BaseAgent.__init__` accepts a
`prompt_builder` kwarg. But no production code path ever instantiates
the builder, so the persona-files feature documented in
`MemoryFilesConfig` (SOUL.md / MEMORY.md / USER.md) had no effect.
Users could write a fully-customized `~/.openjarvis/SOUL.md` and the
file was never read.
This PR wires it up in `_run_agent` (called by `jarvis ask --agent
<name>` and the new fallback from #294):
```python
if "prompt_builder" in inspect.signature(agent_cls.__init__).parameters:
agent_kwargs["prompt_builder"] = SystemPromptBuilder(
agent_template=config.agent.default_system_prompt or "",
memory_files_config=config.memory_files,
system_prompt_config=config.system_prompt,
)
```
The `inspect` guard means agents that override `__init__` without
forwarding `prompt_builder` (e.g. OrchestratorAgent, which has its
own tool-aware system prompt) opt out automatically and keep
working unchanged. SimpleAgent and any future agent that inherits
`BaseAgent.__init__` directly picks up the persona files.
Also fixes a latent bug in `SystemPromptBuilder._load_file`: it
called `path.read_text()` with no encoding, which on Windows falls
back to the system code page (cp950 / cp932 / cp949) and raises
`UnicodeDecodeError` on any non-ASCII persona content. Pin to UTF-8.
## Tests
- Add `test_soul_md_content_reaches_engine_in_simple_agent` — writes
sentinel SOUL.md / MEMORY.md / USER.md to tmp_path, runs the
command, and asserts each sentinel appears in the SYSTEM message
passed to engine.generate.
- Add `test_orchestrator_keeps_its_own_system_prompt` — exercises
the `inspect`-based opt-out so OrchestratorAgent doesn't crash
on the unexpected kwarg.
Run: `pytest tests/cli/test_ask_agent.py tests/cli/test_ask_router.py
tests/cli/test_ask_e2e.py tests/agents/ tests/prompt/`
The 6 remaining failures (test_base_agent, test_loop_guard,
test_manager, test_native_openhands) are pre-existing on
origin/main and unrelated to this change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`jarvis ask "..."` (no `--agent` flag) routed straight to
`engine.generate()` regardless of `agent.default_agent` in the user's
config. As a result the persona stack — `agent.default_system_prompt`,
SOUL.md, MEMORY.md, USER.md — was silently bypassed for the most
common command.
Behavior now:
- `--agent X` → use agent X (unchanged)
- `--agent ""` (empty) → explicit opt-out, direct-to-engine mode
- (omitted) → fall back to `config.agent.default_agent`,
which dataclass-defaults to `"simple"`,
so persona settings finally take effect
Tests:
- Rename `test_no_agent_uses_direct_mode` to
`test_no_agent_flag_falls_back_to_config_default_agent` and update
its docstring to document the new behavior.
- Add `test_explicit_empty_agent_opts_out_of_agent_mode`.
- Add `test_no_agent_with_blank_config_default_uses_direct_mode` to
cover the case where the user clears `default_agent`.
- Update `_patch_engine` (test_ask_router) and `_patch_ask`
(test_ask_e2e) to re-register `SimpleAgent` after the autouse
`_clean_registries` conftest fixture, since the agent path now
runs in tests that previously short-circuited to direct mode.
- Add explicit `cfg.agent.default_agent = ""` to two
`test_ask_router` tests that mock load_config with a MagicMock
(so `cfg.agent.default_agent` doesn't auto-create as a truthy mock).
Note: `tests/cli/test_ask_context.py` has 2 unrelated pre-existing
failures on origin/main (memory backend returns None on Windows);
those are out of scope for this PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
On Windows the default Python stdout encoding follows the system
ANSI code page (cp950 for zh-TW, cp932 for ja, cp949 for ko).
`click.echo()` then raises `UnicodeEncodeError` whenever a CJK
character lands in CLI output — `jarvis ask` returning Chinese
crashes with `'cp950' codec can't encode character '义'`.
Reconfigure `sys.stdout` and `sys.stderr` to UTF-8 with
`errors='replace'` at the `main()` entry point. Scoped to
`win32` so other platforms are untouched. Two unit tests verify
the reconfigure happens on Windows and doesn't on Linux.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(mining): add Pearl model conversion workflow
* test(mining): tolerate missing Docker device request type
* docs(mining): record Gemma and Qwen conversion evidence
* docs(mining): record Qwen local validation evidence
---------
Co-authored-by: krypticmouse <herumbshandilya123@gmail.com>
- openjarvis.__version__ now comes from importlib.metadata.version("openjarvis")
instead of a hardcoded string, so it stays in sync with pyproject.toml on every
release. Tests assert against openjarvis.__version__ rather than a literal.
- Bump actions/checkout@v4 → @v6 and astral-sh/setup-uv@v4 → @v8 across every
workflow ahead of the 2026-06-02 Node.js 20 deprecation on GitHub Actions runners.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tests were storing digests with local time but get_today() filters
by UTC date, causing failures when local date != UTC date.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Incorporates the useful new features from PR #135 (by @gridworks) on top
of the existing PrivacyScanner implementation:
- Add DNS configuration check (macOS, via scutil --dns)
- Add --json flag to `jarvis scan` for machine-readable output
- Add --no-scan flag to `jarvis init` to skip the post-init audit
- Expand remote-access process list (ngrok, tailscaled, cloudflared, ZeroTier)
- Upgrade `jarvis scan` output from plain text to Rich table
- Add GET /v1/security/scan API endpoint
- Add tests for all new features (30 tests, all passing)
Closes#133
Co-Authored-By: gridworks <5502067+gridworks@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auto-detects local sources (Apple Notes, iMessage, Obsidian), ingests
them into ~/.openjarvis/knowledge.db via IngestionPipeline + SyncEngine,
and launches an interactive Deep Research chat session with Qwen3.5 via
Ollama.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The model catalog listed non-existent Qwen3.5 sizes (3B, 8B, 14B) and
pointed to MLX community repos that don't exist, causing `jarvis init`
to recommend models that cannot be downloaded on Apple Silicon.
Replace with the actual Qwen3.5 model family sizes (0.8B, 2B, 9B, 27B)
and verified mlx-community repo URLs from HuggingFace.
Fixes#129
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds connect_cmd.py with --list, --disconnect, --sync, and positional source
routing (filesystem vs OAuth), registers it in the CLI, and covers all
behaviours with 5 passing unit tests.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The test_init_creates_config test in test_cli.py was not updated
when the download prompt and privacy hook were added to jarvis init.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wraps long CliRunner.invoke() calls, removes unused pytest imports,
fixes import ordering, and removes unused variables in test_scan.py.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add slots=True to ScanResult dataclass
- Fix extra space in IPv6 port f-string
- Add missing tests: check_remote_access, check_icloud_sync, run_quick
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Registers the new scan command in the CLI. Adds a lightweight
privacy check at the end of jarvis init that runs disk encryption
and cloud sync checks, with pointer to jarvis scan for full audit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Prompts user to download recommended model during jarvis init.
Adds --no-download flag for CI. Shows helpful message when no
model fits available memory. Adds exo/nexa next-steps text.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Refactors model pull into reusable ollama_pull() function. Adds
--engine flag to support llamacpp (GGUF) and mlx (HuggingFace)
downloads via huggingface-cli, with FileNotFoundError handling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New PrivacyScanner class with checks for disk encryption (FileVault/LUKS),
MDM profiles, cloud sync agents, network exposure, and screen recording.
Supports macOS and Linux with graceful skip on missing tools.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add query complexity analyzer with CLI and UI integration
Classify incoming queries by difficulty (trivial→very_complex) to
suggest appropriate token budgets for local vs. cloud routing.
- Add score_complexity() with weighted signals (length, code, math,
reasoning, multi-step, creative) and token tier mapping
- Wire into `jarvis ask`: auto-suggest max_tokens when not set by user,
show complexity in --profile output, log at DEBUG level
- Add complexity metadata to /v1/chat/completions API response
- Display complexity tier and score in frontend XRayFooter
- Extend RoutingContext with complexity_score, suggested_max_tokens,
has_reasoning fields
- Update HeuristicRouter to route on complexity_score instead of
raw query_length
- Remove duplicated regex patterns from router.py (use complexity
module as single source of truth)
- Add 30 unit tests for complexity module
- Fix existing router tests for new complexity-based routing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: pass temperature and max_tokens from UI settings to backend
The settings page stores temperature and max_tokens in the frontend
store, but these values were never included in the chat API request.
The backend Pydantic model defaults max_tokens to 1024 when the field
is absent, which is too low for thinking models like qwen3.5 — they
consume all tokens on reasoning and return empty content.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: pass UI settings to backend and auto-bump max_tokens from complexity
- Pass temperature and max_tokens from frontend Settings store to the
backend API (cherry-picked from fix/ui-max-tokens-passthrough)
- Server-side: bump max_tokens when the complexity analyzer suggests a
higher budget (e.g. for thinking models on complex queries), never
reduce below the client-requested value
- Fixes empty responses with thinking models (e.g. qwen3.5) that
consumed all tokens on internal reasoning
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: raise token budget tiers to prevent empty responses on thinking models
Double all tier budgets (trivial: 512→1024, simple: 1024→2048, etc.)
so that thinking models like qwen3.5 have enough headroom for internal
chain-of-thought plus visible output, even on simple queries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: CI lint and test failures
- Break long lines in routes.py to satisfy 88-char limit (E501)
- Add intelligence.max_tokens and temperature to mocked config in
test_ask_router.py so complexity analyzer can compare against int
instead of MagicMock
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: line too long in test_complexity.py (E501)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add CLI commands: config, registry, tool
* fix: address review feedback on CLI config/registry/tool commands
- Replace non-existent `create_config_template` with `generate_default_toml`
to fix ImportError when config file is missing
- Deduplicate registry maps into shared `_load_registry_map()` helper
- Remove dead `_get_registry_class()` function and its tests
- Route JSON output to stdout for pipeability (`jarvis config show json | jq`)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jon Saad-Falcon <41205309+jonsaadfalcon@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Engine discovery (#73): get_engine() now falls back to any healthy
engine when the explicitly-requested key fails, instead of returning
None. Fixes LM Studio (and other non-default engines) not being found
when deploying agents.
- FTS5 case sensitivity & scoring (#67): Add explicit unicode61 tokenizer
to the FTS5 virtual table for case-insensitive search. Replace ambiguous
`rank` column with explicit `bm25()` call with column weights (id=0,
content=1, source=0.5) to produce correct positive scores. Includes
auto-migration for existing databases.
- Interactive engine selection (#72): `jarvis init` now detects running
engines and presents an interactive picker. Also accepts `--engine`
flag to skip the prompt. Engine choice flows through to config
generation.
- Windows desktop support (#68): Add RAM detection via wmic, Windows
binary paths (LOCALAPPDATA, ProgramFiles, cargo), port cleanup via
netstat+taskkill, and USERPROFILE fallback for HOME env var.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add interactive agent confirmation mode and fix tool loading
- Add `interactive` and `confirm_callback` params to ToolUsingAgent and
NativeReActAgent so CLI sessions can prompt user before tool execution
- Fix tool resolution in `ask` and `chat` commands to fall back to
config.tools.enabled when no --tools flag is provided
- Register shell_exec tool in tools/__init__.py auto-discovery
- Add dspy and gepa as optional learning extras (learning-dspy, learning-gepa)
- Fix openjarvis-rust lock version (1.0.0 → 0.1.0)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(cli): support configured tool defaults and confirmations
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
